Kubernetes Pod Disruption Budgets: Maximize HA

Introduction

In the dynamic world of Kubernetes, maintaining high availability for your applications is paramount. While Kubernetes excels at orchestrating workloads, ensuring your services remain operational during voluntary disruptions – like node draining for upgrades or scaling down – requires a more nuanced approach. Without proper safeguards, such operations can inadvertently take down too many instances of a critical application, leading to outages and frustrated users.

This is where Pod Disruption Budgets (PDBs) come into play. PDBs are a crucial Kubernetes API object that allows you to specify the minimum number or percentage of healthy pods that must be maintained for a given workload during a voluntary disruption. They act as a guardrail, preventing cluster administrators or automated tools from accidentally violating an application’s availability requirements. By carefully configuring PDBs, you can ensure that even as your cluster undergoes maintenance or scaling events, your applications gracefully handle the disruptions, preserving service continuity.

This comprehensive guide will walk you through everything you need to know about Kubernetes PDBs. We’ll cover their purpose, how to define them, practical examples, and essential considerations for production environments. By the end, you’ll be well-equipped to implement PDBs and significantly enhance the resilience of your Kubernetes-deployed applications.

Prerequisites

To follow along with this guide, you’ll need the following:

• Kubernetes Cluster: A running Kubernetes cluster (v1.21+ is recommended for policy/v1 PDBs). You can use Minikube, Kind, or a cloud-managed cluster like GKE, EKS, or AKS.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. You can find installation instructions in the official Kubernetes documentation.
• Basic Kubernetes Knowledge: Familiarity with core Kubernetes concepts such as Pods, Deployments, ReplicaSets, and Labels/Selectors.
• Text Editor: Any text editor for creating YAML manifest files.

Step-by-Step Guide

1. Understanding Voluntary vs. Involuntary Disruptions

Before diving into PDBs, it’s crucial to distinguish between two types of disruptions:

• Involuntary Disruptions: These are unexpected events outside of your control, such as node failures, kernel panics, or network outages. Kubernetes automatically tries to recover from these by rescheduling pods, but PDBs do not prevent them. For mitigating involuntary disruptions, strategies like ReplicaSets, Pod Anti-Affinity, and Deployments are used.
• Voluntary Disruptions: These are planned actions initiated by a human or an automated process. Examples include:
  • Node draining for maintenance, upgrades, or decommissioning.
  • Cluster upgrades.
  • Scaling down a cluster (e.g., with Cluster Autoscaler).
  • Resource preemption by higher-priority pods.

  PDBs specifically address and protect against these voluntary disruptions. They ensure that during such events, a minimum number of healthy pods for a given application remain running. For advanced networking and traffic management during such events, consider exploring tools like Istio Ambient Mesh or the Kubernetes Gateway API.

PDBs work by interacting with the Kubernetes API server’s eviction subresource. When a voluntary disruption attempts to evict a pod, the API server checks if the proposed eviction would violate any PDBs. If it would, the eviction is denied, and the disruptive operation (e.g., kubectl drain) pauses until enough pods can be safely evicted without violating the PDB.

2. Deploying a Sample Application

First, let’s deploy a simple Nginx application that we can protect with a PDB. This will be a standard Deployment with 3 replicas.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
      terminationGracePeriodSeconds: 30 # Allow time for graceful shutdown

Apply this Deployment to your cluster:

kubectl apply -f deployment.yaml

Verify

Check that the Deployment and its pods are running:

kubectl get deployment nginx-deployment
kubectl get pods -l app=nginx

Expected Output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           <some-time>

NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-55d65c49f6-2h53m   1/1     Running   0          <some-time>
nginx-deployment-55d65c49f6-d7c7h   1/1     Running   0          <some-time>
nginx-deployment-55d65c49f6-l8k7j   1/1     Running   0          <some-time>

3. Defining a Pod Disruption Budget (PDB)

Now, let’s create a PDB for our Nginx application. PDBs can specify either minAvailable or maxUnavailable, but not both.

• minAvailable: The minimum number or percentage of pods that must remain available.
• maxUnavailable: The maximum number or percentage of pods that can be unavailable.

It’s generally recommended to use minAvailable for critical services to ensure a baseline of operational pods. For example, if you have 3 replicas and set minAvailable: 2, only one pod can be disrupted at any given time. If you set minAvailable: 50%, then 50% of your 3 pods (which is 1.5, rounded up to 2) must remain available.

Let’s define a PDB that ensures at least 2 Nginx pods are always available.

# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 2 # At least 2 pods must be available
  selector:
    matchLabels:
      app: nginx # This must match the labels on your pods

Important: The selector in the PDB must exactly match the labels of the pods it’s intended to protect. In our case, it matches the app: nginx label defined in the Deployment’s pod template.

Apply the PDB:

kubectl apply -f pdb.yaml

Verify

Check the status of your PDB:

kubectl get pdb nginx-pdb -o yaml

Expected Output (snippet):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
  ...
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: nginx
status:
  currentHealthy: 3
  desiredHealthy: 2
  disruptionsAllowed: 1 # This is the key field!
  expectedPods: 3
  observedGeneration: 1

The disruptionsAllowed: 1 indicates that one pod can be safely evicted without violating the PDB, given that we have 3 running pods and require 2 to be available (3 – 2 = 1).

4. Simulating a Voluntary Disruption (Node Drain)

Now that our PDB is in place, let’s simulate a voluntary disruption by draining a node. This will attempt to evict pods from the node, and our PDB should prevent too many Nginx pods from being evicted simultaneously.

First, identify the node where your Nginx pods are running. You might need to scale up your Deployment or have multiple nodes to see the effect clearly.

kubectl get pods -o wide -l app=nginx

Pick one of the nodes hosting an Nginx pod (let’s say node01) and attempt to drain it:

kubectl drain node01 --ignore-daemonsets --delete-emptydir-data --force

Note: The --force flag is often needed for kubectl drain to evict pods that are not managed by a controller (like a plain Pod) or to proceed if there are issues. The --ignore-daemonsets flag is crucial because DaemonSet pods are typically not evicted during a drain. --delete-emptydir-data is needed if you have pods with EmptyDir volumes.

Verify

Observe the output of the drain command. It should show that pods are being evicted, but eventually, it might get stuck if it hits the PDB limit for our Nginx pods.

# Example output when drain is blocked by PDB
node/node01 drained
error: unable to drain node "node01", as it would violate the PodDisruptionBudget "nginx-pdb" (would make 1 available, but requires at least 2)
There are pending evictions that could not be completed.
... (The command will hang or report an error if it can't evict further pods)

While the drain is in progress (or stuck), open another terminal and check the PDB status:

kubectl get pdb nginx-pdb

Expected Output:

NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
nginx-pdb   2               N/A               0                     <some-time>

Notice that ALLOWED DISRUPTIONS has dropped to 0. This means no further Nginx pods can be evicted until the current ones are rescheduled and become healthy again, or until the total number of healthy pods increases. The drain operation will wait.

5. Cleaning Up and Observing PDB Behavior

To allow the drain to complete, you would either wait for the evicted pods to reschedule and become healthy on other nodes, or manually scale down your Deployment to release the PDB constraint.

Let’s scale down the Nginx Deployment to 2 replicas, which is our minAvailable threshold. This will effectively satisfy the PDB and allow the drain to continue.

kubectl scale deployment nginx-deployment --replicas=2

Verify

Check the PDB status again. The disruptionsAllowed should still be 0 because we are now at the minimum healthy count.

kubectl get pdb nginx-pdb

Expected Output:

NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
nginx-pdb   2               N/A               0                     <some-time>

Now, if you were to scale down to 1 replica, the PDB would prevent this:

kubectl scale deployment nginx-deployment --replicas=1

Expected Output:

error: The HPA controller was unable to update the target resource: deployments/nginx-deployment.apps "nginx-deployment" is forbidden: PodDisruptionBudget nginx-pdb would be violated if the deployment "nginx-deployment" were to scale down.

This demonstrates how PDBs protect against scaling operations that would violate availability guarantees. For deeper insights into cluster behavior during such events, eBPF Observability with Hubble can provide valuable network and application metrics.

6. Using maxUnavailable

Alternatively, you can define a PDB using maxUnavailable. This specifies the maximum number or percentage of pods that can be unavailable at any given time. For instance, maxUnavailable: 1 means at most one pod can be down. If you have 3 replicas, this is equivalent to minAvailable: 2.

# pdb-maxunavailable.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb-maxunavailable
spec:
  maxUnavailable: 1 # At most 1 pod can be unavailable
  selector:
    matchLabels:
      app: nginx

Apply this PDB (after deleting the previous one if you want to test this in isolation):

kubectl delete -f pdb.yaml
kubectl apply -f pdb-maxunavailable.yaml

Verify

kubectl get pdb nginx-pdb-maxunavailable

Expected Output:

NAME                       MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
nginx-pdb-maxunavailable   N/A             1                 1                     <some-time>

The ALLOWED DISRUPTIONS field will again show 1, indicating that one pod can be disrupted.

Production Considerations

Implementing PDBs effectively in production requires careful planning:

1. Scope and Granularity:
   • Apply PDBs to critical, highly available applications. Not every workload needs a PDB.
   • Use specific selectors to avoid unintended overlaps or exclusions. A PDB should typically target pods managed by a single Deployment, StatefulSet, or ReplicaSet.
2. Percentage vs. Absolute Numbers:
   • For applications with a fixed number of replicas (e.g., StatefulSets), minAvailable or maxUnavailable as an absolute number is straightforward.
   • For Deployments that scale frequently, percentages (e.g., minAvailable: 90%) are more flexible as they adapt to the current replica count. Remember that percentages are rounded up for minAvailable and down for maxUnavailable.
3. Interaction with Cluster Autoscaler and HPA:
   • PDBs can interfere with the Cluster Autoscaler (CA)‘s ability to scale down nodes, as CA respects PDBs. If CA cannot evict pods from a node due to PDBs, the node may remain idle. This can impact cost optimization efforts, potentially negating benefits from tools like Karpenter if PDBs are too restrictive.
   • PDBs also influence how the Horizontal Pod Autoscaler (HPA) scales down. If an HPA attempts to scale down a Deployment, and doing so would violate a PDB, the scale-down operation will be blocked.
4. Graceful Shutdown:
   • Ensure your applications handle SIGTERM signals gracefully and have a sufficient terminationGracePeriodSeconds defined in their pod spec. This allows pods to finish in-flight requests and clean up before being terminated, preventing data loss or service interruptions during eviction.
5. Monitoring and Alerting:
   • Monitor PDB status, especially the disruptionsAllowed field. If it consistently stays at 0 for critical applications, it might indicate a problem with your application’s replica count or an overly restrictive PDB.
   • Alert on situations where PDBs are preventing critical operations (e.g., node drains taking too long).
6. Testing:
   • Always test PDBs in a staging environment by simulating node drains and other voluntary disruptions. This helps validate their effectiveness and identify any unintended consequences.
7. Security Contexts:
   • While not directly related to PDBs, ensuring your pods run with appropriate security contexts and adhere to principles like those enforced by Sigstore and Kyverno for supply chain security, indirectly contributes to overall application resilience and stability. Well-secured pods are less prone to unexpected failures.

Troubleshooting

1. PDB is not preventing disruptions.

   Issue: You’ve created a PDB, but voluntary disruptions (like kubectl drain) are still evicting too many pods.

   Solution:

   1. Selector Mismatch: The most common issue. Ensure the PDB’s selector precisely matches the labels on the pods it’s supposed to protect.

      kubectl get pdb <your-pdb-name> -o yaml | grep "matchLabels"
      kubectl get pods -l <your-pdb-selector-key>=<your-pdb-selector-value> --show-labels

      Verify that all target pods have these labels.

   2. Incorrect API Version: Ensure you’re using policy/v1 and not an older, deprecated version like policy/v1beta1.
   3. PDB Status: Check the disruptionsAllowed field. If it’s not 0 when you expect it to be, it means the PDB currently permits disruptions.

      kubectl get pdb <your-pdb-name>

   4. Involuntary Disruptions: Remember PDBs only protect against voluntary disruptions. If pods are disappearing due to node crashes, OOMKills, or other unexpected events, PDBs won’t intervene.

2. kubectl drain is stuck or fails with a PDB error.

   Issue: When trying to drain a node, the command hangs indefinitely or reports an error about violating a PDB.

   Solution:

   1. This is expected behavior! The PDB is doing its job. The drain will wait until enough healthy pods are available on other nodes to satisfy the PDB.
   2. Check PDB Status:

      kubectl get pdb <your-pdb-name>

      If disruptionsAllowed is 0, the drain is blocked.

   3. Wait for Rescheduling: Allow Kubernetes controllers to reschedule the evicted pods onto other nodes. Once they become healthy, the disruptionsAllowed count will increase, and the drain can proceed.
   4. Scale Up Temporarily: If you need to drain the node quickly, consider temporarily scaling up your Deployment/StatefulSet to provide more available replicas, then scale down after the drain.
   5. Check for Stuck Pods: Ensure pods aren’t stuck in a pending or error state on other nodes, which would prevent them from becoming healthy and satisfying the PDB.

3. PDB prevents HPA from scaling down.

   Issue: Your Horizontal Pod Autoscaler (HPA) tries to scale down a Deployment, but the operation is blocked, and you see errors related to PDBs.

   Solution:

   1. Review PDB Configuration: Your PDB might be too restrictive for the HPA’s scaling behavior. If minAvailable is set to an absolute number, and the HPA wants to scale below that, it will be blocked. Consider using a percentage-based minAvailable or maxUnavailable if your application scales frequently.
   2. Align HPA and PDB: Ensure the HPA’s minimum replica count is compatible with the PDB’s minAvailable. For example, if HPA min is 2, and PDB minAvailable is 3, the HPA will never reach its minimum.
   3. Monitor HPA Events: Check HPA events for more details:

      kubectl describe hpa <your-hpa-name>

4. PDB is not working for StatefulSets.

   Issue: PDBs seem to be ignored for StatefulSet pods during disruptions.

   Solution:

   1. PDBs work with StatefulSets: PDBs are fully compatible with StatefulSets. Double-check the selector in your PDB to ensure it matches the labels of your StatefulSet pods. StatefulSet pods often have labels like app: my-app and statefulset.kubernetes.io/pod-name: my-app-0. The PDB selector should generally match the app label.
   2. Graceful Shutdown: Ensure your StatefulSet pods have proper terminationGracePeriodSeconds and handle signals correctly. This is crucial for StatefulSets, especially if they manage persistent storage.

5. “No matching pods found” for PDB.

   Issue: When creating a PDB, its status shows No matching pods found or expectedPods is 0.

   Solution:

   1. Incorrect Selector: The selector in your PDB does not match any existing pods in the namespace. Double-check the labels on your pods and the selector in the PDB.

      kubectl get pods --show-labels
      kubectl get pdb <your-pdb-name> -o yaml

      Make sure the matchLabels section of the PDB is correct.

   2. Namespace Mismatch: Ensure the PDB is created in the same namespace as the pods it is intended to protect.

FAQ Section

1. What is the difference between minAvailable and maxUnavailable?

   minAvailable specifies the minimum number or percentage of pods that must be healthy and available. For example, minAvailable: 2 means at least two pods must be running. maxUnavailable specifies the maximum number or percentage of pods that can be unavailable. For example, maxUnavailable: 1 means at most one pod can be down. You can only specify one of these fields in a PDB.

2. Do PDBs protect against involuntary disruptions?

   No, PDBs only protect against voluntary disruptions, which are planned actions like node drains or cluster upgrades. They do not prevent pods from being terminated due to involuntary disruptions such as node failures, kernel panics, or OOMKills. For protection against involuntary disruptions, you rely on Kubernetes’ inherent self-healing capabilities (e.g., ReplicaSets, Deployments) and strategies like pod anti-affinity to spread pods across nodes.

3. Can I use PDBs with any type of workload?

   PDBs are primarily designed for workloads managed by controllers that manage multiple identical pods, such as Deployments, StatefulSets, and ReplicaSets. They are less relevant for single-instance Pods or DaemonSets (as DaemonSet pods are usually ignored by drain operations). For more robust network isolation for your workloads, regardless of their type, consider implementing Kubernetes Network Policies.

4. What happens if I delete a PDB while a disruption is ongoing?

   If you delete a PDB while an operation like kubectl drain is blocked by it, the block will be lifted. The disruptive operation will then proceed, potentially evicting more pods than you intended and violating your application’s availability requirements. Therefore, you should exercise caution when deleting PDBs in a live cluster.

5. How do PDBs affect cluster upgrades?

   PDBs are critical during cluster upgrades, especially when nodes are being drained and cordoned. They ensure that as nodes are taken offline for updates, your applications maintain their minimum healthy replica count. Without PDBs, an aggressive upgrade strategy could lead to significant downtime for your services. They complement other high-availability features like topology spread constraints and node anti-affinity, which help distribute pods across failure domains from the start.

Cleanup Commands

To remove the resources created during this tutorial:

# Delete the PDBs
kubectl delete pdb nginx-pdb
kubectl delete pdb nginx-pdb-maxunavailable

# Delete the Deployment
kubectl delete deployment nginx-deployment

# If you drained a node, uncordon it
# Replace node01 with the actual node name you drained
kubectl uncordon node01

Next Steps / Further Reading

• Official Kubernetes Documentation: Dive deeper into the Pod Disruption Budget concept and the task of configuring PDBs.
• Node Draining: Understand the intricacies of safely draining a node.
• High Availability Strategies: Explore other Kubernetes features for high availability, such as Rolling Updates, Pod Anti-Affinity, and Topology Spread Constraints.
• Advanced Networking: For complex traffic management and resilience patterns, investigate Istio Ambient Mesh Production Guide or how to migrate to the Kubernetes Gateway API.
• Cost Optimization: Learn how PDBs can interact with Karpenter for Kubernetes cost optimization and how to balance availability with efficiency.
• Security Best Practices: Enhance your cluster’s resilience by securing your container supply chain with Sigstore and Kyverno.

Conclusion

Pod Disruption Budgets are an indispensable tool for any organization running critical applications on Kubernetes. By providing a declarative way to specify application availability during voluntary disruptions, PDBs act as a vital safety net, preventing accidental outages during routine cluster operations like upgrades, scaling, or maintenance. Mastering PDBs ensures that your Kubernetes cluster remains a robust and reliable platform for your most important services, allowing you to perform necessary administrative tasks with confidence while maintaining a high level of service availability. Integrate PDBs into your deployment strategies today to fortify your applications against the inevitable disruptions of a dynamic cloud-native environment.