Orchestration

Kubernetes Taints & Tolerations: Master Node Control

June 20, 2026 Kubezilla Team 17 min read

Introduction

In the dynamic world of Kubernetes, efficient resource management and workload isolation are paramount. Imagine a scenario where you have specialized nodes – perhaps equipped with GPUs for machine learning tasks, or high-memory instances for data analytics – and you want to ensure that only specific pods run on them, or conversely, prevent certain pods from running on particular nodes. This is where Kubernetes Taints and Tolerations come into play, offering a powerful mechanism to guide the scheduler in placing pods.

Taints are applied to nodes, marking them as “undesirable” for certain pods. Tolerations, on the other hand, are applied to pods, signaling that they “tolerate” a specific taint. When a node has a taint, the Kubernetes scheduler will not place any pod on that node unless the pod has a corresponding toleration. This allows for flexible and fine-grained control over pod placement, enabling specialized clusters, ensuring resource availability, and enhancing overall cluster stability and security.

TL;DR: Taints and Tolerations

Kubernetes Taints and Tolerations are mechanisms to control pod scheduling on nodes.

  • Taints are applied to nodes to repel pods.
  • Tolerations are applied to pods to attract them to tainted nodes.
  • They work together to ensure pods are scheduled only on appropriate nodes.
  • Common uses: dedicated nodes, nodes with special hardware (GPUs), managing node failures.

Key Commands:


# Taint a node
kubectl taint nodes <node-name> <key>=<value>:<effect>

# Example: Taint a node to repel all pods
kubectl taint nodes node1 dedicated=gpu:NoSchedule

# Describe a node to see its taints
kubectl describe node <node-name>

# Apply a toleration to a pod (in YAML)
# tolerations:
# - key: "dedicated"
#   operator: "Equal"
#   value: "gpu"
#   effect: "NoSchedule"

# Remove a taint from a node
kubectl taint nodes <node-name> <key>=<value>:<effect>-

Prerequisites

Before diving into Taints and Tolerations, ensure you have the following:

  • A running Kubernetes cluster (e.g., Minikube, kind, or a cloud-managed cluster like GKE, EKS, AKS).
  • kubectl command-line tool configured to communicate with your cluster. You can find installation instructions in the official Kubernetes documentation.
  • Basic understanding of Kubernetes concepts like Pods, Nodes, and the Scheduler.
  • A text editor for creating Kubernetes manifest files (YAML).

Step-by-Step Guide to Taints and Tolerations

Step 1: Understanding Taints and Their Effects

A taint is a property applied to a node that prevents pods without a matching toleration from being scheduled on it. Each taint consists of a key, a value, and an effect. The effect determines what happens to pods that do not tolerate the taint. Kubernetes supports three main taint effects:

  • NoSchedule: The scheduler will not place any new pods on the node unless they tolerate this taint. Existing pods on the node are not affected. This is the most common and least disruptive effect.
  • PreferNoSchedule: The scheduler will *try* to avoid placing pods on the node, but it’s not a hard requirement. If no other suitable nodes are available, pods without tolerations might still be scheduled here. This is a “soft” version of NoSchedule.
  • NoExecute: This is the most aggressive effect. If a pod does not tolerate a NoExecute taint, it will not be scheduled on the node. More importantly, if a NoExecute taint is added to a node, any pods *already running* on that node that do not tolerate the taint will be evicted. This is crucial for scenarios like node maintenance or when a node becomes unhealthy.

Let’s begin by tainting a node with the NoSchedule effect. We’ll simulate a dedicated node for specialized workloads.

Action: Taint a node

First, identify one of your cluster’s nodes. You can get a list of nodes using kubectl get nodes. Then, apply a taint to it. We’ll use dedicated=gpu:NoSchedule to signify a node intended for GPU workloads.


# Get the name of your node
kubectl get nodes

# Example output:
# NAME         STATUS   ROLES           AGE   VERSION
# kind-control-plane   Ready    control-plane   23h   v1.28.0
# kind-worker          Ready    <none>          23h   v1.28.0
# kind-worker2         Ready    <none>          23h   v1.28.0

# Choose one of your worker nodes, e.g., 'kind-worker'
kubectl taint nodes kind-worker dedicated=gpu:NoSchedule

Verify: Node Taint

You can verify that the taint has been applied by describing the node. Look for the “Taints” section in the output.


kubectl describe node kind-worker

Expected Output:

(Scroll down to find the Taints section, it might be near the end)


...
Taints:             dedicated=gpu:NoSchedule
Unschedulable:      false
...

Step 2: Observing Pod Scheduling Without Tolerations

Now that we have a tainted node, let’s try to schedule a regular pod (without any tolerations) and observe what happens. The Kubernetes scheduler should avoid placing this pod on our tainted kind-worker node. It will either schedule it on another available node or, if kind-worker is the only available node, the pod will remain in a Pending state.

This demonstrates the core function of taints: repelling pods that don’t explicitly tolerate them. This is essential for scenarios like keeping critical infrastructure pods separate or ensuring resource-intensive applications have dedicated environments. For advanced network isolation, you might combine this with Kubernetes Network Policies to further secure communication.

Action: Deploy a Pod without Tolerations

Create a simple Nginx deployment that doesn’t specify any tolerations.


# untolerated-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: untolerated-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Apply this deployment:


kubectl apply -f untolerated-nginx.yaml

Verify: Pod Scheduling

Check where the pod has been scheduled. If you have multiple worker nodes, it should land on any node *other* than kind-worker. If kind-worker is your only schedulable node, the pod will remain in a Pending state.


kubectl get pods -l app=nginx -o wide

Expected Output (example, assuming other nodes exist):


NAME                                 READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
untolerated-nginx-7c7d4c8c7c-abcde   1/1     Running   0          5s    10.244.1.4   kind-worker2   <none>           <none>

Notice that the pod is running on kind-worker2, not our tainted kind-worker.

Step 3: Applying Tolerations to Pods

Now, let’s create a pod that *can* be scheduled on our tainted kind-worker node by adding a matching toleration. A toleration in a pod’s specification matches a taint if the key, value, and effect fields are identical. If the taint’s operator is Exists, then the value field can be omitted, and it will match any taint with that key.

This step is crucial for enabling specialized workloads to run on their designated nodes. For instance, if you have nodes with GPUs for AI/ML tasks, you’d taint those nodes and add tolerations to your LLM GPU Scheduling pods.

Action: Deploy a Pod with Tolerations

Create a new Nginx deployment with a toleration that matches the dedicated=gpu:NoSchedule taint.


# tolerated-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tolerated-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tolerated-nginx
  template:
    metadata:
      labels:
        app: tolerated-nginx
    spec:
      tolerations:
      - key: "dedicated"
        operator: "Equal" # Can be "Equal" or "Exists"
        value: "gpu"
        effect: "NoSchedule"
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Apply this deployment:


kubectl apply -f tolerated-nginx.yaml

Verify: Pod Scheduling with Tolerations

Check the pod’s status and node assignment. It should now be scheduled on the kind-worker node.


kubectl get pods -l app=tolerated-nginx -o wide

Expected Output:


NAME                                 READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
tolerated-nginx-676cdc49f4-vwxyz     1/1     Running   0          5s    10.244.2.5   kind-worker   <none>           <none>

Success! The pod with the toleration is now running on the tainted kind-worker node.

Step 4: Using the NoExecute Taint Effect

The NoExecute effect is powerful because it can evict *already running* pods that do not tolerate it. This is particularly useful for scenarios where a node’s state changes critically, or during maintenance. For instance, if a node becomes unhealthy, you might want to automatically evict non-critical pods to allow critical pods (with appropriate tolerations) to continue operating or be rescheduled elsewhere. This kind of automatic remediation can be part of a broader resilience strategy, perhaps even within a service mesh like Istio Ambient Mesh which handles traffic routing during such events.

Let’s demonstrate this by adding a NoExecute taint to our node and observing the eviction of the untolerated-nginx pod.

Action: Add a NoExecute Taint

We’ll add a new taint with the NoExecute effect to kind-worker. Note that a node can have multiple taints.


kubectl taint nodes kind-worker critical=true:NoExecute

Verify: Node Taints

Check the node’s taints again.


kubectl describe node kind-worker

Expected Output:


...
Taints:             dedicated=gpu:NoSchedule
                    critical=true:NoExecute
Unschedulable:      false
...

Verify: Pod Eviction

Observe the status of the untolerated-nginx pod. It should be evicted from kind-worker. Since untolerated-nginx has no tolerations, it won’t be able to run on kind-worker with the NoExecute taint. If there are other schedulable nodes, it will be rescheduled. If not, it will remain in a Pending state.


kubectl get pods -l app=nginx -o wide

Expected Output (after a short delay):

If it was on kind-worker, it will be evicted. If it was already on another node, it will remain there.


NAME                                 READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
untolerated-nginx-7c7d4c8c7c-abcde   1/1     Running   0          10m     10.244.1.4   kind-worker2   <none>           <none>

The pod which was originally on kind-worker (if it was) would have been evicted and rescheduled to kind-worker2 or terminated.

Now, let’s create a new pod with a toleration for the NoExecute taint.

Action: Deploy a Pod Tolerating NoExecute


# critical-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: critical-app
  template:
    metadata:
      labels:
        app: critical-app
    spec:
      tolerations:
      - key: "critical"
        operator: "Equal"
        value: "true"
        effect: "NoExecute"
      containers:
      - name: busybox
        image: busybox:latest
        command: ["sh", "-c", "echo 'I am a critical app!' && sleep 3600"]

Apply this deployment:


kubectl apply -f critical-app.yaml

Verify: NoExecute Toleration

Check the pod’s status. It should be scheduled on kind-worker.


kubectl get pods -l app=critical-app -o wide

Expected Output:


NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
critical-app-56789abcd-efghj     1/1     Running   0          5s    10.244.2.6   kind-worker   <none>           <none>

Step 5: Taint with TolerationSeconds

For NoExecute taints, you can specify a tolerationSeconds field in the toleration. This tells Kubernetes how long a pod can remain running on a node after a NoExecute taint is added, before being evicted. This is useful for graceful shutdown or allowing time for short-lived tasks to complete.

Action: Apply a NoExecute Taint with tolerationSeconds

First, remove the previous critical=true:NoExecute taint to avoid conflicts. Then, we’ll demonstrate a pod with tolerationSeconds.


# Remove the existing NoExecute taint
kubectl taint nodes kind-worker critical=true:NoExecute-

# Describe node to confirm removal
kubectl describe node kind-worker

Now, create a new pod with tolerationSeconds.


# tolerant-for-a-while.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tolerant-for-a-while
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tolerant-for-a-while
  template:
    metadata:
      labels:
        app: tolerant-for-a-while
    spec:
      tolerations:
      - key: "evict-me-later"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 30 # Pod will be evicted after 30 seconds
      containers:
      - name: busybox
        image: busybox:latest
        command: ["sh", "-c", "echo 'I will be evicted in 30 seconds!' && sleep 3600"]

Apply the deployment:


kubectl apply -f tolerant-for-a-while.yaml

Wait for the pod to be scheduled.


kubectl get pods -l app=tolerant-for-a-while -o wide

Now, apply the matching NoExecute taint to the node.


kubectl taint nodes kind-worker evict-me-later=true:NoExecute

Verify: Delayed Eviction

Immediately after tainting, check the pod’s status. It should still be running.


kubectl get pods -l app=tolerant-for-a-while -o wide

After 30 seconds, check again. The pod should be evicted and either rescheduled or terminated.


kubectl get pods -l app=tolerant-for-a-while -o wide

Expected Output (after 30 seconds):

The pod will likely be terminated or rescheduled to another node if available.


# If rescheduled:
NAME                                 READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
tolerant-for-a-while-xxxxxxxxx-yyyyy 1/1     Running   0          40s     10.244.1.5   kind-worker2   <none>           <none>

# If no other node, it might show as Pending or be recreating

Step 6: Removing Taints from Nodes

To revert a node to its original state or change its specialization, you can remove taints. Removing a taint is straightforward and uses the same kubectl taint command with a hyphen (-) at the end of the taint specification.

Action: Remove Taints

Remove all taints we applied to kind-worker.


kubectl taint nodes kind-worker dedicated=gpu:NoSchedule-
kubectl taint nodes kind-worker evict-me-later=true:NoExecute- # Ensure this one is removed too

Verify: Taints Removed

Describe the node to confirm that no taints are present.


kubectl describe node kind-worker

Expected Output:

The “Taints” section should either be absent or empty.


...
Taints:             <none>
Unschedulable:      false
...

Now kind-worker is a general-purpose node again, and any pod can be scheduled on it.

Production Considerations

Implementing Taints and Tolerations effectively in a production environment requires careful planning and consideration.

  1. Node Specialization and Hardware:
    Use taints to dedicate nodes with special hardware (GPUs, NVMe drives) or specific configurations. This ensures that resource-intensive workloads like LLM GPU Scheduling or big data processing only run on suitable nodes, preventing resource contention on general-purpose nodes.
  2. Node Isolation and Security:
    Taints can isolate sensitive workloads to specific nodes, enhancing security. Combined with Kubernetes Network Policies and potentially Cilium WireGuard Encryption, you can create highly secure and isolated environments for critical applications.
  3. Cluster Autoscaling Integration:
    When using cluster autoscalers like Karpenter or the Kubernetes Cluster Autoscaler, ensure your node groups or instance types are configured with appropriate taints. This allows the autoscaler to provision new nodes that are pre-tainted for specific workloads, optimizing cost and resource allocation.
    Refer to the Kubernetes documentation on Node Controller taints for details on how the system automatically taints unhealthy nodes.
  4. System Pods and Critical Addons:
    Many Kubernetes system pods (e.g., Kube-proxy, CoreDNS, CNI plugins) have built-in tolerations for common taints like node.kubernetes.io/not-ready or node.kubernetes.io/unreachable. When tainting nodes, be mindful that critical cluster add-ons might need explicit tolerations to ensure they can run everywhere.
  5. High Availability and Eviction Strategy:
    For NoExecute taints, carefully consider the tolerationSeconds parameter. This can prevent critical applications from being immediately evicted during transient node issues, allowing them time to gracefully shut down or for the node to recover. However, for truly critical applications, ensure they have sufficient replicas and are spread across multiple availability zones.
  6. Monitoring and Observability:
    Monitor node taints and pod scheduling events closely. Tools like eBPF Observability with Hubble or standard Prometheus/Grafana setups can help visualize scheduling decisions and identify if pods are stuck in a pending state due to unmatched taints.
  7. Automated Tainting:
    Beyond manual tainting, consider automated solutions. For example, a custom controller could automatically taint nodes based on their labels (e.g., gpu-enabled=true) or health status. Cloud providers often taint nodes automatically during maintenance or failure events.
  8. Documentation:
    Clearly document your cluster’s taint strategy. Which nodes are tainted, for what purpose, and what tolerations are required for specific applications? This is crucial for maintainability and onboarding new team members.

Troubleshooting

Here are some common issues you might encounter with Taints and Tolerations, along with their solutions.

  1. Pod remains in Pending state after applying toleration.

    Issue: You’ve applied a toleration to your pod, but it’s still not scheduling on the tainted node.

    Solution:

    • Mismatched Taint/Toleration: Double-check the key, value, operator, and effect fields in both the node’s taint and the pod’s toleration. They must match precisely (unless using operator: Exists).
    • Other Scheduling Constraints: Taints and tolerations are just one part of the scheduler’s logic. Check for other constraints like nodeSelector, node affinity/anti-affinity, resource requests/limits, or pod anti-affinity that might be preventing the pod from landing on the desired node.
    • Insufficient Resources: Even with a matching toleration, if the node doesn’t have enough CPU, memory, or other requested resources, the pod won’t schedule.
    • Describe the Pod: Use kubectl describe pod <pod-name> and look at the “Events” section for detailed scheduling failures.
    
kubectl describe pod <pod-name>
  2. Pod unexpectedly evicted or not scheduled on a node with NoExecute taint.

    Issue: A pod you expected to remain on a node (or schedule there) is evicted or stuck in pending after a NoExecute taint is applied.

    Solution:

    • Missing NoExecute Toleration: Ensure the pod has a toleration specifically for the NoExecute effect. Remember, NoSchedule tolerations don’t cover NoExecute.
    • tolerationSeconds Expired: If the pod has a tolerationSeconds field, it will be evicted after that duration. If you want it to stay indefinitely, remove tolerationSeconds or ensure it’s a very large number.
    • Multiple Taints: The node might have multiple NoExecute taints. The pod needs a toleration for *all* NoExecute taints present on the node to remain there.
  3. Taint not applied or removed correctly.

    Issue: You tried to taint/untaint a node, but kubectl describe node doesn’t show the expected change.

    Solution:

    • Syntax Error: Double-check the kubectl taint command syntax. Ensure the key=value:effect format is correct for applying, and key=value:effect- for removing.
    • Incorrect Node Name: Verify the node name is accurate using kubectl get nodes.
    • Permissions: Ensure your Kubernetes user has the necessary permissions to modify nodes (e.g., edit role or specific RBAC for nodes).
    
# Example of correct removal
kubectl taint nodes my-node dedicated=gpu:NoSchedule-
  4. System pods (e.g., CoreDNS, Kube-proxy) are stuck in Pending.

    Issue: After tainting a node, critical system pods are no longer scheduling or are restarting frequently.

    Solution:

    • Overly Aggressive Tainting: You may have applied a taint that system pods don’t tolerate. Most system pods have built-in tolerations for common taints (like node.kubernetes.io/not-ready), but not for arbitrary custom taints.
    • Add Tolerations to System Pods: If you must apply custom taints cluster-wide, consider modifying the manifests of your system pods (e.g., Kube-proxy, CoreDNS) to include the necessary tolerations. This is generally discouraged unless absolutely necessary, as it can complicate upgrades.
    • Use Node Labels/Affinity Instead: For general-purpose separation where system pods should still run, consider using nodeSelector or node affinity on your application pods instead of tainting nodes.
  5. Pods are being scheduled on a tainted node without a matching toleration.

    Issue: You’ve tainted a node with NoSchedule, but some pods without the corresponding toleration are still running on it.

    Solution:

    • Existing Pods: Remember that NoSchedule only prevents *new* pods from being scheduled. It does not evict existing pods. If you want existing pods to be evicted, use the NoExecute effect (or manually drain the node).
    • PreferNoSchedule vs. NoSchedule: If you used PreferNoSchedule, the scheduler might still place pods there if no better option exists. Always use NoSchedule for strict exclusion.
    • Taint Was Removed: Check the node’s taints to ensure it’s still present.

FAQ Section

  1. What is the difference between Taints/Tolerations and Node Selectors/Node Affinity?

    Taints and Tolerations are a negative constraint system. Taints *repel* pods from nodes unless the pods have a matching toleration. They are primarily used for node specialization, isolation, or handling node failures.
    Node Selectors and Node Affinity are a positive constraint system. They *attract* pods to nodes that match specific labels. They are used to ensure pods run on nodes with certain characteristics (e.g., specific hardware, region).
    You can use both together for very precise control. For example, use a node selector to *attract* pods to GPU nodes, and a taint on those GPU nodes to *repel* all other non-GPU pods.

  2. Can a node have multiple taints?

    Yes, a node can have multiple taints. A pod must then have tolerations for *all* of the taints that it wishes to ignore. If a node has three taints, and a pod only tolerates two of them, the scheduler will not place that pod on the node (for NoSchedule or NoExecute effects).

  3. What happens if a node with running pods gets a new NoExecute taint?

    If a node acquires a new NoExecute taint, any pods currently running on that node that do *not* have a matching toleration for that specific taint will be evicted. If they have a toleration with tolerationSeconds, they will be evicted after that duration. This is a key difference from NoSchedule, which only affects new pod scheduling.

  4. Are Taints and Tolerations considered for DaemonSets?

    Yes, DaemonSets respect taints and tolerations. By default, DaemonSets have a special toleration for node.kubernetes.io/unschedulable and node.kubernetes.io/not-ready, which allows them to run on nodes that might be temporarily unhealthy. If you apply custom taints, you’ll need to add corresponding tolerations to your DaemonSet manifests if you want them to run on those tainted nodes.
    For more details, see the DaemonSet tolerations documentation.

  5. How do I use Taints and Tolerations with Kubernetes Gateway API or Ingress controllers?

    If your Kubernetes Gateway API controller (or Ingress controller) pods are deployed as deployments, you can add tolerations to their pod templates just like any other application. If they are deployed as DaemonSets, they might already have some default tolerations, but you can add more if you want them to run on specific tainted infrastructure nodes (e.g., edge nodes). Ensure your controller pods can always run on appropriate nodes to maintain traffic flow.

Cleanup Commands

After you’ve finished experimenting, it’s good practice to clean up the resources you’ve created.


# Delete the deployments
kubectl delete deployment untolerated-nginx
kubectl delete deployment tolerated-nginx
kubectl delete deployment critical-app
kubectl delete deployment tolerant-for-a-while

# Remove any remaining taints from the node (replace 'kind-worker' with your node name)
kubectl taint nodes kind-worker dedicated=gpu:NoSchedule-
kubectl taint nodes kind-worker critical=true:NoExecute-
kubectl taint nodes kind-worker evict-me-later=true:NoExecute-

# Confirm all resources are deleted and taints are removed
kubectl get deployments
kubectl get pods -o wide
kubectl describe node kind-worker

Next Steps / Further Reading

Taints and Tolerations are a fundamental aspect of advanced Kubernetes scheduling. To deepen your understanding and explore related concepts, consider these topics:

Leave a comment