Optimize Cloud Costs with Karpenter Consolidation

Karpenter Consolidation: Mastering Cost Reduction in Kubernetes

Running Kubernetes in the cloud offers unparalleled scalability and flexibility, but it often comes with a significant challenge: cost optimization. As workloads fluctuate and cluster nodes are provisioned and de-provisioned, idle resources and fragmented capacity can quickly inflate your cloud bill. This is where Karpenter, an open-source, high-performance Kubernetes cluster autoscaler built by AWS, truly shines. While Karpenter is well-known for its rapid node provisioning, its consolidation features are equally, if not more, critical for maintaining a lean and efficient cluster.

Karpenter’s consolidation capabilities are designed to proactively identify and terminate underutilized nodes, replacing them with more appropriately sized instances or packing pods onto fewer, larger nodes. This intelligent re-arrangement of workloads minimizes wasted capacity, reduces operational overhead, and ultimately slashes your infrastructure costs. In this comprehensive guide, we’ll dive deep into Karpenter’s consolidation strategies, walk through practical examples, and equip you with the knowledge to significantly optimize your Kubernetes cloud spending.

Prerequisites

Before we embark on our cost-saving journey with Karpenter consolidation, ensure you have the following:

• A Running Kubernetes Cluster: Preferably on AWS, as Karpenter is optimized for it. While it can run on other clouds, this guide focuses on AWS.
• kubectl Configured: Access to your Kubernetes cluster.
• helm Installed: For easy installation of Karpenter. Refer to the Helm documentation for installation instructions.
• Karpenter Installed and Configured: If you haven’t installed Karpenter yet, please follow the official Karpenter Getting Started guide. For a deeper dive into initial Karpenter setup and cost optimization, you can check out our dedicated guide: Reduce Kubernetes Costs by 60% with Karpenter.
• IAM Permissions: Ensure your Karpenter controller has the necessary IAM permissions to launch and terminate EC2 instances, create/delete launch templates, and interact with other AWS services.

Step-by-Step Guide to Karpenter Consolidation

Karpenter employs several strategies for consolidation, primarily focusing on identifying and terminating underutilized nodes. We’ll cover the most common and impactful methods.

1. Install Karpenter (if not already installed)

If you haven’t already, install Karpenter into your cluster. This involves creating an IAM Role for Karpenter, attaching policies, and then deploying Karpenter via Helm.

First, set up the IAM role and service account. This example uses an OIDC provider for EKS. Replace ${CLUSTER_NAME} and ${AWS_ACCOUNT_ID} with your actual values.

# Set environment variables
export AWS_REGION="us-east-1" # e.g., us-west-2
export CLUSTER_NAME="my-karpenter-cluster"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export KARPENTER_VERSION="v0.32.0" # Use the latest stable version

# Create an IAM OIDC provider for your EKS cluster
eksctl utils associate-iam-oidc-provider --region ${AWS_REGION} --cluster ${CLUSTER_NAME} --approve

# Create an IAM Role for Karpenter
cat < karpenter-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.${AWS_REGION}.amazonaws.com/id/${AWS_OIDC_ID}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.${AWS_REGION}.amazonaws.com/id/${AWS_OIDC_ID}:sub": "system:serviceaccount:karpenter:karpenter"
                }
            }
        }
    ]
}
EOF

# Replace AWS_OIDC_ID with your cluster's OIDC ID
export AWS_OIDC_ID=$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | sed 's/https:\/\/oidc.eks\.us-east-1\.amazonaws\.com\///')
sed -i "s/AWS_OIDC_ID/${AWS_OIDC_ID}/g" karpenter-trust-policy.json

aws iam create-role --role-name Karpenter-${CLUSTER_NAME} --assume-role-policy-document file://karpenter-trust-policy.json
aws iam attach-role-policy --role-name Karpenter-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/KarpenterControllerPolicy-EKS

# Create a Service Account for Karpenter
kubectl create namespace karpenter
kubectl create serviceaccount karpenter --namespace karpenter

# Annotate the Service Account with the IAM Role
kubectl annotate serviceaccount karpenter \
    -n karpenter \
    eks.amazonaws.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/Karpenter-${CLUSTER_NAME}

# Deploy Karpenter using Helm
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace \
    --set serviceAccount.create=false \
    --set serviceAccount.name=karpenter \
    --set settings.aws.clusterName=${CLUSTER_NAME} \
    --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
    --wait

Verify: After installation, ensure Karpenter pods are running.

kubectl get pods -n karpenter

Expected Output:

NAME                         READY   STATUS    RESTARTS   AGE
karpenter-xxxxxxxxx-yyyyy   1/1     Running   0          2m

2. Enable Consolidation in a Provisioner

Karpenter’s consolidation logic is configured at the Provisioner level. A Provisioner defines the types of nodes Karpenter can provision and how it should manage them. To enable consolidation, you simply set consolidation.enabled: true.

There are two main types of consolidation strategies Karpenter employs:

1. Empty Node Consolidation: This is the simplest form. If a node becomes empty (no user pods running on it), Karpenter will terminate it after a specified time. This prevents idle resources from accumulating.
2. Consolidate-by-Rebalancing: Karpenter attempts to find more optimal node configurations for existing pods. This could mean replacing multiple small nodes with a single larger node, or replacing an expensive instance type with a cheaper one if performance characteristics allow. This is driven by the general consolidation.enabled: true flag.

Let’s create a Provisioner that enables consolidation and sets a short ttlSecondsAfterEmpty for demonstration purposes.

apiVersion: karpenter.k8s.aws/v1beta1
kind: Provisioner
metadata:
  name: default
spec:
  # Enable consolidation for this provisioner
  consolidation:
    enabled: true
    # Optional: Wait time before consolidating nodes that are candidates for rebalancing
    # consolidateAfter: 30s 

  # Terminate nodes after 60 seconds if they become empty
  ttlSecondsAfterEmpty: 60

  # Example provisioner configuration (adjust as needed)
  requirements:
    - key: karpenter.k8s.aws/instance-family
      operator: In
      values: ["t3", "m5", "c5"]
    - key: karpenter.k8s.aws/instance-size
      operator: NotIn
      values: ["nano", "micro"]
    - key: kubernetes.io/arch
      operator: In
      values: ["amd64"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"] # Or "spot" for cost savings

  limits:
    resources:
      cpu: 1000
      memory: 1000Gi

  providerRef:
    name: default

---

apiVersion: karpenter.k8s.aws/v1beta1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  amiFamily: AL2 # Or Bottlerocket, Ubuntu, etc.
  instanceProfile: KarpenterNodeInstanceProfile-${CLUSTER_NAME} # Replace with your instance profile
  subnetSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME} # Replace with your cluster name tag
  securityGroupSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME} # Replace with your cluster name tag
  tags:
    karpenter.sh/cluster: ${CLUSTER_NAME}
    karpenter.sh/provisioner-name: default

Explanation:
In this YAML, we define a Provisioner named `default` and an associated AWSNodeTemplate. The critical part for consolidation is within the spec.consolidation block. Setting enabled: true activates Karpenter’s consolidation mechanisms. Additionally, ttlSecondsAfterEmpty: 60 tells Karpenter to terminate any node provisioned by this Provisioner if it remains empty for 60 seconds. This is a very aggressive setting for demonstration; in production, you might want a longer duration (e.g., 5-10 minutes) to account for transient pod restarts or rapid scaling events.

The requirements section defines constraints on the types of instances Karpenter can provision, allowing you to choose certain instance families or exclude others. The providerRef links to an AWSNodeTemplate, which specifies AWS-specific settings like AMI family, instance profile, subnets, and security groups. Ensure your instanceProfile, subnetSelector, and securityGroupSelector tags are correctly configured for your environment.

kubectl apply -f karpenter-provisioner-consolidation.yaml

Verify: Check your Provisioner configuration.

kubectl get provisioners default -o yaml | grep -A 3 "consolidation"

Expected Output:

  consolidation:
    enabled: true
  ttlSecondsAfterEmpty: 60

3. Test Empty Node Consolidation

Let’s deploy a workload that will trigger node provisioning, then scale it down to demonstrate empty node consolidation.

First, deploy a simple Nginx deployment to create demand for a new node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consolidation-test
spec:
  replicas: 5
  selector:
    matchLabels:
      app: consolidation-test
  template:
    metadata:
      labels:
        app: consolidation-test
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources:
          requests:
            cpu: 200m
            memory: 256Mi

kubectl apply -f consolidation-test-deployment.yaml

Verify: Wait for Karpenter to provision a node and for the pods to run on it.

kubectl get nodes -l karpenter.sh/provisioner-name=default
kubectl get pods -l app=consolidation-test -o wide

You should see one or more Karpenter-provisioned nodes and your Nginx pods running on them.

Now, scale down the deployment to zero replicas. This will free up the nodes.

kubectl scale deployment consolidation-test --replicas=0

Verify: Monitor the Karpenter logs and your nodes. You should see Karpenter terminating the nodes after ttlSecondsAfterEmpty (60 seconds in our example).

kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

Look for log entries similar to these:

controller/disruption "level"="debug" "disruption-method"="empty" "type"="consolidation" "node"="ip-xxx-xxx-xxx-xxx.ec2.internal" "provisioner"="default" "zone"="us-east-1a" "reason"="empty for 60s"
controller/disruption "level"="info" "disruption-method"="empty" "type"="consolidation" "node"="ip-xxx-xxx-xxx-xxx.ec2.internal" "provisioner"="default" "zone"="us-east-1a" "id"="xxx-xxx-xxx" "reason"="empty for 60s, terminating node"

And eventually, the node should disappear from your cluster:

kubectl get nodes -l karpenter.sh/provisioner-name=default

Expected Output (after node termination):

No resources found in default namespace.

4. Drift Consolidation

Drift consolidation is a powerful feature that identifies nodes whose configurations (AMI, instance type, tags, etc.) no longer align with the specifications defined in their Provisioner or AWSNodeTemplate. This is crucial for security patching, compliance, and keeping your infrastructure up-to-date. For instance, if you update the AMI referenced in your AWSNodeTemplate, Karpenter can detect that existing nodes are “drifted” and replace them.

Let’s simulate a drift scenario. First, ensure you have a node provisioned by Karpenter.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: drift-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drift-test
  template:
    metadata:
      labels:
        app: drift-test
    spec:
      containers:
      - name: pause
        image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

kubectl apply -f drift-test-deployment.yaml

Verify: A new node should be provisioned.

kubectl get nodes -l karpenter.sh/provisioner-name=default

Now, let’s update the AWSNodeTemplate to use a different AMI family. For example, change AL2 to Bottlerocket. This will cause the existing node to be considered “drifted.”

apiVersion: karpenter.k8s.aws/v1beta1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  amiFamily: Bottlerocket # Changed from AL2
  instanceProfile: KarpenterNodeInstanceProfile-${CLUSTER_NAME}
  subnetSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME}
  securityGroupSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME}
  tags:
    karpenter.sh/cluster: ${CLUSTER_NAME}
    karpenter.sh/provisioner-name: default

kubectl apply -f karpenter-provisioner-consolidation.yaml # Apply the updated NodeTemplate

Explanation: Karpenter continuously monitors nodes against their Provisioner and AWSNodeTemplate definitions. When a mismatch (drift) is detected, and consolidation is enabled, Karpenter will mark the drifted node for termination. It will provision a new, compliant node, cordon and drain the old node, and then terminate it. This ensures your cluster nodes always adhere to your latest specifications.

Verify: Monitor Karpenter logs and node status. You should see a new node being provisioned, pods migrating, and the old node terminating.

kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

Look for log entries indicating drift detection and node replacement:

controller/disruption "level"="debug" "disruption-method"="drift" "type"="consolidation" "node"="ip-xxx-xxx-xxx-xxx.ec2.internal" "provisioner"="default" "zone"="us-east-1a" "reason"="drifted"
controller/disruption "level"="info" "disruption-method"="drift" "type"="consolidation" "node"="ip-xxx-xxx-xxx-xxx.ec2.internal" "provisioner"="default" "zone"="us-east-1a" "id"="xxx-xxx-xxx" "reason"="drifted, terminating node"

You can also check the node events:

kubectl describe node <old-node-name>

You should see events related to cordoning, draining, and eventual deletion.

For more advanced node management and lifecycle, especially concerning network configurations and security, consider exploring tools like Kubernetes Network Policies or even Cilium WireGuard Encryption for Pod-to-Pod Traffic to ensure your node’s network setup is always secure and compliant.

Production Considerations

While consolidation is a powerful cost-saving feature, enabling it in production requires careful planning to avoid service disruptions.

1. PodDisruptionBudgets (PDBs): Ensure all critical applications have PodDisruptionBudgets defined. PDBs tell Kubernetes (and Karpenter) how many disruptions an application can tolerate simultaneously, preventing Karpenter from terminating too many pods at once during consolidation.
2. ttlSecondsAfterEmpty and consolidateAfter:
   • ttlSecondsAfterEmpty: For empty node termination, use a value that balances cost savings with the potential for “node flapping” (nodes terminating and reprovisioning frequently). 5-10 minutes is often a good starting point.
   • consolidateAfter: This setting (part of consolidation) specifies how long Karpenter waits after a node is created before it considers that node for rebalancing consolidation. A longer duration (e.g., 30 minutes to an hour) can prevent newly provisioned nodes from being immediately consolidated if Karpenter later finds a slightly more optimal configuration.
3. Resource Requests and Limits: Accurate resource requests and limits are paramount. Karpenter relies on these to make intelligent scheduling and consolidation decisions. Under-requesting can lead to performance issues, while over-requesting leads to wasted resources and hinders consolidation effectiveness.
4. Node Lifecycle Hooks: For complex shutdown procedures (e.g., gracefully draining stateful applications, uploading logs), consider using Karpenter’s termination hooks (via AWS Launch Templates and EC2 Lifecycle Hooks) to delay node termination until custom scripts complete.
5. Observability: Robust monitoring and logging are essential. Use tools like Prometheus and Grafana to track Karpenter’s metrics (e.g., nodes provisioned, nodes terminated, consolidation events). Review Karpenter controller logs regularly. For deeper insights into your cluster’s networking during consolidation, especially when using advanced CNI like Cilium, consider eBPF Observability with Hubble.
6. Spot Instances: Leverage spot instances with Karpenter. Consolidation works seamlessly with spot, ensuring that if a spot instance is interrupted, Karpenter replaces it, and if it’s underutilized, it will be consolidated. This combination is a major cost-saver.
7. Cluster Autoscaler Coexistence: If migrating from Cluster Autoscaler, ensure it’s fully disabled or configured not to interfere with Karpenter. Running both can lead to unpredictable behavior and increased costs.
8. Testing in Staging: Always test consolidation strategies thoroughly in a staging environment before deploying to production. This helps identify any unexpected behavior or disruptions.

Troubleshooting

Karpenter consolidation can be complex. Here are common issues and their solutions:

1. Karpenter is not consolidating empty nodes.
   • Issue: Nodes remain up even after all pods have been removed.
   • Solution:
     1. Check ttlSecondsAfterEmpty in your Provisioner. Ensure it’s set to a value greater than 0.
     2. Verify that the node truly has no user pods. Sometimes, daemonsets or critical system pods might prevent a node from being considered “empty.” Karpenter only considers pods without node-role.kubernetes.io/control-plane or node-role.kubernetes.io/master labels, and pods with karpenter.sh/do-not-disrupt: "true" annotation are ignored for consolidation.
     3. Check Karpenter controller logs for any errors or warnings related to the node.
2. Karpenter is not performing rebalancing consolidation (e.g., not replacing multiple small nodes with one large one).
   • Issue: Your cluster has fragmented capacity, but Karpenter isn’t taking action.
   • Solution:
     1. Ensure consolidation.enabled: true in your Provisioner.
     2. Check consolidateAfter. If it’s set to a very high value, Karpenter might be waiting before considering nodes for rebalancing. Remove it or set it to a reasonable value for testing.
     3. Verify your Provisioner’s limits and requirements. If they are too restrictive, Karpenter might not find a suitable replacement node.
     4. Karpenter considers the cost of disruption versus the potential savings. If the cost of draining and rescheduling pods outweighs the savings from consolidation (e.g., minimal wasted capacity), it might not consolidate.
     5. Check for PodDisruptionBudgets (PDBs). Restrictive PDBs can prevent consolidation actions.
3. Drift consolidation is not working after updating AMI/NodeTemplate.
   • Issue: Nodes are not being replaced after their underlying configuration has changed.
   • Solution:
     1. Ensure consolidation.enabled: true in your Provisioner.
     2. Verify that the AWSNodeTemplate was successfully updated and that Karpenter has picked up the change.
     3. Check Karpenter logs for messages indicating drift detection. Look for “drifted” in the logs.
     4. Ensure there are no PDBs or other constraints preventing the eviction of pods from the drifted node.
4. Pods are stuck in Pending or are being evicted without new nodes.
   • Issue: During consolidation, pods are evicted, but new nodes don’t appear, or pods remain unschedulable.
   • Solution:
     1. Check Karpenter controller logs for provisioning errors (e.g., “no capacity found,” “insufficient instance capacity,” “invalid launch template”). This often points to issues with your AWSNodeTemplate (subnet, security group, IAM role) or AWS account limits.
     2. Review your Provisioner’s requirements and limits. Are they too restrictive? Does Karpenter have enough options to provision a new node?
     3. Ensure your AWS account has sufficient EC2 instance limits for the instance types Karpenter is trying to provision.
5. Consolidation is too aggressive or not aggressive enough.
   • Issue: Karpenter is terminating nodes too quickly, or not quickly enough to save costs.
   • Solution:
     1. Adjust ttlSecondsAfterEmpty: Lower for more aggressive empty node termination, higher for more grace period.
     2. Adjust consolidateAfter: Lower for more aggressive rebalancing, higher to give nodes more time before considering them for rebalancing.
     3. Review your workload’s resource requests and limits. Accurate requests lead to better packing and more efficient consolidation.
     4. Consider using different Provisioners for different workload types, each with its own consolidation settings.

FAQ Section

1. What is the difference between ttlSecondsAfterEmpty and consolidation.enabled?

   ttlSecondsAfterEmpty is a specific consolidation strategy that terminates nodes after they have been empty for a specified duration. consolidation.enabled: true enables Karpenter’s broader consolidation logic, which includes empty node termination, as well as more intelligent rebalancing strategies (e.g., replacing multiple small nodes with a single larger one, or replacing drifted nodes) to optimize cluster resource usage.

2. Does Karpenter consolidation disrupt running workloads?

   Yes, consolidation, especially rebalancing or drift consolidation, involves terminating existing nodes. Karpenter attempts to gracefully drain nodes by evicting pods according to their PodDisruptionBudgets (PDBs). However, applications without PDBs or with insufficient replica counts may experience brief disruptions. Always define PDBs for critical applications.

3. How can I prevent Karpenter from consolidating a specific node or pod?

   You can prevent a node from being considered for disruption by adding the annotation karpenter.sh/do-not-disrupt: "true" to the node. For pods, adding the same annotation to a pod will prevent Karpenter from evicting it during consolidation, effectively making the node non-empty and delaying its termination. This is useful for sensitive workloads that cannot tolerate any disruption.

4. Can Karpenter consolidate nodes provisioned by the Kubernetes Cluster Autoscaler?

   No, Karpenter only manages nodes it provisions itself. If you are migrating from Cluster Autoscaler to Karpenter, you should ensure that Cluster Autoscaler is disabled or configured not to manage the same node groups/instance types that Karpenter is managing. Running both simultaneously can lead to conflicts and inefficient scaling.

5. How does Karpenter determine the “best” node configuration for consolidation?

   Karpenter considers several factors, including the resource requests of pending pods, the cost of different instance types, and the constraints defined in your Provisioner (e.g., instance families, architectures, capacity types). It aims to find the most cost-effective combination of nodes that can satisfy current and anticipated demand, while minimizing fragmentation and underutilization. This intelligent decision-making is a core strength of Karpenter.

Cleanup Commands

To clean up the resources created during this guide:

# Delete the test deployments
kubectl delete deployment consolidation-test drift-test

# Delete the Karpenter Provisioner and AWSNodeTemplate
kubectl delete provisioner default
kubectl delete awsnodetemplate default

# Uninstall Karpenter Helm chart
helm uninstall karpenter --namespace karpenter

# Delete the Karpenter namespace
kubectl delete namespace karpenter

# Delete the IAM Role and policy created for Karpenter (replace with your role name)
aws iam detach-role-policy --role-name Karpenter-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/KarpenterControllerPolicy-EKS
aws iam delete-role --role-name Karpenter-${CLUSTER_NAME}

# If you created a custom instance profile for Karpenter, delete it
aws ec2 delete-instance-profile --instance-profile-name KarpenterNodeInstanceProfile-${CLUSTER_NAME}

Next Steps / Further Reading

You’ve mastered the basics of Karpenter consolidation! To further enhance your Kubernetes cost optimization and operational efficiency, consider exploring:

• Advanced Provisioner Configuration: Dive deeper into Karpenter’s Provisioner documentation to explore various constraints, taints, and tolerations for fine-grained control over node provisioning.
• Spot Instance Optimization: Combine Karpenter with spot instances for maximum cost savings. Learn how to configure Provisioners to prioritize spot instances.
• Node Termination Handlers: For complex shutdown scenarios, research how to integrate AWS EC2 instance termination handlers with Karpenter to ensure graceful application shutdown.
• Karpenter Metrics and Observability: Set up monitoring for Karpenter using Prometheus and Grafana to track its performance and cost-saving impact. Our guide on eBPF Observability with Hubble can provide insights into network-level metrics, which can be crucial when nodes are frequently churned.
• Kubernetes Workload Resiliency: As consolidation involves node disruption, ensure your applications are resilient. Explore concepts like ReplicaSets, StatefulSets, and Pod Topology Spread Constraints.
• Service Mesh Integration: If you’re running a service mesh like Istio, understand how node lifecycle events interact with sidecar injection and traffic routing. Our Istio Ambient Mesh Production Guide offers insights