Optimize Cloud Costs: Karpenter Consolidation

Karpenter Consolidation: Mastering Cost Reduction in Kubernetes

In the dynamic world of cloud-native computing, managing Kubernetes clusters efficiently is paramount, especially when it comes to controlling infrastructure costs. Traditional cluster autoscalers often struggle with optimizing node utilization, leading to fragmented resources and unnecessary expenses. This is where Karpenter, an open-source, high-performance Kubernetes cluster autoscaler built by AWS, revolutionizes the game. While Karpenter is well-known for its rapid provisioning capabilities, its consolidation feature is a true game-changer for cost optimization.

Karpenter’s consolidation intelligently identifies opportunities to reduce cluster costs by either removing underutilized nodes or replacing a group of nodes with a smaller, more cost-effective set that can still accommodate all running workloads. This proactive approach ensures that your cluster is always running on the optimal infrastructure, eliminating waste and significantly lowering your cloud bill. In this comprehensive guide, we’ll dive deep into Karpenter consolidation, exploring its mechanisms, configuration, and best practices to help you achieve unparalleled cost savings.

# Install Karpenter (if not already installed)
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version <LATEST_VERSION> \
    --namespace karpenter --create-namespace \
    --set serviceAccount.create=false \
    --set serviceAccount.name=karpenter \
    --wait # Replace <LATEST_VERSION> with the actual version

# Example Provisioner with consolidation enabled
kubectl apply -f - <<EOF
apiVersion: karpenter.sh/v1beta1
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default
  consolidation:
    enabled: true
    garbageCollectionPeriod: 30s # Optional: defines how often consolidation runs
  requirements:
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values: ["c", "m", "r"]
    - key: karpenter.k8s.aws/instance-generation
      operator: Gt
      values: ["5"]
  limits:
    resources:
      cpu: "1000"
      memory: 1000Gi
  provider:
    tags:
      karpenter.sh/discovery: my-cluster
EOF

# Observe Karpenter logs for consolidation events
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

# Check Karpenter events for consolidation details
kubectl get events -n karpenter --field-selector reason=ConsolidateAffected

Prerequisites

Before you embark on optimizing your Kubernetes costs with Karpenter consolidation, ensure you have the following:

• A Running Kubernetes Cluster: This guide assumes you have a functional Kubernetes cluster, preferably on AWS, as Karpenter is highly optimized for AWS environments. While Karpenter supports other clouds, its most mature features are on AWS.
• Karpenter Installed: Karpenter must be installed and configured in your cluster. If you haven’t done so, refer to the official Karpenter documentation for installation instructions. You’ll need an IAM role for Karpenter and the necessary permissions.
• kubectl Configured: The Kubernetes command-line tool (kubectl) should be installed and configured to communicate with your cluster.
• Helm Installed: Helm is the package manager for Kubernetes and is commonly used for installing Karpenter. You can find installation instructions on the Helm website.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Nodes, and Custom Resources (CRDs) is essential.
• AWS CLI Configured (Optional but Recommended): If you’re on AWS, having the AWS CLI configured will be helpful for verifying cloud resources.

Step-by-Step Guide to Karpenter Consolidation

1. Understanding Karpenter Consolidation Mechanisms

Karpenter’s consolidation works by continuously evaluating the cluster’s node utilization and identifying opportunities to save costs. It employs two primary strategies:

• Empty Node Consolidation: This is the simplest form. If a node becomes completely empty (no pods scheduled on it, excluding Karpenter’s own pods or DaemonSets), Karpenter will terminate it. This is crucial for avoiding costs from idle resources.
• Multi-Node Consolidation: This is where Karpenter truly shines. It analyzes groups of nodes and determines if their workloads can be rescheduled onto a smaller, more cost-effective set of nodes. For example, if you have two m5.large instances running a few pods each, Karpenter might decide to replace them with a single m5.xlarge instance, saving you money by reducing the number of running instances. This involves gracefully draining pods from the old nodes before terminating them and provisioning new, optimized nodes.

Karpenter achieves this by simulating potential consolidation actions and selecting the one that offers the highest cost savings while ensuring all pods remain schedulable. This intelligent decision-making process is what makes Karpenter so effective at cost optimization.

2. Installing Karpenter with Consolidation Enabled

If you haven’t already installed Karpenter, you can do so using Helm. Ensure you have the necessary IAM roles and policies set up for Karpenter to interact with your cloud provider (e.g., EC2, ASG on AWS). The service account for Karpenter needs permissions to manage EC2 instances, launch templates, and potentially other resources. For a detailed setup, refer to our guide on Reducing Kubernetes Costs by 60% with Karpenter.

When installing, ensure the correct IAM role is associated with the Karpenter service account. The example below assumes you’ve already created an IAM role named karpenter-controller and associated it with the service account.

# Add Karpenter Helm chart repository
helm repo add karpenter https://charts.karpenter.sh/

# Update Helm repositories
helm repo update

# Install Karpenter (replace <LATEST_VERSION> with the actual version, e.g., v0.32.0)
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version <LATEST_VERSION> \
    --namespace karpenter --create-namespace \
    --set serviceAccount.create=false \
    --set serviceAccount.name=karpenter \
    --set settings.aws.clusterName=my-cluster \
    --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile \
    --set settings.aws.interruptionQueue=my-cluster-sqs-queue \
    --wait

Verify: Check if Karpenter pods are running.

kubectl get pods -n karpenter

Expected Output:

NAME                         READY   STATUS    RESTARTS   AGE
karpenter-7b8c7d9f7-abcde    1/1     Running   0          5m

3. Configuring a Provisioner with Consolidation

The core of Karpenter’s operation lies in its Provisioner Custom Resource. This resource defines how Karpenter provisions nodes, including instance types, labels, taints, and, crucially, consolidation settings. To enable consolidation, you simply set consolidation.enabled: true within your Provisioner specification. You can also define an optional garbageCollectionPeriod to control how frequently Karpenter runs its consolidation logic.

The requirements section is vital, as it dictates what types of instances Karpenter can provision. Be judicious with your instance type choices to allow Karpenter flexibility in consolidation. For instance, using instance-category: ["c", "m", "r"] gives Karpenter a wide range of compute, memory, and general-purpose instances to choose from, which is beneficial for consolidation.

# karpenter-provisioner.yaml
apiVersion: karpenter.sh/v1beta1
kind: Provisioner
metadata:
  name: default
spec:
  providerRef:
    name: default # Refers to a NodeClass for AWS specific configurations
  consolidation:
    enabled: true
    # Optional: How often Karpenter should attempt to consolidate. Default is 30s.
    # A longer period might reduce API calls but delay cost savings.
    garbageCollectionPeriod: 30s 
  requirements:
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values: ["c", "m", "r"] # Allow compute, memory, and general purpose instances
    - key: karpenter.k8s.aws/instance-generation
      operator: Gt
      values: ["5"] # Prefer newer generation instances
    - key: karpenter.k8s.aws/instance-type
      operator: Exists # Allow any instance type that meets other requirements
    - key: topology.kubernetes.io/zone
      operator: In
      values: ["us-east-1a", "us-east-1b", "us-east-1c"] # Restrict to specific AZs
    - key: kubernetes.io/arch
      operator: In
      values: ["amd64"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"] # Use on-demand instances for this provisioner
  limits:
    resources:
      cpu: "1000" # Max total CPU for nodes provisioned by this provisioner
      memory: 1000Gi # Max total memory
  provider:
    tags:
      karpenter.sh/discovery: my-cluster # Tag for Karpenter to discover nodes
---
# nodeclass.yaml (for AWS specific configurations)
apiVersion: karpenter.k8s.aws/v1beta1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  amiFamily: AL2 # Amazon Linux 2
  subnetSelector:
    karpenter.sh/discovery: my-cluster # Select subnets tagged for Karpenter
  securityGroupSelector:
    karpenter.sh/discovery: my-cluster # Select security groups tagged for Karpenter
  instanceProfile: KarpenterNodeInstanceProfile # IAM instance profile for nodes
  tags:
    environment: production
    managed-by: karpenter

kubectl apply -f karpenter-provisioner.yaml
kubectl apply -f nodeclass.yaml

Verify: Check if your Provisioner and AWSNodeTemplate are created.

kubectl get provisioner
kubectl get awsnodetemplate

Expected Output:

NAME      AGE
default   2m

NAME      AGE
default   2m

4. Deploying Workloads to Trigger Provisioning and Consolidation

To see consolidation in action, you need to have workloads that periodically scale up and down, or workloads that are initially spread across many small nodes and could be consolidated onto fewer, larger ones. Let’s deploy a simple Nginx deployment and then scale it down to observe empty node consolidation, followed by deploying several smaller pods to demonstrate multi-node consolidation.

First, deploy a workload that will cause Karpenter to provision a node.

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: consolidation-test
spec:
  replicas: 10
  selector:
    matchLabels:
      app: consolidation-test
  template:
    metadata:
      labels:
        app: consolidation-test
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

kubectl apply -f nginx-deployment.yaml

Verify: Watch for new nodes being provisioned and pods scheduling.

watch kubectl get nodes
watch kubectl get pods -o wide

You should see Karpenter provisioning new nodes (e.g., ip-10-0-x-x.ec2.internal) and your Nginx pods being scheduled on them. Depending on your resource requests, Karpenter might provision one or two nodes.

5. Observing Empty Node Consolidation

Now, let’s scale down our deployment to zero replicas. This will leave the provisioned node(s) empty, triggering Karpenter’s empty node consolidation.

kubectl scale deployment/consolidation-test --replicas=0

Verify: Observe Karpenter terminating the empty node(s).

kubectl get nodes
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

In the Karpenter controller logs, you should see messages indicating consolidation. Look for lines similar to:

...
{"level":"info","ts":"2023-10-27T10:00:00.000Z","caller":"controller/consolidation.go:123","msg":"Consolidating nodes","nodes":1}
{"level":"info","ts":"2023-10-27T10:00:00.000Z","caller":"controller/consolidation.go:198","msg":"Consolidation did not find a consolidation option","nodes":1}
# ... or if it finds a node to terminate
{"level":"info","ts":"2023-10-27T10:00:00.000Z","caller":"controller/consolidation.go:234","msg":"Consolidation succeeded, terminating node","node":"ip-10-0-x-x.ec2.internal"}

The node should eventually disappear from kubectl get nodes output.

6. Observing Multi-Node Consolidation

Multi-node consolidation is more complex. It happens when Karpenter can fit the pods from several existing nodes onto a new, more cost-efficient node (or fewer nodes). Let’s simulate this by deploying multiple smaller pods. Karpenter might initially spread these across a few smaller instances, then consolidate them onto a single larger instance.

# multi-pod-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-pod-consolidation
spec:
  replicas: 20 # Enough pods to potentially spread across multiple nodes
  selector:
    matchLabels:
      app: multi-pod-consolidation
  template:
    metadata:
      labels:
        app: multi-pod-consolidation
    spec:
      containers:
      - name: busybox
        image: busybox:latest
        command: ["sh", "-c", "echo 'Hello from consolidation pod!'; sleep 3600"]
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
          limits:
            cpu: "150m"
            memory: "150Mi"

kubectl apply -f multi-pod-deployment.yaml

Verify: Watch for Karpenter provisioning nodes and then consolidating.

watch kubectl get nodes -o wide
watch kubectl get pods -o wide
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter
kubectl get events -n karpenter --field-selector reason=ConsolidateAffected

Initially, you might see Karpenter provision 2-3 nodes. After a few minutes (depending on your garbageCollectionPeriod and resource usage patterns), Karpenter’s logs and events should show consolidation attempts. Look for:

• Consolidation did not find a consolidation option (if conditions aren’t met yet)
• Consolidation succeeded, terminating nodes (followed by a list of nodes being drained and terminated)
• New nodes being provisioned with a different instance type or fewer nodes overall.

You might see events like:

LAST SEEN   TYPE     REASON                 OBJECT                                    MESSAGE
...
10s         Normal   ConsolidateAffected    node/ip-10-0-x-x.ec2.internal             Node "ip-10-0-x-x.ec2.internal" was consolidated. Terminating.
10s         Normal   ConsolidateAffected    node/ip-10-0-y-y.ec2.internal             Node "ip-10-0-y-y.ec2.internal" was consolidated. Terminating.
5s          Normal   Provisioning         provisioner/default                       Provisioned 1 node(s) for 20 pod(s) with instance type(s) m5.xlarge.

This indicates that Karpenter terminated two nodes and provisioned one new, larger node to host the workloads, successfully consolidating your resources and reducing costs.

Production Considerations

Implementing Karpenter consolidation in a production environment requires careful planning and monitoring to ensure stability and continued cost savings.

1. Pod Disruption Budgets (PDBs): Critical for preventing service outages during consolidation. PDBs define the minimum number or percentage of available pods for a workload. Karpenter respects PDBs during node draining, ensuring your applications remain highly available. For more on ensuring application resilience, consider exploring topics like Kubernetes Network Policies for securing inter-pod communication.
2. Graceful Shutdown: Ensure your applications handle SIGTERM signals gracefully. When Karpenter drains a node, it sends a SIGTERM to pods, giving them time to shut down before termination. Configure appropriate terminationGracePeriodSeconds in your Pod specs.
3. Monitoring and Alerting: Monitor Karpenter’s logs and events closely. Integrate with your observability stack (e.g., Prometheus, Grafana) to track node changes, consolidation events, and resource utilization. Tools like eBPF Observability with Hubble can provide deeper insights into network and application performance during such events.
4. Instance Type Selection: Be strategic with your requirements in the Provisioner. Allowing a wide range of instance types (e.g., multiple instance families and generations) gives Karpenter more flexibility for consolidation. However, be mindful of specific application requirements (e.g., GPU instances for AI workloads, as discussed in LLM GPU Scheduling Guide).
5. Spot Instances: Leverage AWS Spot Instances with Karpenter to further reduce costs. Karpenter handles Spot interruptions gracefully by rescheduling pods.
6. Resource Requests and Limits: Accurate resource requests and limits are crucial. Under-requesting can lead to over-provisioning or poor scheduling, while over-requesting can hinder consolidation by making pods appear larger than they are, making it harder to fit them onto fewer nodes.
7. Node Taints and Tolerations: If you use taints to dedicate nodes for specific workloads, ensure your Provisioners are configured correctly. Karpenter will respect these taints during consolidation.
8. Cluster Autoscaler Coexistence: If you’re migrating from Cluster Autoscaler, ensure it’s fully disabled or configured not to interfere with Karpenter, especially regarding node group management. Running both simultaneously can lead to conflicts and suboptimal behavior.
9. Node Termination Handlers: For certain integrations (e.g., custom metrics agents, security agents), you might need to ensure they gracefully handle node termination. On AWS, Karpenter integrates with AWS Node Termination Handler for managing EC2 Spot interruptions and scheduled maintenance events.

Troubleshooting

Even with a robust tool like Karpenter, issues can arise. Here are common troubleshooting scenarios and their solutions:

1. Karpenter is not consolidating nodes.

   Cause: Consolidation might be disabled, or there are no viable consolidation options.

   Solution:

   • Check Provisioner: Ensure consolidation.enabled: true in your Provisioner.

   kubectl get provisioner default -o yaml | grep -A2 consolidation
   

   • Review Karpenter Logs: Look for messages like “Consolidation did not find a consolidation option”. This usually means Karpenter couldn’t find a cheaper way to run your pods given the current state and Provisioner constraints.

   kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter
   

   • Check PDBs: Strict Pod Disruption Budgets (PDBs) can prevent consolidation if draining pods would violate them.
   • Resource Requests: Ensure your pods have reasonable resource requests. Over-requesting can make it seem like more capacity is needed, preventing consolidation.
2. Karpenter is consolidating too aggressively or too slowly.

   Cause: The garbageCollectionPeriod or other settings are not tuned correctly.

   Solution:

   • Adjust garbageCollectionPeriod: A shorter period (e.g., 15s) will make consolidation more frequent, potentially saving more but increasing API calls. A longer period (e.g., 5m) will be less aggressive.

   # Example: Set consolidation to run every 1 minute
   apiVersion: karpenter.sh/v1beta1
   kind: Provisioner
   metadata:
     name: default
   spec:
     consolidation:
       enabled: true
       garbageCollectionPeriod: 1m0s # 1 minute
   

   • Review Consolidation Limits: If you have ttlSecondsAfterEmpty or ttlSecondsAfterCreation set on your Provisioner, these can also influence node lifecycle.
3. Pods are stuck in Pending during consolidation.

   Cause: Karpenter is draining nodes, but new nodes are not being provisioned quickly enough, or there are issues with the new node’s availability.

   Solution:

   • Check Karpenter Logs: Look for errors during node provisioning (e.g., “failed to launch instance”). This could indicate IAM permission issues, insufficient capacity in an AZ, or incorrect subnet/security group configurations.
   • Monitor AWS Console: Check EC2 instances for launch failures or pending state.
   • Review Provisioner Requirements: Ensure your requirements for instance types, zones, etc., are broad enough to allow Karpenter to find suitable instances.
   • PDBs: Verify PDBs are not overly restrictive, which can prevent pods from being rescheduled.
4. Karpenter deletes nodes but doesn’t replace them with more optimal ones (only empty node consolidation).

   Cause: Karpenter might not be identifying multi-node consolidation opportunities due to constraints or lack of cost-saving potential.

   Solution:

   • Check Provisioner Requirements: Ensure you’re allowing a diverse set of instance types (e.g., instance-category: ["c", "m", "r"]) and generations. If you restrict it too much, Karpenter has fewer options for consolidation.
   • Resource Requests and Limits: If pods have very specific or large requests, it might be harder to pack them efficiently onto fewer nodes.
   • Node Templates/Node Classes: Ensure your AWSNodeTemplate (or equivalent for other clouds) is correctly configured and not imposing unintended restrictions.
   • Karpenter Version: Ensure you are running a recent version of Karpenter, as consolidation logic is continuously improved. Refer to the Karpenter GitHub Releases.
5. Karpenter is not respecting Spot instances or interruption handling.

   Cause: Incorrect configuration for Spot instances or interruption queues.

   Solution:

   • Provisioner Capacity Type: Ensure your Provisioner includes karpe.sh/capacity-type: "spot" for Spot instances.

   apiVersion: karpenter.sh/v1beta1
   kind: Provisioner
   metadata:
     name: spot-provisioner
   spec:
     requirements:
       - key: karpenter.sh/capacity-type
         operator: In
         values: ["spot"]
     # ... other requirements
   

   • Interruption Queue: For AWS, ensure you have an SQS queue configured for interruption events and that Karpenter’s Helm values include settings.aws.interruptionQueue=<your-sqs-queue-name>.
   • IAM Permissions: Karpenter’s IAM role needs permissions to receive messages from the SQS queue.

FAQ Section

1. What is the difference between Karpenter and Cluster Autoscaler?

   Karpenter and Cluster Autoscaler (CA) both scale Kubernetes nodes, but they operate differently. CA works with cloud provider Auto Scaling Groups (ASGs) and scales nodes within pre-defined groups. Karpenter, on the other hand, directly provisions nodes from the cloud provider (e.g., EC2 on AWS) based on individual pod requirements, without needing ASGs. This allows Karpenter to be much faster, more flexible, and more efficient at choosing the right node size and type, leading to better cost optimization, especially with consolidation. For more details on the general concepts of autoscaling, you might find our Karpenter Cost Optimization article useful.

2. How does Karpenter consolidation save money?

   Karpenter saves money primarily in two ways through consolidation:

   • Empty Node Deletion: It identifies and terminates nodes that are no longer hosting any pods, eliminating costs for idle resources.
   • Multi-Node Replacement: It analyzes groups of partially utilized nodes and replaces them with a single, more cost-effective node (or fewer nodes) that can still accommodate all existing workloads. This reduces the number of running instances and often leverages larger, more efficient instance types.

   This proactive optimization ensures you’re only paying for the resources your applications truly need.

3. Can I disable consolidation for specific nodes or workloads?

   Yes, you can. Consolidation is enabled at the Provisioner level. If you have specific workloads or node groups that you don’t want Karpenter to consolidate, you can create a separate Provisioner for them with consolidation.enabled: false. Then, use node selectors or taints and tolerations to direct those specific workloads to nodes provisioned by the non-consolidating Provisioner.

4. What happens to my running pods during consolidation?

   During a multi-node consolidation event, Karpenter performs a graceful draining process. It first provisions new, optimized nodes. Once the new nodes are ready, it cordons the old nodes and then evicts pods from them. Karpenter respects Pod Disruption Budgets (PDBs) to ensure minimum availability for your applications. Pods are then rescheduled onto the newly provisioned nodes. This process aims to minimize downtime, but applications should be designed for resilience to node failures and evictions.

5. Are there any risks to enabling Karpenter consolidation in production?

   While highly beneficial, there are potential risks if not configured carefully:

   • Application Downtime: If PDBs are not configured or are too permissive, consolidation could lead to temporary service disruptions.
   • Resource Starvation: If your Provisioner’s limits are too low or requirements are too restrictive, Karpenter might struggle to find suitable replacement nodes, leading to pods getting stuck in a pending state.
   • Increased API Calls: Frequent consolidation attempts (especially with a very short garbageCollectionPeriod) can lead to increased API calls to your cloud provider.

   It’s crucial to test consolidation thoroughly in a staging environment and monitor its behavior in production. Consider pairing this with robust network configurations, perhaps even with advanced solutions like Cilium WireGuard Encryption for secure pod-to-pod communication on new nodes.

Cleanup Commands

To clean up the resources created during this tutorial, follow these steps:

# 1. Delete the deployments
kubectl delete deployment consolidation-test
kubectl delete deployment multi-pod-consolidation

# 2. Delete the Karpenter Provisioner and AWSNodeTemplate
kubectl delete provisioner default
kubectl delete awsnodetemplate default

# 3. Uninstall Karpenter (if you wish to remove it entirely)
helm uninstall karpenter --namespace karpenter
kubectl delete namespace karpenter

# 4. (Optional) Manually terminate any remaining nodes
# If Karpenter hasn't terminated all nodes, you might need to do this manually
# Use `kubectl get nodes` to find node names and then terminate them via AWS console or CLI.
# Example for AWS CLI:
# aws ec2 terminate-instances --instance-ids <instance-id-1> <instance-id-2>

Next Steps / Further Reading

Congratulations! You’ve successfully explored Karpenter’s consolidation capabilities. To further enhance your Kubernetes cost optimization and operational efficiency, consider delving into these topics:

• Advanced Karpenter Provisioner Configuration: Explore advanced features like ttlSecondsAfterEmpty and ttlSecondsAfterCreation for proactive node cycling and even greater cost control.
• Spot Instance Optimization: Deepen your understanding of how Karpenter leverages Spot instances to save up to 90% on compute costs, and how to combine them with on-demand instances for reliability.
• Cost Monitoring Tools: Integrate Karpenter with cost monitoring tools like Kubecost or AWS Cost Explorer to get granular insights into your Kubernetes spending.
• Node Termination Handlers: Learn more about how Karpenter works with AWS Node Termination Handler to manage EC2 Spot interruptions and scheduled maintenance events gracefully.
• Service Mesh Integration: If you’re running a service mesh like Istio, understand how Karpenter’s rapid node provisioning interacts with sidecar injection and traffic routing. Our Istio Ambient Mesh Production Guide offers insights into modern service mesh deployments.
• Security Best Practices: Review your cluster’s security posture