Cut Cloud Costs: Karpenter Consolidation

Introduction

In the dynamic world of cloud-native computing, optimizing resource utilization is paramount for controlling infrastructure costs. Kubernetes, while powerful, doesn’t inherently manage the underlying compute instances. This is where node autoscalers like Karpenter step in, revolutionizing how we provision and manage nodes. Traditional cluster autoscalers react to pending pods by adding new nodes, but often leave underutilized nodes running, leading to unnecessary expenditure. This is especially true when workloads fluctuate, leaving behind a trail of fragmented and inefficiently packed resources.

Karpenter, an open-source node provisioner built by AWS, takes a more intelligent, event-driven approach. Beyond just provisioning, one of its most powerful features is consolidation. Consolidation actively works to reduce costs by identifying underutilized nodes, rescheduling their pods onto other existing or newly provisioned, more efficient nodes, and then terminating the original nodes. This proactive optimization ensures your cluster is always running on the most cost-effective infrastructure, preventing resource waste and significantly impacting your cloud bill. This guide will walk you through setting up and leveraging Karpenter’s consolidation capabilities to achieve substantial savings.

Prerequisites

Before diving into Karpenter consolidation, ensure you have the following:

• Kubernetes Cluster: An existing Kubernetes cluster (preferably EKS, as Karpenter is heavily optimized for AWS). For this guide, we’ll assume an EKS cluster. Refer to the AWS EKS documentation for cluster creation.
• IAM Permissions: An IAM role for Karpenter with the necessary permissions to launch and terminate EC2 instances, create launch templates, and interact with other AWS services. This role should be associated with Karpenter’s service account. Details can be found in the Karpenter official documentation.
• kubectl: Command-line tool for interacting with your Kubernetes cluster, configured to connect to your EKS cluster.
• helm: Package manager for Kubernetes, used to install Karpenter.
• Basic understanding of Karpenter: Familiarity with Karpenter’s core concepts, such as Provisioners and NodePools (in v0.32.0+), is beneficial. If you’re new to Karpenter, consider reading our guide on Karpenter Cost Optimization.
• AWS CLI: Configured with appropriate credentials to verify AWS resources.

Step-by-Step Guide to Karpenter Consolidation

Step 1: Install Karpenter

First, we need to install Karpenter into your Kubernetes cluster. We’ll use Helm for this. Ensure you replace placeholders like <VERSION>, <ACCOUNT_ID>, <CLUSTER_NAME>, <INSTANCE_PROFILE_NAME>, and <SQS_QUEUE_NAME> with your specific values. The interruptionQueue is crucial for Karpenter to receive Spot Interruption Notifications, enabling it to gracefully handle Spot instance terminations.

The serviceAccount.annotations."eks.amazonaws.com/role-arn" links the Kubernetes Service Account to an IAM Role, an AWS best practice for granting permissions using IAM Roles for Service Accounts (IRSA).

# Add the Karpenter Helm repository
helm repo add karpenter https://charts.karpenter.sh/

# Update Helm repositories
helm repo update

# Install Karpenter (replace placeholders with your values)
# For Karpenter v0.32.0 and later, the Provisioner API has been replaced by NodePool and EC2NodeClass.
# This example uses the older Provisioner API for broader compatibility but note the API change.
# We will use a version that supports Provisioners (e.g., v0.31.x or older) for this example's YAML.
# If using v0.32.0+, refer to Karpenter's official documentation for NodePool/EC2NodeClass.
# For this guide, let's assume a pre-0.32.0 version for Provisioner CRD.
# Example version: v0.31.1

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version v0.31.1 --namespace karpenter --create-namespace \
  --set serviceAccount.annotations."eks.amazonaws.com/role-arn"="arn:aws:iam::<ACCOUNT_ID>:role/KarpenterControllerRole" \
  --set settings.aws.clusterName="<CLUSTER_NAME>" \
  --set settings.aws.defaultInstanceProfile="KarpenterNodeInstanceProfile" \
  --set settings.aws.interruptionQueue="<SQS_QUEUE_NAME>" \
  --wait

Verify:

Check if Karpenter pods are running in the karpenter namespace.

kubectl get pods -n karpenter

Expected Output:

NAME                         READY   STATUS    RESTARTS   AGE
karpenter-xxxxxxxxx-yyyyy    1/1     Running   0          2m

Step 2: Create an IAM Role for Karpenter Nodes

Karpenter needs an IAM instance profile that will be associated with the EC2 instances it launches. This profile grants permissions to the nodes (EC2 instances) themselves, allowing them to join the cluster and perform actions like sending logs to CloudWatch.

Create an IAM policy (e.g., KarpenterNodePolicy) and an IAM role (e.g., KarpenterNodeRole) with an instance profile (e.g., KarpenterNodeInstanceProfile). The policy should include permissions like ec2:DescribeImages, ec2:RunInstances, ec2:TerminateInstances, etc., as well as permissions for EKS to join the cluster. A detailed policy can be found in the Karpenter documentation.

# Example IAM Policy (create this in AWS IAM console or via CLI)
# Policy Name: KarpenterNodePolicy
cat <<EOF > karpenter-node-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AssociateAddress",
                "ec2:AssociateIamInstanceProfile",
                "ec2:AttachVolume",
                "ec2:CreateFleet",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateTags",
                "ec2:DeleteLaunchTemplate",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeImages",
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeInstances",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSpotPriceHistory",
                "ec2:DescribeSubnets",
                "ec2:DescribeTags",
                "ec2:DetachVolume",
                "ec2:DisassociateAddress",
                "ec2:ModifyInstanceAttribute",
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "ssm:GetParameter",
                "iam:PassRole",
                "eks:DescribeCluster"
            ],
            "Resource": "*"
        }
    ]
}
EOF

# Create the policy
aws iam create-policy --policy-name KarpenterNodePolicy --policy-document file://karpenter-node-policy.json

# Create the instance profile (if not already created) and attach the role
# First, create the IAM role
aws iam create-role --role-name KarpenterNodeRole --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }'

# Attach the policy to the role
aws iam attach-role-policy --role-name KarpenterNodeRole --policy-arn arn:aws:iam::<ACCOUNT_ID>:policy/KarpenterNodePolicy

# Create the instance profile
aws iam create-instance-profile --instance-profile-name KarpenterNodeInstanceProfile

# Add the role to the instance profile
aws iam add-role-to-instance-profile --instance-profile-name KarpenterNodeInstanceProfile --role-name KarpenterNodeRole

# Verify:
aws iam get-instance-profile --instance-profile-name KarpenterNodeInstanceProfile

Verify:

Confirm the instance profile exists and has the role attached.

aws iam get-instance-profile --instance-profile-name KarpenterNodeInstanceProfile

Expected Output: (partial)

{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "KarpenterNodeInstanceProfile",
        "InstanceProfileId": "AIPAXxxxxxxxxxxxxxxx",
        "Arn": "arn:aws:iam::<ACCOUNT_ID>:instance-profile/KarpenterNodeInstanceProfile",
        "CreateDate": "2023-10-27T10:00:00Z",
        "Roles": [
            {
                "Path": "/",
                "RoleName": "KarpenterNodeRole",
                "RoleId": "AROAxxxxxxxxxxxxxxx",
                "Arn": "arn:aws:iam::<ACCOUNT_ID>:role/KarpenterNodeRole",
                "CreateDate": "2023-10-27T09:55:00Z",
                "AssumeRolePolicyDocument": {
                    // ...
                }
            }
        ]
    }
}

Step 3: Define a Karpenter Provisioner with Consolidation

The Provisioner (or NodePool in newer versions) is Karpenter’s custom resource that defines the types of nodes it can provision. To enable consolidation, you simply set consolidation.enabled: true within your Provisioner specification. Karpenter will then actively monitor your nodes for consolidation opportunities.

Consolidation works in a few ways:

• empty: Terminates nodes that have no running pods (except daemonsets, etc.). This is enabled by ttlSecondsAfterEmpty.
• drift: Replaces nodes that have drifted from their desired state (e.g., AMI updates, security group changes).
• consolidation (general): Identifies opportunities to pack pods more densely onto fewer nodes or replace existing nodes with smaller, cheaper instance types if possible. This is the primary cost-saving mechanism.

For more advanced node management, you might consider how Karpenter interacts with other networking solutions like Cilium WireGuard Encryption, ensuring that consolidation doesn’t disrupt network policies or security configurations.

apiVersion: karpenter.sh/v1beta1
kind: Provisioner
metadata:
  name: default
spec:
  # Enable consolidation for cost optimization
  consolidation:
    enabled: true
    # Optionally, specify specific consolidation types (all are enabled by default if 'enabled: true')
    # gc: # Garbage collection (similar to empty, but more comprehensive)
    #   enabled: true
    # empty: # Terminates empty nodes
    #   enabled: true
    # drift: # Replaces drifted nodes
    #   enabled: true
  requirements:
    - key: kubernetes.io/arch
      operator: In
      values: ["amd64"]
    - key: kubernetes.io/os
      operator: In
      values: ["linux"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand", "spot"] # Allow both on-demand and spot instances
    - key: karpenter.sh/provisioner-name
      operator: In
      values: ["default"] # Link to this provisioner
  limits:
    resources:
      cpu: 1000 # Max 1000 CPU cores for this provisioner
      memory: 1000Gi # Max 1000 GiB memory for this provisioner
  providerRef:
    name: default # Refers to the EC2NodeClass (if using v0.32.0+) or implicit provider for older versions
  ttlSecondsAfterEmpty: 30 # Terminate empty nodes after 30 seconds
  ttlSecondsAfterNotReady: 300 # Terminate not-ready nodes after 5 minutes
  # You can also set specific instance families or types here, e.g.
  # instanceFamily: c5,m5,r5
  # instanceType: c5.large,m5.large,r5.large
  # amiSelector:
  #   aws-ids: "ami-xxxxxxxxxxxxxxxxx" # Specify custom AMI if needed
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2 # Amazon Linux 2
  role: KarpenterNodeRole # IAM role for the node
  securityGroupSelector:
    karpenter.sh/discovery: <CLUSTER_NAME> # Select security groups tagged for your cluster
  subnetSelector:
    karpenter.sh/discovery: <CLUSTER_NAME> # Select subnets tagged for your cluster
  tags:
    karpenter.sh/provisioner-name: default
    environment: production
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 20Gi
        volumeType: gp3
        deleteOnTermination: true

Apply this configuration:

kubectl apply -f provisioner.yaml

Verify:

Check if the Provisioner is created.

kubectl get provisioners

Expected Output:

NAME      AGE
default   1m

Step 4: Deploy a Workload to Trigger Scaling and Consolidation

To see consolidation in action, we need to deploy a workload that will first trigger Karpenter to provision nodes, and then scale it down or introduce an inefficient state to allow consolidation to occur. Let’s deploy a simple Nginx deployment and then scale it down.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consolidation-test
spec:
  replicas: 10 # Start with 10 replicas to ensure multiple nodes are provisioned
  selector:
    matchLabels:
      app: consolidation-test
  template:
    metadata:
      labels:
        app: consolidation-test
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: nginx
        image: nginx
        resources:
          requests:
            cpu: 200m
            memory: 200Mi
      # Optional: Add a toleration to allow it to run on Karpenter-provisioned nodes
      # You might also want to add nodeSelectors if you have specific node requirements
      tolerations:
      - key: karpenter.sh/provisioner-name
        operator: Exists
      - key: karpenter.sh/capacity-type
        operator: Exists

Apply the deployment:

kubectl apply -f deployment.yaml

Verify:

Watch for Karpenter to provision new nodes and for your pods to schedule.

watch kubectl get nodes,pods -o wide

You should see new nodes (e.g., ip-xxx-xxx-xxx-xxx.ec2.internal) appear, provisioned by Karpenter, and your consolidation-test pods running on them. Look for the karpenter.sh/provisioner-name=default label on the nodes.

Step 5: Trigger Consolidation

Now, let’s scale down the deployment to create underutilized nodes. Karpenter will detect this and initiate consolidation.

kubectl scale deployment/consolidation-test --replicas=1

Verify:

Monitor Karpenter logs and cluster nodes. You should see Karpenter rescheduling pods and eventually terminating the underutilized nodes.

kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

Look for log entries indicating consolidation actions, such as “Discovered empty node,” “Consolidating nodes,” or “Terminating node.”

# Example log output indicating consolidation
# ...
# 2023-10-27T10:30:05.123Z INFO controller.consolidation Discovered 1 empty nodes, attempting to terminate {"nodes": ["ip-172-20-x-x.ec2.internal"]}
# 2023-10-27T10:30:10.567Z INFO controller.consolidation Triggering termination for node ip-172-20-x-x.ec2.internal
# 2023-10-27T10:30:15.987Z INFO controller.provisioning Provisioned new node ip-172-20-y-y.ec2.internal (t3.medium) for consolidation
# 2023-10-27T10:30:20.111Z INFO controller.consolidation Successfully consolidated 2 nodes into 1 node. Terminating [ip-172-20-a-a.ec2.internal ip-172-20-b-b.ec2.internal] and launching [ip-172-20-y-y.ec2.internal]
# ...

Also, observe your nodes:

watch kubectl get nodes -o wide

You should see some nodes enter a NotReady state and then be removed, while others might be replaced by different instance types if Karpenter finds a more efficient packing.

This proactive management of nodes is a cornerstone of cost optimization, especially when combined with advanced networking features like those discussed in Kubernetes Network Policies: Complete Security Hardening Guide, ensuring that your leaner infrastructure remains secure.

Production Considerations

Implementing Karpenter consolidation in production requires careful planning and testing to ensure stability and efficiency.

1. Pod Disruption Budgets (PDBs): Ensure your critical applications have Pod Disruption Budgets configured. PDBs protect your applications during voluntary disruptions, including those caused by Karpenter’s consolidation. Without PDBs, Karpenter might evict too many pods simultaneously, leading to application downtime.
2. Graceful Termination: Configure appropriate terminationGracePeriodSeconds for your pods. This gives your applications time to shut down gracefully before a node is terminated, preventing data loss or abrupt service interruptions. Karpenter respects this setting during node termination.
3. Resource Requests and Limits: Accurate resource requests and limits are crucial. Karpenter relies on these values to make intelligent scheduling and consolidation decisions. Under-requesting can lead to overprovisioning of nodes, while over-requesting can lead to underutilization.
4. Node Taints and Tolerations: Be mindful of how taints and tolerations interact with Karpenter. If you have custom taints on your nodes, ensure your pods have corresponding tolerations. Karpenter itself adds taints during node initialization and termination to prevent new pods from scheduling on nodes that are about to be terminated.
5. Monitoring and Alerting: Implement robust monitoring for Karpenter itself (e.g., its metrics via Prometheus) and your cluster’s node utilization. Set up alerts for unexpected node terminations, provisioner failures, or high pod pending rates, which could indicate issues with consolidation or provisioning. Tools like eBPF Observability with Hubble can provide deep insights into network and system activity during node lifecycle events.
6. Cost Monitoring: Integrate Karpenter’s cost-saving efforts with your cloud cost management tools. Monitor the actual cost savings achieved through consolidation to validate its effectiveness. AWS Cost Explorer, for instance, can be used to track EC2 instance costs.
7. Security Groups and Subnets: Ensure your EC2NodeClass (or Provisioner) correctly references the security groups and subnets required for your nodes to operate within your VPC and communicate with the EKS control plane and other services. Incorrect configuration can lead to nodes failing to join the cluster or pods failing to communicate.
8. AMI Management: If you use custom AMIs, ensure your amiSelector in the EC2NodeClass is correctly configured. Keep your AMIs updated for security patches and performance improvements. Karpenter supports AMI drift detection, which can automatically replace nodes using outdated AMIs.
9. Testing Consolidation: Before rolling out to production, thoroughly test consolidation in a staging environment. Simulate various workload patterns (scale up, scale down, fluctuating loads) to observe how Karpenter reacts and ensure it meets your performance and cost objectives.
10. Karpenter Version Management: Stay updated with Karpenter releases. New versions often bring performance improvements, bug fixes, and new features. However, always review release notes for breaking changes, especially regarding API versions (e.g., the transition from Provisioner to NodePool/EC2NodeClass in v0.32.0).

Troubleshooting

Here are some common issues you might encounter with Karpenter consolidation and their solutions:

1. Issue: Nodes are not consolidating, even when underutilized.

   Solution:

   1. Check Karpenter Logs: The most crucial step is to examine Karpenter controller logs for insights. Look for messages related to consolidation.

      kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

   2. Verify Provisioner Configuration: Ensure consolidation.enabled: true is set in your Provisioner (or NodePool).

      kubectl get provisioner default -o yaml | grep -A 2 consolidation

   3. Pod Disruption Budgets (PDBs): PDBs can sometimes prevent consolidation if they’re too restrictive, not allowing enough pods to be evicted. Check your PDBs and ensure they allow for some disruption.

      kubectl get pdb -A

   4. Pods Preventing Eviction: Some pods might have pod.kubernetes.io/do-not-evict: "true" annotation or have a PDB that prevents their eviction. Check pods on the node Karpenter is trying to consolidate.
   5. ttlSecondsAfterEmpty: For empty node termination, ensure ttlSecondsAfterEmpty is set to a reasonable value (e.g., 30s-300s). If it’s not set or too high, empty nodes will take longer to terminate.
2. Issue: Karpenter is terminating nodes but not replacing them with more efficient ones (or the cluster becomes under-resourced).

   Solution:

   1. Provisioner Constraints: Review your Provisioner’s requirements and limits. If they are too restrictive (e.g., only allowing specific instance types that are currently unavailable or too expensive), Karpenter might not find suitable replacement nodes.

      kubectl get provisioner default -o yaml

   2. AWS Resource Limits: Check your AWS account’s EC2 instance limits. If you hit a limit, Karpenter cannot provision new nodes.
   3. Subnet/Security Group Issues: Ensure the subnets and security groups referenced in your EC2NodeClass (or Provisioner) are correctly configured and have sufficient capacity/IP addresses.
3. Issue: Pods are stuck in Pending state after consolidation attempts.

   Solution:

   1. Karpenter Logs: Check Karpenter logs for any errors during provisioning or scheduling.

      kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter

   2. Pod Events: Describe the pending pods to see their events. This will often indicate why they can’t be scheduled (e.g., insufficient CPU/memory, taints, volume issues).

      kubectl describe pod <POD_NAME> -n <NAMESPACE>

   3. Provisioner Limits: Ensure your Provisioner’s limits are not preventing Karpenter from provisioning enough resources.
   4. Node Taints: If Karpenter provisions nodes with specific taints, ensure your pending pods have corresponding tolerations.
4. Issue: Karpenter is terminating nodes with critical pods, causing downtime.

   Solution:

   1. PDBs are Essential: This is almost always due to missing or incorrectly configured Pod Disruption Budgets. Create or adjust PDBs for your critical applications to ensure a minimum number of replicas are available. For example:

      apiVersion: policy/v1
      kind: PodDisruptionBudget
      metadata:
        name: my-critical-app-pdb
      spec:
        minAvailable: 1 # Or a percentage like 50%
        selector:
          matchLabels:
            app: my-critical-app

   2. terminationGracePeriodSeconds: Ensure your critical pods have a sufficient terminationGracePeriodSeconds to gracefully shut down.
5. Issue: Karpenter is provisioning too many nodes or nodes that are too large/small.

   Solution:

   1. Resource Requests/Limits: Re-evaluate your pod’s resource requests and limits. Karpenter provisions nodes based on these. If requests are too low, Karpenter might provision smaller nodes that quickly fill up, leading to more nodes than necessary. If requests are too high, Karpenter might provision larger nodes than needed.
   2. Provisioner requirements: Adjust your Provisioner’s requirements to guide Karpenter towards more optimal instance types. For instance, restrict it to specific instance families (e.g., c5.*, m5.*) or exclude very small/large types if they don’t fit your workload profile.

        requirements:
          - key: karpenter.sh/instance-family
            operator: In
            values: ["c5", "m5"] # Only allow C5 and M5 instance families

   3. Consolidation Thresholds: While not directly configurable for consolidation, Karpenter’s internal algorithms consider node utilization. If you consistently see sub-optimal packing, it might point to issues with pod requests or node sizing preferences.

FAQ Section

1. What is Karpenter consolidation and how does it save costs?

   Karpenter consolidation is a feature that proactively optimizes node utilization in your Kubernetes cluster. It saves costs by identifying underutilized nodes (e.g., nodes with few pods, low CPU/memory usage, or nodes that can be replaced by a smaller, cheaper instance type), rescheduling their pods onto other existing or newly provisioned, more efficient nodes, and then terminating the original, inefficient nodes. This ensures your cluster runs on the minimum necessary and most cost-effective infrastructure.

2. What’s the difference between Karpenter consolidation and a traditional cluster autoscaler’s scale-down?

   Traditional cluster autoscalers primarily focus on scaling up when pods are pending and scaling down empty nodes after a certain grace period. Karpenter’s consolidation goes further. Besides terminating empty nodes (via ttlSecondsAfterEmpty), it can also:
   a) De-fragment: Consolidate pods from multiple partially utilized nodes onto fewer, more densely packed nodes.
   b) Replace: Replace larger, more expensive nodes with smaller, cheaper ones if the workload can fit.
   c) Drift: Replace nodes that have configuration drift (e.g., outdated AMI). This proactive and intelligent optimization is a key differentiator for cost savings.

3. How can I ensure my applications are not disrupted during consolidation?

   The most important mechanism is to implement Pod Disruption Budgets (PDBs) for your critical applications. PDBs specify the minimum number or percentage of pods that must be available during voluntary disruptions. Additionally, configure appropriate terminationGracePeriodSeconds for your pods to allow them to shut down gracefully. Karpenter respects these Kubernetes-native mechanisms.

4. Can Karpenter consolidate Spot Instances?

   Yes, Karpenter is highly effective with AWS Spot Instances. It can provision Spot Instances and also consolidate them. When a Spot Instance receives an interruption notice, Karpenter attempts to drain the node and provision a replacement (either Spot or On-Demand, depending on