Optimize Karpenter Costs: Spot & On-Demand Mix

Prerequisites

Before we begin, ensure you have the following:

• An Amazon EKS cluster running. Karpenter works best with EKS.
• kubectl configured to communicate with your EKS cluster.
• Helm 3 installed.
• AWS CLI v2 installed and configured with appropriate credentials.
• An IAM OIDC provider associated with your EKS cluster. This is standard for EKS and required for IRSA (IAM Roles for Service Accounts).
• Familiarity with Kubernetes concepts like Deployments, Pods, and Custom Resources.

Cluster and Account Setup

First, let's define some environment variables for convenience.

export CLUSTER_NAME="karpenter-spot-demo" # Replace with your EKS cluster name
export AWS_REGION="us-east-1" # Replace with your AWS region
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

echo "Cluster Name: $CLUSTER_NAME"
echo "AWS Region: $AWS_REGION"
echo "AWS Account ID: $ACCOUNT_ID"

Verify that your OIDC provider is set up. If not, you can create it with the AWS CLI:

aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text
# Expected output: https://oidc.eks.${AWS_REGION}.amazonaws.com/id/EXAMPLED999999999999999999999999999999

# If not present, create it:
# eksctl utils associate-iam-oidc-provider --cluster ${CLUSTER_NAME} --approve

Step-by-Step Guide

1. Create Karpenter IAM Roles and Instance Profile

Karpenter requires specific IAM permissions to interact with AWS services like EC2, EC2 Spot, and IAM. We'll create an IAM Role for Karpenter itself (for IRSA) and an Instance Profile that Karpenter will attach to the nodes it provisions. This Instance Profile grants permissions to the EC2 instances.

First, create an IAM role for the Karpenter controller. This role will be assumed by the Karpenter service account in your cluster.

# Create a trust policy for the Karpenter controller role
cat < karpenter-trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.${AWS_REGION}.amazonaws.com/id/$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.${AWS_REGION}.amazonaws.com/id/$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5):sub": "system:serviceaccount:karpenter:karpenter",
          "oidc.eks.${AWS_REGION}.amazonaws.com/id/$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5):aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-role --role-name KarpenterControllerRole-${CLUSTER_NAME} --assume-role-policy-document file://karpenter-trust-policy.json

# Attach the Karpenter controller policy
aws iam attach-role-policy --role-name KarpenterControllerRole-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/KarpenterControllerPolicy

Next, create the EC2 Instance Profile. Karpenter will use this to launch new instances. This profile needs permissions for EC2, ECR, and to register with the EKS cluster.

# Create the instance profile role
aws iam create-role --role-name KarpenterNodeRole-${CLUSTER_NAME} --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}'

# Attach necessary policies
aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore # Optional, for SSM access
aws iam attach-role-policy --role-name KarpenterNodeRole-${CLUSTER_NAME} --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy # If using AWS VPC CNI

# Create the instance profile
aws iam create-instance-profile --instance-profile-name KarpenterNodeInstanceProfile-${CLUSTER_NAME}

# Add the role to the instance profile
aws iam add-role-to-instance-profile --instance-profile-name KarpenterNodeInstanceProfile-${CLUSTER_NAME} --role-name KarpenterNodeRole-${CLUSTER_NAME}

Verify

Confirm the roles and instance profile are created:

aws iam get-role --role-name KarpenterControllerRole-${CLUSTER_NAME} --query Role.Arn --output text
aws iam get-instance-profile --instance-profile-name KarpenterNodeInstanceProfile-${CLUSTER_NAME} --query InstanceProfile.Arn --output text

# Expected output (ARNs will vary):
# arn:aws:iam::123456789012:role/KarpenterControllerRole-karpenter-spot-demo
# arn:aws:iam::123456789012:instance-profile/KarpenterNodeInstanceProfile-karpenter-spot-demo

2. Install Karpenter with Helm

Now we'll install Karpenter into your cluster using Helm. We'll specify the IAM role for the service account and the instance profile for the nodes. Karpenter will also need to know your cluster name and an SQS queue for interruption events.

First, create an SQS queue for Spot interruption and health events. This allows Karpenter to gracefully drain nodes before they are interrupted or terminated.

aws sqs create-queue --queue-name karpenter-${CLUSTER_NAME}

export KARPENTER_IAM_ROLE_ARN=$(aws iam get-role --role-name KarpenterControllerRole-${CLUSTER_NAME} --query Role.Arn --output text)
export INSTANCE_PROFILE_NAME=KarpenterNodeInstanceProfile-${CLUSTER_NAME}
export QUEUE_NAME=karpenter-${CLUSTER_NAME}
export KARPENTER_VERSION="0.33.0" # Check https://github.com/kubernetes-sigs/karpenter/releases for the latest stable version

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="${KARPENTER_IAM_ROLE_ARN}" \
  --set settings.aws.clusterName="${CLUSTER_NAME}" \
  --set settings.aws.defaultInstanceProfile="${INSTANCE_PROFILE_NAME}" \
  --set settings.aws.interruptionQueue="${QUEUE_NAME}" \
  --set settings.aws.defaultInstanceProfile="${INSTANCE_PROFILE_NAME}" \
  --wait

Verify

Check if Karpenter pods are running and the service account has the correct annotation:

kubectl get pods -n karpenter
kubectl get sa karpenter -n karpenter -o yaml | grep "eks.amazonaws.com/role-arn"

# Expected output for pods:
# NAME                         READY   STATUS    RESTARTS   AGE
# karpenter-6789abcd-efghj     1/1     Running   0          2m

# Expected output for service account:
#   eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/KarpenterControllerRole-karpenter-spot-demo

3. Configure AWSNodeTemplate

The AWSNodeTemplate custom resource defines the cloud provider-specific settings for the nodes Karpenter provisions. This includes things like AMI selection, subnets, and security groups. We'll create a generic template that Karpenter can use to provision instances into your EKS cluster.

Karpenter automatically discovers subnets and security groups tagged with karpenter.sh/discovery: ${CLUSTER_NAME}. Ensure your subnets and security groups are correctly tagged. You can do this manually or via eksctl during cluster creation.

# Tag your subnets (example for public subnets)
# Replace subnet-xxxxxxxxxxxxxxxxx with your actual subnet IDs
# aws ec2 create-tags --resources subnet-xxxxxxxxxxxxxxxxx subnet-yyyyyyyyyyyyyyyyy --tags Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}

# Tag your security groups (example for cluster security group)
# Replace sg-xxxxxxxxxxxxxxxxx with your actual security group ID
# aws ec2 create-tags --resources sg-xxxxxxxxxxxxxxxxx --tags Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}

# Create a default AWSNodeTemplate
kubectl apply -f - <

kubectl get awsmodetemplate default -o yaml

# Expected output (truncated):
# apiVersion: karpenter.k8s.aws/v1beta1
# kind: AWSNodeTemplate
# metadata:
#   name: default
# spec:
#   amiSelector:
#     aws.amazon.com/bottlerocket-ami: "true"
#   securityGroupSelector:
#     karpenter.sh/discovery: karpenter-spot-demo
#   subnetSelector:
#     karpenter.sh/discovery: karpenter-spot-demo

kubectl apply -f - <

kubectl get nodepool default -o yaml

# Expected output (truncated):
# apiVersion: karpenter.sh/v1beta1
# kind: NodePool
# metadata:
#   name: default
# spec:
#   disruption:
#     consolidationPolicy: WhenUnderutilized
#     expireAfter: 720h0m0s
#   limits:
#     cpu: "1000"
#     memory: "2000Gi"
#   template:
#     spec:
#       nodeClassRef:
#         name: default
#       requirements:
#       - key: karpenter.sh/capacity-type
#         operator: In
#         values:
#         - spot
#         - on-demand
#       ...

kubectl apply -f - <

kubectl get pods -w # Watch pods until they are running
kubectl get nodes -l karpenter.sh/nodepool=default -o custom-columns=NAME:.metadata.name,CAPACITY_TYPE:.metadata.labels.'karpenter.sh/capacity-type',INSTANCE_TYPE:.metadata.labels.'node.kubernetes.io/instance-type'

# Expected output (example):
# NAME                                                          CAPACITY_TYPE   INSTANCE_TYPE
# ip-10-0-10-100.us-east-1.compute.internal                     spot            c6i.large
# ip-10-0-20-200.us-east-1.compute.internal                     spot            m5.xlarge
# ip-10-0-30-300.us-east-1.compute.internal                     on-demand       r6i.large

kubectl logs -f -n karpenter $(kubectl get pod -n karpenter -l app.kubernetes.io/name=karpenter -o name)