Kubernetes Spot Instances: Scale Cheaply

Introduction

In the dynamic world of cloud-native applications, cost optimization is often as critical as performance and reliability. Kubernetes, while offering unparalleled scalability and resilience, can quickly become a significant line item in your cloud bill. This is particularly true for workloads that can tolerate interruptions, such as batch jobs, stateless services, or development/testing environments. Enter Kubernetes Spot Instances – a powerful yet often underutilized strategy to dramatically reduce your infrastructure costs.

Spot Instances (or Spot VMs, Preemptible VMs, etc., depending on your cloud provider) offer steep discounts compared to on-demand instances, sometimes up to 90% off. The catch? They can be reclaimed by the cloud provider with short notice (typically 30 seconds to 2 minutes). Successfully leveraging these ephemeral resources in a Kubernetes cluster requires careful planning, robust eviction handling, and intelligent scheduling. This guide will walk you through the process of integrating Spot Instances into your Kubernetes cluster, ensuring you can harness their cost-saving potential without sacrificing application stability for suitable workloads.

By the end of this tutorial, you’ll understand how to configure your cluster to intelligently use Spot Instances, how to make your applications resilient to preemption, and how to monitor your cost savings. We’ll focus on practical, actionable steps that you can implement today to start optimizing your Kubernetes infrastructure spend.

Prerequisites

Before diving into the implementation, ensure you have the following:

• A running Kubernetes cluster: This guide assumes you have an existing cluster on a major cloud provider (AWS EKS, GCP GKE, Azure AKS). While the concepts are universal, the implementation details for provisioning Spot instances will be cloud-specific.
• kubectl configured: Command-line tool for interacting with your Kubernetes cluster.
• Cloud Provider CLI: AWS CLI, gcloud CLI, or Azure CLI installed and configured for your respective cloud.
• Basic understanding of Kubernetes concepts: Pods, Deployments, Services, Nodes, Taints, and Tolerations.
• Familiarity with your cloud provider’s Spot instance offerings: Understand their pricing models, interruption policies, and provisioning methods.
• Administrative access to your Kubernetes cluster and cloud account to create/modify node groups and IAM policies.

Step-by-Step Guide: Integrating Spot Instances into Kubernetes

Step 1: Provisioning Spot Node Groups

The first step is to create dedicated node groups composed entirely of Spot Instances. It’s generally recommended to isolate Spot Instances from your on-demand instances to manage their lifecycle and workload scheduling more effectively. Most cloud providers offer mechanisms to do this.

For AWS EKS, you can use managed node groups with a Spot allocation strategy or use tools like Karpenter for more advanced, JIT provisioning. For GCP GKE, you’d create a node pool with Preemptible VMs. Azure AKS allows you to create node pools with Spot instances.

Example: AWS EKS Managed Node Group (Spot)

This example uses eksctl to create a managed node group consisting of Spot instances. We’ll also add a custom label and a taint to these nodes, which will be crucial for scheduling workloads later.

# Replace with your cluster name, region, and desired instance types
CLUSTER_NAME="my-kubezilla-cluster"
REGION="us-east-1"
NODEGROUP_NAME="spot-nodes"
INSTANCE_TYPES="m5.large,m5.xlarge" # Recommend multiple instance types for better Spot availability

eksctl create nodegroup \
  --cluster=$CLUSTER_NAME \
  --region=$REGION \
  --name=$NODEGROUP_NAME \
  --node-type=$INSTANCE_TYPES \
  --nodes-min=0 \
  --nodes-max=10 \
  --nodes=0 \
  --spot \
  --label "lifecycle=Spot" \
  --taint "spot-instance=true:NoSchedule" \
  --ssh-access # Optional, for debugging

Example: GCP GKE Node Pool (Preemptible VMs)

For GKE, you can add a new node pool with preemptible VMs using gcloud. Similar to AWS, we’ll add a label and a taint.

# Replace with your cluster name, zone, and project ID
CLUSTER_NAME="my-gke-cluster"
ZONE="us-central1-c"
PROJECT_ID="your-gcp-project-id"
NODEPOOL_NAME="spot-pool"
MACHINE_TYPE="e2-medium"

gcloud container node-pools create $NODEPOOL_NAME \
  --cluster=$CLUSTER_NAME \
  --zone=$ZONE \
  --project=$PROJECT_ID \
  --machine-type=$MACHINE_TYPE \
  --num-nodes=0 \
  --min-nodes=0 \
  --max-nodes=10 \
  --preemptible \
  --node-labels="lifecycle=Spot" \
  --node-taints="spot-instance=true:NoSchedule" \
  --enable-autoscaling

Verify: Node Group Creation

After running the command, it might take a few minutes for the node group to provision (though it will start with 0 nodes if autoscaling is enabled). You can check its status using your cloud provider’s console or CLI. Once ready, Kubernetes should eventually recognize any nodes that spin up.

kubectl get nodes -l lifecycle=Spot

Expected Output (initially empty if nodes=0, or showing nodes if created):

# If no nodes are currently running in the spot pool (due to min-nodes=0 and no pending pods)
No resources found

# If nodes have spun up
NAME                                         STATUS   ROLES    AGE   VERSION
ip-192-168-XX-XXX.us-east-1.compute.internal Ready    <none>   5m    v1.27.x

Step 2: Configuring Cluster Autoscaler or Karpenter

To dynamically scale your Spot node groups based on workload demand, you’ll need an autoscaling solution. The Kubernetes Cluster Autoscaler is the traditional choice, while Karpenter is a newer, more efficient alternative, especially powerful for heterogeneous node types and cost optimization. We highly recommend exploring Karpenter for advanced cost optimization, but we’ll cover Cluster Autoscaler for broader applicability.

Using Cluster Autoscaler with Spot Node Groups

The Cluster Autoscaler needs to be configured to recognize your Spot node groups and respect their properties (like the spot-instance=true:NoSchedule taint). When a pod requires a node with specific tolerations or node selectors, and no existing nodes satisfy the request, the Cluster Autoscaler will attempt to scale up an appropriate node group.

Deploy Cluster Autoscaler into your cluster. The deployment manifest varies by cloud provider. Here’s a conceptual example for AWS EKS. You’d typically get the full manifest from the official Cluster Autoscaler GitHub repository.

# Partial example: Cluster Autoscaler Deployment for AWS EKS
# Full manifest available at: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
        - image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.27.x # Use appropriate version for your K8s
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-system-pods=false # Important for certain setups
            - --expander=least-waste # Or random, most-pods
            # Add discovery for your node groups
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-kubezilla-cluster
            # Ensure proper IAM role is attached to the service account
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux

Verify: Cluster Autoscaler Logs

Check the Cluster Autoscaler logs to ensure it’s running correctly and detecting your node groups.

kubectl logs -f deployment/cluster-autoscaler -n kube-system

Expected Output (look for lines indicating node group discovery):

I1026 14:30:00.123456       1 aws_manager.go:456] Registering ASG: my-kubezilla-cluster-spot-nodes-XXXXXXXX
I1026 14:30:00.123456       1 aws_manager.go:456] Registering ASG: my-kubezilla-cluster-on-demand-nodes-YYYYYYYY

Step 3: Scheduling Workloads onto Spot Instances

Now that you have Spot node groups configured with taints and labels, you need to tell your Kubernetes workloads to use them. This is done using tolerations for the taint and nodeSelector or nodeAffinity for the label.

Using Taints and Tolerations

The spot-instance=true:NoSchedule taint on your Spot nodes prevents pods without the corresponding toleration from being scheduled there. This ensures your critical, non-interruptible workloads don’t accidentally land on Spot instances.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-job-processor
  labels:
    app: batch-job-processor
spec:
  replicas: 3
  selector:
    matchLabels:
      app: batch-job-processor
  template:
    metadata:
      labels:
        app: batch-job-processor
    spec:
      containers:
      - name: processor
        image: busybox:1.36
        command: ["sh", "-c", "echo 'Processing data on a spot instance...'; sleep 3600"]
      tolerations:
      - key: "spot-instance"
        operator: "Exists" # Matches any value for key "spot-instance"
        effect: "NoSchedule"
      # Optionally, use nodeSelector for more precise targeting
      nodeSelector:
        lifecycle: Spot

Explanation: This Deployment defines a simple batch job. The tolerations section allows this pod to be scheduled on nodes that have the spot-instance=true:NoSchedule taint. The nodeSelector further refines this by ensuring it only lands on nodes explicitly labeled with lifecycle: Spot. This dual approach provides robust control.

Verify: Pod Scheduling

Apply the Deployment and observe where the pods are scheduled. If no Spot nodes are available, the Cluster Autoscaler should provision new ones.

kubectl apply -f batch-job-processor.yaml
kubectl get pods -l app=batch-job-processor -o wide

Expected Output (showing pods running on Spot nodes):

NAME                                  READY   STATUS    RESTARTS   AGE   IP             NODE                                         NOMINATED NODE   READINESS GATES
batch-job-processor-7c6f8f55c-abcde   1/1     Running   0          2m    192.168.1.10   ip-192-168-XX-XXX.us-east-1.compute.internal   <none>           <none>
batch-job-processor-7c6f8f55c-fghij   1/1     Running   0          2m    192.168.1.11   ip-192-168-YY-YYY.us-east-1.compute.internal   <none>           <none>

Step 4: Making Applications Resilient to Preemption

The core challenge with Spot Instances is their ephemeral nature. Your applications must be designed to handle sudden interruptions gracefully. This involves two main aspects: graceful shutdown and checkpointing/state management.

Graceful Shutdown (SIGTERM Handling)

When a Spot Instance is marked for termination, the cloud provider typically sends a notification (e.g., EC2 Spot Instance Interruption Notice, GCP Preemptible VM Termination Notice). The Kubernetes node agent (kubelet) receives this and initiates a graceful shutdown process for pods on that node. It sends a SIGTERM signal to containers, giving them a configurable terminationGracePeriodSeconds (default 30 seconds) to shut down cleanly.

Your application code should listen for SIGTERM and:

• Stop accepting new connections/tasks.
• Finish any in-progress work.
• Flush logs and metrics.
• Release resources.
• Exit.

Example: Python Application with SIGTERM Handling

import signal
import sys
import time
import os

running = True

def signal_handler(signum, frame):
    global running
    print(f"[{os.getpid()}] Received signal {signum}. Initiating graceful shutdown...")
    running = False

signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler) # Also handle Ctrl+C for local testing

print(f"[{os.getpid()}] Application started, doing some work...")
while running:
    # Simulate doing some work
    print(f"[{os.getpid()}] Working hard...")
    time.sleep(5) # Simulate work interval
    if not running:
        break # Exit if signal received during sleep

print(f"[{os.getpid()}] Graceful shutdown complete. Exiting.")
sys.exit(0)

Kubernetes Deployment Configuration for Graceful Shutdown

Ensure your Kubernetes Deployment provides enough terminationGracePeriodSeconds for your application to shut down gracefully. For very long-running tasks, you might need more than the default 30 seconds, but keep it reasonable to avoid delaying node termination.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: resilient-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: resilient-worker
  template:
    metadata:
      labels:
        app: resilient-worker
    spec:
      terminationGracePeriodSeconds: 60 # Give the app 60 seconds to shut down
      containers:
      - name: worker
        image: my-repo/resilient-python-app:latest # Replace with your image
        command: ["python", "app.py"]
      tolerations:
      - key: "spot-instance"
        operator: "Exists"
        effect: "NoSchedule"
      nodeSelector:
        lifecycle: Spot

Checkpointing and State Management

For applications that maintain state or perform long computations, graceful shutdown might not be enough. You need to consider:

• Externalizing State: Avoid in-memory state. Use external databases (e.g., PostgreSQL, MongoDB), message queues (e.g., Kafka, RabbitMQ), or distributed caches (e.g., Redis).
• Checkpointing: Regularly save progress to persistent storage (e.g., S3, GCS, network file systems like EFS) so that a new instance can resume from the last checkpoint.
• Idempotency: Design operations to be idempotent, meaning performing them multiple times has the same effect as performing them once. This prevents issues if a task is restarted and partially re-processed.

For more details on persistent storage in Kubernetes, refer to the official Kubernetes Persistent Volumes documentation.

Step 5: Monitoring and Cost Analysis

Once you have Spot Instances integrated, it’s crucial to monitor their performance, preemption rates, and actual cost savings. Cloud providers offer tools for this, and Kubernetes provides metrics.

Cloud Provider Cost Explorer/Billing Dashboards:

Regularly check your cloud provider’s cost management tools (e.g., AWS Cost Explorer, GCP Billing Reports, Azure Cost Management). Filter by instance type, lifecycle (on-demand vs. spot), and tags (e.g., kubernetes.io/cluster/my-kubezilla-cluster) to see the savings generated by Spot Instances.

Kubernetes Metrics:

Monitor pod restarts, node evictions, and application availability using Prometheus/Grafana or your preferred observability stack. Increased restarts on Spot nodes might indicate your applications are not handling preemption well or that the workload is not suitable for Spot.

You can use eBPF Observability with Hubble to gain deeper insights into network traffic and application behavior, which can help in identifying issues related to preemption.

Tracking Preemptions:

You can often find preemption events in your cloud provider’s logs (e.g., AWS CloudTrail for EC2 Spot Instance Interruption Warnings, GCP Stackdriver logs for Preemptible VM events). You can also look at Kubernetes events:

kubectl get events --field-selector reason=NodeNotReady,reason=NodeLost,reason=Terminating --all-namespaces

This command might show events related to nodes becoming unavailable, which could be due to preemption. More specific events are usually found in cloud provider logs.

Production Considerations

• Workload Suitability: Not all workloads are suitable for Spot Instances. Critical, stateful applications that cannot tolerate interruptions (e.g., databases, core APIs) should always run on on-demand instances. Ideal candidates include batch processing, stateless microservices, CI/CD pipelines, development/test environments, and large-scale data processing that can restart from checkpoints.
• Diversify Instance Types: To maximize Spot availability and reduce preemption rates, use a diverse set of instance types and sizes within your Spot node groups. This allows the autoscaler to pick from a wider pool of available capacity.
• Preemption Handling: Emphasize graceful shutdown and checkpointing in your application development lifecycle. Test preemption scenarios regularly.
• Monitoring and Alerting: Set up robust monitoring for Spot instance interruptions and application restarts. Alert on high preemption rates or sustained application unavailability on Spot nodes.
• Node Taints and Tolerations Strategy: Use taints and tolerations rigorously to ensure critical workloads never land on Spot instances. Consider having a “no-spot-toleration” policy by default for production workloads.
• Cost Allocation Tags: Use cloud provider tags to properly attribute costs to teams or projects. This helps in understanding the true savings and optimizing further.
• Security: Ensure your IAM roles and policies for Spot node groups are aligned with your overall Kubernetes Network Policies and security posture.
• Tooling: Investigate advanced autoscaling solutions like Karpenter, which is specifically designed to optimize node provisioning for cost and performance, including intelligent Spot instance usage.

Troubleshooting

1. Pods not scheduling on Spot nodes (pending state):

   Issue: Your pods are stuck in a Pending state, even though you have Spot node groups configured.

   Solution:

   • Check Taints/Tolerations: Ensure your pods have the correct tolerations for the taint on your Spot nodes (e.g., spot-instance=true:NoSchedule).
   • Check Node Selectors/Affinity: Verify that your nodeSelector (e.g., lifecycle: Spot) or nodeAffinity rules correctly match the labels on your Spot nodes.
   • Cluster Autoscaler Logs: Examine Cluster Autoscaler logs (kubectl logs -f deployment/cluster-autoscaler -n kube-system). Look for messages indicating why it’s not scaling up (e.g., “no unstopped ASG found for pod,” “cannot scale up due to constraints”).
   • Resource Requests: Ensure your pods have appropriate resource requests (CPU/memory) that can be satisfied by your Spot instance types.
2. Spot nodes not scaling down:

   Issue: Spot nodes remain in the cluster even after workloads have finished, incurring unnecessary costs.

   Solution:

   • Cluster Autoscaler Configuration: Verify your Cluster Autoscaler is configured with proper --scale-down-delay-after-add, --scale-down-unneeded-time, and --scale-down-utilization-threshold parameters.
   • Pod Disruption Budgets (PDBs): PDBs can prevent Cluster Autoscaler from draining nodes if they protect too many pods on a single node. Review your PDBs.
   • System Pods: Ensure --skip-nodes-with-system-pods=false (or similar) is set if you expect Cluster Autoscaler to move system pods for scale-down.
   • Local Storage: Pods using emptyDir or other local storage can prevent node scale-down.
3. Frequent Spot instance interruptions:

   Issue: Your Spot instances are being preempted too often, leading to application instability.

   Solution:

   • Diversify Instance Types: As mentioned, use multiple instance types (e.g., m5.large, m5a.large, m5n.large, c5.large) in your Spot node group to increase the chance of finding available capacity.
   • Consider Different Regions/Availability Zones: Spot market prices and availability can vary.
   • Adjust Workload Tolerance: If preemption is still too high, the workload might not be suitable for Spot, or you might need to increase terminationGracePeriodSeconds and improve application resilience.
   • Monitor Spot Advisor/Pricing History: Cloud providers offer tools (e.g., AWS Spot Instance Advisor) to check interruption rates and pricing history.
4. Applications not shutting down gracefully:

   Issue: Pods on Spot instances are being killed abruptly, leading to data loss or inconsistent state.

   Solution:

   • Implement SIGTERM Handling: Ensure your application code explicitly catches and handles the SIGTERM signal.
   • Increase terminationGracePeriodSeconds: Give your pods more time to shut down by increasing this value in your Deployment manifest.
   • Check Liveness/Readiness Probes: Misconfigured probes can interfere with graceful shutdown. Ensure they don’t fail immediately during shutdown.
   • Review Application Logs: Check application logs during a simulated shutdown to see if it’s attempting to exit gracefully.
5. Misconfigured IAM/Service Account permissions:

   Issue: Cluster Autoscaler cannot manage node groups, or nodes fail to join the cluster.

   Solution:

   • Cloud Provider IAM Roles: Ensure the IAM role attached to your Cluster Autoscaler Service Account (or the EC2 instance profile for self-managed nodes) has the necessary permissions to describe, launch, and terminate instances/node groups. Refer to your cloud provider’s documentation (e.g., AWS EKS Cluster Autoscaler IAM requirements).
   • Node Join Permissions: Verify the IAM role used by the Spot instances themselves has permissions to join the Kubernetes cluster (e.g., eks:DescribeCluster for AWS EKS).

FAQ Section

1. What’s the difference between Spot Instances, Preemptible VMs, and Low-priority VMs?

   These are different cloud providers’ terms for the same concept: virtual machines offered at a significantly reduced price, but which can be reclaimed by the cloud provider with short notice. AWS calls them Spot Instances, Google Cloud Platform uses Preemptible VMs, and Azure refers to them as Spot VMs (formerly Low-priority VMs).

2. Can I run stateful applications on Spot Instances?

   Generally, it’s not recommended for critical, stateful applications that cannot tolerate interruptions or data loss. However, if your stateful application has robust checkpointing, replication, and can quickly recover from an instance termination, it might be feasible. For most production stateful workloads, on-demand instances or dedicated nodes are safer. For distributed stateful applications, consider solutions like Istio Ambient Mesh which can provide traffic resilience, but won’t solve the underlying data persistence issues.

3. How much can I save using Spot Instances?

   Savings vary significantly based on cloud provider, region, instance type, and current market demand. Discounts can range from 50% to 90% compared to on-demand prices. For example, AWS EC2 Spot Instances documentation provides details on potential savings: AWS EC2 Spot Pricing.

4. What happens to my application when a Spot Instance is terminated?

   When a Spot Instance receives a termination notice, the Kubernetes kubelet on that node will attempt to gracefully terminate all pods running on it by sending a SIGTERM signal. If your application handles SIGTERM, it can perform cleanup tasks within the terminationGracePeriodSeconds. After this period, any remaining processes are forcefully killed. The Kubernetes scheduler will then reschedule the evicted pods onto available nodes, potentially provisioning new Spot instances if needed by the autoscaler.

5. Should I use Cluster Autoscaler or Karpenter for Spot instances?

   Both can manage Spot instances effectively. Cluster Autoscaler is a mature project that works with existing node groups (Auto Scaling Groups, Node Pools). Karpenter is a newer, more advanced node provisioner that directly interacts with the cloud provider to launch exactly the right size and type of node needed. Karpenter often provides faster scaling and better cost optimization by making more intelligent decisions about instance types and consolidation. For a deep dive, check out our guide on Reducing Kubernetes Costs by 60% with Karpenter.

Cleanup Commands

To avoid incurring further costs, clean up the resources created during this tutorial.

1. Delete the Deployment:

kubectl delete -f batch-job-processor.yaml
kubectl delete -f resilient-worker.yaml

2. Delete the Spot Node Group/Node Pool:

For AWS EKS:

eksctl delete nodegroup --cluster=$CLUSTER_NAME --region=$REGION --name=$NODEGROUP_NAME

For GCP GKE:

gcloud container node-pools delete $NODEPOOL_NAME --cluster=$CLUSTER_NAME --zone=$ZONE --project=$PROJECT_ID

3. (Optional) Uninstall Cluster Autoscaler:

If you installed Cluster Autoscaler specifically for this tutorial and don’t need it for other purposes, you can remove its deployment and associated resources. The exact commands depend on how you installed it.

kubectl delete deployment cluster-autoscaler -n kube-system
kubectl delete serviceaccount cluster-autoscaler -n kube-system
# ... and any associated ClusterRole, ClusterRoleBinding, etc.

Next Steps / Further Reading

• Explore Karpenter: For advanced and highly efficient node provisioning, especially with Spot instances, dive deeper into Karpenter.
• Advanced Scheduling: Learn more about Kubernetes scheduling, including node affinity, anti-affinity, and taints/tolerations.
• Workload Resilience: Deepen your understanding of designing fault-tolerant applications. Consider patterns like circuit breakers, retries, and queues.
• Cloud Provider Spot Best Practices: Consult your cloud provider’s official documentation for their latest Spot instance best practices:
  • AWS EC2 Spot Instances
  • GCP Preemptible VMs
  • Azure Spot Virtual Machines
• Observability: Enhance your cluster’s observability to quickly detect and diagnose issues with Spot instances and application resilience. Consider tools like Prometheus, Grafana, and tracing solutions