Redis Cluster on K8s: HA Setup Guide

Introduction

Running stateful applications like Redis in a Kubernetes environment can often feel like navigating a minefield. While Kubernetes excels at orchestrating stateless microservices, persistent data stores introduce unique challenges related to high availability, data persistence, and cluster management. Redis, a popular open-source, in-memory data structure store, is no exception. A single Redis instance is a single point of failure, making a clustered setup essential for any production workload requiring resilience and scalability.

This guide will walk you through the process of deploying a highly available Redis Cluster on Kubernetes. We’ll leverage Kubernetes’ powerful orchestration capabilities, including StatefulSets for stable, unique network identities, and PersistentVolumes for durable storage. By the end of this tutorial, you’ll have a robust Redis Cluster capable of surviving node failures and scaling to meet your application’s demands, all while adhering to best practices for production-grade deployments. Get ready to unlock the full potential of Redis on Kubernetes!

# Create Namespace
kubectl create namespace redis-cluster

# Apply Headless Service
kubectl apply -f redis-headless-service.yaml -n redis-cluster

# Apply StatefulSet
kubectl apply -f redis-statefulset.yaml -n redis-cluster

# Initialize Cluster (run from one pod)
# Replace  with actual pod IPs: e.g., 10.42.0.4:6379 10.42.0.5:6379 ...
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli --cluster create  --cluster-replicas 1

# Verify Cluster Status
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli cluster info
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli cluster nodes
    

Prerequisites

Before diving into the deployment, ensure you have the following:

• A Kubernetes Cluster: A working Kubernetes cluster (version 1.18+ recommended) is essential. This can be a local cluster like Kind, Minikube, or a cloud-managed service like AWS EKS, GCP GKE, or Azure AKS.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• helm (Optional, but recommended for advanced setups): While we’ll use raw YAML for this guide, Helm is a popular package manager for Kubernetes.
• StorageClass: Your Kubernetes cluster must have a default StorageClass configured or you’ll need to specify one for dynamic PersistentVolume provisioning. You can check available StorageClasses with kubectl get sc.
• Basic Kubernetes Knowledge: Familiarity with concepts like Pods, Services, StatefulSets, and PersistentVolumes will be helpful.

Step-by-Step Guide: Deploying a Redis Cluster

1. Create a Dedicated Namespace

It’s good practice to isolate your Redis Cluster components within their own Kubernetes namespace. This helps with organization, resource management, and access control. We’ll create a namespace called redis-cluster.

kubectl create namespace redis-cluster

Verify:

kubectl get namespace redis-cluster

NAME            STATUS   AGE
redis-cluster   Active   Xs

2. Define a Headless Service for Redis

A Headless Service is crucial for StatefulSets. Unlike a regular Service that provides a single stable IP address, a Headless Service doesn’t have a cluster IP. Instead, it allows for direct DNS lookups of individual Pods. This is vital for Redis Cluster, as each Redis node needs a stable, unique network identity (hostname) to communicate with other nodes and form the cluster correctly. The StatefulSet will use this service to assign predictable hostnames like redis-cluster-0.redis-headless.redis-cluster.svc.cluster.local.

apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  namespace: redis-cluster
  labels:
    app: redis-cluster
spec:
  ports:
  - port: 6379
    name: redis
  - port: 16379 # Port for cluster bus communication
    name: cluster-bus
  clusterIP: None # This makes it a headless service
  selector:
    app: redis-cluster

Apply the headless service:

kubectl apply -f redis-headless-service.yaml -n redis-cluster

Verify:

kubectl get svc -n redis-cluster

NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
redis-headless   ClusterIP   None         <none>        6379/TCP,16379/TCP           Xs

3. Deploy the Redis StatefulSet

The StatefulSet is the backbone of our Redis Cluster. It manages the deployment and scaling of a set of Pods, guaranteeing stable network identities, stable persistent storage, and ordered, graceful deployment and scaling. Each Redis Pod will get its own PersistentVolumeClaim (PVC) for data persistence, ensuring that data is not lost if a Pod restarts or moves to a different node. We’ll deploy 6 Redis instances, which will later be configured into 3 master nodes and 3 replica nodes (--cluster-replicas 1 means one replica per master).

Important considerations in the StatefulSet:

• serviceName: redis-headless: Links the StatefulSet to our headless service for stable hostnames.
• volumeClaimTemplates: Dynamically provisions PersistentVolumes for each Pod. Ensure your cluster has a StorageClass.
• Redis Configuration: The command and args configure Redis to run in cluster mode and bind to the Pod’s IP address. Using --protected-mode no is generally discouraged in production but used here for simplicity during initial setup. For production, consider robust authentication and Network Policies.
• Resource Limits: Set appropriate CPU and memory requests/limits for stability.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
  namespace: redis-cluster
  labels:
    app: redis-cluster
spec:
  serviceName: redis-headless
  replicas: 6 # 3 masters, 3 replicas
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:6.2-alpine
        command: ["redis-server"]
        args:
        - "/etc/redis/redis.conf"
        - "--bind"
        - "0.0.0.0"
        - "--protected-mode"
        - "no" # For simplicity in this tutorial. Use authentication in production!
        - "--port"
        - "6379"
        - "--cluster-enabled"
        - "yes"
        - "--cluster-config-file"
        - "nodes.conf" # This file is rewritten by Redis Cluster
        - "--cluster-node-timeout"
        - "5000"
        - "--appendonly"
        - "yes" # Enable AOF persistence
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        ports:
        - containerPort: 6379
          name: redis
        - containerPort: 16379 # Redis Cluster bus port
          name: cluster-bus
        volumeMounts:
        - name: redis-data
          mountPath: /data
        - name: redis-config
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        readinessProbe:
          tcpSocket:
            port: 6379
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          tcpSocket:
            port: 6379
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
      volumes:
      - name: redis-config
        configMap:
          name: redis-config
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard # Replace with your StorageClass name if different
      resources:
        requests:
          storage: 1Gi

We also need a ConfigMap for the Redis configuration file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-config
  namespace: redis-cluster
data:
  redis.conf: |
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    appendonly yes
    protected-mode no
    # Uncomment and set a password in production
    # requirepass your_strong_password
    # masterauth your_strong_password

Save the ConfigMap as redis-configmap.yaml and the StatefulSet as redis-statefulset.yaml. Then apply them:

kubectl apply -f redis-configmap.yaml -n redis-cluster
kubectl apply -f redis-statefulset.yaml -n redis-cluster

Verify: Wait for all 6 Pods to be running.

kubectl get pods -n redis-cluster -w

NAME              READY   STATUS    RESTARTS   AGE
redis-cluster-0   1/1     Running   0          2m
redis-cluster-1   1/1     Running   0          2m
redis-cluster-2   1/1     Running   0          2m
redis-cluster-3   1/1     Running   0          1m
redis-cluster-4   1/1     Running   0          1m
redis-cluster-5   1/1     Running   0          1m

And check the PersistentVolumeClaims:

kubectl get pvc -n redis-cluster

NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-data-redis-cluster-0   Bound    pvc-12345678-abcd-1234-abcd-1234567890ab   1Gi        RWO            standard       2m
redis-data-redis-cluster-1   Bound    pvc-abcdefgh-efgh-abcd-abcd-1234567890ab   1Gi        RWO            standard       2m
# ... (and so on for all 6 pods)

4. Initialize the Redis Cluster

Now that all Redis Pods are running, they are independent instances. We need to tell them to form a cluster. We’ll use redis-cli --cluster create. This command needs the IPs and ports of all initial master nodes. We’ll create 3 masters and 3 replicas.

First, get the IP addresses of your Redis pods:

kubectl get pods -n redis-cluster -o wide

NAME              READY   STATUS    RESTARTS   AGE     IP            NODE
redis-cluster-0   1/1     Running   0          10m     10.42.0.4     kind-worker
redis-cluster-1   1/1     Running   0          10m     10.42.0.5     kind-worker2
redis-cluster-2   1/1     Running   0          10m     10.42.0.6     kind-worker3
redis-cluster-3   1/1     Running   0          9m      10.42.0.7     kind-worker
redis-cluster-4   1/1     Running   0          9m      10.42.0.8     kind-worker2
redis-cluster-5   1/1     Running   0          9m      10.42.0.9     kind-worker3

Now, execute the redis-cli --cluster create command from one of the Pods (e.g., redis-cluster-0), providing all Pod IPs. The --cluster-replicas 1 option tells Redis to create one replica for each master. This will result in 3 masters and 3 replicas, ensuring high availability.

Note: Replace the IPs below with the actual IPs from your kubectl get pods -o wide output.

# Example IPs: 10.42.0.4, 10.42.0.5, 10.42.0.6, 10.42.0.7, 10.42.0.8, 10.42.0.9
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli \
  --cluster create \
  10.42.0.4:6379 10.42.0.5:6379 10.42.0.6:6379 \
  10.42.0.7:6379 10.42.0.8:6379 10.42.0.9:6379 \
  --cluster-replicas 1

You will be prompted to confirm the cluster creation. Type yes and press Enter.

>>> Performing hash slots distribution on 6 nodes...
Master nodes:
  10.42.0.4:6379
  10.42.0.5:6379
  10.42.0.6:6379
Replica nodes:
  10.42.0.7:6379 will be replica of 10.42.0.4:6379
  10.42.0.8:6379 will be replica of 10.42.0.5:6379
  10.42.0.9:6379 will be replica of 10.42.0.6:6379
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a set of slots to the new masters...
>>> Assigning slots 0-5460 to 10.42.0.4:6379
>>> Assigning slots 5461-10922 to 10.42.0.5:6379
>>> Assigning slots 10923-16383 to 10.42.0.6:6379
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 10.42.0.4:6379)
M: 1234567890abcdef1234567890abcdef12345678 10.42.0.4:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: fedcba9876543210fedcba9876543210fedcba98 10.42.0.7:6379
   replicates 1234567890abcdef1234567890abcdef12345678
M: 9876543210abcdef9876543210abcdef98765432 10.42.0.5:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: badf00dbeefeada1badf00dbeefeada1badf00d 10.42.0.8:6379
   replicates 9876543210abcdef9876543210abcdef98765432
M: 0123456789fedcba0123456789fedcba01234567 10.42.0.6:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: deadbeefdeadbeefdeadbeefdeadbeefdeadbeef 10.42.0.9:6379
   replicates 0123456789fedcba0123456789fedcba01234567
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

5. Verify Cluster Health

After initialization, it’s crucial to verify that the cluster is healthy and all nodes are communicating correctly. You can check the cluster information and node status from any of the Redis Pods.

# Check cluster info
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli cluster info

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:0
cluster_stats_messages_ping_sent:159
cluster_stats_messages_pong_sent:161
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:321
cluster_stats_messages_ping_received:160
cluster_stats_messages_pong_received:159
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:321

The key indicators here are cluster_state:ok, cluster_slots_assigned:16384, and cluster_slots_ok:16384. This confirms the cluster is operational and all hash slots are assigned.

# Check cluster nodes
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli cluster nodes

1234567890abcdef1234567890abcdef12345678 10.42.0.4:6379@16379 master - 0 1678890123456 1 connected 0-5460
fedcba9876543210fedcba9876543210fedcba98 10.42.0.7:6379@16379 replica 1234567890abcdef1234567890abcdef12345678 0 1678890123456 1 connected
9876543210abcdef9876543210abcdef98765432 10.42.0.5:6379@16379 master - 0 1678890123456 2 connected 5461-10922
badf00dbeefeada1badf00dbeefeada1badf00d 10.42.0.8:6379@16379 replica 9876543210abcdef9876543210abcdef98765432 0 1678890123456 2 connected
0123456789fedcba0123456789fedcba01234567 10.42.0.6:6379@16379 master - 0 1678890123456 3 connected 10923-16383
deadbeefdeadbeefdeadbeefdeadbeefdeadbeef 10.42.0.9:6379@16379 replica 0123456789fedcba0123456789fedcba01234567 0 1678890123456 3 connected

This output shows all 6 nodes, their roles (master/replica), and which master each replica is serving. All nodes should be in the connected state.

6. Testing the Redis Cluster

To ensure our cluster is working, we can write and read some data. When connecting to a Redis Cluster, you typically connect to any node, and the client will automatically redirect you to the correct node for the key’s hash slot. Most modern Redis client libraries support cluster mode and handle redirections automatically.

# Connect to one of the masters (e.g., redis-cluster-0)
kubectl exec -it redis-cluster-0 -n redis-cluster -- redis-cli -c

# Set a key (the -c flag enables cluster mode, allowing redirections)
10.42.0.4:6379> set mykey "Hello Kubezilla"
-> Redirected to slot 7186 residing at 10.42.0.5:6379
OK

# Get the key
10.42.0.5:6379> get mykey
"Hello Kubezilla"

# Set another key, it might go to a different slot/master
10.42.0.5:6379> set anotherkey "Kubernetes is awesome"
-> Redirected to slot 15849 residing at 10.42.0.6:6379
OK

# Get the second key
10.42.0.6:6379> get anotherkey
"Kubernetes is awesome"

# Exit redis-cli
10.42.0.6:6379> exit

The redirections indicate that the cluster is correctly distributing hash slots and the client is interacting with it as expected. Congratulations, you now have a functional Redis Cluster on Kubernetes!

Production Considerations

While the above guide sets up a basic Redis Cluster, a production deployment requires additional thought:

• Security:
  • Authentication: Enable requirepass and masterauth in your Redis configuration. Store passwords securely using Kubernetes Secrets.
  • Network Policies: Restrict network access to your Redis Pods. Only necessary application Pods and management tools should be able to connect. For a deeper dive, check out our Kubernetes Network Policies: Complete Security Hardening Guide.
  • TLS/SSL: For highly sensitive data, consider encrypting Redis traffic using TLS. This often involves a sidecar proxy or building Redis with TLS support.
• Monitoring and Alerting:
  • Integrate with Prometheus and Grafana for metrics collection (e.g., Redis exporter).
  • Set up alerts for critical metrics like memory usage, connection count, and cluster state changes.
  • Consider eBPF Observability: Building Custom Metrics with Hubble for deep network insights if you’re using a CNI like Cilium.
• Backup and Restore:
  • Regularly back up your Redis data. This can be done by taking snapshots of the PersistentVolumes or using Redis’s built-in RDB/AOF persistence and backing up those files.
  • Test your restore process periodically to ensure data integrity.
• Resource Management:
  • Precisely define CPU and memory requests and limits based on performance testing.
  • Monitor resource utilization to prevent OOMKills or performance bottlenecks.
  • Consider using Karpenter for Cost Optimization to efficiently manage node resources if your cluster scales dynamically.
• Upgrades:
  • Plan for rolling upgrades of Redis versions and Kubernetes components to minimize downtime.
  • Test upgrades in a staging environment first.
• Client Configuration:
  • Ensure your application’s Redis client is configured for cluster mode, enabling it to handle redirections automatically.
  • Use connection pooling to manage connections efficiently.
• Networking:
  • If using a CNI like Cilium, you might explore features like Cilium WireGuard Encryption for secure pod-to-pod communication.

Troubleshooting

Here are some common issues you might encounter and their solutions:

1. Pods not starting (Pending/CrashLoopBackOff):

   Issue: Your Redis Pods are stuck in a Pending state or repeatedly crashing.

   Solution:

   • Check Events: kubectl describe pod <pod-name> -n redis-cluster for storage issues (e.g., no StorageClass, PV not bound), image pull errors, or scheduling problems.
   • Check Logs: kubectl logs <pod-name> -n redis-cluster for Redis-specific errors (e.g., config file issues, port conflicts, memory allocation failures).
   • StorageClass: Ensure your cluster has a default StorageClass or that the storageClassName in your StatefulSet matches an existing one.
2. Redis Cluster initialization fails (ERR Slot X is already busy or ERR Invalid or out of range slot number):

   Issue: When running redis-cli --cluster create, you get errors related to slots.

   Solution: This often happens if a previous cluster was partially created and some nodes still retain cluster metadata.

   • Clean Data: The easiest way to fix this in a test environment is to delete the StatefulSet, its PVCs, and then recreate them. This ensures fresh data directories.
   • Manual Reset (Careful!): For individual nodes, you can exec into the Pod and run redis-cli cluster reset, but this should be done with extreme caution in a production environment.
3. redis-cli --cluster create hangs or fails to connect:

   Issue: The cluster creation command can’t reach all specified Redis instances.

   Solution:

   • Verify Pod IPs: Double-check that the IP addresses you provided to redis-cli --cluster create are correct and belong to running Pods.
   • Network Connectivity: Ensure there are no network policies or firewall rules blocking communication between the Pods on ports 6379 (client) and 16379 (cluster bus).
   • Headless Service: Confirm the Headless Service is correctly configured and its selector matches the StatefulSet’s labels.
   • Pod Readiness: Make sure all Redis Pods are in the Running and Ready state before attempting cluster creation.
4. (error) CLUSTERDOWN The cluster is down:

   Issue: The cluster reports itself as down, often after a node failure or network partition.

   Solution:

   • Check Node Status: Use kubectl exec -it <pod> -n redis-cluster -- redis-cli cluster nodes to identify which nodes are failing or disconnected.
   • Review Logs: Check logs of affected Pods for specific error messages.
   • Quorum: Ensure enough master nodes are available to form a quorum (more than half of the masters). If too many masters are down, the cluster will halt.
   • Automatic Failover: If replicas are configured, Redis Cluster should automatically failover. If not, investigate why the replica didn’t promote.
5. Clients not connecting or getting redirection errors:

   Issue: Your application clients are having trouble connecting to or interacting with the Redis Cluster.

   Solution:

   • Client Library: Ensure your Redis client library explicitly supports Redis Cluster mode. Older libraries might not handle redirections.
   • Initial Connection Points: Provide your client with multiple seed nodes (IPs and ports of your Redis master pods) so it can discover the cluster topology.
   • Service Discovery: Consider creating a Kubernetes Service (e.g., a ClusterIP service) that selects all Redis Pods. Your application can connect to this service, and the client library will then discover the cluster topology from there.

FAQ Section

1. What is the minimum number of nodes for a Redis Cluster?

   For a production-ready, highly available Redis Cluster, you need at least 3 master nodes, each with at least one replica. This means a minimum of 6 Redis instances (3 masters,