Unlock Multi-Cluster Networking with Cilium

Introduction

As organizations scale their Kubernetes deployments, the need for robust multi-cluster networking solutions becomes paramount. Whether for disaster recovery, geographical distribution, or organizational segmentation, managing services across multiple Kubernetes clusters presents a unique set of challenges. Traditional approaches often involve complex VPNs, load balancers, and intricate routing rules, leading to operational overhead and increased latency. This complexity can hinder agility and make it difficult to maintain consistent security policies across your distributed infrastructure.

Enter Cilium Cluster Mesh: a powerful and elegant solution that extends Cilium’s eBPF-powered networking capabilities across multiple Kubernetes clusters. Cluster Mesh enables seamless service discovery, load balancing, and policy enforcement for workloads distributed across disparate clusters, making them behave as a single, logical network. By leveraging eBPF, Cilium provides high-performance data plane capabilities, advanced security features like identity-aware network policies, and unparalleled observability. This guide will walk you through setting up a Cilium Cluster Mesh, demonstrating how to achieve resilient and efficient multi-cluster communication.

# Install Cilium with Cluster Mesh enabled (example for Cluster1)
helm install cilium cilium/cilium --version 1.15.5 \
  --namespace kube-system \
  --set cluster.name=cluster1 \
  --set cluster.id=1 \
  --set kubeProxyReplacement=strict \
  --set ipam.mode=kubernetes \
  --set egressGateway.enabled=true \
  --set tunnel=vxlan \
  --set enableIPv4BIGTCP=true \
  --set clusterMesh.enabled=true \
  --set gatewayAPI.enabled=true \
  --set hubble.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.relay.enabled=true

# Enable Cluster Mesh services on a cluster (e.g., Cluster1)
cilium clustermesh enable --context kind-cluster1

# Connect Cluster2 to Cluster1
cilium clustermesh connect --context kind-cluster2 --destination-context kind-cluster1

# Verify Cluster Mesh status
cilium status --cluster-mesh --context kind-cluster1

Prerequisites

Before diving into the Cilium Cluster Mesh setup, ensure you have the following:

• Kubernetes Clusters: At least two Kubernetes clusters. For this guide, we’ll use Kind (Kubernetes in Docker) for local testing, but the principles apply to any cloud provider (AWS EKS, GCP GKE, Azure AKS) or on-premise setup.
• kubectl: The Kubernetes command-line tool, configured to access both clusters. Refer to the official Kubernetes documentation for installation.
• helm: The Kubernetes package manager. Install it from the Helm website.
• cilium-cli: The Cilium CLI tool, essential for managing Cilium deployments and Cluster Mesh. Download it from the Cilium installation guide.
• Network Connectivity: Your clusters must have network connectivity between their nodes. For Kind, this is handled by Docker’s networking. For cloud providers, ensure appropriate security groups and routing tables allow traffic between nodes of different clusters on the necessary ports (typically 4240 for Cilium Cluster Mesh).
• Unique Cluster Identifiers: Each cluster participating in the mesh must have a unique cluster.name and cluster.id.

Step-by-Step Guide: Setting up Cilium Cluster Mesh

Step 1: Create Kubernetes Clusters (Kind Example)

First, let’s create two Kind clusters. This setup simulates two distinct Kubernetes environments. We’ll name them cluster1 and cluster2.

This step involves defining two Kind cluster configurations. For each cluster, we expose a port on the host machine (e.g., 8080 for cluster1 and 8081 for cluster2) which will be used later to access services via a LoadBalancer type service. This is particularly useful for local development and testing Cilium’s external connectivity features. The extraPortMappings ensure that traffic to these host ports gets routed into the cluster’s control plane node. The --name flag assigns a unique name to each cluster, which will correspond to our Cilium cluster.name later.

# Create cluster1
cat <<EOF | kind create cluster --name cluster1 --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 8080
    listenAddress: "127.0.0.1"
    protocol: TCP
- role: worker
- role: worker
EOF

# Create cluster2
cat <<EOF | kind create cluster --name cluster2 --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30001
    hostPort: 8081
    listenAddress: "127.0.0.1"
    protocol: TCP
- role: worker
- role: worker
EOF

# Verify clusters are running
kubectl config get-contexts

Verify: You should see two contexts, kind-cluster1 and kind-cluster2, in your kubectl configuration.

CURRENT   NAME                 CLUSTER              AUTHINFO             NAMESPACE
*         kind-cluster1        kind-cluster1        kind-cluster1
          kind-cluster2        kind-cluster2        kind-cluster2

Step 2: Install Cilium on Both Clusters

Now, install Cilium on both clusters, enabling the Cluster Mesh feature and setting unique cluster names and IDs. We’ll also enable other useful features like Hubble for observability and Gateway API support. For a deeper dive into Cilium’s eBPF capabilities, check out our guide on eBPF Observability with Hubble.

This step is crucial. We install Cilium using Helm, which is the recommended way. Key parameters include cluster.name and cluster.id, which must be unique for each cluster. clusterMesh.enabled=true activates the core functionality. We also set kubeProxyReplacement=strict for optimal performance, ipam.mode=kubernetes for IP address management, and enable hubble for network observability. The gatewayAPI.enabled=true flag is forward-looking, allowing us to leverage the Kubernetes Gateway API for advanced traffic management in future setups.

# Add Cilium Helm repository
helm repo add cilium https://helm.cilium.io/

# Update Helm repositories
helm repo update

# Install Cilium on cluster1
helm install cilium cilium/cilium --version 1.15.5 \
  --namespace kube-system \
  --set cluster.name=cluster1 \
  --set cluster.id=1 \
  --set kubeProxyReplacement=strict \
  --set ipam.mode=kubernetes \
  --set egressGateway.enabled=true \
  --set tunnel=vxlan \
  --set enableIPv4BIGTCP=true \
  --set clusterMesh.enabled=true \
  --set gatewayAPI.enabled=true \
  --set hubble.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.relay.enabled=true \
  --context kind-cluster1

# Install Cilium on cluster2
helm install cilium cilium/cilium --version 1.15.5 \
  --namespace kube-system \
  --set cluster.name=cluster2 \
  --set cluster.id=2 \
  --set kubeProxyReplacement=strict \
  --set ipam.mode=kubernetes \
  --set egressGateway.enabled=true \
  --set tunnel=vxlan \
  --set enableIPv4BIGTCP=true \
  --set clusterMesh.enabled=true \
  --set gatewayAPI.enabled=true \
  --set hubble.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.relay.enabled=true \
  --context kind-cluster2

Verify: Check Cilium’s status on both clusters. All pods should be running and healthy.

cilium status --context kind-cluster1
# Expected Output (truncated):
# Cilium:             OK
# Hubble:             OK
# ClusterMesh:        disabled (disabled, use "cilium clustermesh enable")

cilium status --context kind-cluster2
# Expected Output (truncated):
# Cilium:             OK
# Hubble:             OK
# ClusterMesh:        disabled (disabled, use "cilium clustermesh enable")

Step 3: Enable Cluster Mesh Connectivity

With Cilium installed, the next step is to enable Cluster Mesh connectivity on each cluster. This involves exposing the Cilium Cluster Mesh API securely. For production environments, you’d typically use a LoadBalancer service provided by your cloud provider. For local Kind clusters, we’ll use NodePort, leveraging the extra port mappings we configured earlier. For enhanced security, you might consider integrating with solutions like Cilium WireGuard Encryption to secure inter-cluster traffic.

The cilium clustermesh enable command performs several actions: it creates a Kubernetes Secret containing the necessary TLS certificates for secure communication between clusters, and it sets up a Service (either LoadBalancer or NodePort) to expose the Cilium Cluster Mesh API endpoints to other clusters. The --service-type NodePort is crucial for our Kind setup, mapping to the host ports 8080 and 8081 we defined.

# Enable Cluster Mesh on cluster1
cilium clustermesh enable \
  --context kind-cluster1 \
  --service-type NodePort

# Enable Cluster Mesh on cluster2
cilium clustermesh enable \
  --context kind-cluster2 \
  --service-type NodePort

Verify: Confirm the Cluster Mesh service and secrets are created.

kubectl get svc -n kube-system --context kind-cluster1 | grep clustermesh
# Expected Output:
# clustermesh-apiserver   NodePort    10.43.140.160   <none>        4240:30000/TCP   2m5s

kubectl get secrets -n kube-system --context kind-cluster1 | grep clustermesh
# Expected Output:
# cilium-clustermesh-apiserver-client-tls   kubernetes.io/tls                     3      2m20s
# cilium-clustermesh-apiserver-server-tls   kubernetes.io/tls                     3      2m20s
# cilium-clustermesh-apiserver-etcd-tls     kubernetes.io/tls                     3      2m20s

# Repeat for cluster2, expecting port 30001
kubectl get svc -n kube-system --context kind-cluster2 | grep clustermesh
# Expected Output:
# clustermesh-apiserver   NodePort    10.43.140.160   <none>        4240:30001/TCP   2m5s

Step 4: Connect Clusters to the Mesh

Now that both clusters have Cilium with Cluster Mesh enabled and exposed, we can connect them. This involves securely sharing the connectivity information between the clusters. The cilium clustermesh connect command simplifies this process significantly.

This command automates the process of exchanging secrets between clusters. When connecting cluster2 to cluster1, the Cilium CLI fetches the necessary connectivity information (like certificate authorities and API endpoint details) from cluster1 and configures cluster2 to establish a secure connection. This creates a mesh where services can discover and communicate across cluster boundaries. You only need to run this command once for each pair of clusters you wish to connect in a direct mesh. For a full mesh, you’d run it for all pairs.

# Connect cluster2 to cluster1
cilium clustermesh connect \
  --context kind-cluster2 \
  --destination-context kind-cluster1

# Connect cluster1 to cluster2 (optional, but good for full mesh)
# This step ensures cluster1 also has cluster2's connectivity info
cilium clustermesh connect \
  --context kind-cluster1 \
  --destination-context kind-cluster2

Verify: Check the Cluster Mesh status on both clusters. You should see both clusters listed as connected.

cilium status --cluster-mesh --context kind-cluster1
# Expected Output:
# ClusterMesh:        connected
#   Cluster "cluster1" (id: 1)
#   Cluster "cluster2" (id: 2)

cilium status --cluster-mesh --context kind-cluster2
# Expected Output:
# ClusterMesh:        connected
#   Cluster "cluster1" (id: 1)
#   Cluster "cluster2" (id: 2)

Step 5: Deploy Cross-Cluster Services

With the Cluster Mesh established, we can now deploy applications and expose services that are discoverable and accessible across clusters. We’ll deploy a simple Nginx application in cluster1 and try to access it from cluster2.

This demonstrates the core value of Cluster Mesh: seamless cross-cluster service discovery and load balancing. The Service in cluster1 is marked with io.cilium/global-service: "true". This annotation tells Cilium to export this service to the entire mesh. When a pod in cluster2 tries to resolve nginx-service.default.svc.cluster.local, Cilium’s DNS proxy will intercept the request and return the IP addresses of the Nginx pods in cluster1, allowing direct communication. This works because Cilium replaces kube-proxy and handles inter-cluster routing at the eBPF layer.

# service-cluster1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
  annotations:
    io.cilium/global-service: "true" # This makes the service discoverable across clusters
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

# Deploy Nginx on cluster1
kubectl apply -f service-cluster1.yaml --context kind-cluster1

# Deploy a client pod on cluster2 to test connectivity
kubectl run --context kind-cluster2 -it --rm --restart=Never debug-client --image=busybox -- /bin/sh

Verify: From the debug-client pod in cluster2, try to access the Nginx service in cluster1.

# Inside the debug-client pod (cluster2)
/ # wget -O- nginx-service.default.svc.cluster.local
# Expected Output:
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>
# ... (Nginx welcome page HTML)

If you see the Nginx welcome page, congratulations! You have successfully set up Cilium Cluster Mesh and demonstrated cross-cluster service communication.

Production Considerations

Deploying Cilium Cluster Mesh in production requires careful planning beyond a local Kind setup:

• Network Connectivity: Ensure robust and low-latency network connectivity between your data centers or cloud regions. This might involve direct connect, VPNs, or peering connections. For cloud environments, configure security groups and network ACLs to allow traffic on port 4240 (Cilium Cluster Mesh API) and the encapsulation ports (e.g., VXLAN 8472, Geneve 6081) between nodes of different clusters.
• Load Balancer for Cluster Mesh API: Instead of NodePort, use a cloud provider’s LoadBalancer service (e.g., AWS NLB, GCP TCP Proxy Load Balancer, Azure Standard Load Balancer) to expose the clustermesh-apiserver for high availability and external accessibility.
• Security and Encryption: While Cilium Cluster Mesh uses TLS for its control plane, consider encrypting the data plane traffic between clusters. Cilium WireGuard Encryption is an excellent option for this, providing performant, kernel-level encryption for pod-to-pod traffic across the mesh. Also, implement strict Kubernetes Network Policies to control what services can communicate across clusters.
• Identity Management: Cilium’s identity-based security is a key feature. Ensure your identity allocation ranges are properly configured and don’t overlap if you’re managing multiple independent Cilium deployments.
• Observability: Leverage Hubble for deep visibility into cross-cluster traffic. Integrate Hubble with your existing monitoring stack (Prometheus, Grafana, ELK) to gain insights into latency, errors, and security events. Our guide on eBPF Observability with Hubble provides more details.
• DNS Resolution: Cilium’s transparent DNS proxy handles cross-cluster service discovery. Be aware of any custom DNS configurations or external DNS services that might interfere.
• IP Address Management (IPAM): While ipam.mode=kubernetes is fine for many cases, consider ipam.mode=clusterPool for more advanced IP allocation strategies, especially if you have very large clusters or specific IP range requirements.
• Gateway API Integration: If you’re using Kubernetes Gateway API for ingress and traffic management, Cilium’s integration allows for advanced routing policies that can span the cluster mesh.
• Cost Optimization: While Cilium itself is efficient, managing multiple clusters can incur costs. Consider using tools like Karpenter for Kubernetes cost optimization to efficiently manage node resources if your clusters are dynamic.

Troubleshooting

1. Issue: cilium status --cluster-mesh shows “disabled” or “connecting” indefinitely.

   Solution:

   First, ensure clusterMesh.enabled=true was set during Cilium installation. If not, you’ll need to reinstall Cilium with this flag. Then, check if the clustermesh-apiserver service is running and accessible from other clusters. For NodePort, confirm the host port is open and reachable. For LoadBalancer, ensure the LB is provisioned and healthy. Check logs of cilium-operator pods for errors related to Cluster Mesh.

   kubectl get svc -n kube-system | grep clustermesh-apiserver
   kubectl logs -n kube-system -l k8s-app=cilium-operator --tail=100
   

2. Issue: Cross-cluster DNS resolution fails (e.g., wget nginx-service.default.svc.cluster.local doesn’t work).

   Solution:

   Verify that the service in the source cluster has the io.cilium/global-service: "true" annotation. Check that Cilium’s DNS proxy is intercepting requests. You can exec into a pod and try to resolve the service name. Ensure that the Cilium pods themselves are healthy and not experiencing any DNS-related issues. If using CoreDNS, ensure it’s not interfering with Cilium’s DNS proxy.

   kubectl describe svc nginx-service -n default --context kind-cluster1
   kubectl exec -it <pod-in-cluster2> -- nslookup nginx-service.default.svc.cluster.local
   

3. Issue: Traffic between clusters is dropped or timing out.

   Solution:

   This often points to network connectivity or firewall issues. Check your cloud provider’s security groups, network ACLs, or on-premise firewall rules to ensure traffic is allowed on the necessary ports (VXLAN 8472, Geneve 6081, Cilium Cluster Mesh API 4240, etc.) between the nodes of different clusters. Use tcpdump on the node interfaces if possible. Also, check Cilium Network Policies; they might be inadvertently blocking cross-cluster traffic. For a detailed guide on securing traffic, refer to our Kubernetes Network Policies Security Guide.

   # Example: Check security groups in AWS for relevant ports
   # Use tcpdump on a node to see traffic
   sudo tcpdump -i any host <remote-node-ip> and port 8472
   

4. Issue: cilium clustermesh connect fails with TLS errors.

   Solution:

   This indicates an issue with the Cluster Mesh API server certificates or connectivity. Ensure that cilium clustermesh enable completed successfully on both clusters and that the secrets (cilium-clustermesh-*tls) exist. Check the logs of the clustermesh-apiserver pod for any certificate-related errors or connectivity problems. Sometimes, recreating the Cluster Mesh secrets (by disabling and re-enabling) can resolve transient issues.

   kubectl get pods -n kube-system -l k8s-app=clustermesh-apiserver --context kind-cluster1
   kubectl logs -n kube-system -l k8s-app=clustermesh-apiserver --context kind-cluster1
   

5. Issue: Cilium agent pods are in a CrashLoopBackOff state after enabling Cluster Mesh.

   Solution:

   Review the logs of the crashing Cilium agent pods. Common causes include incorrect configuration (e.g., overlapping cluster.id), insufficient resources, or network plugin conflicts. Ensure your cluster.id is unique across all clusters. If you’re switching CNI, make sure the old CNI is fully removed. Increase resource limits for Cilium pods if they are being OOMKilled.

   kubectl logs -n kube-system -l k8s-app=cilium --previous
   kubectl describe pod <crashing-cilium-pod> -n kube-system
   

6. Issue: Cilium Cluster Mesh not load balancing correctly across clusters.

   Solution:

   Ensure that the io.cilium/global-service: "true" annotation is correctly applied to the service you expect to be load-balanced. Verify that all backend pods are healthy and running in the source cluster. Use Hubble to visualize the traffic flow and see if requests are reaching the desired backends. Sometimes, stale DNS entries can cause issues; restarting client pods might help refresh their DNS cache.

   # Install Hubble CLI if not already
   export PATH=$PATH:/usr/local/bin # Assuming /usr/local/bin is where hubble was installed
   hubble observe --context kind-cluster2
   hubble observe --context kind-cluster1
   

FAQ Section

1. What is the difference between Cilium Cluster Mesh and a Service Mesh like Istio?

   Cilium Cluster Mesh focuses on providing a seamless network fabric and identity-aware security across multiple Kubernetes clusters at the CNI layer (Layer 3/4). It enables cross-cluster service discovery and load balancing. A service mesh like Istio Ambient Mesh operates at a higher layer (Layer 7), providing advanced traffic management (e.g., retries, circuit breakers, traffic splitting), policy enforcement, and observability for application-level protocols like HTTP/gRPC. While they solve different problems, they are complementary. Cilium can provide the underlying network for Istio, and Istio can then manage the application traffic over Cilium’s mesh.

2. Does Cilium Cluster Mesh encrypt traffic between clusters by default?

   No, the data plane traffic between clusters is not encrypted by default. Cilium Cluster Mesh secures its control plane (the communication between Cilium agents across clusters) using TLS. To encrypt the actual pod-to-pod data plane traffic, you need to explicitly enable features like Cilium WireGuard Encryption or IPSec.

3. How does Cilium Cluster Mesh handle overlapping IP addresses between clusters?

   Cilium Cluster Mesh explicitly supports overlapping IP CIDRs between clusters. It achieves this by using a global identity for pods that is unique across the mesh, regardless of their local IP address. When traffic traverses clusters, Cilium encapsulates it (e.g., using VXLAN or Geneve) and uses the global identity to route and enforce policies, effectively abstracting away the underlying IP addresses. This is a significant advantage over traditional networking solutions.

4. What are the network requirements for Cilium Cluster Mesh?

   The primary requirement is direct IP connectivity between the nodes of all participating clusters. This means network routes must exist, and firewalls must permit traffic for Cilium’s control plane (port 4240 for the Cluster Mesh API) and data plane encapsulation (e.g., UDP 8472 for VXLAN, UDP 6081 for Geneve). For cloud environments, this typically means configuring appropriate security groups and VPC peering or transit gateways.

5. Can I use Cilium Cluster Mesh with different cloud providers (e.g., AWS EKS and GCP GKE)?

   Yes, absolutely! Cilium Cluster Mesh is cloud-agnostic. As long as there is network connectivity between the nodes of the clusters (e.g., via VPN or direct connect between cloud providers), you can create a mesh across heterogeneous environments. This makes it a powerful tool for hybrid cloud or multi-cloud strategies.

Cleanup Commands

To tear down the environment and remove all resources created during this guide:

# Disconnect clusters from the mesh (optional, but good practice)
cilium clustermesh disconnect --context kind-cluster1 --destination-context kind-cluster2
cilium clustermesh disconnect --context kind-cluster2 --destination-context kind-cluster1

# Disable Cluster Mesh (removes services and secrets)
cilium clustermesh disable --context kind-cluster1
cilium clustermesh disable --context kind-cluster2

# Delete Nginx deployment and service from cluster1
kubectl delete deploy nginx-deployment --context kind-cluster1
kubectl delete svc nginx-service --context kind-cluster1

# Delete Cilium from both clusters
helm uninstall cilium --namespace kube-system --context kind-cluster1
helm uninstall cilium --namespace kube-system --context kind-cluster2

# Delete Kind clusters
kind delete cluster --name cluster1
kind delete cluster --name cluster2

Next Steps / Further Reading

• Explore Cilium’s official Cluster Mesh documentation for advanced configurations and use cases.
• Dive deeper into Kubernetes Network Policies to secure your cross-cluster communication.
• Learn about securing your container supply chain with Sigstore and Kyverno.
• Investigate Cilium WireGuard Encryption to add data plane encryption to your cluster mesh.
• Understand how to monitor your distributed environment using eBPF Observability with Hubble.
• Consider using the Kubernetes Gateway API for advanced traffic management across your mesh.
• For large-scale AI/ML workloads, explore running LLMs on Kubernetes with GPU scheduling best practices.

Conclusion

Cilium Cluster Mesh provides a robust, high-performance, and secure foundation for multi-cluster Kubernetes deployments. By leveraging the power of eBPF, it simplifies complex networking challenges, enables seamless cross-cluster service discovery and load balancing, and extends Cilium’s advanced security policies across your entire distributed infrastructure. This guide has walked you through the essential steps to set up and verify a basic Cluster Mesh, demonstrating its core capabilities. As your Kubernetes footprint grows, Cilium Cluster Mesh will be an indispensable tool for building resilient, scalable, and manageable cloud-native applications across multiple clusters.