Manage Kubernetes Across Clusters

Introduction

As organizations scale their Kubernetes deployments, managing a single cluster often becomes insufficient. Factors like disaster recovery, geographic distribution, regulatory compliance, and multi-cloud strategies necessitate the deployment of applications across multiple Kubernetes clusters. However, this multi-cluster reality introduces significant operational complexity: how do you consistently deploy applications, manage configurations, and ensure seamless communication across disparate clusters? Manually replicating deployments and configurations across dozens or hundreds of clusters is not only error-prone but also a massive drain on engineering resources.

Enter Kubernetes Federation, a powerful concept designed to address these multi-cluster challenges. While the original Kubernetes Federation project (Federation v1) faced adoption hurdles and has since been deprecated, its successor, Kubernetes Federation v2 (also known as KubeFed), offers a more robust and flexible approach. KubeFed provides a mechanism to coordinate and manage resources across multiple Kubernetes clusters from a single control plane. It allows you to define a resource once and propagate it to selected member clusters, enabling consistent application deployment, service discovery, and policy enforcement across your entire Kubernetes fleet. This guide will walk you through the process of setting up and utilizing KubeFed to streamline your multi-cluster management.

Prerequisites

Before diving into KubeFed, ensure you have the following:

• Multiple Kubernetes Clusters: At least two, preferably three, operational Kubernetes clusters. One will serve as the “host” cluster for the KubeFed control plane, and the others will be “member” clusters. These can be on any cloud provider (AWS EKS, GCP GKE, Azure AKS, etc.) or on-premises.
• kubectl: Configured to access all your clusters. Ensure your ~/.kube/config file contains contexts for all clusters you intend to use. You can verify this with kubectl config get-contexts.
• Helm 3: Installed on your local machine for deploying KubeFed. Refer to the Helm installation guide if you don’t have it.
• kubefedctl: The KubeFed command-line tool. We’ll install this in the first step.
• Basic Kubernetes Knowledge: Familiarity with Deployments, Services, Namespaces, and RBAC is assumed.
• Network Connectivity: Ensure that the host cluster can reach the API servers of the member clusters. This might involve configuring firewall rules or VPC peering, depending on your cloud environment. For advanced networking across clusters, consider solutions like Cilium WireGuard Encryption.

Step-by-Step Guide: Setting Up KubeFed and Federating Resources

Step 1: Install kubefedctl and KubeFed Control Plane

First, we need to install the KubeFed command-line utility, kubefedctl, which will be used to interact with the KubeFed control plane. Then, we’ll deploy the KubeFed control plane itself onto one of your Kubernetes clusters, designated as the “host” cluster. This host cluster will manage the federation of resources across all other “member” clusters.

The KubeFed control plane consists of several components, including controllers that watch for federated resources and propagate them to the registered member clusters. We’ll use Helm for a straightforward installation.

# 1. Install kubefedctl (Linux/macOS example)
# For other OS, refer to KubeFed's official installation guide.
# Find the latest release tag from https://github.com/kubernetes-sigs/kubefed/releases
KUBEFED_VERSION=v0.9.0 # Use the latest stable version
OS=$(go env GOOS)
ARCH=$(go env GOARCH)
curl -LO https://github.com/kubernetes-sigs/kubefed/releases/download/${KUBEFED_VERSION}/kubefedctl-${KUBEFED_VERSION}-${OS}-${ARCH}.tgz
tar -zxvf kubefedctl-${KUBEFED_VERSION}-${OS}-${ARCH}.tgz
sudo mv kubefedctl /usr/local/bin/

# Verify installation
kubefedctl version

# 2. Add the KubeFed Helm repository
helm repo add kubefed-charts https://kubernetes-sigs.github.io/kubefed/charts
helm repo update

# 3. Install KubeFed control plane on your designated host cluster
# Replace `host-cluster-context` with the actual kubectl context for your host cluster.
# We'll install KubeFed into the `kubefed-system` namespace.
kubectl config use-context host-cluster-context

helm install kubefed kubefed-charts/kubefed --namespace kubefed-system --create-namespace

# Verify KubeFed deployment
kubectl get pods -n kubefed-system

Verify:

You should see output similar to this, indicating kubefedctl is installed and the KubeFed pods are running in the kubefed-system namespace:

# kubefedctl version output
kubefedctl version
Client Version: version.BuildInfo{Version:"v0.9.0", GitCommit:"e80a061b7f2b1d7d0a7a3a8b46c6b3e8c9c6f2a2", GitTreeState:"clean", GoVersion:"go1.19.4"}

# kubectl get pods -n kubefed-system output
NAME                             READY   STATUS    RESTARTS   AGE
kubefed-controller-manager-xxx   1/1     Running   0          2m
kubefed-webhook-xxx              1/1     Running   0          2m

Step 2: Register Member Clusters with KubeFed

With the KubeFed control plane running on your host cluster, the next step is to register your other Kubernetes clusters as “member” clusters. This allows the KubeFed control plane to communicate with these clusters and propagate resources to them. For each member cluster, KubeFed will create the necessary RBAC roles and service accounts to establish secure communication. You’ll need the kubectl context for each member cluster and the host cluster.

# Replace with your actual cluster contexts and desired names
HOST_CLUSTER_CONTEXT="host-cluster-context"
MEMBER_CLUSTER_1_NAME="cluster-dev"
MEMBER_CLUSTER_1_CONTEXT="dev-cluster-context"
MEMBER_CLUSTER_2_NAME="cluster-prod"
MEMBER_CLUSTER_2_CONTEXT="prod-cluster-context"

# Join the first member cluster
echo "Joining ${MEMBER_CLUSTER_1_NAME}..."
kubefedctl join ${MEMBER_CLUSTER_1_NAME} \
    --cluster-context ${MEMBER_CLUSTER_1_CONTEXT} \
    --host-cluster-context ${HOST_CLUSTER_CONTEXT} \
    --kubefed-namespace kubefed-system

# Join the second member cluster
echo "Joining ${MEMBER_CLUSTER_2_NAME}..."
kubefedctl join ${MEMBER_CLUSTER_2_NAME} \
    --cluster-context ${MEMBER_CLUSTER_2_CONTEXT} \
    --host-cluster-context ${HOST_CLUSTER_CONTEXT} \
    --kubefed-namespace kubefed-system

# Verify registered clusters
kubectl get kubefedclusters -n kubefed-system --context ${HOST_CLUSTER_CONTEXT}

Verify:

You should see both member clusters listed with a Ready status, indicating they have successfully joined the federation:

# kubectl get kubefedclusters -n kubefed-system output
NAME          AGE   READY   STATUS
cluster-dev   1m    True    Cluster is available
cluster-prod  45s   True    Cluster is available

Step 3: Federate a Namespace

Before federating applications, it’s good practice to federate namespaces. Federating a namespace means that KubeFed will ensure this namespace exists in all specified member clusters. This helps in organizing your federated applications and applying consistent policies across clusters. KubeFed uses custom resources like FederatedNamespace to manage these operations.

# my-federated-namespace.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: my-federated-app
  namespace: kubefed-system # Federated resources are often defined in the kubefed-system namespace
spec:
  placement:
    clusters:
    - name: cluster-dev
    - name: cluster-prod
  template:
    metadata:
      labels:
        federated-by: kubefed

# Apply the federated namespace definition to the host cluster
kubectl apply -f my-federated-namespace.yaml --context ${HOST_CLUSTER_CONTEXT}

# Verify the federated namespace
kubectl get federatednamespaces -n kubefed-system --context ${HOST_CLUSTER_CONTEXT}

# Check if the namespace exists in member clusters
kubectl get namespace my-federated-app --context ${MEMBER_CLUSTER_1_CONTEXT}
kubectl get namespace my-federated-app --context ${MEMBER_CLUSTER_2_CONTEXT}

Verify:

You should see the my-federated-app namespace created in both member clusters:

# kubectl get federatednamespaces -n kubefed-system output
NAME                 AGE
my-federated-app     1m

# kubectl get namespace my-federated-app --context cluster-dev-context output
NAME               STATUS   AGE
my-federated-app   Active   45s

# kubectl get namespace my-federated-app --context cluster-prod-context output
NAME               STATUS   AGE
my-federated-app   Active   40s

Step 4: Federate a Deployment

Now, let’s federate a simple Nginx deployment. This means KubeFed will ensure this deployment is created and maintained in the specified member clusters. We’ll use a FederatedDeployment resource, which wraps a standard Kubernetes Deployment and adds placement rules. The override section allows you to customize specific fields for different clusters, which is incredibly powerful for multi-cluster environments.

# my-federated-deployment.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: nginx-deployment
  namespace: my-federated-app # Must be a federated namespace
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 2
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:latest
            ports:
            - containerPort: 80
  placement:
    clusters:
    - name: cluster-dev
    - name: cluster-prod
  overrides:
  - clusterName: cluster-prod
    clusterOverride:
      spec:
        replicas: 3 # Production cluster gets 3 replicas
  - clusterName: cluster-dev
    clusterOverride:
      spec:
        template:
          spec:
            containers:
            - name: nginx
              image: nginx:1.21.6 # Dev cluster uses an older Nginx version

# Apply the federated deployment definition to the host cluster
kubectl apply -f my-federated-deployment.yaml --context ${HOST_CLUSTER_CONTEXT}

# Verify the federated deployment
kubectl get federateddeployments -n my-federated-app --context ${HOST_CLUSTER_CONTEXT}

# Check the deployments in member clusters
echo "Checking deployment in ${MEMBER_CLUSTER_1_NAME} (Dev):"
kubectl get deployment nginx-deployment -n my-federated-app --context ${MEMBER_CLUSTER_1_CONTEXT} -o yaml | grep -E "replicas:|image:"

echo "Checking deployment in ${MEMBER_CLUSTER_2_NAME} (Prod):"
kubectl get deployment nginx-deployment -n my-federated-app --context ${MEMBER_CLUSTER_2_CONTEXT} -o yaml | grep -E "replicas:|image:"

Verify:

You should see the deployments created in both clusters, with the overrides applied:

# kubectl get federateddeployments -n my-federated-app output
NAME             AGE
nginx-deployment 1m

# Checking deployment in cluster-dev (Dev):
  replicas: 2
        image: nginx:1.21.6

# Checking deployment in cluster-prod (Prod):
  replicas: 3
        image: nginx:latest

Step 5: Federate a Service

Federating services is crucial for consistent application access across clusters. A FederatedService ensures that a service is created in the specified member clusters. This can be combined with global load balancing solutions (like DNS-based load balancing) to provide a single entry point for an application spanning multiple clusters. For advanced traffic management, consider integrating with a service mesh like Istio Ambient Mesh or using the Kubernetes Gateway API.

# my-federated-service.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
  name: nginx-service
  namespace: my-federated-app
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      selector:
        app: nginx
      ports:
        - protocol: TCP
          port: 80
          targetPort: 80
      type: LoadBalancer # Or ClusterIP, NodePort depending on your needs
  placement:
    clusters:
    - name: cluster-dev
    - name: cluster-prod
  overrides:
  - clusterName: cluster-prod
    clusterOverride:
      spec:
        type: LoadBalancer # Ensure LoadBalancer in prod
  - clusterName: cluster-dev
    clusterOverride:
      spec:
        type: ClusterIP # Use ClusterIP in dev to save costs

# Apply the federated service definition to the host cluster
kubectl apply -f my-federated-service.yaml --context ${HOST_CLUSTER_CONTEXT}

# Verify the federated service
kubectl get federatedservices -n my-federated-app --context ${HOST_CLUSTER_CONTEXT}

# Check the services in member clusters
echo "Checking service in ${MEMBER_CLUSTER_1_NAME} (Dev):"
kubectl get service nginx-service -n my-federated-app --context ${MEMBER_CLUSTER_1_CONTEXT}

echo "Checking service in ${MEMBER_CLUSTER_2_NAME} (Prod):"
kubectl get service nginx-service -n my-federated-app --context ${MEMBER_CLUSTER_2_CONTEXT}

Verify:

You should see the services created in both clusters, with the correct types applied according to the overrides:

# kubectl get federatedservices -n my-federated-app output
NAME            AGE
nginx-service   1m

# Checking service in cluster-dev (Dev):
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
nginx-service   ClusterIP   10.xx.xx.xx      <none>        80/TCP    45s

# Checking service in cluster-prod (Prod):
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)   AGE
nginx-service   LoadBalancer   10.yy.yy.yy      a123...elb.us-east-1.elb.amazonaws.com   80:3xxxx/TCP   40s

Step 6: Federate Other Resources (Optional)

KubeFed supports federating a wide range of Kubernetes resources. You can federate ConfigMaps, Secrets, Ingresses, Custom Resource Definitions (CRDs), and even Network Policies. The pattern remains the same: create a Federated<ResourceType>, define its template, placement, and optionally overrides. This flexibility allows for comprehensive multi-cluster management.

Here’s an example of federating a ConfigMap:

# my-federated-configmap.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedConfigMap
metadata:
  name: my-app-config
  namespace: my-federated-app
spec:
  template:
    data:
      environment: default
      logLevel: INFO
  placement:
    clusters:
    - name: cluster-dev
    - name: cluster-prod
  overrides:
  - clusterName: cluster-dev
    clusterOverride:
      data:
        environment: development
        logLevel: DEBUG
  - clusterName: cluster-prod
    clusterOverride:
      data:
        environment: production
        logLevel: ERROR

# Apply the federated configmap definition
kubectl apply -f my-federated-configmap.yaml --context ${HOST_CLUSTER_CONTEXT}

# Verify in member clusters
echo "Checking ConfigMap in ${MEMBER_CLUSTER_1_NAME} (Dev):"
kubectl get configmap my-app-config -n my-federated-app --context ${MEMBER_CLUSTER_1_CONTEXT} -o yaml | grep -E "environment:|logLevel:"

echo "Checking ConfigMap in ${MEMBER_CLUSTER_2_NAME} (Prod):"
kubectl get configmap my-app-config -n my-federated-app --context ${MEMBER_CLUSTER_2_CONTEXT} -o yaml | grep -E "environment:|logLevel:"

Verify:

The ConfigMap data should reflect the overrides in each cluster:

# Checking ConfigMap in cluster-dev (Dev):
  environment: development
  logLevel: DEBUG

# Checking ConfigMap in cluster-prod (Prod):
  environment: production
  logLevel: ERROR

Production Considerations

Deploying KubeFed in a production environment requires careful planning and consideration beyond the basic setup:

1. Host Cluster Resiliency: The KubeFed host cluster becomes a critical component. Ensure it is highly available, perhaps using a managed Kubernetes service (EKS, GKE, AKS) with multiple availability zones. Consider backup strategies for its etcd data.
2. Security and RBAC:
   • Host Cluster Access: Restrict access to the KubeFed host cluster. Only authorized personnel or CI/CD systems should be able to create or modify Federated* resources.
   • Member Cluster RBAC: KubeFed creates a ServiceAccount and ClusterRoleBindings in each member cluster. Regularly audit these permissions to ensure the KubeFed control plane has only the necessary privileges (least privilege). For robust security, investigate tools like Sigstore and Kyverno for policy enforcement.
   • Network Security: Secure the communication channels between the host cluster and member cluster API servers. Use private endpoints, VPNs, or VPC peering where possible.
3. Network Latency and Bandwidth: While KubeFed doesn’t route application traffic, it does communicate with member cluster API servers. High latency or low bandwidth between the host and member clusters can impact the responsiveness of federation operations.
4. Observability and Monitoring:
   • Monitor the KubeFed control plane pods and their logs for errors.
   • Monitor the status of KubefedCluster resources to ensure all member clusters are healthy and reachable.
   • Implement cluster-wide logging and monitoring solutions. For advanced eBPF-based observability, check out eBPF Observability with Hubble.
5. Disaster Recovery:
   • What happens if the host cluster goes down? While member clusters will continue to run their applications, you won’t be able to make federated changes. Consider a backup and restore strategy for the KubeFed control plane or a multi-host KubeFed setup (though this adds complexity).
   • For member clusters, ensure your applications are designed for multi-cluster failover, potentially using DNS-based global load balancing.
6. Resource Placement and Scheduling: KubeFed’s placement rules are static. For dynamic, intelligent workload placement based on cluster load, cost, or other metrics, you might need to integrate with external schedulers or consider solutions like Karpenter for node-level autoscaling within each cluster, though this is distinct from cross-cluster workload balancing.
7. Configuration Drift: While KubeFed aims for consistency, manual changes on member clusters can lead to configuration drift. Implement CI/CD pipelines that exclusively manage resources via KubeFed to prevent this.
8. Version Compatibility: Ensure your KubeFed version is compatible with the Kubernetes versions of your host and member clusters. Refer to the KubeFed compatibility matrix.

Troubleshooting

1. Issue: kubefedctl command not found.

   Solution: Ensure kubefedctl is installed correctly and its directory is in your system’s PATH. If you moved it to /usr/local/bin/, verify that directory is in your PATH.

   
   echo $PATH
   ls -l /usr/local/bin/kubefedctl
           

   If not found, re-download and move it, or add the directory where you extracted it to your PATH.

2. Issue: KubeFed pods are not running in kubefed-system namespace.

   Solution: Check the status of the pods and their logs. Common causes include insufficient resources, incorrect RBAC for the Helm deployment, or network issues preventing image pull.

   
   kubectl get pods -n kubefed-system -o wide
   kubectl describe pod <kubefed-controller-manager-pod-name> -n kubefed-system
   kubectl logs <kubefed-controller-manager-pod-name> -n kubefed-system
           

3. Issue: Member cluster shows False for Ready status in KubefedCluster.

   Solution: This indicates KubeFed cannot communicate with the member cluster’s API server.

   • Check network connectivity: Ensure the host cluster can reach the member cluster’s API endpoint (firewalls, security groups).
   • Verify kubeconfig context: Double-check that the --cluster-context and --host-cluster-context provided during kubefedctl join are correct and have appropriate permissions.
   • Inspect KubeFed logs: The controller manager logs on the host cluster might provide specific errors.
   • Check RBAC on member cluster: Ensure the ServiceAccount created by KubeFed (e.g., kubefed-controller-manager-<host-cluster-name> in kube-system namespace of the member cluster) has the necessary permissions to access resources.

   
   kubectl get kubefedclusters -n kubefed-system --context ${HOST_CLUSTER_CONTEXT}
   kubectl logs -f $(kubectl get pod -l app=kubefed-controller-manager -n kubefed-system -o jsonpath='{.items[0].metadata.name}') -n kubefed-system --context ${HOST_CLUSTER_CONTEXT}
           

4. Issue: Federated resource (e.g., FederatedDeployment) is stuck and not propagating.

   Solution:

   • Check the Federated* resource status: It often contains clues about why propagation failed.
   • Verify namespace existence: Ensure the target namespace (e.g., my-federated-app) exists in the member clusters. If it’s a federated namespace, check its status.
   • Inspect KubeFed logs: The controller logs on the host cluster will show errors related to resource propagation.
   • Check RBAC on member cluster: The KubeFed service account on the member cluster might lack permissions to create the specific resource type (e.g., Deployments).
   • Syntax errors: Validate your Federated* YAML against the KubeFed API reference.

   
   kubectl describe federateddeployment nginx-deployment -n my-federated-app --context ${HOST_CLUSTER_CONTEXT}
           

5. Issue: override section not working as expected.

   Solution:

   • Indentation/YAML syntax: YAML is sensitive to indentation. Ensure the override and clusterOverride sections are correctly indented.
   • Path to field: The path within clusterOverride must exactly match the path in the standard Kubernetes resource. For example, spec.replicas for a Deployment.
   • Cluster name: Ensure clusterName in the override exactly matches the name of your KubefedCluster resource (e.g., cluster-prod, not its context name).

   Double-check the YAML definition carefully against the Kubernetes API documentation for the specific resource type.

6. Issue: Cannot delete a federated cluster or resource.

   Solution: When deleting federated resources or unjoining a cluster, KubeFed will attempt to clean up the propagated resources from the member clusters. If this fails (e.g., due to network issues or permissions), the resource might get stuck in a “terminating” state with finalizers.

   • For stuck KubefedCluster: Use kubefedctl unjoin first. If it’s still stuck, you might need to manually remove finalizers on the KubefedCluster resource on the host cluster and then manually clean up KubeFed-related RBAC from the member cluster.
   • For stuck Federated* resource: Check the events and logs. If cleanup on a member cluster fails, you might need to manually delete the underlying resource on the problematic member cluster, then the federated resource on the host cluster.

   
   # Unjoin a cluster cleanly
   kubefedctl unjoin cluster-prod --host-cluster-context ${HOST_CLUSTER_CONTEXT} --kubefed-namespace kubefed-system
   
   # If a resource is stuck and you know it's safe to force-delete (use with caution!)
   kubectl edit federateddeployment nginx-deployment -n my-federated-app --context ${HOST_CLUSTER_CONTEXT}
   # Remove the 'finalizers' section, then save and exit.
           

FAQ Section

1. What is the difference between Kubernetes Federation v1 and KubeFed (v2)?

   Federation v1 was an in-tree solution that tried to mimic the Kubernetes API server for federated resources, leading to complexity and scalability issues. KubeFed (v2) is an out-of-tree project that uses Custom Resource Definitions (CRDs) and controllers. It offers a more modular, flexible, and extensible approach, allowing users to federate specific resource types and apply overrides easily. KubeFed is the recommended approach for multi-cluster management.

2. Does KubeFed provide global load balancing or multi-cluster service discovery?

   KubeFed primarily focuses on resource propagation and configuration. While it can create services in multiple clusters, it doesn’t inherently provide global load balancing or cross-cluster service discovery (e.g., Pod A in Cluster 1 finding Pod B in Cluster 2 by name). For these capabilities, you would typically integrate with external solutions like a global DNS service (e.g., AWS Route 53, GCP Cloud DNS) or a service mesh like Istio Ambient Mesh that supports multi-cluster configurations.

3. Can KubeFed manage resources across different cloud providers?

   Yes, absolutely! KubeFed is cloud-agnostic. As long as the KubeFed host cluster can reach the API servers of your member clusters (regardless of whether they are on AWS, GCP, Azure, or on-premises), it can manage them. Network connectivity and security between these disparate environments are the main considerations.

4. What types of resources can KubeFed federate?

   KubeFed can federate most standard Kubernetes resources like Deployments, Services, ConfigMaps, Secrets, Namespaces, Ingresses, and even Custom Resource Definitions (CRDs). It provides a mechanism to enable federation for any API resource. You can list the available federated types using kubectl get federatedtypes -n kubefed-system.

5. Is KubeFed suitable for all multi-cluster use cases?

   KubeFed excels at consistent configuration and application deployment across multiple clusters. However, it’s not a silver bullet for all multi-cluster challenges. It doesn’t handle application data replication, active-active failover orchestration (beyond resource propagation), or complex cross-cluster networking beyond basic service exposure. For these advanced scenarios, you might need to combine KubeFed with other tools and architectural patterns.

Cleanup Commands

To remove the KubeFed setup and all federated resources, follow these steps. It’s crucial to unjoin clusters before uninstalling KubeFed to ensure resources are properly cleaned up from member clusters.

HOST_CLUSTER_CONTEXT="host-cluster-context"

MEMBER_CLUSTER_1_NAME="cluster-dev"

MEMBER_CLUSTER_1_CONTEXT="dev-cluster-context"

MEMBER_CLUSTER_2_NAME="cluster-prod"

MEMBER_CLUSTER_2_CONTEXT="prod-cluster-context"

# 1. Unjoin all member clusters

echo "Unjoining ${MEMBER_CLUSTER_1_NAME}..."

kubefedctl unjoin ${MEMBER_CLUSTER_1_NAME} \

    --host-cluster-context ${HOST_CLUSTER_CONTEXT} \

    --kubefed-namespace kubefed-system
echo "Unjoining ${MEMBER_CLUSTER_2_NAME}..."

kubefedctl unjoin ${MEMBER_CLUSTER_2_NAME