Connect Clusters: Linkerd Multi-Cluster Power

Prerequisites

Before diving into the multi-cluster setup, ensure you have the following:

* Kubernetes Clusters: At least two Kubernetes clusters. For this guide, we'll refer to them as `cluster-a` and `cluster-b`. You can use local clusters (like Kind or Minikube) or cloud-managed clusters (EKS, GKE, AKS). Ensure they are running a compatible Kubernetes version (refer to the Linkerd documentation for specifics).
* `kubectl` configured: Your `kubectl` command-line tool must be configured to interact with both clusters. You should have separate `kubeconfig` contexts for each, for example, `kubectl --context=cluster-a` and `kubectl --context=cluster-b`.
* `linkerd` CLI: The Linkerd command-line interface (CLI) must be installed on your local machine.
* Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Services, Deployments, and Namespaces.
* (Optional) Helm: While not strictly required for the basic Linkerd install, Helm can be used for more advanced configurations.

Step-by-Step Guide: Setting up Linkerd Multi-Cluster

This guide will walk you through preparing your clusters, installing Linkerd and its multi-cluster extension, and finally linking them to enable cross-cluster communication.

Step 1: Install Linkerd CLI

The Linkerd CLI is your primary tool for interacting with Linkerd. It allows you to install the service mesh, check its status, inject proxies, and manage multi-cluster configurations. It's essential to have the latest version for compatibility with the Linkerd control plane.

First, download and install the Linkerd CLI. The recommended way is to use the provided `curl` script. Once installed, add it to your system's PATH. After installation, perform a pre-check to ensure your environment meets Linkerd's requirements. This check will verify `kubectl` connectivity, Kubernetes version compatibility, and other prerequisites.

# Download and install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Add Linkerd to your PATH (adjust path if different)
export PATH=$PATH:$HOME/.linkerd2/bin

# Verify installation and environment
linkerd version
linkerd check --pre

Verify:

You should see output similar to this, indicating successful installation and a healthy pre-check:

# linkerd version
Client version: stable-2.14.7
Server version: unavailable

# linkerd check --pre
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the maximum Kubernetes API version

linkerd-version
---------------
√ can fetch current version
√ is running the latest CLI version

Status check results are √

Step 2: Install Linkerd Control Plane on Each Cluster

Now, let's install the Linkerd control plane on both `cluster-a` and `cluster-b`. The control plane consists of various components like the controller, proxy injector, and destination service, which manage and observe the data plane proxies. It's crucial for each cluster to have its own Linkerd control plane to manage local services and enable multi-cluster communication.

We'll first install the Custom Resource Definitions (CRDs) and then the core Linkerd components. Remember to specify the correct `kubeconfig` context for each cluster. For more detailed insights into Linkerd's architecture, you can refer to the official Linkerd architecture documentation.

# --- Install on Cluster-A ---
echo "Installing Linkerd on cluster-a..."
linkerd install --crds | kubectl --context=cluster-a apply -f -
linkerd install | kubectl --context=cluster-a apply -f -

# Verify installation on Cluster-A
echo "Verifying Linkerd on cluster-a..."
linkerd --context=cluster-a check

# --- Install on Cluster-B ---
echo "Installing Linkerd on cluster-b..."
linkerd install --crds | kubectl --context=cluster-b apply -f -
linkerd install | kubectl --context=cluster-b apply -f -

# Verify installation on Cluster-B
echo "Verifying Linkerd on cluster-b..."
linkerd --context=cluster-b check

Verify:

The `linkerd check` command should report that all components are healthy for both clusters. This might take a few moments as the pods spin up.

# Example output for 'linkerd --context=cluster-a check'
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the maximum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd' namespace exists
√ 'linkerd-config' config map exists
√ 'linkerd-identity-issuer' secret exists
√ control plane components are installed
√ control plane proxies are ready
√ control plane webhooks are installed
√ control plane crds are installed

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are valid
√ issuer cert is valid
√ issuer pod has completed startup

linkerd-proxy-injector
----------------------
√ webhook is configured
√ webhook has active endpoints
√ webhook container is healthy
√ webhook pod has completed startup

linkerd-destination
-------------------
√ service profile CRD exists
√ destination service is running
√ destination pod has completed startup

linkerd-controller
------------------
√ controller pod has completed startup

linkerd-tap
-----------
√ tap service is running
√ tap pod has completed startup

Status check results are √

Step 3: Install Linkerd Multi-Cluster Extension on Each Cluster

The Linkerd Multi-Cluster extension provides the necessary components to enable cross-cluster communication, primarily the `linkerd-gateway` and `linkerd-service-mirror` deployments. The gateway acts as an entry point for traffic from other clusters, while the service mirror discovers and replicates service information.

Install the multi-cluster extension on both `cluster-a` and `cluster-b`. This will deploy the `linkerd-multicluster` namespace and its associated resources. After installation, perform a `linkerd check --proxy` to ensure the multi-cluster components, especially their proxies, are running correctly.

# --- Install on Cluster-A ---
echo "Installing Linkerd Multi-Cluster extension on cluster-a..."
linkerd mc install | kubectl --context=cluster-a apply -f -

# Verify Multi-Cluster installation on Cluster-A
echo "Verifying Linkerd Multi-Cluster on cluster-a..."
linkerd --context=cluster-a check --proxy # Check for proxy health

# --- Install on Cluster-B ---
echo "Installing Linkerd Multi-Cluster extension on cluster-b..."
linkerd mc install | kubectl --context=cluster-b apply -f -

# Verify Multi-Cluster installation on Cluster-B
echo "Verifying Linkerd Multi-Cluster on cluster-b..."
linkerd --context=cluster-b check --proxy

Verify:

Look for output indicating the multi-cluster components are healthy. Specifically, the `linkerd-multicluster` section should show successful deployment.

# Example output for 'linkerd --context=cluster-a check --proxy'
...
linkerd-multicluster
--------------------
√ 'linkerd-multicluster' namespace exists
√ 'linkerd-gateway' deployment exists
√ 'linkerd-gateway' pods are ready
√ 'linkerd-service-mirror' deployment exists
√ 'linkerd-service-mirror' pods are ready
...
Status check results are √

Step 4: Create a Gateway on Each Cluster

The Linkerd Multi-Cluster gateway is a crucial component that exposes a secure entry point for traffic coming from other clusters. It's essentially a load balancer that directs incoming multi-cluster requests to the correct services within its own cluster. Each cluster participating in the multi-cluster setup needs its own gateway.

The `linkerd mc gateway install` command generates the necessary Kubernetes manifests for the gateway, including a `Deployment` and a `Service` of type `LoadBalancer`. This `LoadBalancer` service will expose an external IP address (or hostname) that other clusters will use to connect to this gateway.

# --- Install Gateway on Cluster-A ---
echo "Installing Multi-Cluster Gateway on cluster-a..."
linkerd mc gateway install | kubectl --context=cluster-a apply -f -

# --- Install Gateway on Cluster-B ---
echo "Installing Multi-Cluster Gateway on cluster-b..."
linkerd mc gateway install | kubectl --context=cluster-b apply -f -

Verify:

Wait for the `linkerd-gateway` service to get an external IP address or hostname. This might take a few minutes depending on your cloud provider or local Kubernetes setup.

# For Cluster-A
kubectl --context=cluster-a -n linkerd-multicluster get svc linkerd-gateway

# For Cluster-B
kubectl --context=cluster-b -n linkerd-multicluster get svc linkerd-gateway

Expected output (the `EXTERNAL-IP` will vary):

# Example for cluster-a
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
linkerd-gateway   LoadBalancer   10.43.200.123   192.0.2.100     4143:30000/TCP   2m

# Example for cluster-b
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
linkerd-gateway   LoadBalancer   10.43.150.234   198.51.100.200  4143:30000/TCP   2m

Step 5: Link the Clusters

This is the core step where you establish the connection between your clusters. Linking involves configuring one cluster's service mirror to connect to another cluster's gateway. This creates a secure, mTLS-protected tunnel between the two. The `linkerd mc link` command generates a `ServiceMirror` resource that defines this connection.

You need to run this command *from* the cluster that will mirror services *to* the cluster whose gateway it will connect to. For bidirectional communication, you'll run it twice, once for each direction. For instance, to allow `cluster-a` to see services in `cluster-b`, you'll run the command from `cluster-a` targeting `cluster-b`'s gateway.

# --- Link Cluster-A to Cluster-B ---
# This command generates a ServiceMirror config for cluster-a, allowing it to mirror services from cluster-b.
# The output is applied to cluster-a.
echo "Linking cluster-a to cluster-b..."
linkerd --context=cluster-a mc link --cluster-name cluster-b | kubectl --context=cluster-a apply -f -

# --- Link Cluster-B to Cluster-A ---
# This command generates a ServiceMirror config for cluster-b, allowing it to mirror services from cluster-a.
# The output is applied to cluster-b.
echo "Linking cluster-b to cluster-a..."
linkerd --context=cluster-b mc link --cluster-name cluster-a | kubectl --context=cluster-b apply -f -

Verify:

Check the status of the service mirror in each cluster. You should see it connected to the remote cluster's gateway.

# For Cluster-A (checking connection to cluster-b)
kubectl --context=cluster-a -n linkerd-multicluster get servicemirror

# For Cluster-B (checking connection to cluster-a)
kubectl --context=cluster-b -n linkerd-multicluster get servicemirror

Expected output:

# Example for cluster-a
NAME        TRUSTED_GATEWAY   REMOTE_CLUSTER_NAME   REMOTE_GATEWAY_ADDR     MIRROR_STATUS   AGE
cluster-b   True              cluster-b             198.51.100.200:4143     Connected       1m

# Example for cluster-b
NAME        TRUSTED_GATEWAY   REMOTE_CLUSTER_NAME   REMOTE_GATEWAY_ADDR     MIRROR_STATUS   AGE
cluster-a   True              cluster-a             192.0.2.100:4143        Connected       1m

Step 6: Deploy Sample Applications and Export Services

To demonstrate cross-cluster communication, we'll deploy a simple `hello-world` service in `cluster-a` and a client application in `cluster-b` that calls it. For services to be discoverable across clusters, they need to be "exported." This tells Linkerd's service mirror to replicate their metadata to the linked clusters.

First, deploy a `hello-world` application in `cluster-a`. Then, use `linkerd mc export service` to create a `Service` and `ServiceProfile` in `cluster-b` that mirrors the `hello-world` service from `cluster-a`. The target service name in `cluster-b` will follow the pattern `..svc..linkerd.svc.cluster.local`. This is a fully qualified domain name (FQDN) that allows services in `cluster-b` to resolve and connect to `hello-world` in `cluster-a`.

# hello-world-a.yaml (to be deployed in cluster-a)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
      annotations:
        linkerd.io/inject: enabled # Ensure Linkerd proxy is injected
    spec:
      containers:
      - name: hello-world
        image: ghcr.io/buoyantio/example-hello-world:v0.1.0
        ports:
        - containerPort: 7000
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  namespace: default
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 7000
      targetPort: 7000

# Deploy hello-world in cluster-a
kubectl --context=cluster-a create ns default || true
kubectl --context=cluster-a apply -f hello-world-a.yaml

# Wait for deployment to be ready
kubectl --context=cluster-a -n default rollout status deploy/hello-world

# Export hello-world service from cluster-a to cluster-b
# This generates the mirrored service definition for cluster-b
linkerd --context=cluster-a mc export service hello-world --namespace default \
  | kubectl --context=cluster-b apply -f -

Verify:

Check for the mirrored service in `cluster-b`.

# Check for the mirrored service in cluster-b
kubectl --context=cluster-b -n default get service hello-world-cluster-a

Expected output:

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hello-world-cluster-a   ClusterIP   10.43.123.45    <none>        7000/TCP   1m

Step 7: Deploy Client Application and Test Cross-Cluster Communication

Now, let's deploy a client application in `cluster-b` that attempts to connect to the `hello-world` service running in `cluster-a`. The client will use the FQDN generated by the service mirror to resolve the remote service.

We'll use a simple `curl` pod to make the request. Notice how the `CURL_TARGET` environment variable points to `hello-world.default.svc.cluster-a.linkerd.svc.cluster.local`. This FQDN is crucial for Linkerd's multi-cluster DNS resolution.

# client-b.yaml (to be deployed in cluster-b)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
      annotations:
        linkerd.io/inject: enabled # Ensure Linkerd proxy is injected
    spec:
      containers:
      - name: client
        image: curlimages/curl:7.83.1
        command: ["sh", "-c", "while true; do curl -s $CURL_TARGET; echo; sleep 5; done"]
        env:
        - name: CURL_TARGET
          value: http://hello-world.default.svc.cluster-a.linkerd.svc.cluster.local:7000

# Deploy client in cluster-b
kubectl --context=cluster-b create ns default || true
kubectl --context=cluster-b apply -f client-b.yaml

# Wait for client deployment to be ready
kubectl --context=cluster-b -n default rollout status deploy/client

Verify:

Check the logs of the `client` pod in `cluster-b`. You should see successful responses from the `hello-world` service in `cluster-a`.

# Get the name of the client pod in cluster-b
CLIENT_POD_B=$(kubectl --context=cluster-b -n default get pod -l app=client -o jsonpath='{.items[0].metadata.name}')

# Stream logs from the client pod
kubectl --context=cluster-b -n default logs $CLIENT_POD_B -f

Expected output from the `client` pod logs:

# Example from client pod logs in cluster-b
Hello from hello-world (cluster-a)!
Hello from hello-world (cluster-a)!
...

This confirms that the client in `cluster-b` is successfully communicating with the `hello-world` service in `cluster-a` through the Linkerd Multi-Cluster gateway and service mirror.

Production Considerations

Deploying Linkerd Multi-Cluster in a production environment requires careful planning beyond a basic setup. Here are key considerations:

1. Security and Trust Anchors: Linkerd uses mTLS by default for all communication. For multi-cluster, it's critical that all clusters share a common trust anchor for their identity certificates. This can be achieved by distributing the same trust anchor certificate to all Linkerd control planes. Consider using a centralized certificate management solution like Vault or cert-manager for managing and rotating these trust anchors. For enhancing your supply chain security, integrating with tools like Sigstore and Kyverno can further harden your cluster.
2. Network Connectivity: Ensure that the `linkerd-gateway` LoadBalancer IP addresses are reachable from all participating clusters. This might involve configuring firewall rules, VPNs, or direct connect links between your cloud environments or on-premises data centers. For secure pod-to-pod encryption within and across clusters, solutions like Cilium WireGuard Encryption can be highly beneficial.
3. Latency and Performance: Cross-cluster communication inherently introduces network latency. Design your applications to be resilient to this. Place services that communicate frequently in the same cluster or geographically closer clusters. Linkerd's metrics can help you observe and diagnose latency issues.
4. Observability: Linkerd provides rich observability with its `tap`, `stat`, and `dashboard` features. Ensure these are configured and monitored for your multi-cluster setup. You can use the Linkerd dashboard to visualize traffic flows and health across your linked clusters. For deeper insights, consider integrating with eBPF-based tools like eBPF Observability with Hubble.
5. DNS Resolution: Understand how Linkerd handles DNS resolution for mirrored services. The FQDN format (`..svc..linkerd.svc.cluster.local`) is specific. Ensure your applications use this format or are configured to use a service discovery mechanism that translates to it.
6. Resource Management: Monitor the resource consumption of Linkerd control plane and data plane proxies. Adjust resource requests and limits as needed. For node scaling and cost optimization, tools like Karpenter Cost Optimization can automatically provision and de-provision nodes based on workload demands.
7. Traffic Management: While Linkerd provides basic traffic routing, for advanced scenarios (e.g., A/B testing, canary deployments across clusters), you might integrate with solutions like the Kubernetes Gateway API or even a full-fledged service mesh like Istio Ambient Mesh (though Linkerd aims for simplicity over such complexity).
8. Network Policies: Even with Linkerd's mTLS, Kubernetes Network Policies remain a crucial layer of defense. Define granular network policies to restrict communication between namespaces and pods, adding an extra layer of security. Our Network Policies Security Guide can provide further insights.
9. Disaster Recovery: Multi-cluster is often used for DR. Plan your failover strategies and ensure your service mirroring and client configurations can gracefully handle the loss of an entire cluster.
10. Upgrade Strategy: Plan for rolling upgrades of Linkerd control planes and proxies across your clusters to minimize downtime. Always test upgrades in a staging environment first.

Troubleshooting

Here are common issues you might encounter and their solutions when setting up Linkerd Multi-Cluster:

1. Issue: `linkerd check` fails after installation.
* Problem: Control plane pods are not ready, or webhooks are not configured correctly.
* Solution:
* Check pod status in the `linkerd` and `linkerd-multicluster` namespaces: `kubectl get pods -n linkerd`, `kubectl get pods -n linkerd-multicluster`.
* View logs of failing pods: `kubectl logs -n linkerd `.
* Ensure your Kubernetes cluster meets the minimum version requirements.
* If webhooks are failing, ensure your `kube-apiserver` can reach the webhook service. This might require network policy adjustments or ensuring `enable-admission-plugins` includes `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook`.

2. Issue: `linkerd mc link` fails or `ServiceMirror` shows `Disconnected`.
* Problem: The remote gateway is unreachable, or there's a certificate mismatch.
* Solution:
* Verify the `linkerd-gateway` service in the remote cluster has an external IP and is accessible from the local cluster. Use `ping` or `nc -vz 4143` from a pod in the local cluster.
* Check firewall rules between clusters.
* Ensure both clusters are using the same trust anchor for Linkerd identity. If not, re-install Linkerd with a shared trust anchor. You can inspect trust anchors with `linkerd identity trust-anchors`.
* Check the logs of the `linkerd-service-mirror` pod in the cluster experiencing the `Disconnected` status: `kubectl --context= -n linkerd-multicluster logs `.

3. Issue: Mirrored service not appearing in the remote cluster.
* Problem: The `linkerd mc export service` command was not run, or the `ServiceMirror` configuration is incorrect.
* Solution:
* Ensure you ran `linkerd mc export service --namespace | kubectl --context= apply -f -` for the service you want to mirror.
* Verify the `ServiceMirror` resource in the mirroring cluster is `Connected`.
* Check the logs of the `linkerd-service-mirror` pod in the *mirroring* cluster for errors related to service discovery.

4. Issue: Client application cannot resolve or connect to the remote service.
* Problem: Incorrect FQDN, missing proxy injection, or network issues.
* Solution:
* Verify the client application is using the correct FQDN format: `..svc..linkerd.svc.cluster.local`.
* Ensure the client pod has the Linkerd proxy injected (`linkerd.io/inject: enabled` annotation). Check with `linkerd --context= check --proxy`.
* Confirm the mirrored service exists in the client's cluster: `kubectl --context= get svc `.
* Check `linkerd tap deploy/` to see if requests are being initiated by the proxy and where they are going.
* Check the logs of the `linkerd-proxy` sidecar in the client pod for connection errors.

5. Issue: High latency or dropped connections for cross-cluster traffic.
* Problem: Network congestion, misconfigured network, or overloaded gateways.
* Solution:
* Use `linkerd stat` and `linkerd tap` to observe traffic metrics (latency, success rate) on both the client and gateway sides.
* Check underlying network infrastructure for congestion or misconfigurations (e.g., firewalls, routing tables).
* Scale up the `linkerd-gateway` deployment in the destination cluster if it's becoming a bottleneck. You can increase replicas: `kubectl --context= -n linkerd-multicluster scale deploy/linkerd-gateway --replicas=3`.
* Consider the geographical distance between clusters.

6. Issue: `linkerd dashboard` not showing multi-cluster metrics.
* Problem: Dashboard installation missing or misconfigured, or metrics are not being collected.
* Solution:
* Ensure the Linkerd dashboard extension is installed: `linkerd install --dashboard | kubectl apply -f -`.
* Verify the `linkerd-prometheus` and `linkerd-grafana` pods are running in the `linkerd` namespace.
* Check the `linkerd-proxy` logs on your service mirror and gateway pods for any Prometheus scraping errors.
* Ensure `linkerd check` passes for all components, including the dashboard.

FAQ Section

1. What is the difference between Linkerd Multi-Cluster and a Global Service Mesh?
Linkerd Multi-Cluster focuses on connecting services across distinct Kubernetes clusters, each with its own Linkerd control plane. It's a "mesh of meshes" approach, where each cluster maintains its autonomy. A "Global Service Mesh" often implies a single, highly centralized control plane spanning multiple clusters, which can introduce complexity and single points of failure. Linkerd's approach prioritizes simplicity and resilience by keeping control planes localized.

2. Does Linkerd Multi-Cluster support different cloud providers or on-premises clusters?
Yes, Linkerd Multi-Cluster is cloud-agnostic. As long as your Kubernetes clusters can establish network connectivity to each other's `linkerd-gateway` LoadBalancer IP addresses (or hostnames), Linkerd can connect them. This often involves configuring VPNs, cloud peering, or direct connect solutions between heterogeneous environments.

3. How does Linkerd handle DNS resolution for cross-cluster services?
Linkerd uses a specialized DNS suffix for mirrored services: `..svc..linkerd.svc.cluster.local`. When a service in `cluster-A` tries to resolve `my-app.default.svc.cluster-B.linkerd.svc.cluster.local`, the Linkerd proxy intercepts this DNS query, routes it through the `ServiceMirror` to the `linkerd-gateway` in `cluster-B`, which then resolves it to the actual `my-app` service in `cluster-B`.

4. Is cross-cluster communication secure with Linkerd Multi-Cluster?
Absolutely. All communication between clusters via the Linkerd Multi-Cluster gateway is secured with mutual TLS (mTLS). Each Linkerd proxy has an identity certificate issued by its local Linkerd identity component. When clusters are linked, they exchange trust anchors, allowing proxies to verify each other's identities across cluster boundaries, ensuring encrypted and authenticated communication.

5. Can I selectively mirror services, or does it mirror everything?
You have granular control over which services are mirrored. You explicitly use the `linkerd mc export service` command for each service you wish to expose to other clusters. This prevents unintended exposure of internal services and allows you to manage your cross-cluster attack surface.

Cleanup Commands

To remove the Linkerd Multi-Cluster setup and associated applications from your clusters:

# --- Clean up Cluster-A ---