Just-in

Kagent Tutorial: Build Cloud-Native AI Agents on Kubernetes

Running Ollama LLM on Kubernetes: A Complete Guide

From Docker to Kubernetes: The Orchestration Journey You Didn’t Know You Needed

What is Kubeflow and how can it be used?

Getting Started with Kubeflow: A Complete Beginner’s Guide

Orchestration

Master Linkerd: Your Lightweight Service Mesh Guide

Introduction

In the bustling world of microservices, managing inter-service communication, observability, and reliability can quickly become a daunting task. As applications scale, developers face challenges like traffic management, fault tolerance, security, and consistent metrics collection without burdening application code. This is precisely where a service mesh steps in, providing a dedicated infrastructure layer for handling these concerns.

While powerful, some service meshes can introduce significant operational overhead and resource consumption. Enter Linkerd, a lightweight, ultra-fast, and secure service mesh designed specifically for Kubernetes. Unlike its more feature-heavy counterparts, Linkerd focuses on simplicity, performance, and a minimal resource footprint, making it an excellent choice for organizations seeking robust service mesh capabilities without the complexity. It achieves this by leveraging Rust for its data plane proxies, ensuring exceptional speed and efficiency. This guide will walk you through the process of installing, configuring, and leveraging Linkerd to bring enhanced observability, reliability, and security to your Kubernetes applications.

TL;DR: Linkerd Quickstart

Deploy Linkerd and inject it into your services for immediate benefits.


# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sL https://run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin

# Verify Linkerd CLI installation
linkerd version

# Check your Kubernetes cluster for Linkerd compatibility
linkerd check --pre

# Install Linkerd control plane
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Verify Linkerd control plane installation
linkerd check

# Inject Linkerd proxy into a sample deployment (e.g., `emojivoto`)
kubectl create ns emojivoto
kubectl apply -f https://run.linkerd.io/emojivoto.yml -n emojivoto
kubectl get -n emojivoto deploy -o yaml \
  | linkerd inject - \
  | kubectl apply -f -

# Verify injected services
kubectl get pods -n emojivoto
linkerd -n emojivoto check --proxy

# Access Linkerd Dashboard
linkerd dashboard

Prerequisites

Before diving into Linkerd, ensure you have the following:

  • Kubernetes Cluster: A running Kubernetes cluster (version 1.22 or higher is recommended). You can use Minikube, Kind, or a cloud-managed Kubernetes service like GKE, EKS, or AKS. For cloud-specific instructions, refer to the official Kubernetes documentation.
  • kubectl: The Kubernetes command-line tool, configured to connect to your cluster.
  • helm (Optional but Recommended): For managing Kubernetes applications, although we’ll use linkerd install for the core installation. You can find installation instructions on the Helm website.
  • Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, and Namespaces.
  • Internet Connectivity: To download Linkerd CLI and manifests.

Step-by-Step Guide: Installing and Using Linkerd

Step 1: Install the Linkerd CLI

The Linkerd CLI (Command Line Interface) is your primary tool for interacting with Linkerd. It allows you to install the control plane, inspect services, and perform various operational tasks. We’ll download it directly from Linkerd’s official installation script, which places the binary in your ~/.linkerd2/bin directory.


# Download and install the Linkerd CLI
curl --proto '=https' --tlsv1.2 -sL https://run.linkerd.io/install | sh

# Add the Linkerd CLI to your PATH
export PATH=$PATH:$HOME/.linkerd2/bin

Verify CLI Installation

After installation, verify that the CLI is correctly installed and accessible by checking its version. This also confirms that your PATH is set up correctly.


linkerd version

Expected Output:


Client version: stable-2.YY.Z
Server version: unavailable

The “Server version: unavailable” is expected at this stage because the Linkerd control plane has not yet been installed on your cluster.

Step 2: Validate Your Kubernetes Cluster

Before installing the Linkerd control plane, it’s crucial to ensure your Kubernetes cluster meets all the necessary requirements. The linkerd check --pre command performs a series of checks, such as verifying Kubernetes API access, core DNS functionality, and sufficient permissions. This step helps prevent common installation issues.


linkerd check --pre

Verify Cluster Readiness

A successful pre-check should show all checks passing, indicating your cluster is ready for Linkerd installation.


linkerd check --pre

Expected Output:


kubernetes
    [...]
    √ control plane pods are running
    √ no unschedulable pods
    √ linkerd-config config maps exist
    √ linkerd-tap-rbac proxy-injector can create serviceaccounttoken
    [...]
    
Status check results are √

If any checks fail, address the reported issues before proceeding. Common problems include insufficient RBAC permissions or an unhealthy Kubernetes control plane.

Step 3: Install the Linkerd Control Plane

The Linkerd control plane consists of several components that manage and monitor the data plane proxies. These include the controller, proxy injector, identity service, and various metrics components. We’ll first install the Custom Resource Definitions (CRDs) and then the core control plane components. The linkerd install command generates the necessary Kubernetes manifests, which we then apply using kubectl apply -f -.


# Install Linkerd CRDs
linkerd install --crds | kubectl apply -f -

# Wait a moment for CRDs to be registered (optional, but good practice)
sleep 5

# Install the Linkerd control plane
linkerd install | kubectl apply -f -

Verify Control Plane Installation

Once the control plane components are deployed, use linkerd check to verify that all components are running correctly and communicating as expected. This command performs a more comprehensive check, including connectivity to the data plane (even though proxies aren’t yet injected).


linkerd check

Expected Output:


kubernetes
    [...]
    √ control plane pods are running
    √ no unschedulable pods
    [...]
linkerd-control-plane
    √ controller Running
    √ web Running
    √ identity Running
    √ proxy-injector Running
    √ sp-validator Running
    √ tap Running
    √ grafana Running
    √ prometheus Running
    [...]

Status check results are √

If any components are not running or checks fail, investigate the logs of the relevant Linkerd pods in the linkerd namespace.

Step 4: Inject Linkerd into Your Applications

This is where the magic happens! To leverage Linkerd’s capabilities, you need to “mesh” your applications by injecting the Linkerd data plane proxy into their pods. The linkerd inject command modifies your Kubernetes deployment manifests to automatically add an init container and a sidecar proxy container to each pod. This sidecar intercepts all incoming and outgoing network traffic, enabling Linkerd’s features.

For demonstration, we’ll use the Emojivoto sample application, a simple microservices application that showcases various Linkerd features. First, deploy Emojivoto without Linkerd, then inject it.


# Create a namespace for the sample app
kubectl create ns emojivoto

# Deploy Emojivoto application (unmeshed)
kubectl apply -f https://run.linkerd.io/emojivoto.yml -n emojivoto

# Wait for pods to be ready
kubectl rollout status deployment -n emojivoto web
kubectl rollout status deployment -n emojivoto emoji
kubectl rollout status deployment -n emojivoto voting

# Inject the Linkerd proxy into the Emojivoto deployment
# We fetch the current deployment YAML, pipe it to `linkerd inject`,
# and then apply the modified YAML back to the cluster.
kubectl get -n emojivoto deploy -o yaml \
  | linkerd inject - \
  | kubectl apply -f -

Verify Proxy Injection

After injection, check the pods in the emojivoto namespace. You should see that each application pod now has 2/2 ready containers (your application container + the Linkerd proxy container).


kubectl get pods -n emojivoto

Expected Output:


NAME                        READY   STATUS    RESTARTS   AGE
emoji-698f797d84-z8l7c      2/2     Running   0          2m
web-7f8976785b-m5s2n        2/2     Running   0          2m
voting-7975f9797c-q7w8t     2/2     Running   0          2m

You can also use linkerd check --proxy to specifically verify the data plane proxies.


linkerd -n emojivoto check --proxy

Expected Output:


linkerd-proxy-injector
    [...]
linkerd-tap
    [...]
emojivoto namespace proxies
    √ emojivoto-web Pods are injected
    √ emojivoto-emoji Pods are injected
    √ emojivoto-voting Pods are injected
    √ emojivoto-web Deployments are ready
    √ emojivoto-emoji Deployments are ready
    √ emojivoto-voting Deployments are ready

Status check results are √

Step 5: Explore the Linkerd Dashboard

The Linkerd dashboard provides a rich, real-time view into the health, performance, and topology of your meshed services. It’s an invaluable tool for debugging and monitoring. It includes Grafana for metrics visualization and Tap for real-time request inspection.


linkerd dashboard

This command will open a web browser to the Linkerd dashboard. Navigate to the emojivoto namespace and observe the live metrics for your services. You can generate traffic to the Emojivoto application to see metrics populate:

  1. Find the web service’s external IP or port-forward it: 
    
kubectl -n emojivoto port-forward svc/web 8080:80
  2. Access http://localhost:8080 in your browser and click on emojis to generate traffic.
  3. Observe the dashboard for live metrics like success rates, latencies, and requests per second.

The dashboard is a powerful tool for understanding service behavior. For instance, you can use it to pinpoint services with high error rates or latency spikes, which can be critical for maintaining application performance. For more advanced observability, consider integrating with tools like Prometheus and Grafana, which Linkerd provides out-of-the-box. You can also explore eBPF Observability with Hubble for even deeper network insights.

Step 6: Traffic Management with Service Profiles

Linkerd’s traffic management capabilities are primarily driven by Service Profiles. A Service Profile defines expected request patterns for a service, including routes, retries, and timeouts. This allows Linkerd proxies to apply intelligent routing, retry logic, and collect per-route metrics. Without Service Profiles, Linkerd treats all traffic to a service as a single route.

Let’s create a Service Profile for the web service in the Emojivoto application. This will enable Linkerd to track individual routes and apply policies.


# web-service-profile.yaml
apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: web.emojivoto.svc.cluster.local
  namespace: emojivoto
spec:
  routes:
  - name: /api/vote
    condition:
      method: POST
      pathRegex: /api/vote
    responseClasses:
    - condition:
        status:
          range:
            min: 500
            max: 599
      isFailure: true
  - name: /api/emojis
    condition:
      method: GET
      pathRegex: /api/emojis
    responseClasses:
    - condition:
        status:
          range:
            min: 500
            max: 599
      isFailure: true
  # Catch-all for other routes
  - name: default
    condition:
      method: GET
      pathRegex: ".*"
    responseClasses:
    - condition:
        status:
          range:
            min: 500
            max: 599
      isFailure: true
    

# Apply the Service Profile
kubectl apply -f web-service-profile.yaml -n emojivoto

Verify Service Profile

You can verify that the Service Profile is active and examine its configuration using the Linkerd CLI.


linkerd -n emojivoto routes svc/web

Expected Output:


ROUTE         SERVICE   SUCCESS   LATENCY_P50   LATENCY_P95   LATENCY_P99   RPS
/api/emojis   web       100.00%           1ms           2ms           3ms   1.5
/api/vote     web       100.00%           1ms           2ms           3ms   0.5
default       web       100.00%           1ms           2ms           3ms   0.1

Now, when you generate traffic to the Emojivoto app, you’ll see per-route metrics in the dashboard and via the CLI, providing granular insights into your application’s behavior.

Step 7: Enforcing Policy with Server and ServerAuthorization

Linkerd provides powerful policy features to control which services can communicate with each other. This is crucial for security and compliance. It uses Server and ServerAuthorization resources to define how incoming connections to a meshed pod are handled and authorized. This complements Kubernetes Network Policies Security Guide by operating at the application layer.

Let’s create a policy that only allows the web service to call the emoji service.


# emoji-server.yaml
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: emoji-server
  namespace: emojivoto
spec:
  podSelector:
    matchLabels:
      app: emoji
  port: 8080 # The port your application listens on
    

# emoji-server-authorization.yaml
apiVersion: policy.linkerd.io/v1beta2
kind: ServerAuthorization
metadata:
  name: emoji-access-by-web
  namespace: emojivoto
spec:
  server:
    selector:
      matchLabels:
        linkerd.io/protocol-for-port-8080: http # Matches the Server resource
  client:
    meshTLS:
      unauthenticatedRoutes:
        - name: web-service-route
          serviceAccount:
            name: web
            namespace: emojivoto
    

# Apply the Server and ServerAuthorization
kubectl apply -f emoji-server.yaml -n emojivoto
kubectl apply -f emoji-server-authorization.yaml -n emojivoto

Verify Policy Enforcement

After applying the policy, the emoji service should only accept connections from the web service. If another meshed service (hypothetically) tried to connect to emoji, it would be denied.

You can observe this in the dashboard or by checking the success rates for the emoji service. Any unauthorized connection attempts would manifest as failed requests (e.g., 403 Forbidden).

Step 8: Automatic Proxy Injection with Namespace Annotation

Manually injecting Linkerd into each deployment can be tedious. A more efficient way is to annotate a namespace, telling Linkerd’s proxy injector to automatically inject proxies into any new pods created within that namespace.


# Annotate the emojivoto namespace for automatic injection
kubectl annotate namespace emojivoto linkerd.io/inject=enabled

# Confirm the annotation
kubectl get ns emojivoto -o yaml | grep linkerd.io/inject

Expected Output:


    linkerd.io/inject: enabled
    

# Restart the emojivoto deployments to pick up the new annotation
# This will trigger a rolling update, and new pods will be injected automatically
kubectl rollout restart deployment -n emojivoto web emoji voting

Verify Automatic Injection

After restarting the deployments, verify that the new pods still have 2/2 ready containers, indicating successful automatic injection.


kubectl get pods -n emojivoto

Expected Output:


NAME                        READY   STATUS    RESTARTS   AGE
emoji-698f797d84-abcde      2/2     Running   0          30s
web-7f8976785b-fghij        2/2     Running   0          30s
voting-7975f9797c-klmno     2/2     Running   0          30s

This approach simplifies managing Linkerd injection across your applications, especially in dynamic environments. For large-scale deployments, managing injection via annotations is the recommended practice.

Production Considerations

Deploying Linkerd in a production environment requires careful planning and consideration beyond a basic installation:

  • Resource Allocation: While lightweight, Linkerd proxies and the control plane still consume resources. Monitor their CPU and memory usage and adjust requests/limits accordingly. The Linkerd documentation on proxy configuration provides guidance.
  • High Availability: Ensure your Linkerd control plane components are deployed with sufficient replicas and anti-affinity rules to spread them across nodes for high availability. The default installation often uses 1 replica; consider increasing this for critical environments.
  • Observability Integration: Linkerd provides built-in Prometheus and Grafana. Integrate these with your existing monitoring stack. Export metrics to a centralized observability platform like Datadog, New Relic, or an ELK stack.
  • Security Best Practices:
    • Mutual TLS (mTLS): Linkerd automatically enables mTLS between meshed services. Ensure your certificates are properly managed and rotated.
    • Policy Enforcement: Leverage Server and ServerAuthorization for fine-grained access control. This is a critical security layer, much like a firewall at the application level.
    • Image Security: Use trusted container images for Linkerd components. Consider tools like Sigstore and Kyverno for ensuring supply chain security.
  • Upgrade Strategy: Plan for smooth Linkerd upgrades. Test upgrades in a staging environment first. Linkerd generally supports rolling upgrades without downtime. Refer to the official upgrade guide.
  • Traffic Management: Implement Service Profiles for all critical services to gain granular control over retries, timeouts, and per-route metrics. This is essential for building resilient microservices.
  • External Traffic: For ingress traffic, consider using an Ingress Controller or the Kubernetes Gateway API in conjunction with Linkerd. Linkerd can mesh the backend services, while the ingress handles edge routing.
  • Egress Traffic: For controlling and observing egress traffic, especially to external services, consider using Linkerd’s egress features or combining it with network policies. For advanced network control, explore solutions like Cilium WireGuard Encryption for network-level security.
  • Cost Optimization: While Linkerd is lightweight, every component adds to resource usage. Monitor the cost of your Kubernetes cluster, and for general cost optimization, look into tools like Karpenter Cost Optimization for efficient node provisioning.

Troubleshooting

Here are some common issues you might encounter with Linkerd and their solutions:

  1. Issue: linkerd check shows “control plane pods are not running”.

    Solution: This indicates that Linkerd control plane components failed to start.

    
kubectl get pods -n linkerd
kubectl describe pod <failing-pod-name> -n linkerd
kubectl logs <failing-pod-name> -n linkerd

    Check pod events and logs for specific errors (e.g., image pull failures, resource constraints, RBAC issues). Ensure your cluster has enough resources.

  2. Issue: Application pods show 1/2 Ready after injection, or linkerd check --proxy fails.

    Solution: The Linkerd proxy sidecar is likely not starting correctly.

    
kubectl get pods -n <namespace> -o wide
kubectl describe pod <failing-app-pod-name> -n <namespace>
kubectl logs <failing-app-pod-name> -c linkerd-proxy -n <namespace>

    Look for errors in the linkerd-proxy container’s logs. Common causes include:

    • Application not listening on the expected port (Linkerd defaults to redirecting traffic to 8080, 80, 443 etc., or the targetPort of the Service).
    • Insufficient resources for the proxy.
    • Pod security policies preventing sidecar injection.
  3. Issue: Dashboard shows no traffic or “N/A” for success rates/latencies.

    Solution:

    • No traffic generation: Ensure you are actively sending requests to your meshed services.
    • Proxy not injected: Verify pods are 2/2 Ready and linkerd check --proxy passes.
    • Service Profile missing/incorrect: If you expect per-route metrics, ensure you have a correct ServiceProfile deployed for the service.
    • Network issues: Check if the Linkerd control plane components (e.g., linkerd-proxy-api, linkerd-prometheus) can reach the proxies.
    • Firewall rules: Ensure no firewall rules are blocking communication between the proxy and the control plane components.
  4. Issue: Application connections fail after Linkerd injection.

    Solution: This often points to a proxy misconfiguration or an application incompatibility.

    • Application port: Explicitly tell Linkerd which port your application listens on, if it’s not a common one (80, 443, 8080). You can do this by annotating the deployment: linkerd.io/inject: enabled and config.linkerd.io/opaque-ports: "4444,5000" (if ports 4444, 5000 should bypass Linkerd’s protocol detection). Or, specify the port in the Service definition targetPort.
    • Protocol detection: Linkerd automatically detects HTTP/1.x, HTTP/2, and gRPC. If your application uses a non-standard protocol on a standard port, you might need to mark it as opaque.
    • Policy issues: If you have ServerAuthorization rules, ensure the client service’s service account is explicitly authorized.
  5. Issue: linkerd install fails with RBAC errors.

    Solution: Your Kubernetes user or service account lacks the necessary permissions to create or modify cluster-scoped resources.

    
# Ensure your current context has admin privileges or similar
kubectl config current-context
# Try with a user that has cluster-admin role
kubectl create clusterrolebinding <your-user>-cluster-admin --clusterrole=cluster-admin --user=<your-user-name>

    Refer to Kubernetes RBAC documentation.

  6. Issue: High CPU/memory usage by Linkerd proxies or control plane.

    Solution: While Linkerd is lightweight, high traffic or a large number of services can increase resource consumption.

    • Proxy resources: Adjust the CPU and memory limits/requests for the Linkerd proxy. You can configure this via annotations on your deployments: 
      
annotations:
  config.linkerd.io/proxy-cpu-limit: "100m"
  config.linkerd.io/proxy-memory-limit: "128Mi"
  config.linkerd.io/proxy-cpu-request: "10m"
  config.linkerd.io/proxy-memory-request: "16Mi"
    • Control plane resources: Scale up control plane components (e.g., linkerd-controller, linkerd-proxy-injector) by increasing replicas or adjusting their resource requests/limits in the Linkerd Helm chart values or using linkerd install --set <key>=<value>.

FAQ Section

  1. What is the difference between Linkerd and Istio?

    Linkerd and Istio are both service meshes, but they have different philosophies. Linkerd focuses on being lightweight, simple, and performant, with a data plane written in Rust. It’s often preferred for its lower resource footprint and ease of use. Istio is a more feature-rich and complex service mesh, offering advanced traffic management, policy, and security features, often with a higher operational overhead and resource consumption. For a comparison of service mesh features see our Istio Ambient Mesh Production Guide which highlights some advanced capabilities.

  2. Does Linkerd support HTTP/2 and gRPC?

    Yes, Linkerd automatically detects and supports HTTP/1.x, HTTP/2, and gRPC traffic. It provides rich metrics and features for all these protocols, including per-route metrics for gRPC services.

  3. How does Linkerd handle mTLS (mutual TLS)?

    Linkerd automatically enables mutual TLS (mTLS) for all meshed TCP connections. It establishes a cryptographically verifiable identity for each service, encrypting all communication between meshed pods by default. This is transparent to your applications and provides strong security guarantees. The identity component of the control plane manages certificate issuance and rotation.

  4. Can Linkerd mesh services in different namespaces or clusters?

    Linkerd can mesh services across different namespaces within the same Kubernetes cluster. For multi-cluster scenarios, Linkerd offers a multi-cluster feature that allows services in different clusters to communicate securely and transparently as if they were in the same mesh.

  5. Is Linkerd suitable for production environments?

    Absolutely. Linkerd is production-grade and used by many organizations for critical applications. Its focus on performance, low resource usage, and strong security features (like automatic mTLS) makes it a robust choice for production. However, like any infrastructure component, proper planning, monitoring, and testing are essential for a successful production deployment.

Cleanup Commands

To remove Linkerd and the sample application from your cluster:


# Remove Emojivoto application
kubectl delete ns emojivoto

# Uninstall Linkerd control plane
linkerd uninstall | kubectl delete -f -

# Optionally, remove CRDs (be cautious in shared clusters if other users rely on them)
linkerd install --crds | kubectl delete -f -

# Remove Linkerd CLI from your system
rm -rf $HOME/.linkerd2
# You might also want to remove the PATH entry from your shell configuration file (.bashrc, .zshrc, etc.)

Next Steps / Further Reading

Congratulations! You’ve successfully installed and experimented with Linkerd. Here are some resources to deepen your knowledge:

  • Official Linkerd Documentation: The official Linkerd documentation is an excellent resource for detailed information on all features.
  • Linkerd Blog: Follow the Linkerd blog for updates, new features, and use cases.
  • Service Mesh Interface (SMI): Explore the Service Mesh Interface, a standard for interoperability across different service meshes, which Linkerd supports.
  • Advanced Traffic Management: Dive deeper into Linkerd’s traffic management capabilities, including retries, timeouts, and load balancing, by exploring the Linkerd tasks section.
  • Service Mesh Security: Understand how Linkerd’s mTLS and policy features enhance your application’s security posture.

Conclusion

Linkerd offers a compelling solution for bringing robust service mesh capabilities to your Kubernetes applications without the typical complexity and resource overhead. Its focus on simplicity, performance, and security makes it an ideal choice for teams looking to enhance observability, reliability, and security of their microservices. By following this guide, you’ve gained hands-on experience with Linkerd’s core features, from installation and injection to traffic management and policy enforcement. As you continue your journey, remember that a well-implemented service mesh like Linkerd can significantly simplify the operational challenges of distributed systems, allowing your development teams to focus on building great applications.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *