Cilium Hubble: Visualize Network Flows Now

Introduction

In the dynamic world of Kubernetes, understanding the intricate web of network communication between your services is paramount. As applications scale and microservices proliferate, traditional network monitoring tools often fall short, struggling to provide granular, real-time insights into pod-to-pod, pod-to-external, and even internal node traffic. This visibility gap can lead to significant challenges in debugging connectivity issues, enforcing security policies, and optimizing application performance.

Enter Cilium and its powerful companion, Hubble. Cilium, a CNI (Container Network Interface) powered by eBPF, revolutionizes Kubernetes networking by providing advanced features like identity-based security, transparent encryption, and efficient load balancing. Hubble, built on top of Cilium’s eBPF data plane, unlocks unparalleled network observability. It transforms raw network events into actionable insights, offering a comprehensive view of all network flows within your cluster. Imagine a real-time map of your Kubernetes network, showing every connection, its status, and even application-layer protocols – that’s the power of Hubble.

This guide will walk you through the process of installing Cilium with Hubble, exploring its core features, and leveraging its capabilities for network visualization and troubleshooting. We’ll delve into practical examples, from basic flow inspection to advanced policy debugging, ensuring you gain a deep understanding of how to make your Kubernetes network truly transparent. For those interested in deeper eBPF insights, our article on eBPF Observability: Building Custom Metrics with Hubble provides further context.

Prerequisites

Before diving into the installation and configuration of Cilium Hubble, ensure you have the following prerequisites in place:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.20+ recommended). This guide assumes you have kubectl configured and pointing to your cluster. You can use any Kubernetes distribution, including Kind, K3s, Minikube, or a cloud-managed service like GKE, EKS, or AKS.
• Helm: Helm 3 installed on your local machine. We will use Helm for installing Cilium.
• kubectl: The Kubernetes command-line tool (kubectl) installed and configured to interact with your cluster.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts such as Pods, Deployments, Services, and Namespaces.
• Basic Networking Knowledge: An understanding of fundamental networking concepts like TCP/IP, ports, and protocols will be helpful.

Step-by-Step Guide

1. Install Cilium with Hubble

The first step is to install Cilium as your CNI plugin and enable Hubble components during the installation. We’ll use Helm for a straightforward deployment. It’s crucial to enable hubble.enabled and hubble.ui.enabled to get the full observability suite.

For production environments, consider adjusting resource requests/limits and replica counts for the Cilium and Hubble components based on your cluster size and traffic patterns. You can find more advanced configuration options in the official Cilium documentation.

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium --version 1.15.4 \
  --namespace kube-system \
  --set hubble.enabled=true \
  --set hubble.ui.enabled=true \
  --set operator.replicas=1 \
  --set ipam.mode=clusterPool \
  --set cluster.name=kubezilla-cluster \
  --set k8sServiceHost=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}') \
  --set k8sServicePort=$(kubectl get service kubernetes -n default -o jsonpath='{.spec.ports[0].port}')

Explanation:

• helm repo add cilium https://helm.cilium.io/: Adds the Cilium Helm repository.
• helm repo update: Updates your local Helm chart repository cache.
• helm install cilium cilium/cilium --version 1.15.4: Installs the Cilium chart. We specify a version for reproducibility; always check the latest stable version.
• --namespace kube-system: Deploys Cilium into the kube-system namespace, which is standard for CNI plugins.
• --set hubble.enabled=true: Enables the Hubble server, which collects and stores network flow data.
• --set hubble.ui.enabled=true: Deploys the Hubble UI, a web-based graphical interface for visualizing flows.
• --set operator.replicas=1: Sets the Cilium Operator replica count. For high availability, you might increase this.
• --set ipam.mode=clusterPool: Configures Cilium’s IP Address Management mode. clusterPool is a common choice for simplicity.
• --set cluster.name=kubezilla-cluster: Assigns a name to your cluster, useful in multi-cluster environments.
• --set k8sServiceHost and k8sServicePort: These ensure Cilium can connect to the Kubernetes API server. The provided commands dynamically fetch the internal IP and port of the Kubernetes API service.

Verify:

Check if all Cilium and Hubble pods are running in the kube-system namespace. It might take a few minutes for all pods to transition to the Running state.

kubectl get pods -n kube-system -l k8s-app=cilium

Expected Output:

NAME                                   READY   STATUS    RESTARTS   AGE
cilium-xxxx                            1/1     Running   0          2m
cilium-operator-xxxx                   1/1     Running   0          2m
hubble-relay-xxxx                      1/1     Running   0          2m
hubble-ui-xxxx                         1/1     Running   0          2m

You should see cilium daemonset pods (one per node), the cilium-operator, hubble-relay, and hubble-ui pods all in the Running state.

2. Install Hubble CLI

The Hubble CLI (hubble command) is an essential tool for interacting with Hubble. It allows you to query network flows, observe real-time traffic, and debug connectivity directly from your terminal. This client-side tool connects to the hubble-relay service running in your cluster.

curl -L --remote-name-all https://github.com/cilium/hubble/releases/latest/download/hubble-linux-amd64.tar.gz
sudo tar xzF hubble-linux-amd64.tar.gz -C /usr/local/bin
rm hubble-linux-amd64.tar.gz

For macOS, replace hubble-linux-amd64.tar.gz with hubble-darwin-amd64.tar.gz or hubble-darwin-arm64.tar.gz depending on your architecture. Refer to the Hubble installation guide for other architectures.

Explanation:

• curl -L --remote-name-all ...: Downloads the latest Hubble CLI binary for Linux AMD64 from the official Hubble GitHub releases.
• sudo tar xzF ... -C /usr/local/bin: Extracts the downloaded archive and places the hubble executable in /usr/local/bin, making it accessible in your system’s PATH.
• rm ...: Cleans up the downloaded archive.

Verify:

Check if the Hubble CLI is installed correctly by running the hubble version command.

hubble version

Expected Output:

hubble v0.12.0 (hubble-cli)
commit: d8a9f0e
goarch: amd64
goos: linux
built: 2024-04-20T10:00:00Z

The version number might differ based on the latest release, but you should see similar output confirming the CLI is installed.

3. Enable Hubble Relay

While we enabled Hubble during the Cilium installation, the hubble CLI needs to know how to connect to the hubble-relay service. The cilium hubble enable command sets up the necessary port-forwarding or configuration for the CLI to communicate with the relay.

The hubble-relay component aggregates flow data from all Cilium agents in the cluster, providing a single endpoint for the Hubble UI and CLI to query. This is essential for cluster-wide network observability.

cilium hubble enable

Explanation:

• cilium hubble enable: This command ensures that the hubble-relay service is accessible. By default, it might set up a local proxy or configure your ~/.hubble/config.yaml to point to the service.

Verify:

Confirm that Hubble Relay is accessible by listing active flows. It might show no flows initially if there’s no traffic yet.

hubble observe --last 5

Expected Output (example, might be empty if no traffic):

Apr 22 10:30:01.000: default/pod-a:8080 <-> default/pod-b:5432 (TCP)
Apr 22 10:30:02.000: kube-system/hubble-relay:443 <-> kube-system/hubble-ui:8080 (TCP)
...

If you see some output, even internal Cilium/Hubble traffic, it means the relay is working. If you get an error, ensure all Hubble pods are running and try again.

4. Deploy Sample Applications

To generate some network traffic and effectively demonstrate Hubble’s capabilities, let’s deploy a simple client-server application. We’ll use a curl client and a netcat server in separate namespaces to simulate inter-service communication.

This setup will allow us to observe traffic between different namespaces, which is a common scenario in Kubernetes. For more advanced networking scenarios, consider exploring Kubernetes Gateway API for traffic management.

# server.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: server-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netcat-server
  namespace: server-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: netcat-server
  template:
    metadata:
      labels:
        app: netcat-server
    spec:
      containers:
      - name: netcat-server
        image: alpine/git
        command: ["sh", "-c"]
        args: ["apk add --no-cache netcat-openbsd && while true; do echo 'Hello from Netcat Server!' | nc -lp 8080; done"]
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: netcat-service
  namespace: server-ns
spec:
  selector:
    app: netcat-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP

# client.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: client-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl-client
  namespace: client-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl-client
  template:
    metadata:
      labels:
        app: curl-client
    spec:
      containers:
      - name: curl-client
        image: curlimages/curl
        command: ["sleep", "3600"] # Keep the pod running
        ports:
        - containerPort: 80

kubectl apply -f server.yaml
kubectl apply -f client.yaml

Explanation:

• The server.yaml defines a server-ns namespace, a Deployment running an alpine/git image that installs netcat-openbsd and listens on port 8080, and a Service to expose it on port 80.
• The client.yaml defines a client-ns namespace and a Deployment running a curlimages/curl image. This pod will be used to send requests to the server.

Verify:

Check if the pods are running in their respective namespaces.

kubectl get pods -n server-ns
kubectl get pods -n client-ns

Expected Output:

NAME                             READY   STATUS    RESTARTS   AGE
netcat-server-xxxx              1/1     Running   0          1m

NAME                             READY   STATUS    RESTARTS   AGE
curl-client-xxxx                 1/1     Running   0          1m

5. Observe Network Flows with Hubble CLI

Now that we have applications generating traffic, we can use the Hubble CLI to observe these network flows in real-time. This is where Hubble’s power truly shines, providing granular details about every connection.

Let’s first generate some traffic from our client to the server and then observe it.

# Get the client pod name
CLIENT_POD=$(kubectl get pod -n client-ns -l app=curl-client -o jsonpath='{.items[0].metadata.name}')

# Send a request from the client to the server
kubectl exec -it $CLIENT_POD -n client-ns -- curl -m 3 netcat-service.server-ns.svc.cluster.local

Expected Output from curl command:

Hello from Netcat Server!

Now, let’s observe the flows:

hubble observe -f --last 5 --namespace client-ns,server-ns

Explanation:

• hubble observe: Starts observing network flows.
• -f: Follows the output, streaming new flows as they occur.
• --last 5: Shows the last 5 flows when starting.
• --namespace client-ns,server-ns: Filters flows to only show those originating from or destined for pods in these namespaces.

Expected Output (truncated example):

Apr 22 10:35:10.123: client-ns/curl-client:34567 <=> server-ns/netcat-server:8080 L3/L4 ACCEPTED (TCP)
Apr 22 10:35:10.124: client-ns/curl-client:34567 <-> server-ns/netcat-service/80 (TCP)
Apr 22 10:35:10.125: client-ns/curl-client:34567 <-> server-ns/netcat-server:8080 L7 OK (HTTP/1.1)
...

You should see flows indicating the communication between curl-client and netcat-server, potentially including DNS lookups and TCP connections. Hubble shows the source and destination pods, ports, protocols, and the verdict (e.g., ACCEPTED) from Cilium’s policy engine.

6. Visualize Flows with Hubble UI

While the CLI is powerful, the Hubble UI provides an intuitive graphical interface for visualizing network flows, services, and even policy enforcement. It’s excellent for getting a high-level overview or drilling down into specific connections.

hubble ui

Explanation:

• hubble ui: This command automatically sets up a local port-forward to the hubble-ui service in your cluster and opens your web browser to the UI. It typically runs on http://localhost:12000.

Verify:

Your web browser should open to the Hubble UI. You’ll see a dashboard with various views:

• Flows: A real-time stream of network flows, similar to hubble observe but with more details and filtering options.
• Service Map: A graphical representation of services and their dependencies, showing communication patterns. This is incredibly useful for understanding microservice interactions.
• Policies: View active Cilium Network Policies and their enforcement status.

Navigate to the “Flows” tab and observe the traffic between client-ns/curl-client and server-ns/netcat-server. You can filter by namespace, source/destination, and verdict. The “Service Map” will show connections between the curl-client and netcat-service.

For more detailed information on network policies and their security implications, refer to our Kubernetes Network Policies: Complete Security Hardening Guide.

7. Filter and Analyze Flows

Hubble’s filtering capabilities are crucial for pinpointing specific traffic. You can filter by source/destination, namespace, IP, port, protocol, and even DNS names.

Let’s make another request and then filter specifically for flows related to our netcat-server.

# Get the client pod name
CLIENT_POD=$(kubectl get pod -n client-ns -l app=curl-client -o jsonpath='{.items[0].metadata.name}')

# Send another request to generate traffic
kubectl exec -it $CLIENT_POD -n client-ns -- curl -m 3 netcat-service.server-ns.svc.cluster.local

Now, filter flows using the CLI:

hubble observe --last 10 --pod netcat-server -n server-ns

Explanation:

• --pod netcat-server: Filters flows where the source or destination pod has “netcat-server” in its name.
• -n server-ns: Further restricts the search to the server-ns namespace.

Expected Output (example):

Apr 22 10:40:05.123: client-ns/curl-client:45678 <=> server-ns/netcat-server:8080 L3/L4 ACCEPTED (TCP)
Apr 22 10:40:05.124: client-ns/curl-client:45678 <-> server-ns/netcat-service/80 (TCP)
Apr 22 10:40:05.125: client-ns/curl-client:45678 <-> server-ns/netcat-server:8080 L7 OK (HTTP/1.1)
...

You should see flows specifically related to the netcat-server, including the incoming connection from the curl-client. The L7 OK (HTTP/1.1) indicates that Hubble can even parse application-layer protocols, providing deep insights.

8. Debugging Network Policies with Hubble

One of Hubble’s most powerful features is its ability to visualize and debug Cilium Network Policies. When a policy drops traffic, Hubble provides a clear “DENIED” verdict and the reason for the drop. This is invaluable for troubleshooting connectivity issues caused by misconfigured policies.

Let’s create a Cilium Network Policy that denies ingress traffic to our netcat-server from the client-ns namespace.

# deny-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: deny-ingress-to-netcat
  namespace: server-ns
spec:
  endpointSelector:
    matchLabels:
      app: netcat-server
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: client-ns # Selects pods in client-ns
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
          - {} # Deny all HTTP requests to port 8080 from client-ns

kubectl apply -f deny-policy.yaml

Explanation:

• This policy, named deny-ingress-to-netcat, targets pods with the label app: netcat-server in the server-ns namespace.
• The ingress rule specifies that traffic fromEndpoints with the label io.kubernetes.pod.namespace: client-ns (i.e., pods in the client-ns namespace) to port 8080 TCP should be subject to HTTP rules. An empty http: - {} rule effectively denies all HTTP traffic.

Verify:

Now, try to access the server again from the client, and observe the flows with Hubble.

# Get the client pod name
CLIENT_POD=$(kubectl get pod -n client-ns -l app=curl-client -o jsonpath='{.items[0].metadata.name}')

# Attempt to send a request (it should fail)
kubectl exec -it $CLIENT_POD -n client-ns -- curl -m 3 netcat-service.server-ns.svc.cluster.local

Expected Output from curl command:

curl: (28) Connection timed out after 3001 milliseconds

Now, observe the flows with Hubble, specifically looking for denied flows:

hubble observe --last 10 --pod netcat-server -n server-ns --verdict DENIED

Expected Output (example):

Apr 22 10:45:01.123: client-ns/curl-client:56789 <=> server-ns/netcat-server:8080 L3/L4 DENIED (TCP) policy-denied
    Policy: deny-ingress-to-netcat (server-ns)
    Labels: {k8s:app=netcat-server k8s:io.kubernetes.pod.namespace=server-ns}
    Source: {k8s:app=curl-client k8s:io.kubernetes.pod.namespace=client-ns}
...

You should clearly see a DENIED verdict for the connection attempt, with the specific policy (deny-ingress-to-netcat) that caused the denial. This level of detail is invaluable for debugging network policy issues. For more advanced Cilium features like WireGuard encryption, check out our guide on Cilium WireGuard Encryption for Pod-to-Pod Traffic.

Production Considerations

Deploying Cilium Hubble in a production environment requires careful planning to ensure scalability, reliability, and security.

• Resource Allocation: Hubble Relay and Hubble UI can consume significant CPU and memory, especially in large clusters with high traffic. Monitor their resource usage and adjust requests/limits accordingly. Consider increasing hubble-relay replicas for high availability.
• Data Retention: Hubble Relay stores flow data in memory by default. For longer retention, consider integrating with external storage solutions like Prometheus and Grafana or a dedicated flow export mechanism. Cilium can export flows in various formats, including IPFIX.
• Security and Access Control:
  • Hubble UI: Access to the Hubble UI should be restricted. Consider placing it behind an Ingress with authentication (e.g., OAuth2 proxy) or using a VPN.
  • Hubble CLI: Ensure that users with hubble CLI access have appropriate RBAC permissions if they are using a non-admin kubeconfig. The hubble CLI typically uses the current kubeconfig context.
• Scalability: In very large clusters, the amount of flow data can be overwhelming. Hubble provides filtering at the agent level to reduce the data sent to the relay. Configure this carefully to avoid dropping essential data while managing load.
• Monitoring and Alerting: Monitor the health of Cilium and Hubble components using standard Kubernetes monitoring tools. Set up alerts for pod failures, high resource usage, or persistent network policy denials.
• Upgrade Strategy: Always follow the official Cilium upgrade guide. Test upgrades in a staging environment before applying them to production.
• Integration with Observability Stacks: Hubble can complement existing observability tools. Integrate flow data with your SIEM for security analytics or with your metrics/logging platforms for comprehensive operational insights.
• Network Performance: While eBPF is highly efficient, enabling deep packet inspection (e.g., L7 parsing) can introduce a slight overhead. Monitor network latency and throughput after enabling advanced Hubble features.

Troubleshooting

1. Hubble CLI Cannot Connect to Relay

   Issue: When running hubble observe or hubble ui, you get an error like “Error: unable to connect to Hubble Relay: dial tcp 127.0.0.1:4245: connect: connection refused”.

   Solution:

   This usually means the hubble-relay pod is not running, or the CLI cannot establish a connection. First, check the hubble-relay pod status:

   kubectl get pods -n kube-system -l k8s-app=hubble-relay

   If it’s not running, check its logs for errors. If it is running, try re-enabling Hubble to ensure the port-forward or configuration is correctly set up:

   cilium hubble disable
   cilium hubble enable

   Also, ensure no other process is using port 4245 on your local machine (the default port for hubble ui and hubble observe to connect to the local proxy).

2. No Flows Visible in Hubble UI/CLI

   Issue: All Hubble pods are running, but you don’t see any network flows, even after generating traffic.

   Solution:

   Verify that Cilium is correctly installed as the CNI and is handling network traffic. Check the Cilium agent logs for any errors:

   kubectl logs -n kube-system -l k8s-app=cilium --tail 100

   Ensure that hubble.enabled=true was set during Cilium installation. If not, you might need to upgrade your Cilium installation:

   helm upgrade cilium cilium/cilium --version 1.15.4 \
     --namespace kube-system \
     --reuse-values \
     --set hubble.enabled=true \
     --set hubble.ui.enabled=true

   Also, check the hubble-relay logs for any issues receiving flows from the agents.

3. Hubble UI Not Loading or Showing Errors

   Issue: The Hubble UI page is blank, shows a loading spinner indefinitely, or displays connection errors.

   Solution:

   First, check the status of the hubble-ui pod:

   kubectl get pods -n kube-system -l k8s-app=hubble-ui