Cilium Service Mesh: Unleash Sidecar-less Power

Verify:

Attempt to make a request that should be allowed and one that should be denied. You can exec into the tiefighter pod to test.

TIEFIGHTER_POD=$(kubectl get pods -l app=tiefighter -o jsonpath='{.items[0].metadata.name}')
echo "Allowed request (GET /public):"
kubectl exec -it $TIEFIGHTER_POD -- curl -s deathstar/public

echo ""
echo "Denied request (GET /private):"
kubectl exec -it $TIEFIGHTER_POD -- curl -s deathstar/private

echo ""
echo "Denied request (POST /public):"
kubectl exec -it $TIEFIGHTER_POD -- curl -X POST -s deathstar/public

Expected Output:

Allowed request (GET /public):
Hello from deathstar!

Denied request (GET /private):


Denied request (POST /public):

The empty output for denied requests indicates that Cilium successfully blocked them at the network level.

Step 7: Observe Traffic with Hubble UI

Hubble, Cilium's observability platform, provides deep visibility into network flows and policy enforcement. With Hubble UI, you can visualize service dependencies, troubleshoot connectivity issues, and see how your L7 policies are impacting traffic in real-time. This is a critical tool for understanding the behavior of your service mesh. For more on Hubble, refer to our eBPF Observability with Hubble guide.

By port-forwarding the Hubble UI service, we can access its web interface from our local machine. The UI will display a graphical representation of network connections, show details of allowed and denied traffic, and help confirm that our L7 policy is actively working. This visual feedback is invaluable for verifying complex network configurations.

# Port-forward Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 8080:80 &

# Open Hubble UI in your browser
echo "Open Hubble UI in your browser: http://localhost:8080"

Verify:

Open your web browser and navigate to http://localhost:8080. You should see the Hubble UI dashboard, displaying network flows between your application pods. You can filter by namespace, pod, or even L7 protocol to observe the effects of your policies.

Production Considerations

While Cilium Service Mesh offers compelling advantages, deploying it in production requires careful planning:

• eBPF Kernel Compatibility: Ensure your Kubernetes nodes are running a Linux kernel version that fully supports the eBPF features required by Cilium (typically 4.9+ for basic features, 5.4+ for advanced ones). Always check the Cilium system requirements for the version you're deploying.
• Resource Management: Although sidecar-less, Cilium still consumes resources on your nodes. Monitor CPU and memory usage of Cilium agents and the operator. Optimize eBPF map sizes and policy complexity to prevent resource contention.
• Observability Integration: Leverage Hubble for deep observability. Integrate Hubble's metrics and logs with your existing monitoring stack (Prometheus, Grafana, ELK) for comprehensive insights. Consider using eBPF Observability with Hubble for custom metrics.
• Security Best Practices:
  • Implement strong Kubernetes Network Policies in conjunction with Cilium's L7 policies.
  • Enable mTLS across all critical namespaces.
  • Regularly audit Cilium policies and configurations. Consider integrating with tools like Kyverno for policy enforcement, similar to the concepts discussed in Securing Container Supply Chains with Sigstore and Kyverno.
• High Availability: Deploy Cilium Operator in a highly available configuration. Ensure your etcd or Kubernetes API server is robust.
• Upgrades: Plan for rolling upgrades of Cilium components. Always test upgrades in a staging environment first. Refer to the official Cilium upgrade documentation.
• Integration with Cloud Providers: If running on a cloud (AWS EKS, GCP GKE, Azure AKS), follow specific integration guides provided by Cilium for optimal performance and compatibility with cloud-native load balancers and services.
• Traffic Management Complexity: For very complex, multi-cluster, or multi-cloud traffic routing scenarios, evaluate if Cilium's current service mesh features meet all requirements. While powerful, some advanced scenarios might still benefit from supplementary tools or a more feature-rich service mesh like Istio, especially when combined with its Ambient Mesh for sidecar-less operation as discussed in Istio Ambient Mesh Production Guide.
• Cost Optimization: The sidecar-less approach inherently saves resources. Further optimize costs by right-sizing nodes and leveraging autoscaling solutions like Karpenter, as explored in Karpenter Cost Optimization.

Troubleshooting

Here are some common issues you might encounter when working with Cilium Service Mesh and their solutions:

1. Issue: Cilium pods are not starting or are in CrashLoopBackOff.

   Solution: This often indicates a problem with the underlying kernel or required modules. Check the Cilium agent logs for specific error messages and verify kernel compatibility. Ensure CONFIG_BPF and related options are enabled in your kernel. A common cause in Kind is missing Kube-proxy replacement requirements.

   kubectl logs -n kube-system -l k8s-app=cilium --tail 100
   kubectl describe pod -n kube-system 
   

   Ensure your kernel version meets Cilium's requirements. For Kind, double-check the kubeProxyReplacement settings during installation.

2. Issue: Application pods are not getting IP addresses or are in Pending state.

   Solution: If Cilium is acting as the CNI, this means Cilium isn't properly assigning IPs. Check Cilium's status and logs. Ensure the ipam.mode is correctly configured (e.g., kubernetes for Kubernetes-managed IPAM).

   cilium status
   kubectl logs -n kube-system -l k8s-app=cilium-operator --tail 100
   

   Verify that the Cilium operator is healthy and has permissions to allocate IPs.

3. Issue: L7 policies are not being enforced, or traffic is unexpectedly blocked/allowed.

   Solution: Policy enforcement issues can be tricky. Use cilium policy get to verify the policy is correctly applied. Crucially, use Hubble to visualize traffic flows and see if packets are being dropped by a specific policy. The Hubble UI (http://localhost:8080 after port-forwarding) or CLI (cilium hubble observe) can pinpoint the exact policy causing the issue.

   cilium policy get
   cilium hubble observe --type L7 --last 5m
   

   Ensure your endpointSelector and fromEndpoints match the labels of your pods precisely. Remember that CiliumNetworkPolicies are additive, and if no policy explicitly allows traffic, it is implicitly denied.

4. Issue: mTLS is not working, or services cannot communicate after enabling it.

   Solution: Verify that the namespace is correctly annotated for service mesh enablement. Check Cilium agent logs for mTLS-related errors. Ensure that all communicating pods are within the mTLS-enabled namespace or that appropriate policies are in place for cross-namespace communication if needed.

   kubectl get namespace default -o jsonpath='{.metadata.annotations}'
   kubectl logs -n kube-system -l k8s-app=cilium --tail 100 | grep "mTLS"
   

   Sometimes, restarting application pods after enabling mTLS on a namespace can resolve initial connection issues as they pick up new network configurations.

5. Issue: Hubble UI is not accessible or not showing any flows.

   Solution: First, ensure the Hubble UI and Hubble Relay pods are running and healthy. Verify that you are correctly port-forwarding the hubble-ui service. If no flows appear, check if Hubble is enabled in your Cilium installation and if the Cilium agents are correctly configured to send flow data to Hubble Relay.

   kubectl get pods -n kube-system -l k8s-app=hubble-ui
   kubectl get pods -n kube-system -l k8s-app=hubble-relay
   kubectl logs -n kube-system -l k8s-app=cilium --tail 100 | grep "hubble"
   

   Confirm that hubble.enabled=true and hubble.relay.enabled=true were set during Helm installation. Also, check for any firewall rules blocking the port-forwarded traffic.

6. Issue: Performance degradation or high resource usage by Cilium.

   Solution: While sidecar-less, Cilium still uses eBPF programs in the kernel. High policy count, complex regex in L7 policies, or a large number of endpoints can increase resource usage. Use cilium metrics to inspect Cilium's internal metrics and identify bottlenecks. Optimize your policies where possible.

   cilium metrics list
   kubectl top pods -n kube-system -l k8s-app=cilium
   

   Consider simplifying overly complex policies or breaking them down. Ensure your nodes have sufficient CPU and memory resources. For deeper insights, you might export Cilium metrics to Prometheus and visualize them in Grafana.

FAQ Section

1. What is eBPF and how does Cilium use it for service mesh?

   eBPF (extended Berkeley Packet Filter) is a revolutionary technology that allows programs to run in the Linux kernel without changing kernel source code or loading kernel modules. Cilium leverages eBPF to implement networking, security, and observability directly in the kernel's data path. For service mesh, this means it can perform L4/L7 traffic management, mTLS, and policy enforcement at wire speed, without the overhead of injecting sidecar proxies. This is the core of Cilium's sidecar-less architecture, offering significant performance and resource efficiency benefits. You can learn more about eBPF on the eBPF.io website.

2. How does Cilium Service Mesh compare to traditional sidecar-based meshes like Istio?

   The primary difference lies in the architecture. Traditional meshes like Istio inject a proxy (e.g., Envoy) sidecar into every application pod. This sidecar intercepts all network traffic. Cilium, on the other hand, uses eBPF programs loaded into the kernel to achieve the same (and often more) functionality without any sidecar proxies. This results in lower resource consumption, reduced latency, and simplified operations (no sidecar lifecycle management). While Istio offers a broader range of features, especially for complex multi-cluster scenarios, Cilium's approach is often preferred for its efficiency and tight integration with Kubernetes networking. It's worth noting that Istio is also moving towards a sidecar-less model with its Ambient Mesh, which we cover in our Istio Ambient Mesh Production Guide.

3. Can Cilium Service Mesh be used with other CNIs?

   No, Cilium functions as a CNI itself. To utilize Cilium's full capabilities, including its service mesh features, it must be installed as the primary CNI for your Kubernetes cluster. It replaces other CNIs like Calico, Flannel, or the default cloud provider CNI. This allows Cilium to have complete control over the network data path and inject eBPF programs for advanced functionality.

4. What kind of traffic management features does Cil