Cilium Network Policies: Master L3/L4/L7 Filtering

Introduction

In the dynamic world of Kubernetes, securing inter-pod communication is paramount. While Kubernetes’ native Network Policies provide essential L3/L4 (IP address and port) filtering, many modern applications demand more granular control. This is where Cilium, a powerful Container Network Interface (CNI) powered by eBPF, shines. Cilium extends Kubernetes Network Policies with advanced capabilities, including L7 (application-layer) filtering, identity-based security, and transparent encryption, transforming your cluster’s network into a highly secure and observable fabric.

Traditional network policies often struggle with the ephemeral nature of containers and the complexity of microservice architectures. Cilium addresses these challenges by leveraging eBPF to enforce policies directly within the Linux kernel, offering unparalleled performance and security. This deep dive will explore how to harness Cilium’s L3/L4/L7 filtering capabilities, moving beyond basic IP-based rules to fine-grained control over HTTP, DNS, and other application-level protocols. By the end of this guide, you’ll be equipped to build robust and intelligent network security for your Kubernetes workloads, ensuring that only authorized application-layer traffic flows through your cluster.

Prerequisites

Before diving into Cilium’s advanced network policies, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (v1.20+ recommended). This guide assumes you have kubectl configured to communicate with your cluster.
• Helm: Helm v3.x installed for easier Cilium deployment.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, and Labels.
• Basic Networking Knowledge: Understanding of TCP/IP, ports, and network policies.
• Cilium CLI (Optional but Recommended): For easier troubleshooting and status checks. Install it via curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz && sudo tar xzvf cilium-linux-amd64.tar.gz -C /usr/local/bin (adjust for your OS/architecture).

Step-by-Step Guide: Cilium Network Policies

Step 1: Install Cilium with L7 Proxy Enabled

To leverage Cilium’s L7 filtering capabilities, you must install it with the L7 proxy enabled. This ensures that Cilium deploys the necessary proxy components (Envoy) to inspect application-layer traffic. We will use Helm for a straightforward installation, enabling Hubble for observability and optionally Cilium WireGuard Encryption if you need encrypted pod-to-pod traffic.

The l7Proxy=true setting is crucial for enabling HTTP and Kafka policies. We also enable Hubble to visualize traffic flows and policy enforcement, which is incredibly useful for debugging. For production, you might also consider enabling encryption for enhanced security.

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium --version 1.15.4 \
  --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set egressGateway.enabled=true \
  --set hubble.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.relay.enabled=true \
  --set l7Proxy=true \
  --set encryption.enabled=false # Set to true for WireGuard/IPsec, see our guide on Cilium WireGuard Encryption

Verify Installation

After installation, verify that all Cilium components are running correctly. This might take a few minutes for all pods to become ready. The cilium status --wait command is particularly useful as it waits for the health checks to pass.

cilium status --wait

Expected Output:

    /¯¯\
 ¯¯\__/¯¯\    Cilium:         OK
   \__/    Cluster:        OK
 /¯¯\__/¯¯\  Nodes:          OK
 \__/¯¯\__/  Endpoint:       OK
 --\__/¯¯   
    \__/     Controller:     OK
             Proxy:          OK
             Hubble:         OK
             ClusterMesh:    Disabled
             IPAM:           OK
             CNI Chaining:   Disabled
             Host firewall:  Disabled
             ETCD:           OK

You should see “OK” for all components, especially “Proxy” and “Hubble”.

Step 2: Deploy Sample Applications

To demonstrate network policies, we need a simple application setup. We’ll deploy a “web” server and a “client” pod in a dedicated namespace. This setup will allow us to test both ingress and egress policies effectively.

We’ll create a test-policies namespace and deploy a basic Nginx web server and a busybox client. The Nginx server will serve a simple HTML page, and the busybox client will be used to send HTTP requests.

# test-apps.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: test-policies
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
  namespace: test-policies
  labels:
    app: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-config-volume
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-config-volume
        configMap:
          name: nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: test-policies
data:
  default.conf: |
    server {
        listen 80;
        location / {
            root /usr/share/nginx/html;
            index index.html;
        }
        location /public {
            return 200 'This is a public path!';
            add_header Content-Type text/plain;
        }
        location /private {
            return 403 'Access Denied';
            add_header Content-Type text/plain;
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: web-server
  namespace: test-policies
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-pod
  namespace: test-policies
  labels:
    app: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - name: busybox
        image: busybox:latest
        command: ["sh", "-c", "sleep infinity"]

kubectl apply -f test-apps.yaml

Verify Applications

Ensure all pods are running and the service is created.

kubectl get pods -n test-policies
kubectl get svc -n test-policies

Expected Output:

NAME                            READY   STATUS    RESTARTS   AGE
client-pod-77d9c6c547-j7x5p     1/1     Running   0          2m
web-server-67897858d4-p2q4q     1/1     Running   0          2m

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
web-server   ClusterIP   10.100.100.100   <none>        80/TCP    2m

Now, let’s test basic connectivity before applying any policies.

CLIENT_POD=$(kubectl get pod -n test-policies -l app=client -o jsonpath='{.items[0].metadata.name}')
WEB_SERVER_IP=$(kubectl get svc -n test-policies web-server -o jsonpath='{.spec.clusterIP}')

kubectl exec -n test-policies "$CLIENT_POD" -- wget -O- http://"$WEB_SERVER_IP"

Expected Output:

Connecting to 10.100.100.100:80 (10.100.100.100:80)
index.html           100% |********************************|   612  0:00:00 ETA

You should see the Nginx welcome page HTML content, confirming connectivity.

Step 3: Implement L3/L4 Network Policies

Cilium’s L3/L4 policies are a direct enhancement of Kubernetes Network Policies, allowing you to control traffic based on IP addresses, CIDR ranges, and ports. This forms the foundational layer of network security.

We’ll start by creating a default deny policy, which is a best practice for zero-trust environments. Then, we’ll explicitly allow traffic from our client pod to the web-server on port 80. This demonstrates how to build a secure baseline before adding L7 rules.

For more detailed information on general Kubernetes Network Policies, refer to our Kubernetes Network Policies: Complete Security Hardening Guide.

# l3-l4-policy.yaml
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny-all
  namespace: test-policies
spec:
  endpointSelector: {} # Applies to all pods in the namespace
  ingress:
    - {} # Deny all ingress
  egress:
    - {} # Deny all egress
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-client-to-web-l3-l4
  namespace: test-policies
spec:
  endpointSelector:
    matchLabels:
      app: web # Applies to pods with label app=web
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: client # Allow ingress from pods with label app=client
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP # Allow traffic on TCP port 80

kubectl apply -f l3-l4-policy.yaml

Verify L3/L4 Policies

First, confirm that the default deny policy is working. The client should no longer be able to reach the web server without the specific allow rule.

CLIENT_POD=$(kubectl get pod -n test-policies -l app=client -o jsonpath='{.items[0].metadata.name}')
WEB_SERVER_IP=$(kubectl get svc -n test-policies web-server -o jsonpath='{.spec.clusterIP}')

# This should fail due to default-deny-all
kubectl exec -n test-policies "$CLIENT_POD" -- wget -O- -T 2 http://"$WEB_SERVER_IP"

Expected Output (after a timeout):

wget: download timed out

Now, let’s apply the specific allow policy and re-test.

kubectl exec -n test-policies "$CLIENT_POD" -- wget -O- http://"$WEB_SERVER_IP"

Expected Output:

Connecting to 10.100.100.100:80 (10.100.100.100:80)
index.html           100% |********************************|   612  0:00:00 ETA

Connectivity is restored, demonstrating the L3/L4 policy working as expected.

Step 4: Implement L7 HTTP Network Policies

This is where Cilium truly shines. L7 HTTP policies allow you to filter traffic based on HTTP methods (GET, POST), paths, headers, and more. This provides a much finer-grained control than traditional L3/L4 rules, enabling true microservice security.

We will modify our policy to allow only GET requests to the /public path on the web server. All other HTTP requests, including GET / and GET /private, should be denied or result in a 403 error due to the policy.

# l7-http-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-http-get-public-path
  namespace: test-policies
spec:
  endpointSelector:
    matchLabels:
      app: web # Applies to pods with label app=web
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: client # Allow ingress from pods with label app=client
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      rules:
        http: # L7 HTTP rules
        - method: "GET"
          path: "/public"
        # - method: "POST" # Uncomment to allow POST requests
        #   path: "/api"

kubectl apply -f l7-http-policy.yaml

Verify L7 HTTP Policies

Let’s test various HTTP requests from our client to see the policy in action.

CLIENT_POD=$(kubectl get pod -n test-policies -l app=client -o jsonpath='{.items[0].metadata.name}')
WEB_SERVER_IP=$(kubectl get svc -n test-policies web-server -o jsonpath='{.spec.clusterIP}')

echo "--- Testing GET / (should be denied by policy) ---"
kubectl exec -n test-policies "$CLIENT_POD" -- wget -O- -T 2 http://"$WEB_SERVER_IP"/

echo "--- Testing GET /public (should be allowed by policy) ---"
kubectl exec -n test-policies "$CLIENT_POD" -- wget -O- http://"$WEB_SERVER_IP"/public

echo "--- Testing GET /private (should be denied by policy) ---"
kubectl exec -n test-policies "$CLIENT_POD" -- wget -O- -T 2 http://"$WEB_SERVER_IP"/private

Expected Output:

--- Testing GET / (should be denied by policy) ---
wget: download timed out # Or connection refused/reset

--- Testing GET /public (should be allowed by policy) ---
Connecting to 10.100.100.100:80 (10.100.100.100:80)
This is a public path!

--- Testing GET /private (should be denied by policy) ---
wget: download timed out # Or connection refused/reset

Notice that only the request to /public was successful. The other requests were either timed out or refused, demonstrating the L7 policy’s effectiveness. The web server itself would have returned a 403 for /private, but Cilium intercepts and denies the connection before it even reaches the application, which is more secure.

Step 5: Implement L7 DNS Network Policies

Controlling DNS lookups is another critical aspect of network security. Cilium allows you to define policies that restrict which DNS requests are permitted, preventing unauthorized data exfiltration or access to malicious domains. This is particularly important for applications that might attempt to resolve external hostnames.

We’ll create a policy that only allows DNS resolution for domains within the Kubernetes cluster (e.g., *.kube-system.svc.cluster.local), effectively blocking external DNS queries from the client pod. The toEntities: [ "kube-dns" ] selector is a powerful Cilium feature that identifies the cluster’s DNS service by its role, rather than a static IP.

# l7-dns-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-internal-dns-only
  namespace: test-policies
spec:
  endpointSelector:
    matchLabels:
      app: client # Applies to pods with label app=client
  egress:
  - toEntities:
    - kube-dns # Special Cilium entity for the cluster's DNS service
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
      rules:
        dns: # L7 DNS rules
        - matchPattern: "*.kube-system.svc.cluster.local" # Allow internal cluster DNS
        - matchPattern: "*.svc.cluster.local" # Allow all cluster services
  # Deny all other egress traffic (implicitly handled by default-deny-all, but good for explicit clarity)
  - {}

kubectl apply -f l7-dns-policy.yaml

Verify L7 DNS Policies

Now, let’s test DNS resolution from our client pod.

CLIENT_POD=$(kubectl get pod -n test-policies -l app=client -o jsonpath='{.items[0].metadata.name}')

echo "--- Testing internal DNS resolution (should be allowed) ---"
kubectl exec -n test-policies "$CLIENT_POD" -- nslookup kubernetes.default.svc.cluster.local

echo "--- Testing external DNS resolution (should be denied by policy) ---"
kubectl exec -n test-policies "$CLIENT_POD" -- nslookup google.com

Expected Output:

--- Testing internal DNS resolution (should be allowed) ---
Name:      kubernetes.default.svc.cluster.local
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

--- Testing external DNS resolution (should be denied by policy) ---
;; connection timed out; no servers could be reached # Or similar DNS failure

The internal DNS lookup for kubernetes.default.svc.cluster.local should succeed, while the external lookup for google.com should fail, demonstrating the L7 DNS policy in action. This granular control is vital for preventing DNS-based attacks or unauthorized external communication.

Step 6: Leverage Hubble for Observability

Cilium’s Hubble provides unparalleled observability into network traffic and policy enforcement. Since we enabled Hubble during installation, we can now use its UI to visualize connections and policy drops.

cilium hubble enable --ui
cilium hubble port-forward&

Open your browser to http://localhost:12000. You should see the Hubble UI. Try refreshing the page and then perform some of the kubectl exec commands from previous steps. You will observe traffic flows and policy decisions in real-time. Look for red lines indicating denied connections and green lines for allowed ones.

Hubble is an indispensable tool for understanding and debugging complex network policies. It gives you a visual representation of what’s happening at the eBPF layer, which is far more insightful than just logs.

Production Considerations

Deploying Cilium network policies in a production environment requires careful planning and consideration to ensure security, performance, and maintainability.

• Default Deny: Always start with a default deny policy and explicitly allow necessary traffic. This is a fundamental principle of zero-trust security.
• Policy Granularity: Balance granularity with complexity. While L7 policies offer fine control, overly complex policies can be hard to manage and debug. Group related services and apply policies at the appropriate layer.
• Policy Testing: Thoroughly test all policies in a staging environment before deploying to production. Use tools like Hubble, cilium monitor, and cilium connectivity test to validate behavior.
• Observability (Hubble): Hubble is critical for production. Ensure it’s properly configured and integrated with your monitoring stack. Use it to audit traffic, troubleshoot issues, and identify potential policy gaps. For more on eBPF observability, check out eBPF Observability: Building Custom Metrics with Hubble.
• Resource Management: L7 policies, especially those involving HTTP parsing, introduce a small overhead due to the Envoy proxy. Monitor CPU and memory usage of Cilium agents and Envoy proxies, especially under high traffic loads.
• Identity-Based Security: Leverage Cilium’s identity-based security. Instead of relying on volatile IP addresses, policies are based on Kubernetes labels, making them resilient to pod restarts and scaling events.
• External Traffic: Remember that Cilium Network Policies primarily govern inter-pod and pod-to-external traffic. For ingress from outside the cluster, you’ll still need to secure your Ingress or Gateway API controllers.
• Policy as Code: Manage your CiliumNetworkPolicies as code in a version control system. This enables collaboration, auditing, and automated deployment.
• Encryption: For sensitive data, consider enabling Cilium’s encryption capabilities (WireGuard or IPsec) for pod-to-pod traffic, especially across nodes or hybrid environments. Our guide on Cilium WireGuard Encryption for Pod-to-Pod Traffic provides further details.
• Integration with Service Mesh: If you’re using a service mesh like Istio, understand how Cilium policies interact. Cilium can often offload network policy enforcement from the service mesh sidecars, improving performance. For more on Istio, see our Istio Ambient Mesh Production Guide.

Troubleshooting

Working with network policies can be tricky. Here are some common issues and their solutions.

1. Issue: Pods cannot communicate despite policies seemingly being correct.

   Solution:

   • Check cilium status: Ensure all components are healthy.
   • Check cilium endpoint list: Verify that your pods have a Cilium identity and are managed by Cilium.
   • Use cilium monitor --type policy: This command shows real-time policy decisions (allow/deny) and can pinpoint exactly which policy is blocking traffic.
   • Check cilium policy get: Review the effective policies for your endpoints.
   • Hubble UI: Visualize traffic flow and drops. This is often the quickest way to diagnose.
2. Issue: L7 HTTP policy is not being enforced or traffic is unexpectedly blocked/allowed.

   Solution:

   • Verify l7Proxy=true: Ensure Cilium was installed with the L7 proxy enabled. If not, you’ll need to reinstall or upgrade Cilium with this setting.
   • Check for proxy errors: Look at the logs of the Cilium agent pods (kubectl logs -n kube-system -l k8s-app=cilium) for any Envoy proxy related errors.
   • Correct HTTP fields: Double-check the method, path, and headers in your policy YAML. HTTP rules are case-sensitive.
   • Policy Order: While Cilium merges policies, sometimes conflicting broader rules might override more specific ones. Ensure your specific L7 policy is indeed targeting the correct endpoints and ingress/egress.
3. Issue: DNS queries are failing, even with an L7 DNS policy.

   Solution:

   • toEntities: ["kube-dns"]: Ensure your DNS policy uses this entity selector for the cluster’s DNS service. This is more robust than IP-based rules.
   • DNS Port/Protocol: Confirm you’re allowing UDP port 53.
   • matchPattern: Verify the DNS match patterns are correct. For example, *.svc.cluster.local for internal services.
   • Client DNS configuration: Ensure your client pods are configured to use the cluster’s DNS service (/etc/resolv.conf).
4. Issue: After applying a policy, some pods become unreachable or cannot reach external services.

   Solution:

   • Default Deny Impact: If you applied a default deny policy, you must explicitly allow all necessary traffic. This includes DNS to kube-dns, access to the Kubernetes API server, and any external dependencies.
   • Egress Policy: Check your egress policies. Are pods allowed to reach the internet, cloud provider metadata services, or other external endpoints they need?
   • Order of Operations: New policies can take a few seconds to propagate. Give it some time.
   • Rollback: If a policy causes widespread issues, immediately roll back the problematic CiliumNetworkPolicy.
5. Issue: Cilium pods are not coming up or are in a CrashLoopBackOff state.

   Solution:

   • Check logs: kubectl logs -n kube-system -l k8s-app=cilium. Look for errors during startup.
   • Node compatibility: Ensure your kernel version meets Cilium’s requirements (Cilium System Requirements).
   • Resource limits: Increase CPU/memory limits for Cilium pods if they are being OOMKilled.
   • CNI conflicts: Ensure no other CNI is installed or conflicting. Cilium generally expects to be the sole CNI.

FAQ Section

Q1: What is the main difference between Kubernetes Network Policies and Cilium Network Policies?

A1: Kubernetes Network Policies provide basic L3/L4 filtering (IP and port-based). Cilium Network Policies extend this significantly by adding L7 filtering (HTTP, DNS, Kafka, etc.), identity-based security, transparent encryption, and advanced observability through eBPF. Cilium policies offer much finer-grained control over application-layer traffic.

Q2: Do I need to enable the L7 proxy for all Cilium Network Policies?

A2: No, only for policies that require L7 filtering (e.g., HTTP, Kafka, DNS with matchPattern). L3/L4 policies work without the L7 proxy. However, it’s generally recommended to enable l7Proxy=true during installation if you anticipate needing L7 capabilities, as it’s easier to enable it upfront than to reinstall later.

Q3: How does Cilium handle DNS resolution with L7 policies?

A3: Cilium intercepts DNS queries and can enforce policies based on the requested domain name. You can use matchPattern to allow specific domains or wildcards. Cilium also has special entities like kube-dns to easily target the cluster’s DNS service without relying on its IP address.

Q4: Can Cilium Network Policies be used with a Service Mesh like Istio?

A4: Yes, Cilium can complement or even enhance a service mesh. Cilium excels at L3/L4/L7 network policy enforcement at the kernel level, which can offload some policy duties from the service mesh’s sidecars, potentially improving performance. Some organizations use Cilium for network policies and a service mesh for advanced traffic management (e.g., A/B testing, canary deployments). For more on Istio, refer to our Istio Ambient Mesh Production Guide.

Q5: What is eBPF, and why is it important for Cilium?

A5: eBPF (extended Berkeley Packet Filter) is a revolutionary technology that allows programs to run in the Linux kernel without changing kernel source code or loading kernel modules. Cilium leverages eBPF to implement network policies, load balancing, and observability directly in the kernel, offering superior performance, security, and flexibility compared to traditional iptables-based CNIs.

Cleanup Commands

To remove all resources created during this guide, execute the following commands:

# Delete Cilium Network Policies

kubectl delete -f l7-dns-policy.yaml

kubectl delete -