Kubernetes Network Fixes: kubectl & tcpdump

Navigating the intricate network landscape of Kubernetes can often feel like peering into a black box. Applications fail to communicate, services are unreachable, and the dreaded “connection refused” message haunts your logs. When your microservices architecture hits a networking snag, identifying the root cause quickly becomes paramount to maintaining application availability and performance. The sheer number of layers—from CNI plugins and network policies to services, pods, and external firewalls—can make pinpointing the exact failure point a daunting task.

Fortunately, you don’t have to troubleshoot blind. With the right tools and a systematic approach, you can peel back these layers and diagnose network issues effectively. This guide will arm you with two indispensable utilities: kubectl, your primary interface to the Kubernetes API, and tcpdump, the venerable packet analyzer. By combining their powers, you’ll gain visibility into network traffic at various levels, enabling you to trace communication paths, identify dropped packets, and resolve even the most elusive network problems within your clusters.

Whether you’re dealing with a misconfigured Service, an overly aggressive Kubernetes Network Policy, or an underlying CNI issue, this comprehensive guide will walk you through practical techniques to diagnose and fix common Kubernetes networking woes. Get ready to demystify your cluster’s network!

Prerequisites

Before we dive into the nitty-gritty of network troubleshooting, ensure you have the following tools and knowledge at your disposal:

• A Kubernetes Cluster: This can be a local cluster (e.g., Minikube, Kind, Docker Desktop Kubernetes) or a cloud-based cluster (AWS EKS, GCP GKE, Azure AKS).
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. You can find installation instructions in the official Kubernetes documentation.
• Basic Kubernetes Concepts: Familiarity with Pods, Services, Deployments, ReplicaSets, and Namespaces is essential. Understanding how Kubernetes networking works at a high level will be beneficial.
• Basic Networking Knowledge: Concepts like IP addresses, ports, TCP/IP, DNS, and firewalls will be referenced.
• tcpdump: While we’ll show you how to use it within a Pod, having a basic understanding of tcpdump syntax will be helpful.
• Access to Nodes (Optional but Recommended): For deeper troubleshooting, SSH access to your Kubernetes worker nodes can be invaluable, especially for CNI-related issues.
• A Debug Image with Network Tools: We’ll use a container image that includes tools like ping, curl, netcat, and tcpdump. A good general-purpose image is nicolaka/netshoot or alpine/git (then install tools).

Step-by-Step Guide to Kubernetes Network Troubleshooting

This section will guide you through a systematic approach to diagnosing Kubernetes network issues, starting from high-level checks and progressively diving deeper.

Step 1: Check Basic Pod and Service Status

The first step in any troubleshooting scenario is to verify the health and configuration of your Kubernetes resources. Often, a network issue isn’t truly a network issue, but a misconfigured Pod or Service that prevents traffic from ever reaching the network layer.

Use kubectl get to inspect the state of your Pods, Services, and Endpoints. Pay close attention to Pod statuses (Running, Pending, CrashLoopBackOff), their assigned IP addresses, and whether Services have active Endpoints.

# Get all pods in the current namespace with their IP addresses and node assignments
kubectl get pods -o wide

# Get all services in the current namespace
kubectl get svc

# Get all endpoints for services (shows which Pods are backing each Service)
kubectl get endpoints

# Get details about a specific Pod, including events and network configuration
kubectl describe pod <pod-name>

# Get details about a specific Service
kubectl describe svc <service-name>

Verify:

You should see your Pods in a Running state, with assigned IP addresses and node names. Services should have `CLUSTER-IP`s and `EXTERNAL-IP`s (if applicable), and crucially, `ENDPOINTS` should list the IP addresses and ports of the healthy Pods backing the Service. If a Service shows `<none>` for `ENDPOINTS`, it means no Pods are currently selected by its label selector.

# Expected output for kubectl get pods -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES
my-app-deployment-789f5f5c-abcde   1/1     Running   0          5m    10.244.1.10    minikube       <none>           <none>
my-other-app-6789abcd-vwxyz    1/1     Running   0          4m    10.244.2.15    minikube       <none>           <none>

# Expected output for kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP    10d
my-app-svc   ClusterIP   10.100.200.10  <none>        80/TCP     5m

# Expected output for kubectl get endpoints
NAME           ENDPOINTS             AGE
kubernetes     192.168.49.2:8443     10d
my-app-svc     10.244.1.10:8080      5m

Step 2: Test Connectivity from Within a Pod

Once you’ve confirmed your Pods are running and Services have endpoints, the next logical step is to test connectivity from the perspective of an application Pod. This helps isolate whether the issue is internal to the Pod, between Pods, or external.

Use kubectl exec to run commands inside a Pod. We’ll use a temporary debug Pod with network utilities to avoid modifying your application containers. A common debug image is nicolaka/netshoot which comes pre-installed with many useful tools like ping, curl, netcat, and tcpdump. If your cluster version supports it, kubectl debug is even better as it attaches a debug container to an existing Pod’s network namespace.

# Option 1: Create a temporary debug pod (if kubectl debug is not available or preferred)
# Replace <namespace> if not default
kubectl run -it --rm debug-pod --image=nicolaka/netshoot -- /bin/bash

# Inside the debug-pod, test connectivity:
# Ping a Service IP (e.g., my-app-svc's ClusterIP from Step 1)
ping 10.100.200.10
# Ping a Pod IP (e.g., my-app-deployment's Pod IP from Step 1)
ping 10.244.1.10
# Curl a Service (replace with your service's port)
curl http://my-app-svc:80
# Curl a Pod IP directly (replace with your pod's port)
curl http://10.244.1.10:8080
# Test external connectivity
curl https://www.google.com

# Option 2: Use kubectl debug (Kubernetes 1.20+ recommended)
# This attaches a debug container to an existing pod's network namespace
# Replace <pod-name> with the actual pod you want to debug
kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name-in-pod> -- /bin/bash

# If --target is not specified, it may default to the first container or create a new one.
# For network troubleshooting, it's often best to join the network namespace of an existing container.
# Example:
# kubectl debug -it my-app-deployment-789f5f5c-abcde --image=nicolaka/netshoot --target=my-app-container -- /bin/bash

# Once inside, perform the same ping/curl tests as above.

Verify:

You should see successful pings and curl responses. If `ping` fails but `curl` to a Pod IP works, it might indicate an ICMP issue or a higher-level application problem. If `curl` to a Service name fails but `curl` to a Pod IP works, it points to a DNS resolution or Service-level routing issue. If external connectivity fails, inspect your cluster’s egress rules or CNI configuration.

# Expected output for ping 10.100.200.10 (from inside a pod)
PING 10.100.200.10 (10.100.200.10): 56 data bytes
64 bytes from 10.100.200.10: seq=0 ttl=62 time=0.087 ms
64 bytes from 10.100.200.10: seq=1 ttl=62 time=0.076 ms
^C
--- 10.100.200.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.076/0.081/0.087 ms

# Expected output for curl http://my-app-svc:80 (from inside a pod)
<!DOCTYPE html>
<html>
<head>
<title>Welcome to Nginx!</title>
...
</html>

Step 3: Inspect Network Policies

Kubernetes Network Policies are a common culprit for connectivity issues. They act as firewalls for Pods, controlling ingress and egress traffic. A misconfigured or overly restrictive policy can silently block legitimate connections, leading to application failures.

Check all Network Policies in your cluster, especially those in the affected namespaces. Understand which Pods they select and what traffic they allow or deny.

# Get all Network Policies in the current namespace
kubectl get netpol

# Get all Network Policies across all namespaces
kubectl get netpol -A

# Describe a specific Network Policy to see its rules and selectors
kubectl describe netpol <policy-name> -n <namespace>

# Get the YAML definition of a Network Policy for detailed review
kubectl get netpol <policy-name> -n <namespace> -o yaml

Verify:

Review the output carefully. Look for `podSelector`, `ingress`, and `egress` rules. Ensure that the traffic you expect to flow is explicitly allowed by one or more policies. Remember that if a Pod is selected by *any* Network Policy, all traffic not explicitly allowed by that policy (or other policies selecting the same Pod) will be denied. If no policies select a Pod, all traffic is allowed by default.

# Example Network Policy output
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-dns-and-web
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

In the above, `my-app` Pods would only accept ingress from `frontend` Pods on port 8080 and only allow egress to DNS (UDP/TCP 53) and HTTP/HTTPS (TCP 80/443).

Step 4: Capture and Analyze Traffic with tcpdump

When logical checks aren’t enough, it’s time to go to the packet level. tcpdump is an invaluable tool for seeing exactly what traffic is entering or leaving a Pod’s network interface. This can help confirm if packets are being sent, received, or dropped, and at what stage.

We’ll use kubectl debug again to run tcpdump within the network namespace of a target Pod. If kubectl debug isn’t an option, you can create a temporary debug Pod on the same node as the target Pod, but this won’t share the network namespace directly, making it less precise for Pod-specific traffic.

# Replace <pod-name> with the actual pod you want to inspect
# Replace <container-name> with the container whose network namespace you want to join
# Ensure the debug image (nicolaka/netshoot) has tcpdump installed.

# Start a debug session, joining the network namespace of the target container
kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name> -- /bin/bash

# Once inside the debug shell, run tcpdump.
# Common tcpdump filters:
# -i any: Capture on all interfaces
# -nn: Don't resolve hostnames or port names (faster, shows IPs/ports directly)
# -v: Verbose output
# port 80 or port 443: Filter for specific ports
# host <IP>: Filter for traffic to/from a specific IP
# src host <IP> and dst port 80: Filter for traffic from a source IP to a destination port

# Example: Capture all traffic on port 80 or 443
tcpdump -i any -nn -v 'port 80 or port 443'

# Example: Capture traffic to/from a specific Pod IP (e.g., 10.244.1.10)
tcpdump -i any -nn -v host 10.244.1.10

# Example: Capture traffic to a specific service ClusterIP and port
tcpdump -i any -nn -v 'dst host 10.100.200.10 and dst port 80'

# To stop tcpdump, press Ctrl+C.
# To exit the debug shell, type 'exit'.

Verify:

While tcpdump is running, try to initiate the problematic connection (e.g., `curl` from another Pod, or access the service from outside). You should see packets appearing in the tcpdump output if traffic is reaching the Pod’s network interface. Look for:

• SYN/SYN-ACK/ACK handshake: Indicates successful TCP connection establishment.
• RST packets: Often means connection refused, potentially by a firewall or an application not listening on the port.
• No packets at all: Suggests traffic is being dropped earlier, perhaps by a Node’s firewall, CNI, or a Network Policy.

# Expected tcpdump output for a successful HTTP connection
# (truncated for brevity)
09:30:00.123456 IP 10.244.1.11.34567 > 10.244.1.10.80: Flags [S], seq 12345, win 29200, options [mss 1460,sackOK,TS val 12345678 ecr 0,nop,wscale 7], length 0
09:30:00.123500 IP 10.244.1.10.80 > 10.244.1.11.34567: Flags [S.], seq 98765, ack 12346, win 28960, options [mss 1460,sackOK,TS val 87654321 ecr 12345678,nop,wscale 7], length 0
09:30:00.123550 IP 10.244.1.11.34567 > 10.244.1.10.80: Flags [.], ack 98766, win 229, options [nop,nop,TS val 12345680 ecr 87654321], length 0
09:30:00.123600 IP 10.244.1.11.34567 > 10.244.1.10.80: Flags [P.], seq 12346:12390, ack 98766, win 229, options [nop,nop,TS val 12345681 ecr 87654321], length 44: HTTP: GET / HTTP/1.1
09:30:00.123700 IP 10.244.1.10.80 > 10.244.1.11.34567: Flags [P.], seq 98766:98910, ack 12390, win 226, options [nop,nop,TS val 87654322 ecr 12345681], length 144: HTTP: HTTP/1.1 200 OK

Step 5: Inspect Node and CNI Configuration (Advanced)

If traffic isn’t reaching the Pod (as indicated by tcpdump showing no packets), the issue might be at the node level or with your Container Network Interface (CNI) plugin. This step requires SSH access to your Kubernetes worker nodes.

Different CNI plugins (Calico, Cilium, Flannel, etc.) have their own troubleshooting tools. We’ll cover general steps here.

# SSH into the Kubernetes node where your affected Pod is running
ssh <node-ip>

# Check the node's firewall rules (e.g., iptables)
sudo iptables -L -n -v | less
sudo iptables -t nat -L -n -v | less

# Check the CNI network configuration file (location varies by CNI and OS)
# Common paths: /etc/cni/net.d/, /opt/cni/bin/
ls -l /etc/cni/net.d/
cat /etc/cni/net.d/<cni-config-file>

# Inspect network interfaces on the node
ip a
ip r

# Check the Docker/containerd bridge network (if applicable)
brctl show

# For Cilium, use the cilium CLI
# (Requires Cilium CLI installed on node or access to cilium pod)
# kubectl exec -it -n kube-system cilium-<pod-id> -- cilium status
# kubectl exec -it -n kube-system cilium-<pod-id> -- cilium endpoint list
# kubectl exec -it -n kube-system cilium-<pod-id> -- cilium policy get
# For more on Cilium, see our guide on Cilium WireGuard Encryption or eBPF Observability with Hubble.

# For Calico, use the calicoctl CLI
# (Requires calicoctl installed on node or access to calico-node pod)
# kubectl exec -it -n kube-system calico-node-<pod-id> -- calicoctl get ippool -o wide
# kubectl exec -it -n kube-system calico-node-<pod-id> -- calicoctl get felixconfiguration -o yaml

Verify:

Look for `iptables` rules that might be blocking traffic, especially in the `KUBE-SERVICES` or `KUBE-FIREWALL` chains. Ensure your CNI configuration files are correct and point to valid CNI binaries. Check `ip a` output for the `cni0` or similar bridge interface and Pod-specific virtual Ethernet (veth) interfaces. Any misconfiguration here can lead to widespread network outages.

If you’re using a CNI like Cilium, its own CLI tools provide deep insights into network policies, endpoint status, and packet flows, often leveraging the power of eBPF. Similarly, Calico‘s `calicoctl` offers detailed policy and IP pool management.

Production Considerations

Troubleshooting in production requires a more cautious and systematic approach to avoid further disruption. Here are some key considerations:

• Non-Disruptive Tools First: Always start with non-disruptive commands like kubectl get, describe, and logs. Avoid exec or debug into critical production Pods initially unless absolutely necessary and you understand the implications.
• Dedicated Debug Pods: Instead of modifying application containers, deploy a temporary debug Pod with all necessary tools (nicolaka/netshoot is excellent). This keeps your application containers clean and avoids introducing new issues.
• Observability is Key: Invest heavily in Kubernetes observability. Tools like Prometheus for metrics, Grafana for dashboards, and a robust logging solution (e.g., ELK stack, Loki) can often highlight network bottlenecks or errors before they become critical. Consider advanced eBPF-based tools like Hubble for Cilium for unparalleled network visibility.
• Network Policy Auditing: Regularly audit your Network Policies. Tools exist that can visualize and analyze policies to ensure they are not overly permissive or restrictive.
• Resource Limits and Quotas: Ensure that your Pods and Namespaces have appropriate resource limits and quotas. While not directly a network issue, resource starvation can manifest as connectivity problems due to unresponsive applications.
• Rolling Updates for Changes: When applying fixes (e.g., changing Network Policies or Service definitions), always use rolling updates or blue/green deployments to minimize downtime and provide a quick rollback mechanism.
• Understand Your CNI: Each CNI plugin (Calico, Flannel, Weave, Cilium) has its own nuances, troubleshooting guides, and specific commands. Familiarize yourself with your chosen CNI’s operational aspects.
• Cloud Provider Networking: If running on a cloud provider, remember that their network infrastructure (VPC, Security Groups, Network ACLs, Load Balancers) also plays a critical role. Ensure these external components are correctly configured and not blocking traffic.
• Gateway API vs. Ingress: If you’re managing external traffic, be aware of whether you’re using Ingress or the newer Kubernetes Gateway API. Each has its own troubleshooting considerations for external access.
• Service Mesh Considerations: If you’re running a service mesh like Istio Ambient Mesh, traffic routing and policy enforcement can become even more complex. The service mesh’s control plane and data plane proxies (e.g., Envoy) introduce additional layers for troubleshooting.

Troubleshooting Common Issues

Here are some common Kubernetes network issues and their solutions:

1. Issue: Pods in `Pending` or `CrashLoopBackOff` due to network errors.

   Diagnosis: Pods fail to start or repeatedly crash. Logs might show “connection refused,” “failed to lookup host,” or similar network-related errors during initialization.

   Solution:

   • Check Pod Logs: kubectl logs <pod-name>. Look for specific error messages.
   • Describe Pod: kubectl describe pod <pod-name>. Check `Events` for issues like image pull failures (network to registry), volume mounting problems, or CNI errors.
   • Verify DNS: If `failed to lookup host` is seen, test DNS resolution from within a debug Pod.
   • Resource Limits: Ensure the Pod has sufficient CPU/memory. Resource starvation can prevent network initialization.
2. Issue: Service has no Endpoints.

   Diagnosis: kubectl get endpoints <service-name> shows `<none>` or an empty list. Pods are running, but the Service can’t route traffic to them.

   Solution:

   • Service Selector Mismatch: The most common cause. Ensure the `selector` in your Service YAML perfectly matches the `labels` on your Pods.

     # Service
     selector:
       app: my-app # Must match Pod labels
     
     # Pod
     labels:
       app: my-app

   • Target Port Mismatch: The `targetPort` in the Service definition must match the `containerPort` defined in the Pod’s container spec.
   • Pod Readiness: If a Pod’s readiness probe is failing, it won’t be added to Service Endpoints. Check Pod logs and `kubectl describe pod` for readiness probe failures.
3. Issue: Pods on different nodes cannot communicate.

   Diagnosis: `ping` or `curl` between Pods on the same node works, but fails between Pods on different nodes.

   Solution:

   • CNI Plugin Issue: This strongly suggests a problem with your CNI plugin’s cross-node networking.
     • Check CNI Pods: kubectl get pods -n kube-system | grep <your-cni-name>. Ensure they are all `Running`.
     • Check CNI Logs: kubectl logs <cni-pod-name> -n kube-system.
     • Inspect Node Routing Tables: SSH to nodes and use `ip r` to check routes for Pod CIDRs.
     • Firewall: Ensure no host-level firewalls (e.g., `firewalld`, `ufw`) are interfering with CNI traffic.
   • Network ACLs/Security Groups: In cloud environments, ensure your network security rules allow traffic between nodes on the necessary ports for your CNI (e.g., VXLAN, IP-in-IP).
4. Issue: External traffic cannot reach a NodePort or LoadBalancer Service.

   Diagnosis: You can access the Service from within the cluster, but not from outside.

   Solution:

   • Service Type: Confirm the Service type is `NodePort` or `LoadBalancer`.
   • Firewalls/Security Groups: This is the most common external issue. Ensure your cloud provider’s firewall rules (Security Groups, Network ACLs) or your on-prem network firewall allow ingress traffic to the node’s IP on the Service’s `nodePort` (for NodePort) or the LoadBalancer’s external IP on the service’s `port`.
   • LoadBalancer Provisioning: For `LoadBalancer` Services, check `kubectl describe svc <service-name>` to see if an external IP has been provisioned. If not, your cloud controller manager might have issues.
   • Network Gateway: Ensure your network gateway or router is correctly forwarding external traffic to your Kubernetes nodes.
5. Issue: DNS resolution failures inside Pods.

   Diagnosis: `curl` or `ping` by hostname fails, but by IP address works. Pod logs show “Name or service not known” or similar.

   Solution:

   • CoreDNS/Kube-DNS Status: Check the status of your cluster’s DNS Pods (usually CoreDNS) in the `kube-system` namespace. kubectl get pods -n kube-system | grep coredns.
   • CoreDNS Logs: kubectl logs <coredns-pod-name> -n kube-system.
   • `resolv.conf` in Pod: `kubectl exec -it <pod-name> — cat /etc/resolv.conf`. Ensure it points to the `kube-dns` Service IP (typically `10.96.0.10`).
   • Network Policies: Ensure Network Policies are not blocking DNS traffic (UDP/TCP port 53) from your Pods to the CoreDNS Service.
6. Issue: Intermittent connectivity or high latency.

   Diagnosis: Connections drop occasionally, or requests take a long time to complete.

   Solution:

   • Resource Saturation: Check Node and Pod CPU/memory usage. Saturated nodes can lead to network performance degradation. Consider Karpenter for efficient node autoscaling.