Kubernetes Tracing: Jaeger & OpenTelemetry

Verify:

Get the NodePort and a node's IP to access the UI.

kubectl get svc simple-jaeger-query-nodeport

Expected Output:

NAME                           TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
simple-jaeger-query-nodeport   NodePort   10.96.150.150           16686:30080/TCP   1m

Find a node's IP address:

kubectl get nodes -o wide

Then, navigate to `http://:30080/jaeger` in your browser.

Step 6: Instrument Your Application with OpenTelemetry

This is the core step where you modify your application to generate traces. OpenTelemetry provides SDKs for various languages. We'll demonstrate with a simple Python Flask application. The key is to configure the SDK to send traces to the OpenTelemetry Collector we deployed.

The application needs to:

1. Install OpenTelemetry SDKs/instrumentation libraries.
2. Initialize a TracerProvider.
3. Configure an OTLP Span Exporter pointing to the OpenTelemetry Collector.
4. Apply automatic or manual instrumentation to generate spans.

Here's a sample Python Flask application and its Kubernetes Deployment:

Python Application (app.py):

from flask import Flask, request
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
import requests
import os

# 1. Configure OpenTelemetry
resource = Resource.create({
    "service.name": os.environ.get("OTEL_SERVICE_NAME", "my-python-app"),
    "environment": "development"
})

trace.set_tracer_provider(
    TracerProvider(
        resource=resource
    )
)

# Configure the OTLP exporter to send traces to the OpenTelemetry Collector
otlp_exporter = OTLPSpanExporter(
    endpoint=os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector.default.svc.cluster.local:4317"),
    insecure=True # Use insecure for simplicity in this tutorial
)

# Add the exporter to the TracerProvider
trace.get_tracer_provider().add_span_processor(
    SimpleSpanProcessor(otlp_exporter)
)

app = Flask(__name__)

# 2. Instrument Flask and Requests libraries
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

tracer = trace.get_tracer(__name__)

@app.route("/")
def hello_world():
    with tracer.start_as_current_span("hello-endpoint"):
        # Make an internal HTTP request to demonstrate chained traces
        try:
            response = requests.get("http://localhost:5000/internal")
            internal_status = response.status_code
        except requests.exceptions.ConnectionError:
            internal_status = "error"

        return f"Hello, World! Internal call status: {internal_status}"

@app.route("/internal")
def internal_endpoint():
    with tracer.start_as_current_span("internal-endpoint"):
        # Simulate some work
        import time
        time.sleep(0.05)
        return "This is an internal endpoint."

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

requirements.txt:

Flask==2.0.3
opentelemetry-api==1.24.0
opentelemetry-sdk==1.24.0
opentelemetry-exporter-otlp==1.24.0
opentelemetry-instrumentation-flask==0.45b0
opentelemetry-instrumentation-requests==0.45b0
requests==2.28.1

Dockerfile:

FROM python:3.9-slim-buster

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY app.py .

ENV OTEL_SERVICE_NAME=my-python-app
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.default.svc.cluster.local:4317

EXPOSE 5000

CMD ["python", "app.py"]

Build and push the Docker image (replace `your-docker-repo` with your actual repo):

docker build -t your-docker-repo/my-python-app:latest .
docker push your-docker-repo/my-python-app:latest

Kubernetes Deployment (my-app-deployment.yaml):

Note the environment variables `OTEL_SERVICE_NAME` and `OTEL_EXPORTER_OTLP_ENDPOINT` pointing to our collector.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-python-app
  labels:
    app: my-python-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-python-app
  template:
    metadata:
      labels:
        app: my-python-app
    spec:
      containers:
      - name: my-python-app
        image: your-docker-repo/my-python-app:latest # REPLACE WITH YOUR IMAGE
        ports:
        - containerPort: 5000
        env:
        - name: OTEL_SERVICE_NAME
          value: my-python-app
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://otel-collector.default.svc.cluster.local:4317 # Ensure this matches your collector service
---
apiVersion: v1
kind: Service
metadata:
  name: my-python-app-service
spec:
  selector:
    app: my-python-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000
  type: ClusterIP

kubectl apply -f my-app-deployment.yaml

Verify:

Access your application and check Jaeger UI.

kubectl get pods -l app=my-python-app

Expected Output:

NAME                              READY   STATUS    RESTARTS   AGE
my-python-app-76c944f79c-pqrst    1/1     Running   0          1m

Now, send some requests to your application. If you have a NodePort or Ingress for it, use that. Otherwise, you can port-forward:

kubectl port-forward svc/my-python-app-service 8080:80

Then, in another terminal, hit the endpoint:

curl http://localhost:8080

Repeat this a few times. Now, open your Jaeger UI (`http://:30080/jaeger` or your Ingress URL). Select `my-python-app` from the "Service" dropdown and click "Find Traces". You should see traces representing your requests, with spans for "hello-endpoint" and "internal-endpoint".

This demonstrates end-to-end tracing, allowing you to visualize the flow of requests and identify performance characteristics of different parts of your application. For more advanced network security and isolation, consider implementing Kubernetes Network Policies around your tracing components and applications.

Production Considerations

Deploying tracing in production requires more robustness and scalability than our `allInOne` development setup.

• Jaeger Deployment Strategy: Switch from `allInOne` to a `production` strategy for Jaeger. This deploys separate components:
  • Jaeger Collector: Receives traces.
  • Jaeger Query: Serves the UI and API.
  • Jaeger Agent: Can run as a DaemonSet on each node to collect traces from local applications and forward them to the collector. This reduces network overhead and provides a local buffer.
  • Storage Backend: For production, you must use a persistent storage backend like Elasticsearch, Cassandra, or an external database. The `allInOne` uses in-memory storage, which is not durable. Configure this in your `Jaeger` CR. For example, using Elasticsearch:

    spec:
      strategy: production
      storage:
        type: elasticsearch
        options:
          es:
            server-urls: http://elasticsearch-master:9200
    

• OpenTelemetry Collector Deployment:
  • Deployment Strategy: For high-traffic services, consider deploying the collector as a `DaemonSet` on each node or as a sidecar alongside your application pods.
    • DaemonSet: Each node runs a collector. Applications on that node send traces to the local collector. This is efficient for host-level telemetry.
    • Sidecar: Each application pod has its own collector sidecar. Applications send traces to `localhost`. This provides strong isolation and resource management per application.
  • Resource Limits: Set appropriate CPU and memory limits for your collector pods to prevent resource exhaustion.
  • Horizontal Scaling: If using a Deployment for the collector, you might need to scale it horizontally with `HPA` based on CPU or memory usage, especially if it's a central bottleneck.
• Network Security:
  • TLS: Enable TLS for OTLP endpoints on the OpenTelemetry Collector and for communication between the collector and Jaeger.
  • Network Policies: Implement Kubernetes Network Policies to restrict traffic to and from your tracing components, ensuring only authorized services can send or receive tracing data. For example, allow only application pods to send to the collector, and only the collector to send to Jaeger.
  • Ingress/Gateway: Expose the Jaeger UI securely using an Ingress controller or the Kubernetes Gateway API, with proper authentication and authorization.
• Sampling: In high-volume environments, collecting 100% of traces can be costly and generate too much data. Implement sampling strategies in the OpenTelemetry Collector to reduce the volume of traces while still capturing representative samples. This can be configured in the collector's `processors` section.
• Resource Management: Monitor the resource consumption of Jaeger and OpenTelemetry Collector components. Adjust `requests` and `limits` in your Kubernetes manifests. For cost optimization, especially with underlying nodes, tools like Karpenter can help manage node resources dynamically.
• Observability of Observability: Monitor your Jaeger and OpenTelemetry Collector instances themselves. Are they healthy? Are they dropping traces? Integrate their metrics into your existing monitoring solution. You can leverage tools like eBPF Observability with Hubble to gain deeper insights into network interactions if you're using Cilium.
• Authentication/Authorization: Secure access to the Jaeger UI and API with appropriate authentication and authorization mechanisms, especially if exposed externally.
• Sidecar Injection with Service Mesh: If you're already using a service mesh like Istio (e.g., Istio Ambient Mesh), it can often handle automatic instrumentation and trace context propagation, reducing the need for manual application-level changes.

Troubleshooting

Here are some common issues you might encounter and their solutions:

1. No Traces Appearing in Jaeger UI

   Issue: You've instrumented your app and sent requests, but Jaeger UI shows "No traces found."

   Solution:

   1. Check Application Logs: Look for errors in your application's logs related to OpenTelemetry or trace export.
   2. Verify Collector Connectivity: Ensure your application's `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable correctly points to the OpenTelemetry Collector service (e.g., `http://otel-collector.default.svc.cluster.local:4317`).
   3. Check Collector Logs: Inspect the logs of the `otel-collector` pod. Look for