Orchestration

Grafana Tempo: Master Distributed Traces

July 4, 2026 Kubezilla Team 16 min read

Introduction

In the complex, distributed landscape of modern applications, understanding how requests flow through microservices can be a monumental challenge. When a user experiences a slow response or an error, pinpointing the exact service or component responsible becomes a high-stakes detective mission. Traditional logging and metrics offer snapshots, but they often lack the crucial context of a request’s journey across multiple services, databases, and external APIs. This is where distributed tracing shines, providing end-to-end visibility into the lifecycle of a request, detailing latency, errors, and the relationships between services.

Grafana Tempo emerges as a powerful, high-scale, and cost-effective distributed tracing backend designed to store traces without the need for an expensive indexing database. Unlike traditional tracing systems that index every span, Tempo focuses on storing trace IDs and their associated spans, relying on Grafana Loki for log correlation and Prometheus for metrics. This unique architectural approach makes it incredibly efficient for handling massive volumes of trace data, allowing you to observe the full story of your application’s behavior in production. This guide will walk you through deploying Grafana Tempo on Kubernetes, configuring your applications to send traces, and visualizing them in Grafana, transforming your debugging and performance optimization efforts.

TL;DR: Grafana Tempo Distributed Tracing on Kubernetes

Deploy Grafana Tempo and Grafana on Kubernetes using Helm. Configure your applications to send OpenTelemetry traces to Tempo. Visualize traces, logs, and metrics in Grafana for end-to-end observability.

Key Commands:


# Add Grafana Helm repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Grafana Tempo (simple mode)
helm install tempo grafana/tempo -n tempo --create-namespace

# Install Grafana
helm install grafana grafana/grafana -n grafana --create-namespace

# Port-forward Grafana to access UI
kubectl port-forward service/grafana -n grafana 3000:80

# Example: Configure an application to send traces to Tempo
# (Assuming OpenTelemetry Collector is used)
# collector-config.yaml excerpt:
# exporters:
#   otlp:
#     endpoint: "tempo-tempo-distributed-collector.tempo.svc.cluster.local:4317"
#     tls:
#       insecure: true
# service:
#   pipelines:
#     traces:
#       receivers: [otlp]
#       processors: [batch]
#       exporters: [otlp]

Prerequisites

To follow this guide, you’ll need:

  • A running Kubernetes cluster (v1.20+ recommended).
  • Helm (v3.x) installed on your local machine.
  • kubectl configured to connect to your cluster.
  • Basic understanding of Kubernetes concepts (Pods, Services, Deployments, Namespaces).
  • Familiarity with distributed tracing concepts (spans, traces, OpenTelemetry).
  • A web browser to access the Grafana UI.

Step-by-Step Guide: Deploying Grafana Tempo and Tracing an Application

1. Add Grafana Helm Repository and Update

First, we need to add the official Grafana Helm chart repository, which contains charts for Grafana, Tempo, Loki, and other related projects. This ensures we can easily install the latest stable versions of these components. After adding the repository, it’s good practice to update your local Helm repositories to fetch the latest chart information.


helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Verify: You should see output indicating the repositories have been updated.


"grafana" has been added to your repositories
Hang tight while we grab the latest from your configured repos...
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

2. Deploy Grafana Tempo to Kubernetes

Grafana Tempo can be deployed in various modes, including a simple all-in-one mode suitable for development and small clusters, or a distributed mode for production environments. For this guide, we’ll start with the simple mode, which bundles all Tempo components into a single pod, making deployment straightforward. We’ll deploy it into its own namespace for better isolation. For production-grade deployments, consider the distributed mode for scalability and resilience.


helm install tempo grafana/tempo -n tempo --create-namespace

Verify: Check that Tempo pods are running and the service is available.


kubectl get pods -n tempo


NAME                                 READY   STATUS    RESTARTS   AGE
tempo-0                              1/1     Running   0          2m


kubectl get svc -n tempo


NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
tempo-tempo                           ClusterIP   10.101.10.10     <none>        3100/TCP,9095/TCP,4317/TCP,14268/TCP   2m
tempo-tempo-distributed-collector     ClusterIP   10.101.10.11     <none>        4317/TCP,4318/TCP,14250/TCP,14268/TCP  2m

Note the tempo-tempo-distributed-collector service. This is the primary endpoint for sending traces to Tempo using OTLP (OpenTelemetry Protocol).

3. Deploy Grafana for Visualization

Grafana is the primary interface for visualizing traces stored in Tempo, as well as metrics from Prometheus and logs from Loki. We’ll deploy Grafana into its own namespace and then configure it to connect to our Tempo instance.


helm install grafana grafana/grafana -n grafana --create-namespace

Verify: Ensure Grafana pods are running.


kubectl get pods -n grafana


NAME                               READY   STATUS    RESTARTS   AGE
grafana-7bb674d8d4-abcde           1/1     Running   0          1m

4. Access Grafana UI and Configure Tempo Data Source

To access the Grafana UI, we’ll use kubectl port-forward. This temporarily exposes the Grafana service on your local machine. Once accessed, we’ll add Tempo as a data source.


kubectl port-forward service/grafana -n grafana 3000:80

Open your web browser and navigate to http://localhost:3000. You’ll need the Grafana admin password. Retrieve it using:


kubectl get secret grafana -n grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Log in with username admin and the retrieved password.
Once logged in, navigate to Configuration (gear icon) -> Data Sources -> Add data source and select Tempo.

Configure the Tempo data source with the following settings:

  • Name: Tempo (or any descriptive name)
  • URL: http://tempo-tempo-distributed-query-frontend.tempo.svc.cluster.local:3100
  • Service Name: tempo-tempo-distributed-query-frontend.tempo.svc.cluster.local

Click “Save & Test”. You should see “Data source is working” or similar confirmation.

For more details on Grafana data sources, refer to the official Grafana documentation.

5. Instrument an Application to Send Traces

Now, let’s deploy a sample application that is instrumented to send traces using OpenTelemetry. We’ll use a simple Python Flask application for this example. The key is to configure the OpenTelemetry exporter to point to our Tempo collector service. We recommend using an OpenTelemetry Collector as an intermediary for robust trace ingestion.

First, create a Kubernetes namespace for our sample application:


kubectl create namespace sample-app

Next, define the OpenTelemetry Collector configuration. This collector will receive traces from our application and forward them to Tempo.


apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: sample-app
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
        send_batch_size: 100
        timeout: 10s
    exporters:
      otlp:
        endpoint: "tempo-tempo-distributed-collector.tempo.svc.cluster.local:4317" # Tempo OTLP gRPC endpoint
        tls:
          insecure: true # Use insecure for simplicity in this demo. For production, secure communication is essential.
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]


kubectl apply -f otel-collector-config.yaml

Now, deploy the OpenTelemetry Collector as a Deployment:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: sample-app
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.96.0 # Use a recent stable version
        command:
        - "/otelcol-contrib"
        - "--config=/conf/otel-collector-config.yaml"
        ports:
        - name: otlp-grpc
          containerPort: 4317
        - name: otlp-http
          containerPort: 4318
        volumeMounts:
        - name: otel-collector-config-volume
          mountPath: /conf
      volumes:
      - name: otel-collector-config-volume
        configMap:
          name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: sample-app
  labels:
    app: otel-collector
spec:
  selector:
    app: otel-collector
  ports:
  - name: otlp-grpc
    protocol: TCP
    port: 4317
    targetPort: 4317
  - name: otlp-http
    protocol: TCP
    port: 4318
    targetPort: 4318


kubectl apply -f otel-collector-deployment.yaml

Finally, deploy our sample Python Flask application. This application will be configured to send traces to the otel-collector service.


apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
  namespace: sample-app
  labels:
    app: flask-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
    spec:
      containers:
      - name: flask-app
        image: python:3.9-slim-buster
        command: ["/bin/bash", "-c"]
        args:
          - |
            pip install Flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-requests
            cat <<EOF > app.py
            from flask import Flask, request
            import requests
            from opentelemetry import trace
            from opentelemetry.sdk.resources import Resource
            from opentelemetry.sdk.trace import TracerProvider
            from opentelemetry.sdk.trace.export import BatchSpanProcessor
            from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
            from opentelemetry.instrumentation.flask import FlaskInstrumentor
            from opentelemetry.instrumentation.requests import RequestsInstrumentor

            # Service name in traces
            resource = Resource.create(attributes={"service.name": "flask-sample-app"})
            trace.set_tracer_provider(TracerProvider(resource=resource))
            tracer = trace.get_tracer(__name__)

            # Configure OTLP exporter to send to the OpenTelemetry Collector
            otlp_exporter = OTLPSpanExporter(endpoint="otel-collector.sample-app.svc.cluster.local:4317", insecure=True)
            span_processor = BatchSpanProcessor(otlp_exporter)
            trace.get_tracer_provider().add_span_processor(span_processor)

            app = Flask(__name__)
            FlaskInstrumentor().instrument_app(app)
            RequestsInstrumentor().instrument()

            @app.route("/")
            def hello():
                with tracer.start_as_current_span("hello-request"):
                    # Make an internal HTTP call to simulate a distributed transaction
                    response = requests.get("http://localhost:5000/internal")
                    return f"Hello from Flask! Internal call status: {response.status_code}"

            @app.route("/internal")
            def internal():
                with tracer.start_as_current_span("internal-call"):
                    return "This is an internal endpoint."

            if __name__ == "__main__":
                app.run(host="0.0.0.0", port=5000)
            EOF
            python app.py
        env:
        - name: OTEL_SERVICE_NAME
          value: my-flask-app
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://otel-collector.sample-app.svc.cluster.local:4318 # OTLP HTTP endpoint of the collector
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: http/protobuf
        ports:
        - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: flask-app
  namespace: sample-app
spec:
  selector:
    app: flask-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000
  type: ClusterIP


kubectl apply -f flask-app-deployment.yaml

Verify: Check that the application pods are running.


kubectl get pods -n sample-app


NAME                             READY   STATUS    RESTARTS   AGE
otel-collector-798864789-fghij   1/1     Running   0          2m
flask-app-6789abcd-efgh          1/1     Running   0          1m

6. Generate Traffic and View Traces in Grafana

Now that our application is deployed and configured to send traces, we need to generate some traffic to it. We’ll port-forward the Flask application service and then make some requests.


kubectl port-forward service/flask-app -n sample-app 8080:80

In a new terminal, make a few requests:


curl http://localhost:8080
curl http://localhost:8080
curl http://localhost:8080

Now, go back to your Grafana UI (http://localhost:3000).
Navigate to Explore (compass icon). Select the Tempo data source.
In the Tempo query editor, you can search for traces. Since we just generated traffic, you can often click “Run Query” without specific filters to see recent traces. You can also filter by service name (flask-sample-app) or other attributes if you send more complex traces.
You should see a list of traces. Click on one to view its detailed span waterfall diagram, showing the duration of each operation within the request. This provides invaluable insights into latency and dependencies.

For advanced observability, consider integrating Grafana Tempo with eBPF Observability with Hubble to gain deeper network insights, or utilize Istio Ambient Mesh for sidecar-less service mesh capabilities which often include integrated tracing.

Production Considerations

Deploying Grafana Tempo in production requires careful planning for scalability, reliability, and security.

  • Scalability: For high-volume environments, deploy Tempo in its distributed mode. This involves separate deployments for ingesters, distributors, compactors, and queriers, allowing independent scaling of each component. Consider using object storage (S3, GCS, Azure Blob Storage) for long-term trace storage, as it’s highly scalable and cost-effective.
  • Storage Backend: While the simple mode uses ephemeral storage, production deployments require a persistent and scalable backend. Configure Tempo to use S3, GCS, Azure Blob Storage, or a compatible S3-like object storage. This ensures trace data survives pod restarts and can scale with your needs.
  • Resource Limits: Set appropriate CPU and memory limits for Tempo components. Tracing can be resource-intensive, especially for ingesters and compactors. Monitor resource usage closely and adjust limits as needed.
  • Security:
    • TLS: Secure OTLP endpoints with TLS encryption. In our demo, we used `insecure: true` for simplicity, but this is a critical security vulnerability in production.
    • Authentication/Authorization: Implement authentication for Grafana and ensure proper role-based access control (RBAC) for accessing trace data.
    • Network Policies: Restrict network access to Tempo components using Kubernetes Network Policies. Only allow necessary services (like OpenTelemetry Collectors) to send traces to Tempo.
    • Data Encryption: Ensure data at rest in your object storage backend is encrypted.
  • OpenTelemetry Collector: Always use an OpenTelemetry Collector in front of Tempo in production. The collector acts as a buffer, batcher, and can perform various processing steps (like attribute filtering, sampling) before traces hit Tempo, improving efficiency and reliability.
  • Sampling: Distributed tracing can generate a huge amount of data. Implement intelligent sampling strategies (head-based, tail-based) at the OpenTelemetry Collector level to manage data volume and cost without losing critical insights.
  • Monitoring and Alerting: Monitor Tempo’s health and performance using Prometheus and Grafana. Set up alerts for issues like high error rates, slow trace ingestion, or storage backend problems.
  • High Availability: Deploy multiple replicas for critical Tempo components (distributors, queriers) and ensure your object storage backend is highly available.
  • Cost Optimization: Consider the storage costs associated with trace data. Sampling, data retention policies, and choosing the right storage backend (e.g., S3 Intelligent-Tiering) can significantly impact costs. For overall Kubernetes cost optimization, tools like Karpenter can help manage node costs.

Troubleshooting

  1. Issue: Grafana Tempo pods are not starting or are in a CrashLoopBackOff state.

    Solution:

    1. Check pod logs: kubectl logs <tempo-pod-name> -n tempo. Look for error messages related to configuration, storage, or resource issues.
    2. Check pod descriptions: kubectl describe pod <tempo-pod-name> -n tempo. This can reveal issues like insufficient memory/CPU, persistent volume claim (PVC) problems, or image pull errors.
    3. Verify Helm values: Ensure your Helm `values.yaml` (if used) has correct configurations, especially for storage.
  2. Issue: Traces are not appearing in Grafana.

    Solution:

    1. Check application logs: Ensure your application is not reporting errors when sending traces.
    2. Verify OpenTelemetry Collector:
      • Check collector pods: kubectl get pods -n sample-app -l app=otel-collector.
      • Check collector logs: kubectl logs <otel-collector-pod> -n sample-app. Look for errors receiving traces from the app or exporting them to Tempo.
      • Verify collector configuration: Ensure the endpoint for the OTLP exporter points to the correct Tempo service (e.g., `tempo-tempo-distributed-collector.tempo.svc.cluster.local:4317`).
    3. Verify Tempo Ingestion Endpoints:
      • Check Tempo pod logs for any ingestion errors: kubectl logs <tempo-pod-name> -n tempo.
      • Ensure network connectivity from the OpenTelemetry Collector to Tempo. You can test this by exec’ing into the collector pod and trying to resolve the Tempo service name or even a `curl` if HTTP endpoint is used.
    4. Grafana Data Source: Double-check the Tempo data source URL in Grafana (e.g., `http://tempo-tempo-distributed-query-frontend.tempo.svc.cluster.local:3100`).
  3. Issue: High latency in trace ingestion or querying.

    Solution:

    1. Storage Backend: If using a non-local storage, check the performance of your object storage (S3, GCS). Network latency to the storage can impact performance.
    2. Tempo Resources: In a distributed Tempo deployment, ensure you have enough resources (CPU, memory) allocated to distributors (ingestion) and queriers (querying).
    3. Compactor Activity: If traces are slow to appear, the compactor might be overloaded or misconfigured, delaying the availability of traces for querying. Check compactor logs.
    4. Network Issues: Investigate network latency within your Kubernetes cluster. Tools like Cilium WireGuard Encryption can help secure network traffic, but ensure it’s not introducing unexpected latency.
  4. Issue: Grafana UI is not accessible via port-forward.

    Solution:

    1. Ensure the kubectl port-forward command is still running in its terminal.
    2. Verify the Grafana service name: kubectl get svc -n grafana. Ensure you are forwarding the correct service.
    3. Check for local port conflicts: Make sure port 3000 isn’t already in use on your local machine. Try forwarding to a different local port (e.g., kubectl port-forward service/grafana -n grafana 8080:80).
  5. Issue: Missing spans or incomplete traces in Grafana.

    Solution:

    1. Sampling: Check if your OpenTelemetry Collector or application is configured for sampling. If so, some traces might be intentionally dropped.
    2. Context Propagation: Ensure trace context (trace ID, span ID) is correctly propagated across service boundaries (e.g., HTTP headers like `traceparent`).
    3. Instrumentation: Verify that all services involved in a transaction are properly instrumented with OpenTelemetry.
    4. Batching/Buffering: The OpenTelemetry Collector might be batching traces. Ensure the batching configuration (send_batch_size, timeout) is appropriate and not causing delays or drops.
  6. Issue: High resource usage by OpenTelemetry Collector.

    Solution:

    1. Batch Processor Tuning: Adjust send_batch_size and timeout in the collector’s batch processor. Larger batches can reduce CPU overhead but increase latency for traces to appear.
    2. Sampling: Implement head-based or tail-based sampling in the collector to reduce the volume of traces being processed and exported.
    3. Resource Limits: Increase CPU and memory limits for the collector pods if they are consistently hitting limits and being throttled.
    4. Horizontal Scaling: For very high trace volumes, consider deploying multiple collector instances behind a load balancer.

FAQ Section

  1. Q: What is the main difference between Grafana Tempo and other tracing backends like Jaeger or Zipkin?

    A: The primary differentiator is Tempo’s trace-ID-based storage model. Unlike Jaeger or Zipkin, which index every span, Tempo only stores the trace ID and the full trace data. This makes Tempo incredibly cost-effective and scalable for high-volume tracing as it avoids the overhead of an expensive indexing database. It relies on logs (Loki) and metrics (Prometheus) for initial context to find trace IDs, rather than indexing span attributes directly.

  2. Q: How does Grafana Tempo integrate with Grafana Loki and Prometheus?

    A: This is where Tempo’s true power lies. Grafana provides seamless correlation between traces, logs, and metrics. From a trace in Tempo, you can jump directly to relevant logs in Loki (using matching trace IDs) or metrics in Prometheus (using service names or other attributes). This “exemplar” pattern significantly speeds up root cause analysis. For more on correlative observability, consider eBPF Observability with Hubble, which provides deep network metrics.

  3. Q: Is OpenTelemetry required to send traces to Tempo?

    A: While Tempo supports various trace formats (Jaeger, Zipkin, OpenCensus, OpenTelemetry), OpenTelemetry (OTel) is the recommended and future-proof way to instrument your applications and send traces. OTel provides a standardized, vendor-agnostic set of APIs, SDKs, and collectors for telemetry data (traces, metrics, logs).

  4. Q: How can I reduce the amount of trace data sent to Tempo?

    A: The most effective way is through sampling. You can implement sampling at the application level (SDK) or, more commonly and flexibly, within an OpenTelemetry Collector. The Collector supports various sampling strategies, including head-based (e.g., probabilistic, always-on for errors) and tail-based (sampling decisions made after a trace is complete, allowing for more intelligent decisions based on errors or latency). This is crucial for managing costs and storage in production. Also, consider Karpenter for overall cost optimization in your cluster.

  5. Q: Can Tempo store traces from different programming languages and frameworks?

    A: Yes, absolutely. Because Tempo primarily accepts traces via the OpenTelemetry Protocol (OTLP), any application instrumented with OpenTelemetry SDKs for various languages (Python, Java, Go, Node.js, .NET, etc.) can send traces to Tempo. The OTLP format acts as a universal language for telemetry.

Cleanup Commands

To remove all resources deployed in this guide:


# Delete Grafana Tempo
helm uninstall tempo -n tempo
kubectl delete namespace tempo

# Delete Grafana
helm uninstall grafana -n grafana
kubectl delete namespace grafana

# Delete sample application and OpenTelemetry Collector
kubectl delete deployment otel-collector -n sample-app
kubectl delete service otel-collector -n sample-app
kubectl delete configmap otel-collector-config -n sample-app
kubectl delete deployment flask-app -n sample-app
kubectl delete service flask-app -n sample-app
kubectl delete namespace sample-app

# Remove Helm repo (optional)
helm repo remove grafana

Next Steps / Further Reading

Conclusion

Distributed tracing is no longer a luxury but a necessity for understanding the behavior of complex microservices architectures. Grafana Tempo offers a powerful, cost-effective, and scalable solution for storing and querying trace data. By deploying Tempo on Kubernetes and integrating it with Grafana, you gain unparalleled visibility into your application’s request flows, enabling faster debugging, performance optimization, and a deeper understanding of your system’s health. The combination of traces, logs, and metrics within the Grafana ecosystem provides a holistic observability platform, empowering your teams to build and operate robust, high-performance applications with confidence.

Leave a comment