OpenTelemetry Collector: Unified Observability

Welcome, Kubezilla engineers! In the dynamic world of Kubernetes, observability is not merely a feature; it’s a foundational pillar for maintaining healthy, high-performing applications. Yet, achieving a unified view across logs, metrics, and traces can feel like herding cats. Different agents, varying formats, and a multitude of backend systems often lead to a fragmented, complex, and costly observability stack.

Enter the OpenTelemetry Collector – a vendor-agnostic, powerful, and highly flexible component designed to solve this very problem. It acts as a central processing pipeline for your telemetry data, allowing you to collect, process, and export data in a standardized way, regardless of its origin or destination. By deploying the Collector within your Kubernetes clusters, you gain unparalleled control over your observability data, simplifying your architecture and enhancing your ability to understand and troubleshoot your distributed applications.

This guide will walk you through deploying and configuring the OpenTelemetry Collector on Kubernetes, transforming your fragmented observability landscape into a streamlined, efficient, and unified pipeline. We’ll cover everything from basic setup to advanced processing, ensuring your telemetry data flows seamlessly from your applications to your chosen analytics backends. Let’s dive in and unlock the full potential of your Kubernetes observability!

 

Prerequisites

Before we embark on this observability journey, ensure you have the following:

• A Kubernetes Cluster: A running Kubernetes cluster (e.g., Minikube, Kind, GKE, EKS, AKS). This guide assumes you have kubectl configured to interact with your cluster.
• kubectl: The Kubernetes command-line tool, installed and configured. Refer to the official Kubernetes documentation for installation instructions.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, ConfigMaps, and Namespaces.
• Helm (Optional but Recommended): For easier management and deployment of complex applications like the OpenTelemetry Collector, Helm is highly recommended. We will primarily use raw Kubernetes manifests for granular control, but mention Helm where appropriate.
• A Target Observability Backend: While the Collector can send data anywhere, having a target like Jaeger (for traces), Prometheus (for metrics), or an OTLP-compatible backend (e.g., Grafana Cloud, Datadog, New Relic) will allow you to see your data in action. We’ll use a simple local debugging setup for demonstration.

 

Step-by-Step Guide: Deploying the OpenTelemetry Collector on Kubernetes

We’ll deploy the OpenTelemetry Collector in a dedicated namespace, configure it via a ConfigMap, and then deploy it as both a DaemonSet (for node-level collection) and a Deployment (for cluster-level aggregation). This dual approach covers common use cases.

Step 1: Create a Dedicated Namespace

It’s a best practice to isolate infrastructure components like the OpenTelemetry Collector in their own Kubernetes namespace. This helps with organization, resource management, and security. We’ll create a namespace named opentelemetry-collector.

This command creates a new namespace. All our subsequent resources for the Collector will reside within this namespace, making it easy to manage and clean up later. Good namespace hygiene is crucial in any Kubernetes environment, especially when dealing with core infrastructure components.

kubectl create namespace opentelemetry-collector

Verify Namespace Creation

You should see the opentelemetry-collector namespace listed among others.

kubectl get namespaces

NAME                      STATUS   AGE
default                   Active   2d
kube-system               Active   2d
kube-public               Active   2d
kube-node-lease           Active   2d
opentelemetry-collector   Active   5s
...

Step 2: Define the Collector Configuration with a ConfigMap

The OpenTelemetry Collector’s behavior is entirely driven by its configuration file. We’ll store this configuration in a Kubernetes ConfigMap, allowing us to easily update it without redeploying the Collector Pods. This ConfigMap will define receivers (how data is ingested), processors (how data is transformed), and exporters (where data is sent).

For this example, we’ll set up a basic configuration:

• Receivers:
  • otlp: Listens for OpenTelemetry Protocol (OTLP) data over gRPC (port 4317) and HTTP (port 4318). This is the standard way applications instrumented with OpenTelemetry send their telemetry.
  • prometheus: Scrapes metrics from a local Prometheus endpoint (port 8889). This is useful if the collector itself exposes metrics.
• Processors:
  • batch: Batches data before sending, improving efficiency.
  • memory_limiter: Prevents the collector from consuming too much memory, crucial for stability.
• Exporters:
  • logging: Prints all received telemetry data to the Collector’s standard output (useful for debugging and verification).
  • otlp: Exports data to another OTLP endpoint. In a real-world scenario, this would be your observability backend. For now, we’ll configure it to point to a non-existent endpoint or another collector.
  • prometheus: Exposes metrics in Prometheus format at /metrics (port 8889).
• Service Pipelines: These connect the receivers, processors, and exporters for traces, metrics, and logs.

Create a file named otel-collector-config.yaml:

# otel-collector-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: opentelemetry-collector-config
  namespace: opentelemetry-collector
  labels:
    app.kubernetes.io/name: opentelemetry-collector
data:
  # The OpenTelemetry Collector configuration.
  # For more details, see: https://opentelemetry.io/docs/collector/configuration/
  collector.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 10s
              static_configs:
                - targets: ['0.0.0.0:8889'] # Collector's own Prometheus exporter

    processors:
      memory_limiter:
        # 75% of maximum memory up to 1GB
        limit_mib: 1024
        spike_limit_mib: 256
        check_interval: 5s
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      logging:
        loglevel: debug # Useful for debugging, shows all telemetry data
      otlp:
        endpoint: "otel-backend:4317" # Replace with your actual OTLP backend endpoint
        tls:
          insecure: true # Use this for local testing, disable for production
      prometheus:
        endpoint: "0.0.0.0:8889" # Exposes metrics on this port

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging, otlp]
        metrics:
          receivers: [otlp, prometheus]
          processors: [memory_limiter, batch]
          exporters: [logging, otlp, prometheus]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging, otlp]

Apply this ConfigMap to your cluster:

kubectl apply -f otel-collector-config.yaml

Verify ConfigMap Creation

Check if the ConfigMap was created successfully and inspect its contents.

kubectl get configmap opentelemetry-collector-config -n opentelemetry-collector -o yaml

apiVersion: v1
data:
  collector.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 10s
              static_configs:
                - targets: ['0.0.0.0:8889'] # Collector's own Prometheus exporter

    processors:
      memory_limiter:
        # 75% of maximum memory up to 1GB
        limit_mib: 1024
        spike_limit_mib: 256
        check_interval: 5s
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      logging:
        loglevel: debug # Useful for debugging, shows all telemetry data
      otlp:
        endpoint: "otel-backend:4317" # Replace with your actual OTLP backend endpoint
        tls:
          insecure: true # Use this for local testing, disable for production
      prometheus:
        endpoint: "0.0.0.0:8889" # Exposes metrics on this port

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging, otlp]
        metrics:
          receivers: [otlp, prometheus]
          processors: [memory_limiter, batch]
          exporters: [logging, otlp, prometheus]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [logging, otlp]
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"collector.yaml":"..."},"kind":"ConfigMap","metadata":{"labels":{"app.kubernetes.io/name":"opentelemetry-collector"},"name":"opentelemetry-collector-config","namespace":"opentelemetry-collector"}}
  creationTimestamp: "2023-10-27T10:00:00Z"
  labels:
    app.kubernetes.io/name: opentelemetry-collector
  name: opentelemetry-collector-config
  namespace: opentelemetry-collector
  resourceVersion: "12345"
  uid: abcdef-1234-5678-90ab-cdef12345678

Step 3: Deploy the Collector as a DaemonSet (Node-Level Collection)

A DaemonSet ensures that a copy of the OpenTelemetry Collector runs on every (or selected) node in your cluster. This is ideal for collecting host-level metrics, logs from system components, or acting as an agent for applications running on that specific node. It often acts as the first hop in a multi-stage collection pipeline.

In this DaemonSet, we mount the ConfigMap we created earlier. The Collector Pod will read its configuration from /conf/collector.yaml. We also expose the OTLP gRPC port (4317) and HTTP port (4318) for applications to send data to.

Create a file named otel-collector-daemonset.yaml:

# otel-collector-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: opentelemetry-collector-daemonset
  namespace: opentelemetry-collector
  labels:
    app.kubernetes.io/name: opentelemetry-collector-daemonset
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetry-collector-daemonset
  template:
    metadata:
      labels:
        app.kubernetes.io/name: opentelemetry-collector-daemonset
    spec:
      serviceAccountName: opentelemetry-collector # We'll create this in the next step
      containers:
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector-contrib:0.87.0 # Using contrib for more receivers/exporters
          command:
            - "/otelcol-contrib"
            - "--config=/conf/collector.yaml"
          ports:
            - name: otlp-grpc
              containerPort: 4317
              protocol: TCP
            - name: otlp-http
              containerPort: 4318
              protocol: TCP
            - name: prometheus
              containerPort: 8889
              protocol: TCP
          volumeMounts:
            - name: opentelemetry-collector-config
              mountPath: /conf
          livenessProbe:
            httpGet:
              path: /health
              port: 13133 # Health Check Extension
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 13133 # Health Check Extension
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            limits:
              cpu: 200m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi
      volumes:
        - name: opentelemetry-collector-config
          configMap:
            name: opentelemetry-collector-config
      # Optional: Node affinity for specific nodes
      # nodeSelector:
      #   kubernetes.io/os: linux

Before applying the DaemonSet, we need a ServiceAccount for the Collector to interact with the Kubernetes API (e.g., to discover Prometheus targets or collect Kubernetes events). Create otel-serviceaccount.yaml:

# otel-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: opentelemetry-collector
  namespace: opentelemetry-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: opentelemetry-collector-clusterrole
rules:
- apiGroups: [""]
  resources: ["nodes", "nodes/metrics", "nodes/stats", "nodes/proxy", "pods", "services", "endpoints"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["replicasets", "deployments"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources: ["jobs", "cronjobs"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: opentelemetry-collector-clusterrolebinding
subjects:
- kind: ServiceAccount
  name: opentelemetry-collector
  namespace: opentelemetry-collector
roleRef:
  kind: ClusterRole
  name: opentelemetry-collector-clusterrole
  apiGroup: rbac.authorization.k8s.io

Apply the ServiceAccount and then the DaemonSet:

kubectl apply -f otel-serviceaccount.yaml
kubectl apply -f otel-collector-daemonset.yaml

Verify DaemonSet Deployment

Check if the DaemonSet Pods are running on each node.

kubectl get daemonset opentelemetry-collector-daemonset -n opentelemetry-collector
kubectl get pods -l app.kubernetes.io/name=opentelemetry-collector-daemonset -n opentelemetry-collector

NAME                                DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
opentelemetry-collector-daemonset   3         3         3       3            3           <none>          10s

NAME                                          READY   STATUS    RESTARTS   AGE
opentelemetry-collector-daemonset-abcde       1/1     Running   0          10s
opentelemetry-collector-daemonset-fghij       1/1     Running   0          10s
opentelemetry-collector-daemonset-klmno       1/1     Running   0          10s

You can also check the logs to see the collector starting up and printing debug info (due to logging exporter).

kubectl logs -l app.kubernetes.io/name=opentelemetry-collector-daemonset -n opentelemetry-collector

2023-10-27T10:05:00.123Z INFO  service/collector.go:283  "Starting Otelcol...
2023-10-27T10:05:00.124Z INFO  extensions/extensions.go:42 "Loading extensions...
2023-10-27T10:05:00.125Z INFO  service/collector.go:175  "Setting up own Prometheus scrape targets"
...
2023-10-27T10:05:05.123Z DEBUG loggingexporter@v0.87.0/logging_exporter.go:72 "TracesExporter"  {"#spans": 1, "resource spans #0": 1, "scope spans #0": 1}
...

Step 4: Deploy the Collector as a Deployment (Cluster-Level Aggregation)

While DaemonSets are great for node-level collection, a Deployment is more suitable for cluster-level aggregation or as an intermediary hop for data before it leaves the cluster. This allows for scaling the Collector horizontally based on demand and simplifies management compared to managing individual agents on each node.

This Deployment will use the same ConfigMap, but it’s deployed as a standard Deployment resource. We’ll also create a Service to expose the Collector’s OTLP ports, making it easily discoverable by other applications or DaemonSet Collectors sending data to it.

Create a file named otel-collector-deployment.yaml:

# otel-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opentelemetry-collector
  namespace: opentelemetry-collector
  labels:
    app.kubernetes.io/name: opentelemetry-collector
spec:
  replicas: 1 # Start with 1 replica, scale as needed
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetry-collector
  template:
    metadata:
      labels:
        app.kubernetes.io/name: opentelemetry-collector
    spec:
      serviceAccountName: opentelemetry-collector
      containers:
        - name: opentelemetry-collector
          image: otel/opentelemetry-collector-contrib:0.87.0
          command:
            - "/otelcol-contrib"
            - "--config=/conf/collector.yaml"
          ports:
            - name: otlp-grpc
              containerPort: 4317
              protocol: TCP
            - name: otlp-http
              containerPort: 4318
              protocol: TCP
            - name: prometheus
              containerPort: 8889
              protocol: TCP
          volumeMounts:
            - name: opentelemetry-collector-config
              mountPath: /conf
          livenessProbe:
            httpGet:
              path: /health
              port: 13133
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 13133
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 256Mi
      volumes:
        - name: opentelemetry-collector-config
          configMap:
            name: opentelemetry-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: opentelemetry-collector
  namespace: opentelemetry-collector
  labels:
    app.kubernetes.io/name: opentelemetry-collector
spec:
  selector:
    app.kubernetes.io/name: opentelemetry-collector
  ports:
    - name: otlp-grpc
      protocol: TCP
      port: 4317
      targetPort: 4317
    - name: otlp-http
      protocol: TCP
      port: 4318
      targetPort: 4318
    - name: prometheus
      protocol: TCP
      port: 8889
      targetPort: 8889

Apply this Deployment and Service:

kubectl apply -f otel-collector-deployment.yaml

Verify Deployment and Service Creation

Check if the Deployment and Service are running correctly.

kubectl get deployment opentelemetry-collector -n opentelemetry-collector
kubectl get service opentelemetry-collector -n opentelemetry-collector
kubectl get pods -l app.kubernetes.io/name=opentelemetry-collector -n opentelemetry-collector

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
opentelemetry-collector   1/1     1            1           10s

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
opentelemetry-collector   ClusterIP   10.96.10.123     <none>        4317/TCP,4318/TCP,8889/TCP            10s

NAME                                       READY   STATUS    RESTARTS   AGE
opentelemetry-collector-789abcd-efghj      1/1     Running   0          10s

Again, you can check the logs:

kubectl logs -l app.kubernetes.io/name=opentelemetry-collector -n opentelemetry-collector

2023-10-27T10:10:00.123Z INFO  service/collector.go:283  "Starting Otelcol...
...

Step 5: Test the Collector with a Sample Application

To see the Collector in action, we need an application that sends OpenTelemetry data. We’ll use a simple Go application that generates traces and metrics and is configured to send them to our Collector.

First, let’s create a Service for our DaemonSet Collectors to expose their OTLP ports to the cluster. This is crucial for applications to send data to the DaemonSet. Note that this is a ClusterIP service, as we want internal cluster communication.

Create otel-daemonset-service.yaml:

# otel-daemonset-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: opentelemetry-collector-daemonset-headless
  namespace: opentelemetry-collector
  labels:
    app.kubernetes.io/name: opentelemetry-collector-daemonset
spec:
  clusterIP: None # Headless Service for DaemonSet, so each pod gets its own DNS entry
  selector:
    app.kubernetes.io/name: opentelemetry-collector-daemonset
  ports:
    - name: otlp-grpc
      protocol: TCP
      port: 4317
      targetPort: 4317
    - name: otlp-http
      protocol: TCP
      port: 4318
      targetPort: 4318

Apply the Service:

kubectl apply -f otel-daemonset-service.yaml

Now, let’s deploy a sample application. This application will send OTLP data to the DaemonSet Collector running on its node. We’ll configure the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to point to the DaemonSet’s service.

Create a file named sample-app.yaml:

# sample-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-otel-app
  namespace: default # Deploy in default namespace for demonstration
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-otel-app
  template:
    metadata:
      labels:
        app: sample-otel-app
    spec:
      containers:
        - name: sample-app
          image: otel/otel-go-demo:latest # A simple Go demo application
          env:
            - name: OTEL_SERVICE_NAME
              value: "my-sample-go-app"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "opentelemetry-collector-daemonset-headless.opentelemetry-collector.svc.cluster.local:4317" # Point to the DaemonSet Collector
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: "grpc"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=production,service.version=1.0.0"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: sample-otel-app
  namespace: default
spec:
  selector:
    app: sample-otel-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP

Apply the sample application:

kubectl apply -f sample-app.yaml

Verify Sample Application and Data Flow

Check if the sample application Pods are running.

kubectl get pods -l app=sample-otel-app -n default

NAME                              READY   STATUS    RESTARTS   AGE
sample-otel-app-789abcd-efghj     1/1     Running   0          20s
sample-otel-app-uvwxy-ijklm       1/1     Running   0          20s

Now, the critical part: check the logs of your OpenTelemetry Collector DaemonSet Pods. You should see the traces and metrics generated by the sample application being received and processed by the Collector, thanks to the logging exporter.

kubectl logs -l app.kubernetes.io/name=opentelemetry-collector-daemonset -n opentelemetry-collector

...

2023-10-27T10:15:00.123Z DEBUG loggingexporter@v0.87.0/logging_exporter.go:72 "TracesExporter"  {"#spans": 1, "resource spans #0": 1, "scope spans #0": 1}

2023-10-27T10:15:00.124Z DEBUG loggingexporter@v0.87.0/logging_exporter.go:72 "ResourceSpans #0"  {"resource": {"attributes": [{"key": "deployment.environment", "value": {"stringValue": "production"}}, {"key": "host.arch", "value": {"stringValue": "amd64"}}, {"key": "service.name", "value": {"stringValue": "my-sample-go-app"}}, {"key": "service.version", "value": {"stringValue": "1.0.0"}}]}, "scope_spans": [{"scope": {}, "spans": [{"trace_id": "...", "span_id": "...", "parent_span_id": "...", "name": "doWork", "kind": "SPAN_KIND_INTERNAL", "start_time": "...", "end_time": "...", "attributes": [{"key": "work.amount", "value": {"intValue": 100}}]}]}]}

2023-10-27T10:15:05.123Z DEBUG loggingexporter@v0.87.0/logging_exporter.go:72 "MetricsExporter"  {"#metrics": 2, "resource metrics #0": 1, "scope metrics #0": 2}

2023-10-27T10:15:05.124Z DEBUG loggingexporter@v0.87.0/logging_exporter.go:72 "ResourceMetrics #0"  {"resource": {"attributes": [{"key": "deployment.environment", "value": {"stringValue": "production"}}, {"key": "host.arch", "value": {"stringValue": "amd64"}}, {"key": "service.name", "value": {"stringValue": "my-sample-go-app"}}, {"key": "service.version", "value": {"stringValue": "1.0.0"}}]}, "scope_metrics": [{"scope": {}, "metrics": [{"name": "app.request.count", "description": "Measures the number of incoming requests.", "unit": "{request}", "sum": {"data_points": [{"start_time": "...", "time": "...", "value": 1.0, "attributes": []}], "aggregation_temporality": "AGG