Master Prometheus ServiceMonitor & PodMonitor

In the dynamic world of Kubernetes, ensuring the reliability and performance of your applications hinges on robust monitoring. While Prometheus has emerged as the de-facto standard for cloud-native observability, collecting metrics from hundreds or thousands of ephemeral pods and services can quickly become a management nightmare. Manually configuring Prometheus to scrape these targets is not just tedious; it’s fundamentally at odds with the declarative, automated nature of Kubernetes itself.

Enter the Prometheus Operator, a game-changer that brings the power of Kubernetes to Prometheus management. By introducing custom resource definitions (CRDs) like Prometheus, Alertmanager, ServiceMonitor, and PodMonitor, the Operator allows you to declare your monitoring targets and configurations directly within your cluster. This tutorial will dive deep into the two most crucial CRDs for metric discovery: ServiceMonitor and PodMonitor, equipping you with the knowledge to automatically discover and scrape metrics from your Kubernetes workloads with unparalleled ease.

Whether you’re running a few microservices or a large-scale, multi-tenant platform, understanding how to leverage ServiceMonitor and PodMonitor is essential for building a resilient and observable Kubernetes environment. We’ll walk through their installation, configuration, and practical application, ensuring your Prometheus instance is always up-to-date with your ever-changing application landscape. Let’s transform your monitoring from a manual chore into an automated, declarative superpower.

# Install Prometheus Operator (e.g., via Helm)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

# Example ServiceMonitor for an Nginx deployment
kubectl apply -f <(cat <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-nginx-app
  labels:
    app: my-nginx-app
spec:
  selector:
    matchLabels:
      app: my-nginx-app
  endpoints:
  - port: http-metrics
    path: /metrics
EOF
)

# Example PodMonitor for a custom app
kubectl apply -f <(cat <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-custom-pod
  labels:
    app: my-custom-pod
spec:
  selector:
    matchLabels:
      app: my-custom-pod
  podMetricsEndpoints:
  - port: metrics-port
    path: /prometheus
EOF
)

# Verify Prometheus targets
kubectl port-forward svc/prometheus-kube-prometheus-stack-prometheus 9090:9090 -n monitoring
# Then open http://localhost:9090/targets in your browser

Prerequisites

To follow this guide effectively, you’ll need the following tools and knowledge:

• A running Kubernetes cluster: Any version 1.16+ should work, but newer versions are always recommended. You can use Minikube, Kind, or a cloud-managed cluster (EKS, GKE, AKS).
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• helm: The Kubernetes package manager, used for installing the Prometheus Operator. If you don’t have it, follow the Helm installation guide.
• Basic understanding of Kubernetes concepts: Familiarity with Pods, Deployments, Services, and Labels/Selectors is essential.
• Basic understanding of Prometheus: Knowledge of how Prometheus scrapes metrics and its data model will be beneficial.
• A metrics-emitting application: For demonstration purposes, we’ll use a simple Nginx application configured to expose metrics.

Step-by-Step Guide: Prometheus Operator, ServiceMonitor & PodMonitor

1. Install the Prometheus Operator

The first step is to get the Prometheus Operator up and running in your Kubernetes cluster. The easiest and most recommended way to do this is by using the kube-prometheus-stack Helm chart, which not only installs the Operator but also Prometheus, Alertmanager, Grafana, and default dashboards/rules. This provides a complete monitoring solution out-of-the-box.

We’ll create a dedicated namespace for our monitoring components to keep things organized. This practice is generally recommended for infrastructure-level services. Once the namespace is created, we’ll add the Prometheus Community Helm repository and then install the chart.

# Create a dedicated namespace for monitoring components
kubectl create namespace monitoring

# Add the Prometheus Community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Update your Helm repositories to fetch the latest charts
helm repo update

# Install the kube-prometheus-stack chart into the 'monitoring' namespace
# This will install Prometheus Operator, Prometheus, Alertmanager, and Grafana
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --wait

Verify Installation

After the installation, you should see several pods, services, and other resources created in the monitoring namespace. We can verify the Prometheus Operator pod is running, which is the core component responsible for managing ServiceMonitor and PodMonitor CRDs.

kubectl get pods -n monitoring -l app.kubernetes.io/name=kube-prometheus-stack-operator

Expected Output:

NAME                                                READY   STATUS    RESTARTS   AGE
prometheus-kube-prometheus-stack-operator-7...-xyz   1/1     Running   0          5m

You can also check for the presence of the ServiceMonitor and PodMonitor Custom Resource Definitions (CRDs) to confirm the Operator has successfully registered them.

kubectl get crd servicemonitors.monitoring.coreos.com podmonitors.monitoring.coreos.com

Expected Output:

NAME                                   CREATED AT
servicemonitors.monitoring.coreos.com  2023-10-27T10:00:00Z
podmonitors.monitoring.coreos.com      2023-10-27T10:00:00Z

2. Deploy a Sample Application with Metrics

To demonstrate ServiceMonitor and PodMonitor, we need an application that exposes Prometheus-compatible metrics. For this guide, we’ll deploy a simple Nginx server with the Nginx Prometheus Exporter sidecar. This setup is common for exposing infrastructure metrics.

We’ll create a Deployment for Nginx and its exporter, and then a Service to expose the Nginx port (80) and the exporter’s metrics port (9113). Crucially, we’ll add labels to both the Deployment/Pods and the Service, which the ServiceMonitor and PodMonitor will use for discovery.

# my-nginx-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx-app
  labels:
    app: my-nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-nginx-app
  template:
    metadata:
      labels:
        app: my-nginx-app
        component: webserver
      annotations:
        prometheus.io/scrape: "true" # Optional, but good practice for clarity
        prometheus.io/port: "9113"   # Optional, specify metrics port
        prometheus.io/path: "/metrics" # Optional, specify metrics path
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80
          name: http
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      - name: nginx-exporter
        image: nginx/nginx-prometheus-exporter:0.10.0
        args:
          - -nginx.scrape-uri=http://localhost:80/stub_status
        ports:
        - containerPort: 9113
          name: http-metrics
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config-cm
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config-cm
data:
  nginx.conf: |
    events {
        worker_connections 1024;
    }
    http {
        server {
            listen 80;
            location / {
                return 200 "Hello from Nginx!";
            }
            location /stub_status {
                stub_status on;
                access_log off;
                allow 127.0.0.1; # Allow exporter to access
                deny all;
            }
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-app
  labels:
    app: my-nginx-app
    metrics-target: "true" # Custom label for ServiceMonitor
spec:
  selector:
    app: my-nginx-app
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: http
  - name: http-metrics
    protocol: TCP
    port: 9113
    targetPort: http-metrics

Save the above content as my-nginx-app.yaml and apply it:

kubectl apply -f my-nginx-app.yaml

Verify Application Deployment

Ensure the pods and service are running correctly.

kubectl get pods -l app=my-nginx-app
kubectl get svc my-nginx-app

Expected Output:

NAME                             READY   STATUS    RESTARTS   AGE
my-nginx-app-7...-xyz            2/2     Running   0          2m
my-nginx-app-8...-abc            2/2     Running   0          2m

NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
my-nginx-app     ClusterIP   10.xx.xx.xx    <none>        80/TCP,9113/TCP              2m

You can also verify that the metrics endpoint is accessible by port-forwarding to one of the pods:

# Get a pod name
POD_NAME=$(kubectl get pods -l app=my-nginx-app -o jsonpath='{.items[0].metadata.name}')

# Port-forward to the metrics port (9113)
kubectl port-forward $POD_NAME 9113:9113 &

# Access metrics
curl http://localhost:9113/metrics

You should see Prometheus metrics output, confirming the exporter is working.

3. Configure ServiceMonitor for Service-based Metrics

A ServiceMonitor is used to define how a group of services should be monitored. It tells the Prometheus Operator which Services to look for (based on labels in the selector) and which ports/paths on those Services expose Prometheus metrics. This is the most common and recommended way to scrape metrics in Kubernetes, as it abstracts away individual pod IPs and handles scaling gracefully.

For our Nginx application, we will create a ServiceMonitor that targets the my-nginx-app Service. We’ll use the http-metrics port defined in our Service and the default /metrics path.

# my-nginx-app-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-nginx-app-sm
  labels:
    release: prometheus # This label is crucial for the Prometheus instance to discover this ServiceMonitor
spec:
  selector:
    matchLabels:
      app: my-nginx-app # Select the Service with this label
  endpoints:
  - port: http-metrics # Name of the port in the Service spec
    path: /metrics     # Path to scrape metrics from
    interval: 30s      # Scrape interval (optional)
  namespaceSelector:
    matchNames:
    - default # Only look for services in the 'default' namespace

A critical point here is the metadata.labels.release: prometheus. This label must match the spec.serviceMonitorSelector.matchLabels defined in your Prometheus CRD (which is automatically configured by the kube-prometheus-stack Helm chart). Without this matching label, your Prometheus instance will not discover the ServiceMonitor.

Apply the ServiceMonitor:

kubectl apply -f my-nginx-app-servicemonitor.yaml

Verify ServiceMonitor Configuration

The Prometheus Operator should now be aware of your ServiceMonitor. You can inspect the ServiceMonitor to ensure it’s correctly configured.

kubectl get servicemonitor my-nginx-app-sm
kubectl describe servicemonitor my-nginx-app-sm

Expected Output (snippet from describe):

...
Spec:
  Endpoints:
    Port:       http-metrics
    Path:       /metrics
    Interval:   30s
  Namespace Selector:
    Match Names:
      default
  Selector:
    Match Labels:
      app:  my-nginx-app
...

Now, let’s verify that Prometheus is actually scraping the targets. We need to access the Prometheus UI. The kube-prometheus-stack creates a Service for Prometheus. We can port-forward to it.

# Get the Prometheus service name (it usually starts with 'prometheus-kube-prometheus-stack-prometheus')
kubectl get svc -n monitoring | grep prometheus-kube-prometheus-stack-prometheus

# Port-forward to the Prometheus UI
kubectl port-forward svc/prometheus-kube-prometheus-stack-prometheus 9090:9090 -n monitoring &

Open your web browser and navigate to http://localhost:9090/targets. You should see entries for your my-nginx-app with a state of “UP”.

4. Configure PodMonitor for Pod-based Metrics

While ServiceMonitor is excellent for services, there are scenarios where you need to scrape metrics directly from individual pods. This might be necessary for:

• Headless services where you need to target specific pod IPs.
• StatefulSets where each pod might have unique metrics.
• When you need more granular control over which pods are scraped, potentially bypassing the Service abstraction.
• Applications that expose metrics on a port not exposed by their Service.

A PodMonitor works similarly to a ServiceMonitor but selects pods directly using labels. We’ll create a PodMonitor that targets our Nginx exporter pods. Remember that our Nginx Deployment pods have the label app: my-nginx-app and expose metrics on port http-metrics (9113) at path /metrics.

# my-nginx-app-podmonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-nginx-app-pm
  labels:
    release: prometheus # This label is crucial for Prometheus discovery
spec:
  selector:
    matchLabels:
      app: my-nginx-app # Select pods with this label
  podMetricsEndpoints:
  - port: http-metrics # Name of the port in the Pod spec (containerPort name)
    path: /metrics     # Path to scrape metrics from
    interval: 15s      # Scrape interval (optional)
  namespaceSelector:
    matchNames:
    - default # Only look for pods in the 'default' namespace

Note the use of podMetricsEndpoints instead of endpoints, and that the port refers to the name of the container port within the Pod’s spec, not the Service port. The release: prometheus label is again vital for discovery by the Prometheus instance.

Apply the PodMonitor:

kubectl apply -f my-nginx-app-podmonitor.yaml

Verify PodMonitor Configuration

Check the PodMonitor resource:

kubectl get podmonitor my-nginx-app-pm
kubectl describe podmonitor my-nginx-app-pm

Expected Output (snippet from describe):

...
Spec:
  Namespace Selector:
    Match Names:
      default
  Pod Metrics Endpoints:
    Port:       http-metrics
    Path:       /metrics
    Interval:   15s
  Selector:
    Match Labels:
      app:  my-nginx-app
...

Now, revisit the Prometheus UI at http://localhost:9090/targets (if you closed the port-forward, restart it). You should now see additional targets, one for each Nginx pod, discovered by the PodMonitor. You’ll likely see both the ServiceMonitor and PodMonitor targets active, each scraping the same metrics but from different discovery mechanisms.

5. Advanced Configuration (Relabeling, TLS, Authentication)

Prometheus Operator’s ServiceMonitor and PodMonitor CRDs offer powerful advanced configuration options to fine-tune metric scraping. These are crucial for handling complex scenarios like modifying metric labels, securing endpoints, or integrating with specific network configurations. For example, if your cluster uses Cilium WireGuard Encryption, you might need to ensure your scraping targets are correctly configured to communicate over the encrypted network.

Relabeling

Relabeling allows you to rewrite target labels before they are scraped, or even to drop targets based on label values. This is incredibly flexible for standardizing labels, adding environment-specific tags, or filtering targets.

Let’s add a relabeling rule to our ServiceMonitor to add a custom label environment: dev and drop any target that doesn’t have the label component: webserver (which our pods do).

# my-nginx-app-servicemonitor-relabel.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-nginx-app-sm-relabel
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: my-nginx-app
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
    relabelings: # Add relabeling rules here
    - sourceLabels: [__address__] # A dummy source label, since we just want to add a static label
      targetLabel: environment
      replacement: dev
    - sourceLabels: [__meta_kubernetes_pod_label_component] # Use a pod label as source
      regex: "webserver" # Match only if component label is "webserver"
      action: keep # Keep targets that match
  namespaceSelector:
    matchNames:
    - default

Apply this new ServiceMonitor:

kubectl apply -f my-nginx-app-servicemonitor-relabel.yaml

Verify in Prometheus UI at http://localhost:9090/targets. You should see the new environment="dev" label on the targets from my-nginx-app-sm-relabel. Also, if you were to remove the component: webserver label from your pods, this ServiceMonitor would stop scraping them.

TLS and Authentication

If your metrics endpoints require TLS or authentication (e.g., basic auth, bearer token), ServiceMonitor and PodMonitor can configure Prometheus to handle this. This is critical for securing your monitoring data, especially when dealing with sensitive application metrics.

Here’s an example snippet for basic authentication and TLS. Note that you would need to create Kubernetes Secrets containing the credentials and CA certs first.

# Example snippet for TLS and Basic Auth
# (This won't be applied as we don't have secrets for it)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-secure-app-sm
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: my-secure-app
  endpoints:
  - port: https-metrics
    path: /metrics
    scheme: https # Specify HTTPS
    tlsConfig:
      ca:
        secret:
          name: my-ca-secret
          key: ca.crt
      cert:
        secret:
          name: my-client-cert-secret
          key: client.crt
      keySecret:
        secret:
          name: my-client-cert-secret
          key: client.key
      insecureSkipVerify: false # Set to true only for testing
    basicAuth: # Basic Authentication
      username:
        name: my-auth-secret
        key: username
      password:
        name: my-auth-secret
        key: password
    bearerTokenSecret: # Bearer Token Authentication (alternative to basicAuth)
      name: my-token-secret
      key: token
  namespaceSelector:
    matchNames:
    - default

This snippet demonstrates how you can reference Kubernetes Secrets to provide TLS certificates, client certificates, and authentication credentials. For more details on these advanced configurations, refer to the Prometheus Operator ServiceMonitor API documentation.

Production Considerations

Deploying Prometheus Operator with ServiceMonitor and PodMonitor in a production environment requires careful planning and adherence to best practices. Observability is a critical pillar for any robust system, and ensuring your monitoring stack is reliable, secure, and scalable is paramount.

1. Namespace Strategy:

   Always deploy the Prometheus Operator and its components (Prometheus, Alertmanager, Grafana) into a dedicated monitoring namespace. This isolates your monitoring infrastructure from application workloads and simplifies management. When defining ServiceMonitor and PodMonitor resources, use namespaceSelector to explicitly define which namespaces they should discover targets from. Avoid scraping all namespaces if not necessary, as this can lead to performance issues and security concerns.

2. Resource Management:

   Prometheus can be resource-intensive, especially in large clusters with many targets and high churn rates. Configure appropriate CPU and memory requests and limits for your Prometheus pods. Monitor their resource usage and scale them horizontally if needed. Consider using Karpenter for Cost Optimization to efficiently provision nodes for your monitoring workloads, ensuring resources are available when needed without over-provisioning.

3. Relabeling for Cost & Performance:

   Excessive labels can lead to high cardinality, increasing Prometheus’s memory and disk usage significantly. Use relabeling rules in your ServiceMonitor and PodMonitor to drop unnecessary labels or rename them for consistency. For instance, drop ephemeral pod IP labels unless absolutely required. This is a crucial step for optimizing Prometheus performance and storage costs.

4. Security Contexts and Network Policies:

   Prometheus Operator components should run with appropriate Kubernetes Security Contexts, restricting privileges to the minimum necessary. Implement Kubernetes Network Policies to control ingress and egress traffic for your Prometheus pods, ensuring they can only scrape authorized targets and communicate with Alertmanager/Grafana, while preventing unauthorized access or data exfiltration. This is a fundamental security hardening step.

5. High Availability and Storage:

   For production, run Prometheus in a highly available configuration (e.g., two replicas managed by the Operator). For long-term storage, integrate Prometheus with a remote storage solution like Thanos, Cortex, or Mimir. The kube-prometheus-stack chart often includes options for this. Ensure your persistent storage for Prometheus’s local TSDB has sufficient IOPS and capacity.

6. Alerting and Dashboards:

   Leverage Alertmanager (installed by the Operator) to send notifications based on Prometheus alerts. Configure robust alerting rules (PrometheusRule CRD) and build comprehensive Grafana dashboards to visualize your metrics. This ensures you’re proactively notified of issues and have visibility into your application’s health.

7. Monitoring the Monitoring System:

   Don’t forget to monitor Prometheus itself! The kube-prometheus-stack typically includes pre-built ServiceMonitor resources for Prometheus, Alertmanager, and the Operator. Ensure these are active and that you have alerts for your monitoring system’s health and performance.

8. Version Management:

   Keep your Prometheus Operator, Prometheus, and related components updated. Regularly review release notes for new features, bug fixes, and security patches. Use Helm for easy upgrades.

9. Scrape Intervals and Timeouts:

   Adjust interval and scrapeTimeout in your ServiceMonitor/PodMonitor based on the criticality and volatility of the metrics. More critical, fast-changing metrics might need shorter intervals (e.g., 5s), while less critical ones can use longer intervals (e.g., 60s) to reduce load on Prometheus and targets. Be mindful of scrape timeouts for slow endpoints.

Troubleshooting

Even with the best planning, issues can arise. Here are some common problems you might encounter when working with Prometheus Operator, ServiceMonitor, and PodMonitor, along with their solutions.

1. Prometheus UI shows “no targets found” or targets are “DOWN”

   Problem: You’ve applied your ServiceMonitor or PodMonitor, but Prometheus isn’t scraping anything, or the targets are showing as down in the Prometheus UI (/targets).

   Solution:

   • Check release label: Ensure your ServiceMonitor/PodMonitor has a metadata.labels.release: prometheus (or whatever label your Prometheus CRD’s serviceMonitorSelector/podMonitorSelector is configured to use). This is the most common reason for targets not showing up.
   • Check namespaceSelector: Verify that the namespaceSelector in your ServiceMonitor/PodMonitor correctly includes the namespace where your application’s Service/Pods reside.
   • Check selector: Double-check that the spec.selector.matchLabels in your monitor resource exactly matches the labels on your target Service or Pods. Labels are case-sensitive.
   • Check port name: For ServiceMonitor, ensure spec.endpoints.port matches the name of the port in your Kubernetes Service spec. For PodMonitor, it must match the name of the containerPort in the Pod spec.
   • Check metrics path: Confirm that spec.endpoints.path (or spec.podMetricsEndpoints.path) is the correct path where your application exposes metrics (e.g., /metrics).
   • Prometheus Operator logs: Check the logs of the Prometheus Operator pod (kubectl logs -n monitoring prometheus-kube-prometheus-stack-operator-...) for any errors related to CRD processing.
   • Prometheus logs: Check the logs of the Prometheus pods (kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-...). They might indicate issues with scraping, like connection refused or HTTP errors.

2. Targets are UP, but no metrics are being collected or expected metrics are missing.

   Problem: Prometheus shows the target as “UP”, but queries for specific metrics return no data, or only a subset of expected metrics are present.

   Solution:

   • Verify metrics endpoint manually: Port-forward to your application pod/service and use curl to access the metrics endpoint directly (e.g., curl http://localhost:9113/metrics). Ensure the metrics are actually being exposed by your application and are in the correct Prometheus text format.
   • Check relabelings: If you have relabelings configured, they might be inadvertently dropping or renaming metrics/labels. Temporarily remove them to see if metrics reappear.
   • Scrape interval/timeout: If the application is slow to respond, increase the scrapeTimeout in the ServiceMonitor/PodMonitor. If metrics are very sparse, check the interval.
   • Application logs: Check your application’s logs for any errors related to exposing metrics.

3. Prometheus Operator pod is crashing or in a degraded state.

   Problem: The Prometheus Operator pod is restarting frequently or not coming up at all.

   Solution:

   • Check logs: Get the logs of the Prometheus Operator pod (kubectl logs -n monitoring prometheus-kube-prometheus-stack-operator-...). Look for error messages, especially related to RBAC permissions or CRD issues.
   • Resource limits: If the pod is getting OOMKilled, increase its memory limits in the Helm chart values.
   • Kubernetes API access: Ensure the service account used by the Operator has sufficient RBAC permissions to list/watch/get ServiceMonitor, PodMonitor, Services, Endpoints, and Pods across the relevant namespaces.

4. Prometheus pods are not coming up or are in a pending state.

   Problem: The Prometheus server pods are stuck or failing to start.

   Solution:

   • Check PVCs: Prometheus requires persistent storage. Ensure PersistentVolumeClaims (PVCs) are bound to PersistentVolumes (PVs). If using dynamic provisioning, check your StorageClass and provisioner logs.
   • Resource limits: Prometheus can be memory-hungry. Ensure sufficient memory and CPU resources are allocated.
   •