Master Prometheus ServiceMonitor & PodMonitor

Prerequisites

Before we embark on this journey, ensure you have the following:

1. Kubernetes Cluster: A running Kubernetes cluster (v1.16+ recommended). Minikube, kind, or a cloud-managed cluster will work.
2. kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official kubectl installation guide if you need assistance.
3. Helm (Optional but Recommended): For easy installation of the Prometheus Operator. Install it from the Helm documentation.
4. Basic Kubernetes Knowledge: Familiarity with Pods, Deployments, Services, and Labels.
5. Basic Prometheus Knowledge: Understanding of what Prometheus is and how it collects metrics.

Step-by-Step Guide: Deploying Prometheus Operator and Monitoring Applications

Step 1: Install the Prometheus Operator

The Prometheus Operator manages the entire Prometheus stack, including Prometheus servers, Alertmanagers, and the custom resources we'll be using. The easiest way to install it is via the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, Alertmanager, and the Operator itself. This provides a complete, production-ready monitoring solution.

First, add the Prometheus Community Helm repository and update it. Then, install the kube-prometheus-stack chart into its own namespace, typically named prometheus or monitoring. This ensures all monitoring components are isolated.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --create-namespace --namespace prometheus

This command will deploy a comprehensive monitoring stack. It might take a few minutes for all components to become ready. You can monitor the progress by checking the pods in the prometheus namespace.

Verify Step 1

Check if the Prometheus Operator and other components are running:

kubectl get pods -n prometheus

Expected Output (truncated):

NAME                                                     READY   STATUS    RESTARTS   AGE
prometheus-kube-prometheus-operator-7c87c7b744-8d99c     1/1     Running   0          2m
prometheus-kube-prometheus-grafana-7945d867c4-h9xdf      1/1     Running   0          2m
prometheus-kube-prometheus-prometheus-0                  2/2     Running   0          2m
prometheus-kube-prometheus-alertmanager-0                2/2     Running   0          2m
...

Step 2: Deploy a Sample Application with Prometheus Metrics

To demonstrate ServiceMonitor and PodMonitor, we need an application that exposes Prometheus-compatible metrics. We'll use a simple Go application that exposes a /metrics endpoint. This application will be deployed as a Kubernetes Deployment and exposed via a Service.

First, create a Deployment and a Service for our sample application. Note the labels (app: demo-app) and the named port (http-metrics) in the Service, as these will be crucial for our ServiceMonitor.

# demo-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
      - name: demo-app
        image: prom/demo-app:latest # A simple Go app that exposes /metrics
        ports:
        - name: http-metrics
          containerPort: 8080
        env:
        - name: METRICS_PATH
          value: "/metrics"
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  selector:
    app: demo-app
  ports:
  - name: http-metrics # Important: named port for ServiceMonitor
    protocol: TCP
    port: 80
    targetPort: http-metrics

kubectl apply -f demo-app.yaml

This creates a Deployment running two replicas of our demo application and a ClusterIP Service to expose it. The service port 80 maps to the container port named http-metrics (which is 8080 inside the container).

Verify Step 2

Check if the application Pods and Service are running:

kubectl get deployment,service demo-app

Expected Output:

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-app       2/2     2            2           2m

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/demo-app   ClusterIP   10.101.10.10     <none>        80/TCP         2m

You can also port-forward to the service and check the metrics endpoint:

kubectl port-forward svc/demo-app 8080:80 &
curl http://localhost:8080/metrics

You should see Prometheus metrics output.

Step 3: Create a ServiceMonitor Resource

Now, let's create a ServiceMonitor custom resource. This resource tells the Prometheus Operator which Kubernetes Services to watch for metrics endpoints. The Operator then automatically configures the Prometheus server to scrape these endpoints.

The key fields in a ServiceMonitor are:

• metadata.labels.release: prometheus: This label is critical. The Prometheus instance deployed by the kube-prometheus-stack chart is configured to discover ServiceMonitor resources that have the label release: prometheus. Make sure this matches your Helm release name.
• spec.selector.matchLabels: This defines which Services the ServiceMonitor should target. We use app: demo-app to match our sample application's Service.
• spec.endpoints: This describes how to access the metrics.
  • port: This must match the name of the port defined in your Kubernetes Service (e.g., http-metrics).
  • path: The HTTP path where metrics are exposed (e.g., /metrics).

# demo-app-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-app-monitor
  labels:
    release: prometheus # Crucial: Must match the Prometheus instance's selector
spec:
  selector:
    matchLabels:
      app: demo-app # Targets Services with this label
  endpoints:
  - port: http-metrics # Name of the port in the Service spec
    path: /metrics
    interval: 15s # How often to scrape
    scrapeTimeout: 10s # Timeout for the scrape request

kubectl apply -f demo-app-servicemonitor.yaml

Once applied, the Prometheus Operator will detect this new ServiceMonitor, find the matching demo-app Service, and dynamically update the Prometheus configuration to scrape its endpoints. This declarative approach greatly simplifies managing scrape targets.

Verify Step 3

Access the Prometheus UI to confirm the target is being scraped. You'll need to port-forward to the Prometheus Service:

kubectl -n prometheus port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 &

Open your browser to http://localhost:9090/targets. You should see an entry for demo-app-monitor with a state of "UP".

If you don't see it, check the Prometheus server logs for any errors related to ServiceMonitor discovery:

kubectl logs -f -n prometheus prometheus-kube-prometheus-prometheus-0 -c prometheus

Step 4: Create a PodMonitor Resource (Alternative/Supplemental)

While ServiceMonitor is excellent for services, sometimes you need to scrape metrics directly from Pods without going through a Service. This is common for:

• DaemonSets where each Pod's metrics are unique to its node.
• StatefulSets where Pods might have unique identities and metrics.
• Applications where a Service might aggregate multiple Pods, but you need to inspect individual Pod metrics for specific debugging.
• Scraping sidecar containers within a Pod.

PodMonitor works similarly to ServiceMonitor but selects Pods directly. Let's create a PodMonitor for our demo-app as an example, showcasing how it targets Pods directly.

Key fields in a PodMonitor:

• metadata.labels.release: prometheus: Same as ServiceMonitor, matches the Prometheus instance.
• spec.selector.matchLabels: Targets Pods with these labels. We'll use app: demo-app.
• spec.podMetricsEndpoints: Describes how to access metrics from the Pods.
  • port: This must match the name of the port defined in the Pod's container spec (e.g., http-metrics).
  • path: The HTTP path where metrics are exposed (e.g., /metrics).

# demo-app-podmonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: demo-app-pod-monitor
  labels:
    release: prometheus # Crucial: Must match the Prometheus instance's selector
spec:
  selector:
    matchLabels:
      app: demo-app # Targets Pods with this label
  podMetricsEndpoints:
  - port: http-metrics # Name of the port in the Pod's container spec
    path: /metrics
    interval: 15s
    scrapeTimeout: 10s
    # targetPort: 8080 # Can specify numeric targetPort if port name not available

kubectl apply -f demo-app-podmonitor.yaml

The Prometheus Operator will now discover the individual demo-app Pods and configure Prometheus to scrape each of them directly, in addition to the Service-based scraping.

Verify Step 4

Refresh your browser at http://localhost:9090/targets. You should now see additional targets, one for each demo-app Pod, under the demo-app-pod-monitor job. The state should be "UP" for all of them.

This demonstrates the flexibility of PodMonitor for finer-grained scraping.

Step 5: Advanced Configuration (Relabeling and Metric Relabeling)

Prometheus offers powerful relabeling capabilities to transform labels before or after scraping. Both ServiceMonitor and PodMonitor support these. Relabeling is crucial for:

• Adding or dropping labels based on Kubernetes metadata.
• Renaming labels for consistency.
• Filtering targets or metrics.

Let's modify our ServiceMonitor to add a custom label environment: dev to all scraped metrics and drop any metrics that start with go_ (e.g., Go runtime metrics).

# demo-app-servicemonitor-advanced.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-app-monitor-advanced
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: demo-app
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 15s
    scrapeTimeout: 10s
    # Relabeling: Applied before scraping, affects target discovery and instance labels
    relabelings:
    - sourceLabels: [__meta_kubernetes_service_name] # Example: uses service name
      targetLabel: service_name
      regex: (.*)
      replacement: $1
    - sourceLabels: [__address__] # Add a static label
      targetLabel: environment
      replacement: dev
    # Metric Relabeling: Applied after scraping, affects individual metrics
    metricRelabelings:
    - sourceLabels: [__name__] # Target the metric name
      regex: go_.* # Drop all metrics starting with 'go_'
      action: drop

kubectl apply -f demo-app-servicemonitor-advanced.yaml

In this example:

• relabelings operate on the target's metadata before the scrape. We're adding a service_name label derived from the Kubernetes service name and a static environment: dev label.
• metricRelabelings operate on the scraped metrics themselves. We're dropping all metrics whose name starts with go_.

For more detailed information on relabeling, refer to the Prometheus documentation on relabeling.

Verify Step 5

Check the Prometheus UI at http://localhost:9090/targets. You should see the new demo-app-monitor-advanced target. Click on "Show more" to see the labels applied. You should see environment="dev" and service_name="demo-app".

Then, navigate to the Graph page and query for demo_app_requests_total (a metric from our demo app). You should see the environment="dev" label attached to the metric. Try querying for go_goroutines; you should find no results if the metric relabeling effectively dropped them.

Production Considerations

Deploying Prometheus Operator with ServiceMonitor and PodMonitor in production requires careful planning:

• Resource Management: Prometheus can be resource-intensive. Ensure your Prometheus server deployments have adequate CPU, memory, and storage. Use persistent volumes for Prometheus data. Consider Karpenter for cost optimization on your cluster nodes if resource demands fluctuate.
• High Availability: For critical environments, deploy highly available Prometheus instances. The kube-prometheus-stack chart often includes this configuration, but verify it.
• Remote Write/Storage: For long-term storage and global views, integrate Prometheus with remote storage solutions like Thanos, Cortex, or Mimir. Prometheus itself is designed for short-to-medium term storage.
• Security:
  • Network Policies: Restrict access to Prometheus metrics endpoints using Kubernetes Network Policies. Only allow scraping from the Prometheus Pods.
  • RBAC: Ensure the Prometheus Operator's ServiceAccount has the necessary RBAC permissions to watch Services, Pods, and Endpoints across the namespaces you intend to monitor.
  • TLS/Authentication: For sensitive metrics, configure TLS and authentication for your metrics endpoints. Both ServiceMonitor and PodMonitor support TLS configurations.
  • If you're interested in advanced security practices, explore topics like Securing Container Supply Chains with Sigstore and Kyverno.
• Scalability: As your cluster grows, the number of scrape targets can become very large. Consider sharding Prometheus instances or using hierarchical Prometheus setups.
• Alerting: Configure Alertmanager (also part of the kube-prometheus-stack) with appropriate alerting rules and notification channels (Slack, PagerDuty, etc.).
• Observability Best Practices: Beyond just metrics, consider integrating logs (Loki) and traces (Tempo) for a complete observability picture. Tools like eBPF Observability with Hubble can provide deep network insights.
• Service Mesh Integration: If you're using a service mesh like Istio, you might need to adapt your monitoring strategy. For instance, Istio Ambient Mesh changes how sidecars work, potentially impacting direct Pod monitoring.
• Gateway API: If you're using the Kubernetes Gateway API, ensure your metrics endpoints are correctly exposed and discoverable through your chosen Gateway implementation.

Troubleshooting

1. Issue: Prometheus UI shows targets "DOWN" or "PENDING"

Solution:

1. Check Prometheus Operator Logs:

   kubectl logs -n prometheus -l app.kubernetes.io/name=prometheus-operator
           

   Look for errors related to reconciling ServiceMonitor or PodMonitor resources.

2. Check Prometheus Server Logs:

   kubectl logs -n prometheus prometheus-kube-prometheus-prometheus-0 -c prometheus
           

   Search for messages like "Error scraping target" or "no such host". This indicates Prometheus tried to scrape but failed.

3. Verify ServiceMonitor/PodMonitor Labels: Ensure the metadata.labels.release: prometheus (or whatever your Prometheus instance expects) is correct on your Monitor resource. Also, verify spec.selector.matchLabels correctly identifies your application's Service or Pods.
4. Verify Port Name: Double-check that the port field in your ServiceMonitor or PodMonitor exactly matches the name of the port in your Kubernetes Service or Pod spec. Not the number, but the name (e.g., http-metrics).
5. Network Connectivity: Ensure Prometheus Pods can reach your application Pods/Services. Check Network Policies if any are in place. For example, if you're using Cilium WireGuard Encryption, ensure policies allow traffic between namespaces.
6. Application Exposing Metrics: Confirm your application is actually exposing metrics at the specified path and port. Port-forward to your application Pod or Service and curl the endpoint.

2. Issue: Metrics are scraped, but custom labels are missing or incorrect

Solution:

1. Check Relabeling Configuration: Review the relabelings section in your ServiceMonitor or PodMonitor.
2. Understand Relabeling Phases: Remember that relabelings happen before the scrape, affecting target discovery and instance labels. metricRelabelings happen after the scrape, affecting individual metrics. Ensure you're using the correct one for your desired outcome.
3. Source Labels: Verify the sourceLabels you're using. For Kubernetes metadata, common source labels include __meta_kubernetes_service_name, __meta_kubernetes_pod_name, __meta_kubernetes_pod_label_, etc. See Prometheus Kubernetes SD config for a full list.

3. Issue: Prometheus is scraping too many metrics or specific metrics are missing

Solution:

1. Metric Relabeling: Use metricRelabelings to filter metrics.
   • To drop specific metrics: action: drop with regex matching metric names.
   • To keep specific metrics: action: keep with regex matching metric names.
2. Scrape Interval/Timeout: Adjust interval and scrapeTimeout in your Monitor resource. A very short interval for too many metrics can overload Prometheus.
3. Application-level Filtering: If possible, configure your application to only expose the necessary metrics to reduce scrape load.

4. Issue: Prometheus Operator not creating/updating Prometheus config

Solution:

1. RBAC Permissions: Ensure the Prometheus Operator's ServiceAccount has sufficient RBAC permissions to get, watch, and list services, endpoints, pods, and servicemonitors/podmonitors in the relevant namespaces.
2. Prometheus CRD Version: Confirm the Prometheus Operator CRDs are installed and up-to-date. Sometimes, after an upgrade, CRDs might be outdated.
3. Target Namespace: By default, the Prometheus instance will only discover Monitor resources in its own namespace. If you want to monitor applications in other namespaces, you need to configure serviceMonitorNamespaceSelector and podMonitorNamespaceSelector in your Prometheus custom resource or the Helm chart values. For example:

   # values.yaml for kube-prometheus-stack
   prometheus:
     prometheusSpec:
       serviceMonitorSelectorNilUsesHelmValues: false # Important for global monitoring
       podMonitorSelectorNilUsesHelmValues: false # Important for global monitoring
       serviceMonitorSelector: {} # Selects all ServiceMonitors
       podMonitorSelector: {} # Selects all PodMonitors
       # Or to select specific namespaces:
       # serviceMonitorNamespaceSelector:
       #   matchLabels:
       #     monitoring: "true"
       # podMonitorNamespaceSelector:
       #   matchLabels:
       #     monitoring: "true"
           

   Then, label the namespaces you want to monitor accordingly (e.g., kubectl label namespace <your-app-namespace> monitoring=true).

5. Issue: High CPU/Memory usage by Prometheus

Solution:

1. Reduce Scrape Interval: Increase the interval in your ServiceMonitor/PodMonitor to scrape less frequently.
2. Filter Metrics: Aggressively use metricRelabelings to drop unneeded metrics.
3. Sharding: For very large clusters, consider running multiple Prometheus instances, each responsible for a subset of targets.
4. Resource Limits: Ensure your Prometheus Pods have appropriate resource limits and requests.
5. Storage: If using local storage, ensure sufficient IOPS and throughput. Consider remote storage solutions for long-term data.
6. GPU Scheduling: While Prometheus itself doesn't typically use GPUs, if your monitored applications involve heavy ML workloads, ensure your node autoscaling and scheduling (e.g., for GPUs) are optimized to prevent resource contention.

FAQ Section

Q1: What's the main difference between ServiceMonitor and PodMonitor?

A1: ServiceMonitor discovers metrics endpoints exposed by Kubernetes Services, using the Service's labels to select targets and the named ports within the Service to find the endpoint. It's ideal for applications exposed via a stable Service. PodMonitor, on the other hand, discovers metrics endpoints directly from individual Pods, using Pod labels for selection and named ports in the Pod's container spec. It's useful for cases where Pods have unique metrics or when a Service abstraction isn't desired (e.g., StatefulSets, DaemonSets, or sidecar containers).

Q2: How do I ensure my Prometheus instance picks up my ServiceMonitor/PodMonitor?

A2: The most common reason for a Monitor not being picked up is incorrect labels. Ensure your ServiceMonitor or PodMonitor resource has a label that matches the serviceMonitorSelector or podMonitorSelector defined in your Prometheus custom resource (which is usually configured via the Helm chart). By default, the kube-prometheus-stack looks for release: prometheus. Also, verify the Monitor is in the same namespace as the target Service/Pods, or that the Prometheus instance is configured to monitor multiple namespaces using serviceMonitorNamespaceSelector/podMonitorNamespaceSelector.

Q3: Can I use both ServiceMonitor and PodMonitor for the same application?

A3: Yes, you can. Prometheus will treat them as separate scrape jobs. This can be useful for specific scenarios, but generally, you'd pick one that best suits your discovery needs. For example, you might use ServiceMonitor for primary application metrics and PodMonitor for a specific debug endpoint on individual Pods or a sidecar container within the same Pod.

Q4: My application exposes metrics on a non-standard path or port. How do I configure that?

A4: In both ServiceMonitor and PodMonitor, you can specify the path and port within the endpoints (for ServiceMonitor) or podMetricsEndpoints (for PodMonitor) section. The port field should match the name of the port defined in your Kubernetes Service or Pod spec. The path field should be the HTTP path where your application exposes its metrics (e.g., /my-custom-metrics).

Q5: How do I monitor applications across different namespaces?

A5: By default, the Prometheus Operator's Prometheus instance only discovers ServiceMonitor and PodMonitor resources in its own namespace. To monitor across namespaces, you need to configure the Prometheus custom resource (or the Helm chart values) with serviceMonitorNamespaceSelector and podMonitorNamespaceSelector. You can set them to {} to select all namespaces, or use specific label selectors to target a subset of namespaces. Remember to label your application namespaces accordingly if you use specific selectors.

Cleanup Commands

To remove all resources created during this tutorial:

# Delete sample application and monitors
kubectl delete -f demo-app-servicemonitor.yaml
kubectl delete -f demo-app-podmonitor.yaml
kubectl delete -f demo-app-servicemonitor-advanced.yaml
kubectl delete -f demo-app.yaml

# Uninstall Prometheus Operator and stack
helm uninstall prometheus --namespace prometheus
kubectl delete namespace prometheus

# Kill port-forward processes
killall kubectl

Next Steps / Further Reading

• Explore Prometheus Official Documentation for deeper insights into Prometheus.
• Learn about Prometheus Operator GitHub Repository for advanced configurations and features.
• Dive into Grafana Documentation to build powerful dashboards with your collected metrics.
• Understand Kubernetes Labels and Selectors, which are fundamental to how ServiceMonitor and PodMonitor work.
• Discover how to set up alerting with Alertmanager, which is included in the kube-prometheus-stack.
• For advanced networking and observability, consider integrating with tools like eBPF Observability: Building Custom Metrics with Hubble.
• If you're managing complex traffic routing, understanding the Kubernetes Gateway API can provide context on how your metrics endpoints are exposed externally.

Conclusion

The Prometheus Operator, with its ServiceMonitor and PodMonitor custom resources, fundamentally transforms how monitoring is implemented in Kubernetes. By leveraging these declarative APIs, you can automate the discovery and scraping of metrics from your applications, eliminating manual configuration and reducing operational overhead. This not only simplifies the monitoring setup but also ensures that your monitoring infrastructure