Scale Pods with Kubernetes Custom Metrics HPA

Prerequisites

Before diving into custom metrics with HPA, ensure you have the following:

• A running Kubernetes Cluster: Any version 1.16+ will work, but newer versions offer enhanced HPA API features (autoscaling/v2). You can use Minikube, Kind, or a cloud provider's managed Kubernetes service (EKS, GKE, AKS).
• kubectl configured: Your kubectl command-line tool should be authenticated and connected to your Kubernetes cluster.
• Helm (recommended): For easier deployment of Prometheus and Prometheus Adapter. You can download Helm from its official installation guide.
• Basic understanding of Kubernetes Deployments and Services: You should be familiar with creating and managing these core Kubernetes resources.
• Basic understanding of Prometheus: Knowledge of how Prometheus scrapes metrics and its query language (PromQL) will be beneficial.
• An application that exposes metrics: For this guide, we'll use a simple Nginx deployment configured to export custom metrics, which we'll simulate. In a real-world scenario, your application would expose these metrics itself (e.g., via a /metrics endpoint).

Step-by-Step Guide: HPA with Custom Metrics

This guide will walk you through setting up a simple Nginx application, deploying Prometheus and a Prometheus Adapter, and finally configuring an HPA to scale based on a custom metric representing HTTP requests per second.

Step 1: Deploy a Sample Application

First, let's deploy a basic Nginx application. This application will serve as our scaling target. For demonstration purposes, we'll later simulate custom metrics for it.

We'll create a Deployment and a Service for Nginx. The Deployment defines the desired state for our pods, and the Service provides a stable endpoint to access them.

# my-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "80" # Assuming Nginx metrics (if enabled) are on port 80
    spec:
      containers:
      - name: my-app
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Apply this manifest to your cluster:

kubectl apply -f my-app.yaml

Verify Step 1

Check if the deployment and service are running:

kubectl get deployment my-app

Expected Output:

NAME     READY   UP-TO-DATE   AVAILABLE   AGE
my-app   1/1     1            1           XXs

kubectl get service my-app-service

Expected Output:

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
my-app-service   ClusterIP   XXX.XXX.XXX.XXX           80/TCP    XXs

Step 2: Deploy Prometheus

To collect custom metrics, we need a metrics collection system. Prometheus is the de facto standard for this in the Kubernetes ecosystem. We'll deploy it using Helm for simplicity.

Prometheus will scrape metrics from our application pods. The annotations prometheus.io/scrape: "true" and prometheus.io/port: "80" in our my-app deployment template tell Prometheus how to discover and scrape metrics from our Nginx pods. For more advanced configurations, you might explore custom Prometheus scrape configurations.

# Add Prometheus Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install Prometheus
helm install prometheus prometheus-community/prometheus \
  --set alertmanager.enabled=false \
  --set pushgateway.enabled=false \
  --set server.persistentVolume.enabled=false \
  --set server.service.type=ClusterIP

Verify Step 2

Check if Prometheus pods are running:

kubectl get pods -l app=prometheus -n default

Expected Output:

NAME                                           READY   STATUS    RESTARTS   AGE
prometheus-kube-prometheus-stack-prometheus-0  1/1     Running   0          XXs

You can also port-forward to the Prometheus UI to confirm it's scraping your my-app pods (though this is optional for the HPA to work):

kubectl port-forward svc/prometheus-kube-prometheus-stack-prometheus 9090:9090

Then open http://localhost:9090 in your browser. You should see your my-app pods listed as targets under "Status" -> "Targets".

Step 3: Deploy Prometheus Adapter

The Horizontal Pod Autoscaler doesn't directly query Prometheus. Instead, it relies on the Kubernetes Custom Metrics API. The Prometheus Adapter acts as a bridge, translating Prometheus queries into responses consumable by the Custom Metrics API.

The adapter needs to know where to find Prometheus and how to translate specific custom metrics into a format the HPA can understand. This is configured via its rules. For example, we want to expose a metric like http_requests_total_per_second. The adapter will query Prometheus with a PromQL expression to get this value.

# prometheus-adapter-values.yaml
# This configuration tells the adapter to expose a custom metric
# named 'http_requests_total_per_second' based on a Prometheus query.
rules:
  - seriesQuery: '{__name__=~"nginx_http_requests_total"}' # Adjust this to your actual metric name
    resources:
      overrides:
        namespace: {resource: "namespace"}
        deployment: {resource: "deployment"}
    name:
      matches: "nginx_http_requests_total"
      as: "http_requests_total_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{namespace="<<.Namespace>>", deployment="<<.Deployment>>"}[1m])) by (namespace, deployment)'
    # The 'metricsQuery' calculates the rate of 'nginx_http_requests_total' over 1 minute,
    # grouped by namespace and deployment. This simulates a "requests per second" metric.

# Ensure Prometheus URL is correct
prometheus:
  url: http://prometheus-kube-kube-prometheus-stack-prometheus.default.svc.cluster.local
  port: 9090

Note: The seriesQuery and metricsQuery in the YAML above are placeholders. In a real-world scenario, you'd integrate your application to expose a specific metric (e.g., using a Nginx Prometheus Exporter or by instrumenting your application code). For this tutorial, we'll assume a metric like nginx_http_requests_total exists and simulate its increase.

Now, install the Prometheus Adapter using Helm and the custom values file:

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  -f prometheus-adapter-values.yaml \
  --set prometheus.url=http://prometheus-kube-prometheus-stack-prometheus.default.svc.cluster.local

Verify Step 3

Check if the Prometheus Adapter pod is running:

kubectl get pods -l app=prometheus-adapter -n default

Expected Output:

NAME                                 READY   STATUS    RESTARTS   AGE
prometheus-adapter-XXXXX-XXXXX       1/1     Running   0          XXs

You can also verify that the custom metrics API is available:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

Expected Output (snippet, might be long):

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "pods/http_requests_total_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "deployments/http_requests_total_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    ...
  ]
}

This output confirms that the Prometheus Adapter has registered custom metrics like http_requests_total_per_second for various resource types (pods, deployments, etc.).

Step 4: Simulate Custom Metrics

Since our Nginx application doesn't natively expose a http_requests_total counter, we'll simulate it by manually pushing a metric to Prometheus. In a real-world scenario, your application or an exporter would expose this metric directly.

We'll use a simple curl command to push a metric to Prometheus's Pushgateway, which Prometheus will then scrape. While Pushgateway is not ideal for transient metrics, it serves our simulation purpose well. For production, applications should expose metrics via an HTTP endpoint. For more on eBPF Observability with Hubble, you can see how metrics can be collected in a more distributed way.

First, ensure Pushgateway is enabled and accessible. If you didn't enable it during Prometheus installation, you can install it separately:

helm install prometheus-pushgateway prometheus-community/prometheus-pushgateway \
  --set service.type=ClusterIP

Find the Pushgateway service IP:

kubectl get svc prometheus-pushgateway

Expected Output:

NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
prometheus-pushgateway   ClusterIP   XXX.XXX.XXX.XXX           9091/TCP   XXm

Let's assume the Cluster IP is 10.43.201.123. Now, push a metric. We'll simulate increasing requests over time.

# Initial metric value
curl -X PUT -H "Content-Type: text/plain" --data-binary 'nginx_http_requests_total{instance="my-app-0",job="my-app",namespace="default",deployment="my-app"} 100' http://prometheus-pushgateway.default.svc.cluster.local:9091/metrics/job/my-app/instance/my-app-0

Run this command multiple times, increasing the value each time, to simulate traffic:

# After a few seconds, simulate more requests
curl -X PUT -H "Content-Type: text/plain" --data-binary 'nginx_http_requests_total{instance="my-app-0",job="my-app",namespace="default",deployment="my-app"} 200' http://prometheus-pushgateway.default.svc.cluster.local:9091/metrics/job/my-app/instance/my-app-0

# And again
curl -X PUT -H "Content-Type: text/plain" --data-binary 'nginx_http_requests_total{instance="my-app-0",job="my-app",namespace="default",deployment="my-app"} 350' http://prometheus-pushgateway.default.svc.cluster.local:9091/metrics/job/my-app/instance/my-app-0

Verify Step 4

After pushing metrics, you can verify they are visible in Prometheus. Port-forward to Prometheus (if not already) and check the graph for nginx_http_requests_total.

kubectl port-forward svc/prometheus-kube-prometheus-stack-prometheus 9090:9090

Go to http://localhost:9090/graph and enter nginx_http_requests_total in the expression bar. You should see the values increasing. More importantly, check the custom metric exposed by the adapter:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/deployments/my-app/http_requests_total_per_second" | jq .

Expected Output (value will vary):

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/deployments/my-app/http_requests_total_per_second"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Deployment",
        "namespace": "default",
        "name": "my-app",
        "apiVersion": "apps/v1"
      },
      "metric": {
        "name": "http_requests_total_per_second",
        "selector": null
      },
      "timestamp": "2023-10-27T10:00:00Z",
      "value": "1.5" # This value will reflect the rate of change of your pushed metrics
    }
  ]
}

This confirms the Prometheus Adapter is successfully translating the raw Prometheus metric into a custom metric for the Kubernetes API.

Step 5: Create Horizontal Pod Autoscaler with Custom Metrics

Now that our custom metric is exposed via the Custom Metrics API, we can configure the HPA to use it. We'll create an HPA that scales our my-app deployment based on http_requests_total_per_second.

The HPA resource will target our my-app deployment. We'll specify minReplicas and maxReplicas to define the scaling boundaries. The metrics section is where the magic happens. We define a metric of type: Object, referencing the http_requests_total_per_second metric we configured in the Prometheus Adapter. The describedObject specifies which Kubernetes object this metric applies to (our my-app deployment in this case), and target.value defines the desired value for the metric per object.

For more complex networking scenarios and traffic management that might influence scaling decisions, consider exploring advanced tools like the Kubernetes Gateway API or Istio Ambient Mesh.

# hpa-custom-metrics.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 1
  maxReplicas: 5 # Allow scaling up to 5 pods
  metrics:
  - type: Object # Use 'Object' type for metrics related to a specific Kubernetes object
    object:
      metric:
        name: http_requests_total_per_second
      describedObject:
        apiVersion: apps/v1
        kind: Deployment
        name: my-app
      target:
        type: Value # Target a specific value for the metric
        value: "2" # Target 2 requests per second per deployment

Apply the HPA:

kubectl apply -f hpa-custom-metrics.yaml

Verify Step 5

Watch the HPA status:

kubectl get hpa my-app-hpa -w

Expected Output (initially):

NAME         REFERENCE           TARGETS                 MINPODS   MAXPODS   REPLICAS   AGE
my-app-hpa   Deployment/my-app   /2             1         5         1          XXs

The target is normal initially as the HPA waits for metrics. After a short period (HPA sync interval, typically 15-30 seconds), it should pick up the metric:

NAME         REFERENCE           TARGETS                 MINPODS   MAXPODS   REPLICAS   AGE
my-app-hpa   Deployment/my-app   1.5/2                   1         5         1          XXs

If your simulated metric (e.g., 1.5) is below the target (2), the HPA will keep replicas at minReplicas. Now, let's simulate more traffic to trigger scaling up.

# Push a higher rate of requests
curl -X PUT -H "Content-Type: text/plain" --data-binary 'nginx_http_requests_total{instance="my-app-0",job="my-app",namespace="default",deployment="my-app"} 1000' http://prometheus-pushgateway.default.svc.cluster.local:9091/metrics/job/my-app/instance/my-app-0

Wait for a few HPA cycles. You should see the TARGETS value increase, eventually triggering the HPA to scale up REPLICAS:

NAME         REFERENCE           TARGETS                 MINPODS   MAXPODS   REPLICAS   AGE
my-app-hpa   Deployment/my-app   5.0/2                   1         5         3          XXm

You can also describe the HPA for more details:

kubectl describe hpa my-app-hpa

Expected Output (snippet):

Name:                                                  my-app-hpa
Namespace:                                             default
Labels:                                                
Annotations:                                           
CreationTimestamp:                                     Fri, 27 Oct 2023 10:05:00 +0000
Reference:                                             Deployment/my-app
Metrics:                                               (current / target)
  Resource: cpu on pods                                 (utilized / 80%)
  Object: http_requests_total_per_second on deployment/my-app: 5.0 / 2
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       3 current / 3 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  the HPA is ready to act on a scale request
  ScalingActive   True    ValidMetricFound  the HPA is receiving metrics and is ready to autoscale
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  2m     horizontal-pod-autoscaler  New size: 3; reason: custom metric http_requests_total_per_second above target

This output clearly shows the HPA reacting to the custom metric and scaling the deployment. If you stop pushing metrics, the rate will drop, and the HPA will eventually scale down to minReplicas.

Production Considerations

Implementing HPA with custom metrics in a production environment requires careful planning and robust infrastructure.

1. Reliable Metrics Source: Your metrics collection system (e.g., Prometheus) must be highly available and performant. Metrics data loss or delays can lead to incorrect scaling decisions. Consider Prometheus High Availability setups or managed services.
2. Metric Granularity and Latency: The custom metric should reflect the load accurately and with low latency. If your metric updates infrequently or has significant delays, HPA might react too slowly or too aggressively.
3. Prometheus Adapter Configuration: The PromQL queries in your Prometheus Adapter rules are critical. They must accurately represent the metric you intend to scale on. Test them thoroughly in Prometheus UI before deploying. Ensure the seriesQuery and metricsQuery correctly identify and aggregate metrics from your applications.
4. Throttling and Cooldown Periods: HPA has default scaling up/down stabilization windows to prevent rapid, flapping scaling. For custom metrics, you might need to adjust these values in the HPA definition (spec.behavior in autoscaling/v2) to suit your application's specific needs. Refer to the Kubernetes HPA documentation on configurable scaling behavior.
5. Cost Optimization: While HPA helps optimize costs by scaling down idle resources, ensure your minReplicas and maxReplicas are set appropriately. Overly generous maxReplicas can lead to unexpected cloud spend. Tools like Karpenter can further optimize node provisioning in conjunction with HPA.
6. Monitoring HPA Itself: Monitor the HPA's status, events, and the metrics it's consuming. Alerts should be in place if HPA fails to fetch metrics or if scaling actions are consistently failing.
7. Security: Ensure secure access to your metrics endpoints and the Prometheus Adapter. If you're exposing custom metrics from external systems, consider network policies (e.g., Kubernetes Network Policies) and authentication/authorization.
8. Resource Requests and Limits: Proper resource requests and limits on your application pods are crucial. HPA scales pods, but individual pods still need sufficient resources. If pods are starved, adding more replicas won't solve the problem.
9. Metric Types: Decide between Value (overall value for the object) or AverageValue (average value per pod) for your custom metrics. AverageValue is often preferred for metrics like requests per second where you want to maintain a certain load per replica.
10. External Metrics: For metrics originating outside the cluster (e.g., AWS SQS queue depth, GCP Pub/Sub message count), you'll need an External Metrics Adapter. Prometheus Adapter can also be configured to pull external metrics if they are ingested into Prometheus.

Troubleshooting

Here are common issues you might encounter when setting up HPA with custom metrics and their solutions:

1. HPA shows for TARGETS.

   Problem: The HPA cannot fetch the custom metric value from the Custom Metrics API.

   Solution:

   • Check Prometheus Adapter Pods: Ensure the Prometheus Adapter pods are running and healthy (kubectl get pods -l app=prometheus-adapter).
   • Verify Custom Metrics API: Try to fetch the metric directly from the API (kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/deployments/my-app/http_requests_total_per_second" | jq .). If this fails or shows an empty list, the adapter isn't exposing the metric correctly.
   • Check Prometheus Adapter Logs: Look for errors in the adapter's logs (kubectl logs ). It might indicate issues connecting to Prometheus or parsing PromQL queries.
   • Prometheus Connectivity: Ensure the Prometheus Adapter can reach your Prometheus service (check the prometheus.url in the adapter's configuration).
   • Prometheus Scrapes: Verify Prometheus is scraping your application's metrics (check Prometheus UI -> Targets).
   • Prometheus Adapter Rules: Double-check the seriesQuery and metricsQuery in your Prometheus Adapter configuration. A typo or incorrect PromQL can prevent the metric from being exposed.
2. HPA is not scaling up/down, even if TARGETS show high/low values.

   Problem: The HPA is aware of the metric but isn't taking scaling actions.

   Solution:

   • Check HPA Events: Use kubectl describe hpa and examine the "Events" section. It will often provide a reason for inaction (e.g., "too few replicas," "stabilization window").
   • Min/Max Replicas: Ensure your minReplicas and maxReplicas are set correctly and allow for scaling. If current replicas are already at maxReplicas, it won't scale up.
   • Stabilization Window: HPA has default stabilization windows (5 minutes for scale-down, 3 minutes for scale-up) to prevent rapid fluctuations. You might need to wait, or configure behavior in autoscaling/v2 to adjust these.
   • Metric Value vs. Target: Ensure the current metric value is sufficiently different from the target value to trigger scaling. HPA won't scale for minor deviations.
   • Resource Requests: If your pods don't have resource requests defined, the HPA might struggle to calculate resource-based metrics (though this is less relevant for custom object/pod metrics).
3. HPA scales up/down too aggressively or too slowly.

   Problem: The scaling behavior is not optimal for your application.

   Solution:

   • Adjust Target Value: Fine-tune the target.value in your HPA. A lower target will make HPA scale up earlier, a higher one later.
   • Adjust Stabilization Windows: Use the behavior field in autoscaling/v2 HPA to customize scaleUp.stabilizationWindowSeconds and scaleDown.stabilizationWindowSeconds.
   • PromQL Query: Re-evaluate your Prometheus Adapter's metricsQuery. For instance, using a longer rate window (e.g., [5m] instead of [1m]) can smooth out spikes but make HPA react slower.
   • Metric Granularity: If your metric is too volatile, consider smoothing it out in your PromQL query or by adjusting how your application exposes it.
4. Prometheus Adapter logs show "no metrics found" or similar errors.

   Problem: The adapter cannot find the expected metrics in Prometheus.

   Solution:

   • Check PromQL Query: Log into Prometheus UI, go to the Graph tab, and run the exact PromQL query from your adapter's metricsQuery. Does it return data?
   • Verify Metric Name: Ensure the metric name in your adapter's seriesQuery matches the actual metric name scraped by Prometheus.
   • Labels: Ensure the labels used in your metricsQuery (e.g., namespace="<<.Namespace>>", deployment="<<.Deployment>>") correctly match the labels on your Prometheus metrics.