Scale Kubernetes with Custom Metrics HPA

Prerequisites

Before diving into custom metrics with HPA, ensure you have the following:

• A running Kubernetes cluster: Any version 1.16+ will suffice, but ideally, use a recent stable version (e.g., 1.25+). You can use Minikube, Kind, or a cloud-managed service like GKE, EKS, or AKS.
• kubectl configured: Your kubectl command-line tool should be configured to interact with your cluster.
• Helm (optional but recommended): For easier installation of Prometheus and other ecosystem tools. Download it from the official Helm documentation.
• Metrics Server: This is fundamental for HPA to work, even with custom metrics, as it provides basic CPU/memory metrics. If you don't have it, we'll install it.
• Basic understanding of Kubernetes concepts: Deployments, Pods, Services, and how HPA works with standard metrics. For a refresher, refer to the Kubernetes HPA documentation.
• Prometheus (or similar monitoring system): To collect and store your custom metrics. We'll primarily focus on Prometheus as it's the de-facto standard for this use case.
• Prometheus Adapter (or another custom metrics API adapter): This component is crucial. It translates Prometheus queries into a format the Kubernetes Custom Metrics API can understand, allowing HPA to consume them.

Step-by-Step Guide

Step 1: Deploy a Sample Application

First, let's deploy a simple PHP Apache application. This application is commonly used in Kubernetes HPA examples because it exposes a simple CPU-intensive endpoint, making it easy to generate load and observe scaling behavior. We'll later modify it conceptually to include custom metrics.

kubectl apply -f https://k8s.io/examples/application/php-apache.yaml

This command deploys a Deployment named php-apache and a Service to expose it.

Verify

Check if the deployment and service are running.

kubectl get deployment php-apache

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
php-apache   1/1     1            1           XXs

kubectl get service php-apache

NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
php-apache   ClusterIP   10.96.XX.XX   <none>        80/TCP         XXs

Step 2: Install Metrics Server

The Kubernetes Metrics Server is a cluster-wide aggregator of resource usage data. It collects CPU and memory usage from Kubelets and exposes them via the Kubernetes Metrics API. While we are focusing on custom metrics, the Metrics Server is a foundational component for HPA. If you already have it, you can skip this step.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Verify

Ensure the Metrics Server pod is running and healthy. It might take a minute or two to start.

kubectl get pods -l k8s-app=metrics-server -n kube-system

NAME                            READY   STATUS    RESTARTS   AGE
metrics-server-XXXXX-YYYYY      1/1     Running   0          XXs

You can also check if it can retrieve node and pod metrics.

kubectl top nodes
kubectl top pods

If these commands return metrics, the Metrics Server is working.

Step 3: Deploy Prometheus and Prometheus Adapter

To use custom metrics, you need a monitoring system to collect them and an adapter to expose them to the Kubernetes Custom Metrics API. Prometheus is the most common choice for metric collection in Kubernetes. The Prometheus Adapter then acts as the bridge.

3.1 Install Prometheus

We'll install Prometheus using Helm, which simplifies deployment and configuration.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

This command adds the Prometheus Helm repository, updates it, and then installs Prometheus into a new monitoring namespace.

Verify

Check if Prometheus pods are running in the monitoring namespace.

kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus

NAME                                             READY   STATUS    RESTARTS   AGE
prometheus-kube-state-metrics-XXXXX-YYYYY        1/1     Running   0          XXs
prometheus-prometheus-node-exporter-XXXXX        1/1     Running   0          XXs
prometheus-prometheus-pushgateway-XXXXX-YYYYY    1/1     Running   0          XXs
prometheus-prometheus-server-XXXXX-YYYYY         2/2     Running   0          XXs

3.2 Deploy Prometheus Adapter

The Prometheus Adapter is the key component for custom metrics. It queries Prometheus and exposes those metrics via the Custom Metrics API (custom.metrics.k8s.io). The configuration of the adapter involves defining rules to map Prometheus metrics to Kubernetes custom metrics.

For this example, we'll use a simplified deployment. A real-world scenario would involve a custom values.yaml for the Helm chart to define specific metric mapping rules. For instance, to expose the http_requests_total metric from our sample application, you would configure the adapter to query Prometheus for this metric and expose it under a specific name.

Here's a basic example of how you might install the Prometheus Adapter. Note that the rules.custom and rules.external sections are crucial for defining which metrics are exposed.

# custom-metrics-adapter-values.yaml
rules:
  custom:
  - seriesQuery: '{__name__="http_requests_total", container="php-apache"}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second" # Expose as a rate metric per pod
    metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>)
  - seriesQuery: '{__name__="http_requests_total"}' # Another example, aggregate across all pods
    resources:
      overrides:
        namespace: {resource: "namespace"}
    name:
      matches: "^(.*)_total$"
      as: "${1}_total"
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  external:
  - seriesQuery: '{__name__="queue_length"}' # Example for an external metric
    name:
      matches: "queue_length"
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)

Now, deploy the Prometheus Adapter using Helm with these custom rules.

helm repo add k8s-at-home https://k8s-at-home.com/charts/
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts # Already added, but good practice
helm repo update
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring --create-namespace \
  -f custom-metrics-adapter-values.yaml

Note: The above custom-metrics-adapter-values.yaml is a simplified example. In a real application, you'd need your application to expose Prometheus-compatible metrics (e.g., via a /metrics endpoint), and Prometheus would need to be configured to scrape them. For more details on configuring Prometheus Adapter, refer to its official documentation.

Verify

Check if the Prometheus Adapter pod is running and that the custom metrics API is available.

kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-adapter

NAME                                     READY   STATUS    RESTARTS   AGE
prometheus-adapter-XXXXX-YYYYY           1/1     Running   0          XXs

Test the custom metrics API endpoint:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

You should see a list of available custom metrics, including those defined in your adapter configuration.

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "pods/http_requests_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "deployments/http_requests_total",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/http_requests_total",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Similarly, for external metrics:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "queue_length",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Step 4: Create a HorizontalPodAutoscaler with Custom Metrics

Now that our metrics infrastructure is in place, we can define an HPA that leverages these custom metrics. HPA supports three types of custom metrics:

• Pods metrics: Apply to pods and are averaged across all pods in the scaling target. E.g., "average HTTP requests per second per pod".
• Object metrics: Describe a single Kubernetes object (e.g., a Deployment, Service, Ingress) rather than individual pods. E.g., "total HTTP requests for the entire Deployment".
• External metrics: Metrics coming from outside the Kubernetes cluster, not directly related to a Kubernetes object (e.g., queue length in an AWS SQS queue, database connections).

Let's create an HPA that uses all three types for demonstration. We'll assume our php-apache application (or a proxy in front of it) exposes http_requests_total and that we have an imaginary external queue_length metric.

# hpa-custom-metrics.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache-hpa-custom
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second # This comes from our adapter rule mapping http_requests_total as a rate
      target:
        type: AverageValue
        averageValue: 10 # Scale up if average requests per second per pod exceeds 10
  - type: Object
    object:
      metric:
        name: http_requests_total # This comes from our adapter rule mapping http_requests_total
      describedObject:
        apiVersion: apps/v1
        kind: Deployment
        name: php-apache
      target:
        type: Value
        value: 500 # Scale up if total HTTP requests for the deployment exceeds 500
  - type: External
    external:
      metric:
        name: queue_length # This comes from an external source, exposed by the adapter
      target:
        type: AverageValue
        averageValue: 5 # Scale up if average queue length exceeds 5
  behavior: # Optional: Define scaling behavior (cool-down, stabilization window)
    scaleDown:
      stabilizationWindowSeconds: 300 # Wait 5 minutes before scaling down
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleUp:
      stabilizationWindowSeconds: 0 # Scale up immediately
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15

Apply this HPA definition:

kubectl apply -f hpa-custom-metrics.yaml

Verify

Check the status of your HPA. Initially, it will show "unknown" for custom metrics until the adapter starts providing data.

kubectl get hpa php-apache-hpa-custom

NAME                    REFERENCE               TARGETS                                                MINPODS   MAXPODS   REPLICAS   AGE
php-apache-hpa-custom   Deployment/php-apache   <unknown>/10 (pods), <unknown>/500 (object), <unknown>/5 (external)   1         10        1          XXs

After some time, and assuming metrics are being collected and exposed, you should see actual values.

NAME                    REFERENCE               TARGETS                                                MINPODS   MAXPODS   REPLICAS   AGE
php-apache-hpa-custom   Deployment/php-apache   0/10 (pods), 0/500 (object), 0/5 (external)            1         10        1          XXm

Step 5: Generate Load and Observe Scaling

To see the HPA in action, we need to generate some load against our php-apache application. We'll use a busybox pod for this.

kubectl run -it --rm --restart=Never busybox-load-generator --image=busybox -- /bin/sh

Once inside the busybox container, run a loop to hit the PHP Apache service.

# Inside busybox-load-generator pod
while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done

This will continuously send requests to the php-apache service. Prometheus will scrape metrics from the application (if configured to do so), the Prometheus Adapter will expose them, and the HPA will react.

Verify

Monitor the HPA status and the number of replicas for your deployment.

kubectl get hpa php-apache-hpa-custom -w

You should observe the REPLICAS column increasing as the load generates metrics that exceed your defined targets.

NAME                    REFERENCE               TARGETS                                                MINPODS   MAXPODS   REPLICAS   AGE
php-apache-hpa-custom   Deployment/php-apache   15/10 (pods), 750/500 (object), 0/5 (external)         1         10        1          XXm
php-apache-hpa-custom   Deployment/php-apache   15/10 (pods), 750/500 (object), 0/5 (external)         1         10        2          XXm # Scaled up!
php-apache-hpa-custom   Deployment/php-apache   12/10 (pods), 600/500 (object), 0/5 (external)         2         10        3          XXm # Scaled up again!

You can also watch the pods being created:

kubectl get pods -l app=php-apache -w

Once you stop the load generator, after the stabilizationWindowSeconds (if configured) and metrics drop, the HPA will scale down the replicas back to minReplicas.

For deeper insights into networking and how traffic is routed and observed during scaling, consider exploring tools like Cilium WireGuard Encryption for secure pod-to-pod communication or eBPF Observability with Hubble for detailed network flow visibility.

Production Considerations

Implementing HPA with custom metrics in a production environment requires careful planning and robust infrastructure. Here are key considerations:

1. Reliable Metrics Pipeline: Your entire metrics pipeline (application instrumentation, Prometheus scraping, Prometheus Adapter) must be highly available and performant. Any disruption can lead to incorrect scaling decisions or a complete failure to scale.
2. Metric Granularity and Latency:
   • Scrape Interval: Prometheus scrape intervals directly impact how quickly HPA can react. Shorter intervals (e.g., 5-15s) provide faster reactions but increase Prometheus load.
   • Adapter Refresh Rate: The Prometheus Adapter queries Prometheus periodically. Ensure this interval is appropriate for your scaling needs.
3. Metric Selection and Thresholds:
   • Meaningful Metrics: Choose metrics that truly reflect the load and performance of your application. Avoid "vanity" metrics.
   • Accurate Thresholds: Setting correct targetValue or averageValue is critical. Too low, and you over-scale; too high, and you risk performance degradation. Use historical data and load testing to determine optimal values.
   • Multiple Metrics: HPA can use multiple metrics. It scales based on the metric that suggests the largest number of replicas. This provides resilience, but ensure your metrics don't contradict each other.
4. Scaling Behavior (behavior field):
   • Stabilization Window: Crucial for preventing "thrashing" (rapid scale up/down cycles). Set appropriate stabilizationWindowSeconds for both scaleUp and scaleDown.
   • Scaling Policies: Use policies to control the rate of scaling (e.g., scale up by a percentage of current pods or a fixed number of pods per period). This prevents sudden, massive scaling events that could overwhelm dependent services.
5. Resource Requests and Limits: HPA cannot function effectively without proper resource requests and limits for your pods. These define the baseline resource allocation and help the scheduler place pods efficiently.
6. Pod Startup/Readiness Probes: Ensure your pods have robust readiness probes. HPA counts only ready pods when calculating target metrics. If new pods take a long time to become ready, HPA might over-scale.
7. Cost Optimization: While HPA helps optimize resource usage, consider integrating it with node autoscalers like Karpenter for Cost Optimization. HPA scales pods, but Karpenter ensures that underlying nodes are provisioned or deprovisioned efficiently to support those pods, minimizing infrastructure costs.
8. Security: Ensure your metrics endpoints are secured, especially if they contain sensitive information. For cluster-internal communication, consider solutions like Kubernetes Network Policies to restrict access to metrics endpoints. For comprehensive supply chain security, integrate with tools like Sigstore and Kyverno to ensure only trusted images run in your cluster.
9. Observability: Monitor the HPA itself. Observe its scaling decisions, current and target replicas, and the metrics it's using. Prometheus can scrape metrics from the HPA controller manager to provide insights.
10. Testing: Thoroughly test your HPA configurations under various load conditions (peak, average, sudden spikes) in a staging environment before deploying to production.

Troubleshooting

HPA with custom metrics can be complex. Here are common issues and their solutions:

1. Issue: HPA shows <unknown> for custom metrics.

   Explanation: This typically means the Prometheus Adapter isn't correctly configured, isn't running, or can't fetch metrics from Prometheus.

   Solution:

   • Check Prometheus Adapter logs:

     kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus-adapter

     Look for errors related to Prometheus connectivity or metric queries.

   • Verify Prometheus is running and healthy:

     kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus

   • Ensure Prometheus can scrape your application's metrics. Access the Prometheus UI (port-forward prometheus-prometheus-server service) and check "Status -> Targets".
   • Verify the Prometheus Adapter's configuration (rules.custom and rules.external) in its values.yaml. The seriesQuery must match existing metrics in Prometheus, and metricsQuery must be valid PromQL.
   • Test the custom metrics API directly:

     kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/php-apache/http_requests_per_second" | jq .

     This should return a metric value if everything is working.

2. Issue: HPA is not scaling up despite high load.

   Explanation: Even if metrics are visible, HPA might not scale due to configuration issues, resource constraints, or other factors.

   Solution:

   • Check HPA events:

     kubectl describe hpa php-apache-hpa-custom

     Look for "Events" at the bottom for messages indicating why scaling isn't happening (e.g., too few replicas, scaling active, no metrics available, stabilization window still active).

   • Verify minReplicas, maxReplicas, and target values. Ensure the current metric value exceeds the target.
   • Check the behavior section for any restrictive scaleUp policies or long stabilizationWindowSeconds.
   • Ensure there are enough cluster resources (CPU, Memory) for new pods to be scheduled. If your cluster is resource-constrained, pods might remain in Pending state, preventing HPA from reaching its target.
3. Issue: HPA is scaling down too aggressively or not at all.

   Explanation: Often related to stabilizationWindowSeconds or scaleDown policies.

   Solution:

   • Adjust stabilizationWindowSeconds in the behavior.scaleDown section. A longer window (e.g., 300-600 seconds) prevents rapid scale-down during temporary dips in load.
   • Review scaleDown.policies. Ensure they allow scaling down at a reasonable rate.
   • Make sure your custom metrics are accurately reflecting reduced load. If metrics remain high even after load subsides, Prometheus or the adapter might be misconfigured.
   • Check if pods are draining gracefully. If pods take a long time to terminate, it can interfere with scale-down.
4. Issue: HPA is scaling based on an incorrect metric value.

   Explanation: The PromQL query in the Prometheus Adapter might be incorrect or returning unexpected values.

   Solution:

   • Debug your PromQL query directly in the Prometheus UI. Ensure it returns the expected values for your metric.
   • Verify the seriesQuery and metricsQuery in your Prometheus Adapter configuration. Pay close attention to label matching (<<.LabelMatchers>>) and aggregation (sum by (<<.GroupBy>>)).
   • Ensure the name.as field in the adapter correctly transforms the metric name to what HPA is expecting.
5. Issue: API service custom.metrics.k8s.io or external.metrics.k8s.io is unavailable.

   Explanation: The Prometheus Adapter, which provides these API services, might be down or not registered correctly.

   Solution:

   • Check the Prometheus Adapter pod status and logs.
   • Verify the APIService resource for the custom metrics API:

     kubectl get apiservice v1beta1.custom.metrics.k8s.io -o yaml

     Ensure its status.conditions show True for Available. If not, check the service and certificate configuration of the Prometheus Adapter.

   • Ensure your Kubernetes API server is configured to trust the Prometheus Adapter's serving certificate.
6. Issue: HPA reports "no metrics available" for a specific metric even when others work.

   Explanation: This often points to a problem with that particular metric's configuration in the Prometheus Adapter or its presence in Prometheus.

   Solution:

   • Double-check the specific Prometheus Adapter rule for that metric. Is the seriesQuery correct? Does it match any series in Prometheus?
   • Query Prometheus directly for that metric to confirm it exists and has values for the relevant labels.
   • Ensure the name.as transformation is correct and doesn't conflict with other metrics.
   • If it's an Object metric, ensure the describedObject accurately points to the Kubernetes resource.

FAQ Section

1. What is the difference between Pods, Object, and External custom metrics?

   Pods metrics are associated with