Just-in

Kagent Tutorial: Build Cloud-Native AI Agents on Kubernetes

Running Ollama LLM on Kubernetes: A Complete Guide

From Docker to Kubernetes: The Orchestration Journey You Didn’t Know You Needed

What is Kubeflow and how can it be used?

Getting Started with Kubeflow: A Complete Beginner’s Guide

Orchestration

Kubecost: Monitor Kubernetes Costs Now

Introduction

In the dynamic world of cloud-native applications, Kubernetes has become the de facto standard for orchestrating containers. Its power lies in its ability to automate deployment, scaling, and management of containerized workloads. However, with great power comes great complexity, especially when it comes to understanding and controlling costs. Many organizations find themselves grappling with unexpectedly high cloud bills, struggling to pinpoint exactly which applications, teams, or resources are consuming the most budget. This lack of visibility can lead to inefficient resource allocation, delayed cost-saving initiatives, and a general disconnect between engineering efforts and financial realities.

This is where Kubecost steps in. Kubecost is an open-source tool that provides real-time cost visibility and insights for Kubernetes workloads. It aggregates cost data from your cloud provider (AWS, GCP, Azure, on-prem) and combines it with your Kubernetes resource usage metrics to give you a granular breakdown of expenses. Imagine knowing not just your total cluster cost, but the exact cost of each namespace, deployment, service, or even individual pod. Kubecost empowers engineering and finance teams to collaborate effectively, optimize resource utilization, and make data-driven decisions to reduce their cloud spend without sacrificing performance or reliability. It’s not just about cutting costs; it’s about understanding the value delivered by your Kubernetes infrastructure.

This comprehensive guide will walk you through the process of installing, configuring, and leveraging Kubecost to gain unparalleled insights into your Kubernetes expenses. We’ll cover everything from initial deployment to advanced features, ensuring you have the tools and knowledge to take control of your cloud budget. Whether you’re a FinOps practitioner, a platform engineer, or a developer looking to understand the financial impact of your code, this guide will provide the expertise you need to master Kubernetes cost monitoring.

TL;DR: Kubernetes Cost Monitoring with Kubecost

Quick Summary: Kubecost provides real-time cost visibility and insights for Kubernetes workloads, integrating cloud billing data with resource usage to break down expenses by namespace, deployment, and more. This guide covers installation, configuration, and key features to optimize your cloud spend.

Key Takeaways:

  • Granular Cost Breakdown: Understand costs by cluster, namespace, deployment, service, and pod.
  • Resource Optimization: Identify over-provisioned resources and opportunities for savings.
  • Alerting & Governance: Set up budgets, alerts, and policies to prevent cost overruns.
  • Integrates with Cloud Providers: Pulls billing data from AWS, GCP, Azure for accurate pricing.

Key Commands:

# Add the Kubecost Helm repository
helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm repo update

# Install Kubecost with a custom namespace
helm upgrade --install kubecost kubecost/cost-analyzer \
  --namespace kubecost --create-namespace \
  --set kubecostToken="YOUR_KUBECOST_TOKEN" \
  --set service.type=NodePort \
  --set prometheus.kube-state-metrics.enabled=false \
  --set prometheus.node-exporter.enabled=false \
  --set prometheus.server.enabled=false

# Get Kubecost service URL (example for NodePort)
kubectl get svc -n kubecost kubecost-cost-analyzer
# For port-forwarding (local access)
kubectl port-forward --namespace kubecost deployment/kubecost-cost-analyzer 9090

Prerequisites

Before we dive into the installation and configuration of Kubecost, ensure you have the following prerequisites in place:

  • A Kubernetes Cluster: You’ll need a running Kubernetes cluster (v1.15+ recommended) where you have administrative access. This could be on any cloud provider (AWS EKS, GCP GKE, Azure AKS) or an on-premises cluster. For managing your cluster, refer to the official Kubernetes documentation on setting up clusters.
  • kubectl: The Kubernetes command-line tool must be installed and configured to connect to your cluster. You can find installation instructions on the Kubernetes documentation site.
  • Helm: Kubecost is primarily deployed using Helm, the package manager for Kubernetes. Ensure you have Helm v3.x installed. If not, follow the Helm installation guide.
  • Prometheus (Optional but Recommended): Kubecost integrates seamlessly with an existing Prometheus instance for collecting metrics. While Kubecost can deploy its own embedded Prometheus, using an existing one is often more efficient. If you don’t have one, Kubecost will provision one for you.
  • Cloud Provider Access (for accurate pricing): To get accurate cloud costs, you’ll need to configure Kubecost to access your cloud provider’s billing data. This typically involves creating IAM roles or service accounts with specific permissions. We’ll cover this in detail.
  • Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, Namespaces, and Helm charts will be beneficial.

Step-by-Step Guide: Installing and Configuring Kubecost

Step 1: Add the Kubecost Helm Repository

The first step is to add the official Kubecost Helm repository to your Helm client. This allows you to easily discover and install the Kubecost cost-analyzer chart.

Helm repositories are like app stores for Kubernetes applications, providing a centralized location for charts. Adding the repository ensures that your Helm client knows where to pull the Kubecost chart definitions from. After adding it, an update command refreshes your local cache of charts from all configured repositories, making sure you have the latest versions available.

helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm repo update

Verify: You should see output similar to this, confirming the repository has been added and updated:

"kubecost" has been added to your repositories
Hang tight while we grab the latest from your configured repositories...
...Successfully got an update from the "kubecost" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete.

Step 2: Install Kubecost with Helm

Now that the repository is added, we can proceed with installing Kubecost. We’ll use a Helm upgrade --install command, which will install Kubecost if it doesn’t exist, or upgrade it if it does. This command also allows us to configure various parameters.

For this installation, we’re making a few key configurations:

  • --namespace kubecost --create-namespace: We’re installing Kubecost into its own dedicated namespace called kubecost. This is a best practice for isolating system components.
  • --set kubecostToken="YOUR_KUBECOST_TOKEN": This is a crucial step for accessing the enterprise features and support from Kubecost. You can obtain a free trial token from the Kubecost website. While Kubecost works without a token, many advanced features require it.
  • --set service.type=NodePort: By default, Kubecost might use a LoadBalancer service, which incurs cloud costs. For initial setup or testing, a NodePort makes the service accessible on each node’s IP at a static port. For production, you might switch to an Ingress or LoadBalancer.
  • --set prometheus.kube-state-metrics.enabled=false, --set prometheus.node-exporter.enabled=false, --set prometheus.server.enabled=false: These settings disable the embedded Prometheus components. If you have an existing Prometheus instance, you should disable these to avoid resource duplication. Kubecost can be configured to use your existing Prometheus instance, which is often more robust and cost-effective. For more on Prometheus and observability, check out our guide on eBPF Observability with Hubble, which often integrates with Prometheus.
# Replace YOUR_KUBECOST_TOKEN with your actual token
helm upgrade --install kubecost kubecost/cost-analyzer \
  --namespace kubecost --create-namespace \
  --set kubecostToken="YOUR_KUBECOST_TOKEN" \
  --set service.type=NodePort \
  --set prometheus.kube-state-metrics.enabled=false \
  --set prometheus.node-exporter.enabled=false \
  --set prometheus.server.enabled=false \
  --set global.prometheus.enabled=false \
  --set global.grafana.enabled=false

Verify: After the installation, you should see output indicating a successful deployment. You can then check the pods in the kubecost namespace:

kubectl get pods -n kubecost

Expected output (pod names and statuses may vary slightly, but should be Running):

NAME                                             READY   STATUS    RESTARTS   AGE
kubecost-cost-analyzer-779899f848-j2khc          1/1     Running   0          2m
kubecost-grafana-79cbb45464-h5q8h                1/1     Running   0          2m
kubecost-kubectl-677d9c6665-w7slj                1/1     Running   0          2m
kubecost-prometheus-server-6775676767-f5lq2      2/2     Running   0          2m

Note: If you disable Prometheus components as shown above, you will not see kubecost-prometheus-server or related pods. Instead, Kubecost will expect an external Prometheus to be configured. We will assume for brevity in subsequent steps that an external Prometheus is either already configured or you chose to enable the embedded one.

Step 3: Access the Kubecost UI

Once Kubecost is installed and its pods are running, you can access the web interface. Depending on your service.type setting, the method of access will vary.

If you used NodePort, you can find the port and access it via any node’s IP:

kubectl get svc -n kubecost kubecost-cost-analyzer

Expected output:

NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubecost-cost-analyzer     NodePort   10.100.200.10   <none>        9090:3xxxx/TCP   5m

You would then navigate to http://<ANY_NODE_IP>:3xxxx in your browser.

A more common and convenient way for local access and testing is to use kubectl port-forward. This creates a secure tunnel from your local machine to the Kubecost service.

kubectl port-forward --namespace kubecost deployment/kubecost-cost-analyzer 9090

Verify: Open your web browser and navigate to http://localhost:9090. You should be greeted by the Kubecost dashboard. It might take a few minutes for data to start populating as Kubecost collects metrics and processes billing information.

Step 4: Configure Cloud Integration for Accurate Pricing

For Kubecost to provide truly accurate cost data, it needs to understand the actual pricing of your cloud resources. This means integrating with your cloud provider’s billing APIs. Without this, Kubecost will use default pricing models, which may not reflect your specific discounts or negotiated rates. This step is critical for effective cost optimization efforts.

The configuration varies by cloud provider:

AWS Integration

You need to create an IAM role with specific permissions and configure a Cost and Usage Report (CUR) in S3.

  1. Create S3 Bucket for CUR: 
    aws s3api create-bucket --bucket kubecost-cur-<YOUR_ACCOUNT_ID> --region us-east-1
  2. Configure Cost and Usage Report (CUR):
    Go to the AWS Billing dashboard, enable the Cost and Usage Report, and configure it to deliver to the S3 bucket you just created. Ensure it includes resource IDs. Refer to the AWS CUR documentation for detailed steps.
  3. Create IAM Policy for Kubecost:
    Create an IAM policy with permissions to read the S3 bucket and access billing data.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::kubecost-cur-<YOUR_ACCOUNT_ID>",
                "arn:aws:s3:::kubecost-cur-<YOUR_ACCOUNT_ID>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ce:GetCostAndUsage",
                "pricing:GetProducts"
            ],
            "Resource": "*"
        }
    ]
}
  4. Create IAM Role and Attach Policy:
    Create an IAM role (e.g., kubecost-role) and attach the policy you just created. Configure the trust policy to allow your EKS worker nodes (or wherever Kubecost runs) to assume this role, typically via IAM Roles for Service Accounts (IRSA).
  5. Configure Kubecost Helm Values:
    Update your Kubecost Helm installation to use the ARN of the IAM role.

    helm upgrade --install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --set kubecostToken="YOUR_KUBECOST_TOKEN" \
  --set service.type=NodePort \
  --set prometheus.kube-state-metrics.enabled=false \
  --set prometheus.node-exporter.enabled=false \
  --set prometheus.server.enabled=false \
  --set global.prometheus.enabled=false \
  --set global.grafana.enabled=false \
  --set aws.athena.enabled=false \
  --set aws.spotLabel="eks.amazonaws.com/capacityType" \
  --set aws.spotLabelValue="SPOT" \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"="arn:aws:iam::<YOUR_ACCOUNT_ID>:role/kubecost-role" \
  --set aws.projectID="<YOUR_ACCOUNT_ID>" # This is your AWS Account ID

GCP Integration

For GCP, you’ll need to enable the Billing Export to BigQuery and create a Service Account with appropriate permissions.

  1. Enable Billing Export to BigQuery:
    Follow the GCP documentation to export your billing data to a BigQuery dataset.
  2. Create a Service Account:
    Create a new GCP Service Account with the following roles:

    • BigQuery Data Viewer (on the dataset containing billing data)
    • BigQuery Job User
    • Project Viewer (on the project where the BigQuery dataset resides)
  3. Create a Service Account Key:
    Generate a JSON key for this service account. Save this file securely.
  4. Configure Kubecost Helm Values:
    Create a Kubernetes secret containing your GCP service account key.

    kubectl create secret generic kubecost-gcp-key \
  --from-file=key.json=/path/to/your/gcp-service-account-key.json \
  --namespace kubecost

    Then, update your Kubecost Helm installation:

    helm upgrade --install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --set kubecostToken="YOUR_KUBECOST_TOKEN" \
  --set service.type=NodePort \
  --set prometheus.kube-state-metrics.enabled=false \
  --set prometheus.node-exporter.enabled=false \
  --set prometheus.server.enabled=false \
  --set global.prometheus.enabled=false \
  --set global.grafana.enabled=false \
  --set gcp.serviceKeySecret="kubecost-gcp-key" \
  --set gcp.projectID="<YOUR_GCP_PROJECT_ID>" \
  --set gcp.billingDataset="<YOUR_BIGQUERY_DATASET_NAME>"

Azure Integration

Azure integration involves setting up an Azure Service Principal and configuring Cost Management exports.

  1. Configure Cost Management Export:
    Set up a recurring export of your Azure Cost Management data to a Storage Account. Refer to the Azure documentation.
  2. Create an Azure Service Principal:
    Create an Azure Service Principal with “Reader” permissions on your subscription and “Storage Blob Data Reader” on the storage account where cost exports are stored.

    az ad sp create-for-rbac --name kubecost-sp --role "Reader" --scopes /subscriptions/<SUBSCRIPTION_ID>

    Note the appId, password, and tenant from the output.

  3. Configure Kubecost Helm Values:
    Create a Kubernetes secret for the Azure Service Principal credentials.

    kubectl create secret generic kubecost-azure-secret \
  --from-literal=clientID="<APP_ID>" \
  --from-literal=clientSecret="<PASSWORD>" \
  --from-literal=tenantID="<TENANT_ID>" \
  --namespace kubecost

    Then, update your Kubecost Helm installation:

    helm upgrade --install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --set kubecostToken="YOUR_KUBECOST_TOKEN" \
  --set service.type=NodePort \
  --set prometheus.kube-state-metrics.enabled=false \
  --set prometheus.node-exporter.enabled=false \
  --set prometheus.server.enabled=false \
  --set global.prometheus.enabled=false \
  --set global.grafana.enabled=false \
  --set azure.subscriptionID="<SUBSCRIPTION_ID>" \
  --set azure.resourceGroup="<RESOURCE_GROUP_CONTAINING_STORAGE_ACCOUNT>" \
  --set azure.storageAccount="<STORAGE_ACCOUNT_NAME>" \
  --set azure.container="<CONTAINER_NAME_FOR_EXPORTS>" \
  --set azure.secretName="kubecost-azure-secret"

Verify: After configuring the cloud integration and updating your Helm release, navigate to the Kubecost UI. Go to Settings -> Integrations. You should see your cloud provider listed as “Configured” or “Connected”. It may take some time (up to 24 hours for initial sync) for the data to fully populate and reflect accurate cloud pricing.

Production Considerations

Deploying Kubecost in a production environment requires careful planning beyond a basic installation. Here are key considerations to ensure reliability, scalability, security, and accuracy.

High Availability and Scalability

For production, you’ll want to ensure Kubecost itself is highly available.

  • Replicas: Increase the replica count for the cost-analyzer deployment. 
    # values.yaml snippet for increased replicas
cost-analyzer:
  replicas: 3
  • External Prometheus: As alluded to in the installation steps, using an external, highly available Prometheus instance (e.g., using Prometheus Federation or a managed Prometheus service) is crucial. The embedded Prometheus is suitable for smaller clusters or testing, but an external one offers better scalability, long-term storage, and operational resilience. Ensure your external Prometheus is configured to scrape metrics from all your Kubernetes components.
  • Resource Requests and Limits: Set appropriate CPU and memory requests and limits for Kubecost pods to prevent resource starvation or noisy neighbor issues. Monitor resource usage and adjust as necessary.

Security

Security is paramount for any production system, especially one handling sensitive cost data.

  • Network Access: Restrict access to the Kubecost UI. Instead of NodePort, use an Ingress Controller (e.g., NGINX, Traefik, or an API Gateway) with TLS encryption. Implement authentication (e.g., OAuth2 proxy, OIDC) to secure access for authorized users only. Consider network policies to restrict communication to and from Kubecost pods; our Kubernetes Network Policies: Complete Security Hardening Guide can provide further insights.
  • Cloud Provider Credentials: Use IAM Roles for Service Accounts (IRSA on AWS), Workload Identity (GCP), or Azure AD Pod Identity to provide cloud provider credentials to Kubecost securely, avoiding static credentials in secrets where possible. This aligns with least privilege principles.
  • Kubecost Token: Protect your Kubecost token. Store it as a Kubernetes secret and reference it in your Helm values, rather than hardcoding it directly in the Helm command.
  • Regular Updates: Keep Kubecost and its dependencies (Helm, Kubernetes) updated to patch security vulnerabilities.

Data Persistence and Backups

Kubecost stores its configuration and some aggregated data.

  • Persistent Storage: Ensure the Kubecost deployment uses a PersistentVolumeClaim (PVC) for its data directory, especially if using the embedded Prometheus or if you rely on its internal data for historical analysis.
  • Backup Strategy: Implement a backup strategy for the PersistentVolume used by Kubecost. This could involve volume snapshots provided by your cloud provider or a Kubernetes-native backup solution like Velero.

Monitoring and Alerting

Monitor Kubecost itself to ensure it’s healthy and collecting data.

  • Kubecost Metrics: Kubecost exposes its own metrics via Prometheus. Scrape these metrics with your central Prometheus and set up alerts for issues like data collection failures, high latency, or API errors.
  • Resource Usage: Monitor the resource consumption of Kubecost pods (CPU, memory) to ensure they are operating within their limits and not experiencing performance bottlenecks.

Cost Allocation and Chargeback

For effective FinOps, accurate cost allocation is key.

  • Labeling Strategy: Enforce a strict labeling strategy across your Kubernetes resources. Kubecost can use labels to break down costs by team, project, environment, or application. This is fundamental for chargeback or showback models.
  • Custom Pricing: If you have specific negotiated rates with your cloud provider or want to factor in on-premises hardware costs, Kubecost allows for custom pricing configurations.
  • Budgets and Alerts: Utilize Kubecost’s budgeting features to set thresholds for namespaces, teams, or projects. Configure alerts to notify stakeholders when budgets are approaching or exceeded.

Integration with Other Tools

Kubecost can be part of a larger FinOps and observability ecosystem.

  • CI/CD: Integrate cost checks into your CI/CD pipelines to provide developers with immediate feedback on the cost implications of their deployments.
  • Reporting: Leverage Kubecost’s API to pull cost data into external reporting tools or dashboards for broader organizational visibility.

By addressing these production considerations, you can transform Kubecost from a simple monitoring tool into a robust, integral part of your Kubernetes FinOps strategy.

Troubleshooting

Even with careful installation, you might encounter issues with Kubecost. Here are some common problems and their solutions.

1. Kubecost UI shows “No Data Available” or Incomplete Data

Problem: The Kubecost dashboard is empty, or shows partial data like only cluster costs but no detailed breakdown.

Solution:

  • Wait for Data Collection: Kubecost needs time to collect metrics from Prometheus and process cloud billing data. This can take anywhere from a few minutes to several hours, especially for initial cloud integration.
  • Check Prometheus Connectivity: Ensure Kubecost can reach your Prometheus instance. If you disabled the embedded Prometheus, verify your global.prometheus.external.url setting in Helm values. 
    # Check Kubecost logs for Prometheus connection errors
kubectl logs -n kubecost deployment/kubecost-cost-analyzer -c cost-model | grep -i prometheus
  • Verify Prometheus Scrapes: Ensure Prometheus is correctly scraping metrics from your Kubernetes cluster, including kube-state-metrics and node-exporter.
  • Cloud Integration Status: Go to Settings -> Integrations in the Kubecost UI. Check if your cloud provider is “Configured” and if there are any errors reported.
  • Check Cloud Provider Permissions: Double-check the IAM roles/Service Principals and their associated permissions for your cloud provider integration. Missing permissions are a common cause of data gaps.

2. Incorrect or Inaccurate Cost Data

Problem: Kubecost shows costs, but they don’t match your cloud bill, or specific resources have incorrect pricing.

Solution:

  • Cloud Integration Configuration: Re-verify your cloud provider integration settings (AWS CUR, GCP BigQuery Export, Azure Cost Management Export). Ensure the export includes resource IDs and is generating data correctly.
  • Pricing API Access: Confirm Kubecost has access to the cloud provider’s pricing API (e.g., AWS Pricing API, GCP Cloud Billing API). This is separate from billing export access.
  • Custom Pricing: If you have custom pricing or negotiated rates, ensure you’ve configured them within Kubecost’s settings.
  • Timezone Discrepancy: Ensure your Kubecost instance, Kubernetes cluster, and cloud provider billing data are aligned on timezones.
  • Resource Reconciliation: Kubecost reconciles Kubernetes resources with cloud provider resources. If there are discrepancies in naming conventions or tags, this can lead to inaccuracies. Kubecost’s Assets page can help diagnose this.

3. Kubecost Pods are Crashing or in Pending State

Problem: Kubecost pods are not running correctly (e.g., CrashLoopBackOff, Pending).

Solution:

  • Check Pod Logs: 
    kubectl logs -n kubecost <pod-name>
kubectl describe pod -n kubecost <pod-name>

    Look for error messages, out-of-memory issues, or configuration errors.

  • Resource Constraints: If pods are in Pending, check if your cluster has enough resources (CPU, memory) and if the Kubecost deployment has appropriate resource requests and limits. 
    kubectl describe nodes
  • Image Pull Issues: If pods are in ImagePullBackOff, verify that your cluster nodes can pull images from Docker Hub or your configured registry.
  • Helm Values: Review your Helm values.yaml file or the --set flags used during installation for any misconfigurations.

4. High Resource Consumption by Kubecost Components

Problem: Kubecost pods (especially Prometheus if embedded) are consuming excessive CPU or memory.

Solution:

  • External Prometheus: If you’re using the embedded Prometheus on a large cluster, consider migrating to an external, dedicated Prometheus instance or a managed service. This offloads the metric storage and querying burden.
  • Adjust Resource Limits: Tune the resource requests and limits for the Kubecost deployment and any embedded components. Start with reasonable values and scale up as needed based on monitoring.
  • Data Retention: Adjust the data retention period for Prometheus if it’s storing too much historical data.
  • Scrape Intervals: Optimize Prometheus scrape intervals. Reducing the frequency for less critical metrics can save resources.

5. Unable to Access Kubecost UI Remotely (beyond Port-Forward)

Problem: kubectl port-forward works, but you can’t access Kubecost via a public IP or custom domain.

Solution:

  • Service Type: If you used NodePort, ensure your firewall rules allow traffic to the node’s IP on the assigned NodePort. For public access, you’ll need to configure an external LoadBalancer or Ingress.
  • LoadBalancer Service: If you set service.type=LoadBalancer, check if your cloud provider provisioned an external IP. 
    kubectl get svc -n kubecost kubecost-cost-analyzer

    Look for an EXTERNAL-IP. If it’s <pending>, check your cloud provider’s load balancer quota or logs.

  • Ingress Controller: For production, configure an Ingress Controller (e.g., NGINX Ingress, Traefik, or cloud provider-specific) and create an Ingress resource pointing to the Kubecost service. 
    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubecost-ingress
  namespace: kubecost
spec:
  rules:
  - host: kubecost.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubecost-cost-analyzer
            port:
              number: 9090

    Ensure your DNS points to the Ingress Controller’s IP.

  • Network

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *