OpenCost: Master Kubernetes Costs Now

Introduction

In the dynamic world of cloud-native applications, Kubernetes has emerged as the de facto orchestrator for containerized workloads. While Kubernetes offers unparalleled flexibility, scalability, and resilience, managing its operational costs can be a significant challenge. As organizations scale their Kubernetes deployments, understanding where their cloud spend goes becomes paramount. Without clear visibility into cost allocation, it’s easy for expenses to spiral out of control, leading to budget overruns and inefficient resource utilization.

This is where OpenCost steps in. OpenCost is an open-source, vendor-neutral standard for measuring and allocating infrastructure costs in Kubernetes environments. Developed under the Cloud Native Computing Foundation (CNCF), OpenCost provides granular insights into Kubernetes spend, breaking down costs by various dimensions such as namespaces, deployments, labels, and even individual pods. By offering a standardized approach to cost attribution, OpenCost empowers engineering teams, finance departments, and operations personnel to make informed decisions, optimize resource usage, and ultimately reduce their cloud bills. This guide will walk you through the process of deploying and leveraging OpenCost to gain complete visibility into your Kubernetes expenses.

# Add OpenCost Helm repository
helm repo add opencost https://opencost.github.io/opencost-helm-chart/

# Update Helm repositories
helm repo update

# Install OpenCost with default settings
helm install opencost opencost/opencost -n opencost --create-namespace

# Port-forward to access the UI (replace with your pod name)
kubectl port-forward -n opencost deployment/opencost 9090:9090

# Access the OpenCost UI in your browser: http://localhost:9090

Prerequisites

Before diving into OpenCost, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.16+). This can be on any cloud provider (AWS EKS, GCP GKE, Azure AKS, etc.) or on-premises.
• kubectl: The Kubernetes command-line tool configured to connect to your cluster.
• Helm: Helm 3.x installed for simplified deployment of OpenCost.
• Cloud Provider Credentials (Optional but Recommended): For accurate cost data, OpenCost needs access to your cloud billing data. This typically involves creating specific IAM roles or service accounts with read-only access to billing reports. We will cover this for AWS, GCP, and Azure.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like pods, deployments, namespaces, and services.

Step-by-Step Guide

Step 1: Add OpenCost Helm Repository and Install

The easiest way to deploy OpenCost into your Kubernetes cluster is by using Helm. Helm charts encapsulate all the necessary Kubernetes manifests, making installation and management straightforward. First, you need to add the official OpenCost Helm repository to your Helm configuration. After adding the repository, it’s good practice to update your local Helm cache to ensure you have the latest chart versions available. Finally, we’ll install OpenCost into its own dedicated namespace.

# Add the OpenCost Helm repository
helm repo add opencost https://opencost.github.io/opencost-helm-chart/
helm repo update

# Install OpenCost into a new namespace called 'opencost'
# We use --create-namespace to automatically create the namespace if it doesn't exist.
helm install opencost opencost/opencost -n opencost --create-namespace

Verify Installation

After running the installation command, you should see output indicating that OpenCost has been deployed. You can verify the successful deployment by checking the pods in the opencost namespace. Look for a running pod named opencost-xxxx.

kubectl get pods -n opencost

Expected Output:

NAME                               READY   STATUS    RESTARTS   AGE
opencost-7b9f8f5c9-abcde           1/1     Running   0          2m

This shows that the OpenCost pod is up and running, ready to start collecting cost data.

Step 2: Access the OpenCost UI

Once OpenCost is deployed, you’ll want to access its web-based user interface to visualize your cost data. The simplest way to do this for initial setup and testing is by using kubectl port-forward. This command creates a secure tunnel from your local machine to the OpenCost pod, allowing you to access its UI directly through your browser without exposing it publicly.

First, find the name of your OpenCost pod. Then, forward a local port (e.g., 9090) to the container port (also 9090) where OpenCost serves its UI. After executing the command, open your web browser and navigate to http://localhost:9090.

# Get the OpenCost pod name
OPENCOST_POD=$(kubectl get pods -n opencost -l app.kubernetes.io/name=opencost -o jsonpath='{.items[0].metadata.name}')

# Port-forward to the OpenCost UI
kubectl port-forward -n opencost $OPENCOST_POD 9090:9090

Verify UI Access

Open your browser and go to http://localhost:9090. You should see the OpenCost dashboard. Initially, it might show default or estimated prices if you haven’t configured cloud provider integration. This is normal.

Expected Output: A functional OpenCost UI displaying initial cost metrics. It will look something like this:

Step 3: Configure Cloud Provider Integration (AWS Example)

For OpenCost to provide accurate, real-time cost data based on your actual cloud bill, it needs to integrate with your cloud provider’s billing system. This step is crucial for moving beyond estimated prices to precise cost allocation. We’ll focus on AWS here, but similar principles apply to GCP and Azure.

For AWS, the recommended approach is to use a Cross-Account IAM Role. This involves creating an IAM role in your AWS billing account that grants read-only access to your Cost and Usage Report (CUR) and other necessary billing data. You then configure OpenCost with the ARN of this role. OpenCost assumes this role to fetch the billing data. Ensure you have enabled the AWS Cost and Usage Report in your AWS console and configured it to deliver reports to an S3 bucket.

Detailed AWS Setup:

1. Enable Cost and Usage Report (CUR):
   • Go to the AWS Billing console.
   • Navigate to “Cost & Usage Reports”.
   • Create a new report, ensuring it includes resource IDs and is delivered to an S3 bucket (e.g., opencost-cur-bucket-12345). Note the S3 bucket name and the report path prefix.
2. Create an IAM Policy: This policy grants OpenCost read access to the S3 bucket where your CUR is stored.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::opencost-cur-bucket-12345",
                "arn:aws:s3:::opencost-cur-bucket-12345/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "athena:*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetPartition"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::opencost-cur-bucket-12345"
        }
    ]
}

3. Create an IAM Role:
   • Go to IAM -> Roles -> Create role.
   • Select “AWS account” for trusted entity, then “Another AWS account”.
   • Enter your Kubernetes cluster’s AWS account ID.
   • Attach the policy created in step 2.
   • Add a trust policy that allows the OpenCost service account in your Kubernetes cluster to assume this role. This will typically involve an OIDC provider for your EKS/GKE/AKS cluster.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_K8S_ACCOUNT_ID:oidc-provider/oidc.eks.YOUR_REGION.amazonaws.com/id/EXAMPLED539D4633E53BB260DAF08800000000"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.YOUR_REGION.amazonaws.com/id/EXAMPLED539D4633E53BB260DAF08800000000:sub": "system:serviceaccount:opencost:opencost"
        }
      }
    }
  ]
}

Replace placeholders like YOUR_K8S_ACCOUNT_ID, YOUR_REGION, and the OIDC provider URL with your actual values. The opencost service account is created by the Helm chart.

4. Update OpenCost Helm Chart: Now, re-install or upgrade OpenCost, providing the necessary cloud provider details.

# Create a values.yaml file for OpenCost configuration
cat < opencost-values.yaml
opencost:
  cloudProvider:
    aws:
      athena:
        bucket: "opencost-cur-bucket-12345" # Your CUR S3 bucket
        region: "us-east-1" # Region of your CUR S3 bucket
        externalRoleArn: "arn:aws:iam::YOUR_BILLING_ACCOUNT_ID:role/OpenCostCURAccessRole" # ARN of the role created above
        reportName: "YourCURReportName" # The name of your CUR report
      spotData:
        enabled: true # Fetch spot instance pricing
EOF

# Upgrade OpenCost with the new configuration
helm upgrade opencost opencost/opencost -n opencost -f opencost-values.yaml

Verify Cloud Integration

After upgrading, check the OpenCost pod logs for messages indicating successful connection to AWS billing data. It might take some time for the data to be fetched and processed.

kubectl logs -n opencost $(kubectl get pods -n opencost -l app.kubernetes.io/name=opencost -o jsonpath='{.items[0].metadata.name}') | grep "AWS"

Expected Output (example):

I0825 10:30:00.123456       1 aws.go:123] Successfully fetched AWS pricing data.
I0825 10:30:05.678901       1 aws.go:456] AWS Cost and Usage Report "YourCURReportName" found in S3 bucket "opencost-cur-bucket-12345"

Once integrated, the OpenCost UI will display more accurate costs based on your actual cloud provider rates. This integration is vital for cost optimization efforts, as it provides the real-world financial impact of your Kubernetes workloads.

Step 4: Explore Cost Data in the UI

With OpenCost installed and potentially integrated with your cloud provider, it’s time to explore the wealth of cost data it provides. The OpenCost UI is designed to be intuitive, allowing you to slice and dice your Kubernetes spend in various ways. You can analyze costs by namespace, controller, service, label, and even individual pod.

Reconnect to the OpenCost UI using kubectl port-forward if you’ve closed the session.

# Get the OpenCost pod name
OPENCOST_POD=$(kubectl get pods -n opencost -l app.kubernetes.io/name=opencost -o jsonpath='{.items[0].metadata.name}')

# Port-forward to the OpenCost UI
kubectl port-forward -n opencost $OPENCOST_POD 9090:9090

Navigate to http://localhost:9090 in your browser. You’ll see the main dashboard. Experiment with the filters:

• Date Range: Adjust the time period to see historical costs.
• Aggregate By: Group costs by Namespace, Controller, Label, Service, etc. This is incredibly powerful for understanding cost drivers.
• Cluster: If you have multiple clusters integrated, you can filter by cluster.

Pay attention to the “Cost by Namespace” view, which often highlights the most expensive applications or teams. You can drill down further into a specific namespace to see which deployments or services are consuming the most resources.

Verify Data Exploration

Interact with the UI to generate different cost views. For instance, aggregate by ‘Namespace’ and identify the top spenders. Then, click on a specific namespace to see its breakdown.

Expected Outcome: The UI should dynamically update to show cost breakdowns. You should be able to identify, for example, that your dev namespace spent $X last week, and within that, deployment my-app consumed $Y of the total.

Step 5: Integrate with Prometheus and Grafana (Optional but Recommended)

While the OpenCost UI is great for quick insights, for long-term monitoring, alerting, and custom dashboards, integrating OpenCost with Prometheus and Grafana is highly recommended. OpenCost exposes its metrics in a Prometheus-compatible format, allowing you to scrape this data and visualize it in Grafana alongside other operational metrics.

This setup provides a unified observability platform. For example, you can create Grafana dashboards that show cost alongside CPU utilization, memory usage, and network traffic, helping you correlate resource consumption with actual spend. This is particularly useful when combined with tools like eBPF Observability with Hubble for network visibility or general Kubernetes metrics.

Prometheus Setup (assuming you have Prometheus installed):

Add a scrape configuration to your Prometheus server to target the OpenCost service. The OpenCost Helm chart typically exposes a service named opencost on port 9003 for metrics.

# Example Prometheus scrape configuration for OpenCost
# Add this to your prometheus.yaml or a similar configmap
- job_name: 'opencost'
  scrape_interval: 1m
  static_configs:
    - targets: ['opencost.opencost.svc.cluster.local:9003'] # Adjust if your service name/namespace differs

Grafana Setup:

Once Prometheus is scraping OpenCost metrics, you can create a new Grafana dashboard. OpenCost provides a rich set of metrics, usually prefixed with kubecost_ (because OpenCost originated from Kubecost’s open-source components). You can import pre-built OpenCost dashboards from Grafana Labs or build your own using PromQL queries.

Common metrics to look for:

• kubecost_pod_cpu_allocation_hours
• kubecost_pod_memory_allocation_bytes_hours
• kubecost_pod_cost
• kubecost_namespace_cost

Verify Prometheus/Grafana Integration

Check Prometheus targets to ensure OpenCost is being scraped successfully. Then, in Grafana, create a new dashboard and add a panel querying a kubecost_ metric. You should see data points appearing.

# Access Prometheus UI (e.g., via port-forward) and navigate to Status -> Targets
# You should see 'opencost' listed as 'UP'

Expected Outcome: Grafana dashboard showing cost metrics over time, allowing for custom visualizations and alerting.

Production Considerations

Deploying OpenCost in a production environment requires careful planning beyond the basic installation. Here are key considerations:

• Cloud Provider Integration:
  • Security: Always use dedicated IAM roles/service accounts with the principle of least privilege. Grant only read-only access to billing data. Avoid hardcoding credentials.
  • Data Freshness: Understand the latency of your cloud provider’s billing reports. CUR reports, for instance, can have a delay of several hours. This impacts the real-time accuracy of OpenCost.
  • AWS CUR Recommendations: For AWS, ensure your CUR is configured for hourly granularity and includes resource IDs. Store it in a dedicated S3 bucket.
• Persistence:
  • OpenCost uses a local boltDB for some historical data and configuration. For production, consider configuring a persistent volume (e.g., using an AWS EBS volume, GCP Persistent Disk, or Azure Disk) for the OpenCost pod. This prevents data loss if the pod restarts or moves to a different node.
  • The Helm chart allows configuring opencost.persistentVolume.enabled: true and specifying storage class, size, etc.
• Resource Allocation:
  • Monitor OpenCost’s own resource consumption (CPU, memory). While generally lightweight, it can increase with the size and activity of your cluster. Set appropriate resource requests and limits for the OpenCost deployment.
• High Availability:
  • For mission-critical cost visibility, consider running OpenCost in a highly available setup. This might involve multiple replicas and shared storage, though OpenCost is generally designed as a single instance per cluster for simplicity. For multi-cluster environments, each cluster runs its own OpenCost instance.
• Access Control:
  • UI Access: Do not expose the OpenCost UI directly to the internet without proper authentication and authorization. Use an Ingress controller with authentication (e.g., OAuth2-Proxy) or a VPN/internal network access. For Kubernetes Gateway API users, you can configure similar security policies.
  • API Access: If integrating with other tools, secure the OpenCost API endpoints.
• Monitoring and Alerting:
  • Beyond just viewing dashboards, set up alerts in Grafana (or your preferred monitoring system) for sudden cost spikes, unexpected resource consumption, or if OpenCost fails to collect data.
• Cost Allocation Strategy:
  • Labels: Leverage Kubernetes labels consistently across your cluster. OpenCost can aggregate costs by any label, making them invaluable for allocating costs to teams, projects, or environments.
  • Unallocated Costs: Understand that some costs (e.g., shared services, cluster overhead) might not be directly attributable to a specific pod or namespace. OpenCost provides insights into these “unallocated” costs, which you’ll need to distribute or account for separately.
• Multi-Cluster Deployments:
  • If you operate multiple Kubernetes clusters, deploy OpenCost in each cluster. You can then aggregate this data centrally using custom dashboards in Grafana or by querying each OpenCost instance’s API.
• Integration with FinOps Tools:
  • OpenCost can feed into broader FinOps strategies and tools. Its API allows for programmatic access to cost data, enabling integration with internal billing systems or third-party FinOps platforms.

Troubleshooting

Here are some common issues you might encounter with OpenCost and their solutions:

1. Issue: OpenCost UI shows “Loading…” or “No data available.”

   Solution: This often happens right after deployment or if there’s an issue collecting data.

   1. Check Pod Status: Ensure the OpenCost pod is running:

      kubectl get pods -n opencost

   2. Check Pod Logs: Look for errors in the logs:

      kubectl logs -n opencost $(kubectl get pods -n opencost -l app.kubernetes.io/name=opencost -o jsonpath='{.items[0].metadata.name}')

      This can reveal issues with cloud provider credentials, S3 access, or internal processing.

   3. Wait: If it’s a fresh install, it might take a few minutes for initial data to be collected and processed.
2. Issue: Costs are displayed but seem generic or estimated, not reflecting actual cloud bill.

   Solution: This indicates that the cloud provider integration is either not configured or misconfigured.

   1. Review Cloud Provider Setup: Double-check your AWS CUR, IAM policy, IAM role, and trust policy. Ensure the S3 bucket, report name, and external role ARN in your opencost-values.yaml are correct.
   2. Check OpenCost Logs for Cloud Errors: Look for errors related to AWS, GCP, or Azure in the logs. For AWS, search for “AWS” or “CUR”.
   3. Verify S3 Bucket Access: Ensure the IAM role has correct permissions to s3:GetObject and s3:ListBucket on your CUR S3 bucket.
3. Issue: kubectl port-forward fails or UI is inaccessible.

   Solution:

   1. Correct Pod Name: Ensure you’re using the correct OpenCost pod name.
   2. Port Conflict: Check if local port 9090 is already in use:

      lsof -i :9090

      If so, use a different local port (e.g., kubectl port-forward -n opencost $OPENCOST_POD 8080:9090).

   3. Network Policies: If you have strict Kubernetes Network Policies, ensure they allow traffic to the OpenCost pod on port 9090 from within the cluster.
4. Issue: OpenCost pod is in a CrashLoopBackOff state.

   Solution: This usually points to a critical configuration error or resource issue.

   1. Check Pod Logs: This is the first step. The logs will almost certainly contain the error message causing the crash.
   2. Resource Limits: If OpenCost runs out of memory or CPU, it can crash. Try increasing resource limits in your Helm values:

      
      opencost:
        resources:
          requests:
            cpu: 200m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
                      

   3. Configuration Errors: A malformed opencost-values.yaml or incorrect environment variables can cause crashes. Review your Helm upgrade command and the values file carefully.
5. Issue: Prometheus is not scraping OpenCost metrics.

   Solution:

   1. Prometheus Configuration: Verify your Prometheus scrape_configs. Ensure the job name and target address (opencost.opencost.svc.cluster.local:9003) are correct.
   2. Service and Port: Confirm the OpenCost service exists and exposes port 9003:

      kubectl get svc -n opencost opencost

   3. Network Connectivity: Ensure Prometheus can reach the OpenCost service. Check network policies or firewall rules. If Prometheus is in a different namespace, ensure cross-namespace service discovery is working. For advanced networking, consider tools like Cilium WireGuard Encryption for secure inter-service communication.
   4. Prometheus UI: Check the “Targets” page in the Prometheus UI (usually at http://prometheus-ip:9090/targets) to see the status of the OpenCost target.
6. Issue: Missing or incorrect pricing for specific resources (e.g., GPUs, specific instance types).

   Solution:

   1. Cloud Provider Integration: Re-verify that your cloud provider integration is fully functional and pulling accurate pricing data. Some specific pricing might require additional permissions or specific CUR configurations. For LLM GPU Scheduling, accurate GPU pricing is critical.
   2. Spot Instance Pricing: If using spot instances, ensure opencost.cloudProvider.aws.spotData.enabled: true (or equivalent for other clouds) is set in your Helm values.
   3. Manual Price Adjustments: OpenCost allows for manual price adjustments via its configuration. If a specific resource’s price isn’t being pulled correctly, you can manually set it in the Helm chart values.

FAQ Section

1. What is OpenCost and why do I need it?

   OpenCost is a CNCF-backed open-source project that provides real-time cost visibility and allocation for Kubernetes workloads. You need it to understand where your Kubernetes cloud spend is going, break down costs by team, application, or environment, and identify opportunities for optimization. Without it, managing cloud costs in Kubernetes can be a black box, leading to inefficiencies and unexpected bills.

2. How does OpenCost get its pricing data?

   OpenCost can get pricing data in two main ways:

   1. Estimated Pricing: By default, it uses publicly available cloud provider pricing APIs to estimate the cost of CPU, memory, and storage.
   2. Actual Cloud Billing Data: For accurate costs, it integrates directly with your cloud provider’s billing system (e.g., AWS Cost and Usage Reports, GCP Billing Export, Azure Cost Management). This requires configuring appropriate IAM roles/service accounts with read-only access to your billing data.
3. Is OpenCost the same as Kubecost?

   OpenCost is an open-source project that originated from Kubecost. Kubecost is a commercial product that builds upon the OpenCost core, offering additional enterprise features like advanced reporting, budgeting, anomaly detection, and integrations. OpenCost provides the foundational cost allocation capabilities as a vendor-neutral standard.

4. Can OpenCost track costs across multiple Kubernetes clusters?

   Yes, but typically you would deploy an OpenCost instance in each Kubernetes cluster. Each instance would monitor and report on its respective cluster’s costs. You can then aggregate this data externally, for example, by scraping metrics from all OpenCost instances into a central Prometheus and visualizing them in a multi-cluster Grafana dashboard. This approach helps in managing costs for complex environments, similar to how Istio Ambient Mesh can manage traffic across multiple clusters.

5. How accurate is OpenCost’s cost allocation?

   The accuracy of OpenCost depends heavily on its integration with your cloud provider’s actual billing data. When properly configured with access to your Cost and Usage Reports (CURs), it can provide highly accurate, granular cost breakdowns. Without this integration, it relies on estimated public pricing, which may not perfectly reflect your negotiated rates, discounts, or specific instance types. Consistent use of Kubernetes labels also significantly improves the granularity and accuracy of cost attribution to specific teams or applications.

Cleanup Commands

If you need to remove OpenCost from your cluster, you can do so easily using Helm:

# Uninstall OpenCost
helm uninstall opencost -n opencost

# Delete the OpenCost namespace
kubectl delete namespace opencost

# If you created persistent volumes, you might need to manually delete them
# (e.g., for AWS EBS, GCP PD, Azure Disk) if they were not automatically reclaimed.
# Example for PVCs:
kubectl get pvc -n opencost # Identify PVCs
kubectl delete pvc  -n opencost

Remember to also clean up any cloud provider resources you created for integration (IAM roles, policies, S3 buckets, billing reports) to avoid ongoing costs or security concerns.

Next Steps / Further Reading

You’ve successfully deployed OpenCost and started your journey toward better Kubernetes cost management. Here are some next steps and resources for deeper exploration:

• Explore OpenCost Documentation: Dive deeper into the official OpenCost documentation for advanced configurations, API usage, and more detailed cloud provider integrations.
• Implement Cost Optimization Strategies: Use the insights from OpenCost to inform your optimization efforts. This could involve right-sizing workloads, optimizing resource requests/limits, identifying idle resources, or leveraging spot instances. Check out our guide on Karpenter Cost Optimization for automating node provisioning and reducing costs.
• Integrate with Other Observability Tools: Combine OpenCost data with other observability tools like Prometheus, Grafana, and tracing systems to get a holistic view of performance and cost. For network-level insights, consider eBPF Observability with Hubble.
• Leverage Kubernetes Labels: Develop a robust labeling strategy across your Kubernetes resources. Consistent labels are key to granular cost allocation by team, project, environment, or application.
• Explore FinOps Practices: Learn about FinOps principles and how to integrate cost management into your organizational culture. The FinOps Foundation is a great resource.
• Advanced Security with Sigstore and Kyverno: While not directly related to cost, ensuring the security of your supply chain can prevent costly breaches. Learn more about Securing Container Supply Chains with Sigstore and Kyverno.

Conclusion

Managing costs in Kubernetes is no longer an optional task but a critical component of successful cloud-native operations. OpenCost provides the transparency and granularity needed to transform abstract cloud bills into actionable insights. By deploying OpenCost, integrating it with your cloud provider, and actively using its data, you empower your teams to make data-driven decisions that lead to significant cost savings and improved resource efficiency. Embrace OpenCost as your standard for Kubernetes cost allocation and take control of your cloud spend, turning your Kubernetes clusters from cost centers into optimized and predictable environments.