Kubernetes Quotas & Limits: Master Them Now

Kubernetes Resource Quotas and LimitRanges Best Practices

Managing resources efficiently is paramount in any shared computing environment, and Kubernetes is no exception. Without proper controls, a single rogue application or misconfigured deployment can quickly consume all available cluster resources, leading to performance degradation, instability, and even complete outages for other critical workloads. This is where Kubernetes Resource Quotas and LimitRanges step in – they are your primary tools for enforcing resource constraints and ensuring fair usage across your cluster.

Imagine a multi-tenant Kubernetes cluster where different teams or applications share the same underlying infrastructure. Without a mechanism to limit CPU, memory, or storage consumption, one team’s resource-hungry application could starve others, leading to an unfair distribution of resources and potential service disruptions. Resource Quotas provide a way to define hard limits on resource consumption per namespace, acting as a fence to prevent any single tenant from monopolizing the cluster. Complementing this, LimitRanges ensure that individual Pods within a namespace are provisioned with appropriate resource requests and limits, preventing them from running wild or failing due to underspecification. Together, they form a robust resource governance strategy, crucial for maintaining a healthy, performant, and cost-effective Kubernetes environment.

Prerequisites

Before diving into the practical implementation of Resource Quotas and LimitRanges, ensure you have the following:

• A Kubernetes Cluster: Access to a running Kubernetes cluster (e.g., Minikube, kind, or a cloud-managed cluster like GKE, EKS, AKS). For local development, Minikube or kind are excellent choices.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. You can find installation instructions in the official Kubernetes documentation.
• Basic Kubernetes Knowledge: Familiarity with core Kubernetes concepts like Pods, Deployments, Namespaces, and resource requests/limits.
• Text Editor: Any text editor for creating YAML files (e.g., VS Code, Vim, Nano).

Step-by-Step Guide: Implementing Resource Quotas and LimitRanges

This guide will walk you through setting up and verifying both Resource Quotas and LimitRanges in a Kubernetes cluster.

Step 1: Create a Dedicated Namespace

It’s a best practice to apply Resource Quotas and LimitRanges at the namespace level. This isolates resource management for different teams or applications. Let’s create a new namespace for our demonstration.

Namespaces provide a logical separation for resources within your cluster, making it easier to manage and enforce policies. Creating a dedicated namespace for our examples ensures that our resource constraints only apply to the resources within that specific boundary, preventing unintended side effects on other parts of your cluster. This also aligns with multi-tenancy best practices.

kubectl create namespace dev-team-a

Verify:
Confirm that the namespace has been created successfully.

kubectl get namespaces

Expected Output:

NAME              STATUS   AGE
default           Active   ...
kube-system       Active   ...
kube-public       Active   ...
kube-node-lease   Active   ...
dev-team-a        Active   ...

Step 2: Define and Apply a ResourceQuota

A ResourceQuota object defines aggregate resource limits for a namespace. We’ll create a quota that limits the total CPU, memory, and persistent volume claims within our dev-team-a namespace.

Resource Quotas are critical for preventing resource exhaustion and ensuring fair sharing among different teams or applications. By setting hard limits on total CPU, memory, and even the number of objects (like Pods, Services, or Deployments), you establish a clear boundary for resource consumption. This helps in capacity planning and prevents any single namespace from consuming all available cluster resources. For advanced networking policies that might complement resource management, consider exploring our Kubernetes Network Policies: Complete Security Hardening Guide.

# resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: dev-quota
spec:
  hard:
    requests.cpu: "1"        # Total CPU requested by all Pods
    requests.memory: "1Gi"   # Total Memory requested by all Pods
    limits.cpu: "2"          # Total CPU limits for all Pods
    limits.memory: "2Gi"     # Total Memory limits for all Pods
    pods: "10"               # Maximum number of Pods
    persistentvolumeclaims: "2" # Maximum number of PVCs
    storage.kubernetes.io/persistentvolumeclaims: "5Gi" # Total storage for PVCs

Apply this ResourceQuota to the dev-team-a namespace:

kubectl apply -f resource-quota.yaml -n dev-team-a

Verify:
Check the status of the ResourceQuota. This will show the limits and current usage.

kubectl describe quota dev-quota -n dev-team-a

Expected Output (usage will be 0 initially):

Name:           dev-quota
Namespace:      dev-team-a
Resource        Used    Hard
--------        ----    ----
limits.cpu      0       2
limits.memory   0       2Gi
pods            0       10
persistentvolumeclaims 0       2
requests.cpu    0       1
requests.memory 0       1Gi
storage.kubernetes.io/persistentvolumeclaims 0       5Gi

Step 3: Define and Apply a LimitRange

A LimitRange object enforces default resource requests and limits for Pods and Containers within a namespace, and can also specify minimum and maximum values.

LimitRanges are crucial for ensuring that every Pod and Container in a namespace has appropriate resource requests and limits defined. This prevents “runaway” containers that might consume excessive resources, and also ensures that Pods are scheduled correctly by the Kubernetes scheduler. Without LimitRanges, Pods might get scheduled without any resource requests, making it difficult for the scheduler to make informed decisions, potentially leading to oversubscription or inefficient node utilization. They also prevent users from deploying Pods that request absurdly small or large amounts of resources, maintaining a baseline quality of service.

# limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: dev-limits
spec:
  limits:
  - type: Pod
    max:
      cpu: "1"
      memory: "1Gi"
    min:
      cpu: "100m"
      memory: "100Mi"
  - type: Container
    defaultRequest:
      cpu: "200m"
      memory: "256Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    max:
      cpu: "800m"
      memory: "800Mi"
    min:
      cpu: "50m"
      memory: "64Mi"
    maxLimitRequestRatio:
      cpu: "4" # Limit cannot be more than 4x the request
      memory: "2" # Limit cannot be more than 2x the request

Apply this LimitRange to the dev-team-a namespace:

kubectl apply -f limit-range.yaml -n dev-team-a

Verify:
Check the status of the LimitRange.

kubectl describe limitrange dev-limits -n dev-team-a

Expected Output:

Name:       dev-limits
Namespace:  dev-team-a
Type        Resource        Min     Max     Default Request Default Limit MaxLimitRequestRatio
----        --------        ---     ---     --------------- ------------- --------------------
Pod         cpu             100m    1       -               -             -
Pod         memory          100Mi   1Gi     -               -             -
Container   cpu             50m     800m    200m            500m          4
Container   memory          64Mi    800Mi   256Mi           512Mi         2

Step 4: Deploy an Application to Test ResourceQuota and LimitRange

Now, let’s deploy a simple Nginx application without explicitly defining resource requests and limits. The LimitRange should automatically inject these values.

This step demonstrates the power of LimitRanges. By deploying a standard application without explicit resource definitions, we can observe how the LimitRange we configured automatically applies default CPU and memory requests and limits to the containers. This ensures that even applications not explicitly configured for resource management still adhere to the namespace’s resource policies, contributing to overall cluster stability and predictability.

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

Deploy the application:

kubectl apply -f nginx-deployment.yaml -n dev-team-a

Verify:
Inspect the deployed Pods to see the injected resource requests and limits.

kubectl get pods -n dev-team-a

Expected Output:

NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-7b9c7b649d-abcde   1/1     Running   0          1m
nginx-deployment-7b9c7b649d-fghij   1/1     Running   0          1m

Now, describe one of the Pods to see its resource configuration:

POD_NAME=$(kubectl get pods -n dev-team-a -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl describe pod $POD_NAME -n dev-team-a | grep -E "Requests:|Limits:" -A 3

Expected Output (showing injected values from LimitRange):

    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     200m
      memory:  256Mi

Also, check the ResourceQuota usage again:

kubectl describe quota dev-quota -n dev-team-a

Expected Output (usage updated):

Name:           dev-quota
Namespace:      dev-team-a
Resource        Used    Hard
--------        ----    ----
limits.cpu      1 (500m * 2 pods)       2
limits.memory   1Gi (512Mi * 2 pods)    2Gi
pods            2                       10
persistentvolumeclaims 0                2
requests.cpu    400m (200m * 2 pods)    1
requests.memory 512Mi (256Mi * 2 pods)  1Gi
storage.kubernetes.io/persistentvolumeclaims 0       5Gi

Step 5: Test ResourceQuota Limits

Let’s try to deploy more Pods than allowed by our ResourceQuota, or deploy a Pod that requests resources exceeding the total quota.

This step is crucial for understanding how Resource Quotas enforce their limits. By deliberately attempting to exceed the defined boundaries, we can observe the admission controller in action, rejecting deployments that would violate the quota. This validation mechanism prevents over-provisioning and ensures that your cluster’s resources remain within the predefined limits, critical for stability and predictable performance. It’s a direct way to see the “hard” limits in practice.

First, try to deploy a third Nginx Pod. Our current quota for pods is 10, and we have 2 running. This should succeed. Let’s try to deploy a deployment that would exceed the CPU or Memory quota.

Let’s create a new deployment that would push us over the memory limit (2Gi total for limits.memory). Each Nginx pod uses 512Mi, so 4 pods would use 2Gi. If we try to deploy a 5th, it should fail.

Let’s scale our existing deployment to 4 replicas:

kubectl scale deployment nginx-deployment --replicas=4 -n dev-team-a

Verify:
Check the quota usage. It should now be at the limit for memory.

kubectl describe quota dev-quota -n dev-team-a

Expected Output (showing memory limits.memory at 2Gi):

Name:           dev-quota
Namespace:      dev-team-a
Resource        Used    Hard
--------        ----    ----
limits.cpu      2       2
limits.memory   2Gi     2Gi
pods            4       10
persistentvolumeclaims 0       2
requests.cpu    800m    1
requests.memory 1Gi     1Gi
storage.kubernetes.io/persistentvolumeclaims 0       5Gi

Now, try to scale to 5 replicas. This will attempt to allocate an additional 512Mi memory limit, exceeding the 2Gi quota.

kubectl scale deployment nginx-deployment --replicas=5 -n dev-team-a

Expected Output (error):

error: deployments "nginx-deployment" was not scaled
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [dev-quota] Exceeded quota for: limits.memory, requested: 2560Mi, used: 2048Mi, limited: 2048Mi

Note: The error message might vary slightly depending on your Kubernetes version or if you have admission controllers like Gatekeeper installed. The core message will indicate a quota violation.

Step 6: Test LimitRange Max/Min and MaxLimitRequestRatio

Let’s try to deploy a Pod that violates the max or min constraints set by the LimitRange, or the maxLimitRequestRatio.

This step highlights the granular control provided by LimitRanges over individual Pod and Container resources. By attempting to deploy a Pod with resource specifications that fall outside the defined `min`, `max`, or `maxLimitRequestRatio` values, we can see how the LimitRange acts as a gatekeeper. This ensures that no single container consumes an unreasonable amount of resources (preventing “noisy neighbors”) or requests too little (which might lead to OOMKills or poor performance), thereby maintaining cluster health and application stability. This granular enforcement is key to consistent performance.

First, let’s try to deploy a Pod that requests CPU below the min (50m) for a container:

# pod-too-small.yaml
apiVersion: v1
kind: Pod
metadata:
  name: tiny-pod
spec:
  containers:
  - name: tiny-container
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    resources:
      requests:
        cpu: "10m" # Too small, min is 50m
      limits:
        cpu: "20m"
        memory: "64Mi"

kubectl apply -f pod-too-small.yaml -n dev-team-a

Expected Output (error):

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [dev-limits] CPU request (10m) for container tiny-container is less than the minimum allowed (50m)

Next, let’s try a Pod that requests CPU above the max (800m) for a container:

# pod-too-large.yaml
apiVersion: v1
kind: Pod
metadata:
  name: huge-pod
spec:
  containers:
  - name: huge-container
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    resources:
      requests:
        cpu: "900m" # Too large, max is 800m
      limits:
        cpu: "1"
        memory: "512Mi"

kubectl apply -f pod-too-large.yaml -n dev-team-a

Expected Output (error):

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [dev-limits] CPU request (900m) for container huge-container is greater than the maximum allowed (800m)

Finally, let’s test the maxLimitRequestRatio for CPU. Our ratio is 4. If we request 100m, the limit cannot exceed 400m. Let’s try 100m request with 500m limit.

# pod-bad-ratio.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bad-ratio-pod
spec:
  containers:
  - name: bad-ratio-container
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    resources:
      requests:
        cpu: "100m"
      limits:
        cpu: "500m" # Exceeds maxLimitRequestRatio (4 * 100m = 400m)
        memory: "200Mi"

kubectl apply -f pod-bad-ratio.yaml -n dev-team-a

Expected Output (error):

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [dev-limits] CPU limit (500m) for container bad-ratio-container is greater than the maximum allowed (400m) based on the request (100m) and maxLimitRequestRatio (4)

Production Considerations

Implementing Resource Quotas and LimitRanges in production environments requires careful planning and continuous monitoring. Here are key considerations:

1. Granularity and Scope:

   Decide on the appropriate level of granularity for your quotas. Should they be per team, per application, or per environment? A common approach is to have quotas per team/project namespace. For organizations managing costs, integrating with tools like Karpenter for Kubernetes Cost Optimization can further enhance resource efficiency by ensuring nodes are provisioned optimally based on actual demands.

2. Right-Sizing Applications:

   Encourage development teams to accurately define resource requests and limits for their applications. Over-requesting leads to resource waste and higher costs, while under-requesting can cause performance issues (throttling, OOMKills) and instability. Use historical data from monitoring tools to right-size. Tools like Metrics Server, Prometheus, and Grafana are invaluable here. For advanced observability, consider leveraging eBPF Observability: Building Custom Metrics with Hubble.

3. Monitoring and Alerting:

   Actively monitor ResourceQuota usage. Set up alerts when namespaces approach their defined limits (e.g., 80% or 90% utilization). This allows teams to proactively adjust their quotas or optimize their applications before hitting hard limits and causing outages. Metrics such as kube_resourcequota_hard and kube_resourcequota_used from kube-state-metrics are essential for this.

4. Admission Controllers:

   Resource Quotas and LimitRanges are enforced by admission controllers. Ensure these are enabled in your cluster (they are by default in most managed Kubernetes services). For more advanced policy enforcement, consider using policy engines like Kyverno or Gatekeeper, which can provide more flexible and granular controls than native LimitRanges, such as enforcing specific label requirements or image registries, similar to how Sigstore and Kyverno secure supply chains.

5. Impact on Scheduling:

   Resource requests are crucial for the Kubernetes scheduler. Without them, Pods might be scheduled on nodes with insufficient resources, leading to poor performance. LimitRanges ensure that even Pods without explicit requests get default values, aiding the scheduler. For specific workloads like AI/ML, understanding how resource requests influence node selection is critical, especially when dealing with GPUs, as discussed in our LLM GPU Scheduling Guide.

6. Preventing Quota Exhaustion for Critical Services:

   Be mindful of critical infrastructure components (e.g., monitoring agents, DNS, ingress controllers) that might run in general-purpose namespaces. Exclude them from strict quotas or place them in their own dedicated, less constrained namespaces. Alternatively, consider using Guaranteed QoS classes for these critical Pods by setting requests equal to limits.

7. Handling Persistent Storage:

   Resource Quotas can also limit PersistentVolumeClaims (PVCs) and total storage capacity. This is vital for managing costs, especially with cloud provider storage. Be sure to define appropriate limits for persistentvolumeclaims and storage.<storage-class-name>/persistentvolumeclaims (or storage.kubernetes.io/persistentvolumeclaims for any storage class).

8. Iterative Refinement:

   Start with reasonable, perhaps slightly generous, quotas and LimitRanges, then gradually tighten them based on observed usage patterns and application requirements. This iterative approach minimizes disruption and allows teams to adapt.

Troubleshooting

Here are some common issues you might encounter when working with Resource Quotas and LimitRanges, along with their solutions:

1. Issue: Pods Pending with “FailedScheduling” due to “QuotaExceeded”

   Description: You try to deploy a Pod, but it stays in a Pending state, and kubectl describe pod shows an event indicating “QuotaExceeded” or “resource quota exceeded”.

   Solution: This means your namespace has hit one of its ResourceQuota limits (e.g., pods, requests.cpu, limits.memory).

   1. Check the ResourceQuota usage:

      kubectl describe quota -n <namespace>

   2. Identify which resource is exhausted (Used equals Hard).
   3. Options:
      • Delete existing, unnecessary Pods or deployments in the namespace.
      • Scale down existing deployments.
      • Increase the ResourceQuota limits (requires editing the ResourceQuota object).
      • Optimize your application’s resource requests/limits to consume less.
2. Issue: Pods Failing Creation with “Forbidden” due to LimitRange Violation

   Description: When creating a Pod or Deployment, you get an immediate “Forbidden” error stating that a resource request/limit is outside the allowed range (e.g., “CPU request (10m) for container… is less than the minimum allowed (50m)”).

   Solution: This indicates that the Pod’s (or its containers’) explicit or implicit resource requests/limits violate a LimitRange policy.

   1. Check the LimitRange definition in the namespace:

      kubectl describe limitrange -n <namespace>

   2. Compare the Pod’s resource specifications with the Min, Max, Default Request, Default Limit, and MaxLimitRequestRatio values.
   3. Options:
      • Adjust the Pod/Deployment’s resource requests/limits to fall within the LimitRange.
      • If the LimitRange is too restrictive, modify the LimitRange object.
3. Issue: Pods are created, but resources are not as expected (LimitRange defaults not applied).

   Description: You deployed a Pod without explicit resource requests/limits, but after inspection (kubectl describe pod), you see no defaults were applied from the LimitRange.

   Solution: Ensure the LimitRange is correctly applied to the same namespace where the Pod is being deployed.

   1. Verify the LimitRange exists in the namespace:

      kubectl get limitrange -n <namespace>

   2. Double-check the LimitRange YAML for syntax errors or incorrect type (e.g., Pod vs. Container).
   3. Ensure the Pod does not explicitly define resource requests/limits if you expect the defaults to be injected. Explicit values always override LimitRange defaults.
4. Issue: Applications experiencing OOMKills or CPU Throttling despite having limits.

   Description: Your application containers are getting terminated with OOMKilled status or showing signs of CPU throttling, even though you have set resource limits.

   Solution: This often means your resource limits are set too low for the application’s actual needs.

   1. OOMKills: The container tried to use more memory than its limits.memory.
      • Increase limits.memory for the container.
      • Optimize the application to use less memory.
   2. CPU Throttling: The container tried to use more CPU than its limits.cpu.
      • Increase limits.cpu for the container.
      • Optimize the application to use less CPU.
   3. Use monitoring tools (Prometheus, Grafana) to observe actual CPU and memory usage patterns of your application over time to determine appropriate limits.
5. Issue: ResourceQuota updates are not taking effect.

   Description: You modified a ResourceQuota, but new Pods are still being blocked by the old limits, or the Used values don’t reflect recent changes.

   Solution:

   1. Ensure you applied the updated YAML correctly:

      kubectl apply -f updated-quota.yaml -n <namespace>

   2. Verify the change using kubectl describe quota <quota-name> -n <namespace>. Make sure the Hard limits reflect your changes.
   3. ResourceQuota changes are typically immediate. If you still see issues, ensure there are no other admission controllers or policies (like OPA Gatekeeper or Kyverno) that might be overriding or interfering with the native Kubernetes ResourceQuota.
   4. If the Used values are not updating, it might be a temporary delay in the Kubernetes API server or controller manager, but this is rare. Restarting the affected Pods might trigger a re-evaluation, but usually, it’s not necessary.

FAQ Section

1. What is the difference between Resource Quotas and LimitRanges?

   Resource Quotas set aggregate resource limits for an entire namespace (e.g., “this namespace can use a maximum of 2 CPUs and 4Gi memory across all its Pods”). They prevent a namespace from consuming too many cluster resources. LimitRanges, on the other hand, define default resource requests/limits for individual Pods and Containers within a namespace, and also enforce minimum and maximum values for those resources (e.g., “any container in this namespace must request at least 50m CPU and no more than 1 CPU”). They ensure individual Pods are well-behaved.

2. Can I use Resource Quotas and LimitRanges together?

   Yes, absolutely, and it’s highly recommended! They complement each other perfectly. LimitRanges ensure that all Pods have sensible resource requests/limits, which then contribute to the overall usage tracked by the ResourceQuota. Without LimitRanges, Pods might not have requests/limits, making it harder for the scheduler and potentially leading to unexpected ResourceQuota violations when defaults are applied by other mechanisms (or no limits at all, leading to resource starvation).

3. What happens if a Pod’s resource requests/limits violate a LimitRange or ResourceQuota?

   If a Pod (or its containers) attempts to request or limit resources outside the boundaries defined by a LimitRange, the Pod creation will be immediately rejected by the Kubernetes API server with a “Forbidden” error. If a Pod would cause the total resource consumption of a namespace to exceed a ResourceQuota, its creation will also be rejected with a “Forbidden” or “QuotaExceeded” error. Kubernetes uses Admission