Kubernetes GPU Resource Management Best Practices: A Complete Guide

Managing GPU resources in Kubernetes clusters has become increasingly critical as machine learning, deep learning, and AI workloads continue to dominate modern computing environments. Graphics Processing Units (GPUs) are expensive hardware resources that require careful orchestration to maximize utilization and minimize costs. This comprehensive guide explores the best practices for managing GPU resources in Kubernetes, helping you optimize performance while avoiding common pitfalls.

Understanding GPU Resources in Kubernetes

Kubernetes treats GPUs as extended resources, which means they’re managed differently from standard compute resources like CPU and memory. When you deploy GPU-enabled nodes in your cluster, the kubelet automatically discovers NVIDIA GPUs and exposes them as schedulable resources. This discovery happens through device plugins that implement the Kubernetes device plugin framework.

The most common implementation uses the NVIDIA GPU Operator or the legacy NVIDIA device plugin. These components handle the communication between Kubernetes and the underlying GPU hardware, making GPUs available to containerized workloads. Understanding this architecture is fundamental to implementing effective GPU resource management strategies.

Why Kubernetes Runs Better on GPUs

Kubernetes includes support for GPUs, making it easy to configure and use GPU resources for accelerating workloads such as data science, machine learning, and deep learning. Device plug-ins enable pods access to specialized hardware features such as GPUs and expose them as schedulable resources.  

With the increasing number of AI-powered applications and services and the broad availability of GPUs in public cloud, there’s an increasing need for Kubernetes to be GPU-aware. NVIDIA has been steadily building its library of software to optimize GPUs to use in a container environment. For example, Kubernetes on NVIDIA GPUs enables multi-cloud GPU clusters to be scaled seamlessly with automated deployment, maintenance, scheduling, and operation of GPU-accelerated containers across multi-node clusters. 

Installing and Configuring GPU Support

Before implementing GPU resource management practices, you need to ensure your cluster properly supports GPU workloads. The NVIDIA GPU Operator simplifies this process by managing all necessary components including drivers, container runtime configurations, and device plugins.

To install the GPU Operator using Helm, you can use the following approach:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator

Once installed, verify that your GPU nodes are properly labeled and that GPUs are being advertised as allocatable resources. You can check this by inspecting your nodes:

kubectl get nodes -o json | jq '.items[].status.allocatable'

Look for entries like nvidia.com/gpu in the allocatable resources section.

Resource Allocation and Pod Scheduling

The foundation of effective GPU resource management lies in properly requesting GPU resources in your pod specifications. Unlike CPU and memory, GPUs are discrete resources that cannot be fractionally allocated by default. When a pod requests a GPU, it receives exclusive access to one or more entire GPUs.

Here’s an example of a pod specification requesting GPU resources:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-pod
spec:
  containers:
  - name: tensorflow-training
    image: tensorflow/tensorflow:latest-gpu
    resources:
      limits:
        nvidia.com/gpu: 1
  nodeSelector:
    accelerator: nvidia-tesla-v100

This configuration requests one GPU and uses a node selector to ensure the pod lands on nodes with specific GPU models. This practice is crucial because different GPU models have varying capabilities and performance characteristics.

Implementing Resource Quotas and Limits

Resource quotas help prevent GPU hoarding and ensure fair distribution across namespaces and teams. When multiple teams share a Kubernetes cluster with limited GPU resources, implementing quotas becomes essential for maintaining operational efficiency.

Create a ResourceQuota object to limit GPU consumption per namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"

This quota ensures that the ml-team namespace cannot request more than four GPUs simultaneously. Combine this with LimitRanges to set default GPU requests and prevent users from creating pods without explicit GPU specifications.

GPU Sharing and Time-Slicing

Traditional GPU allocation in Kubernetes provides exclusive access to entire GPUs, which can lead to underutilization when workloads don’t fully saturate GPU capacity. NVIDIA’s time-slicing feature addresses this limitation by allowing multiple workloads to share a single GPU.

Time-slicing works by configuring the device plugin to advertise more GPU resources than physically exist. For example, you can configure a node with four physical GPUs to advertise eight virtual GPUs, allowing two workloads to time-share each physical GPU.

To enable time-slicing, create a ConfigMap for the GPU Operator:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  tesla-t4: |-
    version: v1
    sharing:
      timeSlicing:
        replicas: 2

This configuration allows two containers to share each Tesla T4 GPU. However, time-slicing introduces overhead and may not be suitable for latency-sensitive applications. Carefully evaluate your workload characteristics before implementing GPU sharing strategies.

Multi-Instance GPU (MIG) Configuration

For NVIDIA A100 and H100 GPUs, Multi-Instance GPU technology provides a more robust alternative to time-slicing. MIG partitions a single GPU into multiple isolated instances, each with dedicated memory and compute resources. This approach delivers better isolation and more predictable performance compared to time-slicing.

Configuring MIG requires careful planning because MIG profiles are static and must be configured at the node level. The GPU Operator supports MIG configuration through the following workflow:

First, define your desired MIG strategy in the ClusterPolicy:

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  mig:
    strategy: mixed
  migManager:
    enabled: true

The mixed strategy allows some GPUs to run in MIG mode while others operate in full GPU mode, providing flexibility for different workload requirements. After applying the MIG configuration, the device plugin automatically discovers and advertises the MIG devices as separate schedulable resources.

Monitoring GPU Utilization

Effective GPU resource management requires comprehensive monitoring to identify underutilization, bottlenecks, and optimization opportunities. The NVIDIA Data Center GPU Manager (DCGM) provides detailed metrics about GPU performance, temperature, memory usage, and error rates.

Integrate DCGM with Prometheus to collect GPU metrics:

apiVersion: v1
kind: ConfigMap
metadata:
  name: dcgm-exporter-config
data:
  default-counters.csv: |
    # GPU Utilization
    DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization
    # Memory Usage
    DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization
    DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used

Deploy the DCGM exporter as a DaemonSet on GPU nodes to expose these metrics to your monitoring stack. Create dashboards and alerts to track GPU utilization patterns and identify opportunities for optimization.

Scheduling Strategies and Node Affinity

Kubernetes scheduling decisions significantly impact GPU resource utilization. Implementing intelligent scheduling strategies ensures that GPU workloads land on appropriate nodes while maintaining cluster efficiency.

Use node affinity rules to guide pod placement based on GPU characteristics:

apiVersion: v1
kind: Pod
metadata:
  name: inference-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu-memory
            operator: Gt
            values:
            - "16000"
  containers:
  - name: model-server
    image: your-inference-image:latest
    resources:
      limits:
        nvidia.com/gpu: 1

This configuration ensures the pod only schedules on nodes with GPUs that have more than 16GB of memory, which is crucial for large model inference workloads.

Handling GPU Workload Priorities

In multi-tenant environments, different workloads have varying levels of importance. Implement PriorityClasses to ensure critical GPU workloads receive preferential treatment during resource contention:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-high-priority
value: 1000000
globalDefault: false
description: "High priority for production inference workloads"

Assign this priority class to production inference services while using lower priority classes for experimental training jobs. This approach allows the scheduler to preempt lower-priority pods when high-priority workloads need GPU resources.

Optimizing GPU Memory Management

GPU memory management presents unique challenges because, unlike system RAM, GPU memory errors often lead to immediate pod failures. Implement proactive memory management practices to prevent out-of-memory errors and improve stability.

Configure resource limits carefully to account for framework overhead. For example, PyTorch and TensorFlow maintain their own memory pools, which consume GPU memory beyond your model’s requirements. A good rule of thumb is to request 10-15% more GPU memory than your model theoretically needs.

Use the CUDA_VISIBLE_DEVICES environment variable to control GPU visibility within containers:

env:
- name: CUDA_VISIBLE_DEVICES
  value: "0"

This practice prevents multi-GPU libraries from attempting to use GPUs that aren’t allocated to the container.

Implementing Health Checks for GPU Pods

Standard Kubernetes liveness and readiness probes often don’t capture GPU-specific failures. Implement custom health checks that verify GPU accessibility and functionality:

livenessProbe:
  exec:
    command:
    - python
    - -c
    - |
      import torch
      assert torch.cuda.is_available()
      torch.cuda.get_device_properties(0)
  initialDelaySeconds: 30
  periodSeconds: 60

This probe ensures that CUDA remains accessible throughout the pod’s lifetime and catches GPU-specific failures that standard probes might miss.

Managing GPU Driver Updates

GPU drivers require periodic updates for security patches, performance improvements, and new feature support. However, driver updates on GPU nodes require careful coordination to avoid disrupting running workloads.

Use node cordoning and draining procedures to safely update drivers:

kubectl cordon gpu-node-01
kubectl drain gpu-node-01 --ignore-daemonsets --delete-emptydir-data

After updating drivers, uncordon the node to return it to service. The GPU Operator can automate this process through its upgrade mechanism, but always verify your specific workload requirements before enabling automatic driver updates.

Cost Optimization Strategies

GPU resources represent significant infrastructure costs. Implementing cost optimization strategies helps maximize return on investment without compromising workload performance.

Consider using node autoscaling for GPU workloads, but be aware that GPU nodes typically have longer startup times than CPU-only nodes. Configure appropriate scale-up and scale-down policies:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
data:
  priorities: |-
    10:
      - .*-gpu-spot-.*
    50:
      - .*-gpu-on-demand-.*

This configuration prioritizes spot GPU instances for cost savings while maintaining on-demand capacity for critical workloads.

Troubleshooting Common Issues

GPU resource management in Kubernetes involves several potential failure points. Understanding common issues and their solutions accelerates problem resolution.

When pods remain in a pending state despite available GPUs, verify that the device plugin is running correctly. Check the device plugin logs:

kubectl logs -n gpu-operator -l app=nvidia-device-plugin-daemonset

GPU memory fragmentation can prevent large model allocations even when total available memory appears sufficient. Restart pods periodically to defragment GPU memory or use MIG to provide isolated memory spaces.

If pods experience intermittent CUDA errors, verify driver compatibility between the host drivers and the CUDA version in your containers. The container CUDA version should match or be lower than the host driver version.

Implementing Namespace Isolation

For multi-tenant clusters, implement strict namespace isolation to prevent GPU resource conflicts. Combine ResourceQuotas, LimitRanges, and NetworkPolicies to create isolated execution environments:

apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limits
  namespace: ml-team
spec:
  limits:
  - max:
      nvidia.com/gpu: "2"
    min:
      nvidia.com/gpu: "1"
    type: Container

This LimitRange ensures that every GPU container in the namespace requests at least one GPU and cannot request more than two GPUs, preventing both resource waste and hoarding.

Future-Proofing Your GPU Strategy

The GPU landscape continues to evolve with new technologies like DPU (Data Processing Units), specialized AI accelerators, and improved virtualization capabilities. Design your GPU resource management strategy with flexibility in mind.

Use abstraction layers like the Kubernetes device plugin framework to avoid vendor lock-in. While NVIDIA GPUs dominate the current market, AMD, Intel, and custom accelerators are gaining traction. A well-designed resource management strategy adapts to new hardware types without requiring fundamental architectural changes.

Conclusion

Effective GPU resource management in Kubernetes requires a holistic approach that balances performance, cost, and operational complexity. By implementing these best practices—from proper resource allocation and quotas to monitoring and optimization strategies—you can maximize the value of your GPU infrastructure while maintaining operational excellence.

Start with fundamental practices like explicit resource requests and proper monitoring, then progressively implement advanced features like time-slicing, MIG, and intelligent scheduling as your requirements evolve. Regular performance audits and cost analysis ensure your GPU resource management strategy continues to deliver value as your workloads scale.