Kubernetes Dynamic Resource Allocation (DRA): A Hands-On GPU Tutorial

Kubernetes has scheduled GPUs the same way since 2016: a device plugin advertises nvidia.com/gpu: 8 on a node, you write limits: {nvidia.com/gpu: 1} in your pod spec, and the scheduler treats that GPU like it treats a slice of CPU — an anonymous, interchangeable integer.

That abstraction held up fine when a GPU was a GPU. It does not hold up in 2026, when a single cluster might contain A100s, H100s, L40S and GB200 superchips, when one team needs 80 GB of HBM and another needs a 1g.5gb MIG slice, and when a distributed training job needs its eight GPUs to land on the same NVSwitch domain or it will run at a third of the expected throughput.

Dynamic Resource Allocation (DRA) is the answer upstream Kubernetes settled on. It reached GA in Kubernetes v1.34, and as of v1.35 the DynamicResourceAllocation feature gate is locked on — you cannot turn it off. If you run GPUs on Kubernetes, DRA is no longer a thing to watch. It is the thing to learn.

This tutorial takes you from zero to a working DRA setup, using a mock driver that needs no GPU hardware at all, then shows you the real NVIDIA driver.

What was actually wrong with device plugins

The classic model is deliberately dumb. A device plugin registers with the kubelet, reports a count, and the kubelet exposes that count as an extended resource in node.status.allocatable:

resources:
  limits:
    nvidia.com/gpu: 1

Three rules apply: you may set limits without requests; if you set both they must be equal; you may not set requests without limits. That’s the whole API surface.

Everything else — which GPU model, how much memory, which NUMA node, whether two pods may share a device — has to be smuggled in through node labels and nodeSelector, or through vendor-specific node-level configuration that a workload author cannot see or influence. The Kubernetes docs put it bluntly: device plugins “require per-container device requests, don’t support device sharing, and don’t support expression-based device filtering.”

Here is what that looks like in practice. You want an A100 with at least 40 GB. With device plugins you label your nodes, hope the labels are accurate, and write:

nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-tesla-a100
resources:
  limits:
    nvidia.com/gpu: 1

You are describing a node, not a device. If someone relabels the fleet, your job silently lands on the wrong hardware.

The four objects DRA gives you

DRA replaces the integer with a small set of real API objects, all in the resource.k8s.io/v1 group:

Kind	Who creates it	What it is
ResourceSlice	the driver	The inventory. Each slice lists devices with structured attributes (model, architecture, UUID, NUMA node) and capacities (memory, multiprocessors).
DeviceClass	cluster admin / driver	A named category of device, with a CEL selector that decides which devices belong to it.
ResourceClaim	you	A request for one or more devices, optionally filtered with CEL.
ResourceClaimTemplate	you	A stamp that produces a fresh per-pod ResourceClaim.

The scheduler reads the slices, matches your claim against them, picks devices, and writes the result into ResourceClaim.status.allocation — then places the pod on a node that can reach those devices. This is called structured parameters: the scheduler itself understands the allocation, so there is no vendor controller in the scheduling loop.

The key difference is that you now describe what you need, not where to go:

selectors:
- cel:
    expression: |
      device.attributes['gpu.nvidia.com'].architecture == 'Hopper' &&
      device.capacity['gpu.nvidia.com'].memory.isGreaterThan(quantity("80Gi"))

Version and feature-gate status

Core DRA needs no feature gate and no runtimeConfig on v1.34 and later — resource.k8s.io/v1 is served by default. The sub-features are a moving target, so here is where things stand:

Feature gate	1.34	1.35	1.36
DynamicResourceAllocation	Stable, on	Stable, locked	Stable, locked
DRAAdminAccess	Beta, on	Beta, on	GA
DRAPrioritizedList	Beta, on	Beta, on	GA
DRAPartitionableDevices	Alpha, off	Alpha, off	Beta, on
DRAConsumableCapacity	Alpha, off	Alpha, off	Beta, on
DRAExtendedResource	Alpha, off	Alpha, off	Beta, on
DRADeviceTaints	Alpha, off	Alpha, off	Beta, on
DRAResourceClaimDeviceStatus	Beta, on	Beta, on	Beta, on

One heads-up: the upstream concepts page still files several of these under a heading called “DRA alpha features” while the body text of each correctly calls them beta. Trust the per-gate reference pages, not the heading.

Lab: a DRA cluster with no GPUs

You do not need hardware to learn this. The dra-example-driver project publishes eight mock GPUs and is the fastest honest way to see the whole flow.

git clone https://github.com/kubernetes-sigs/dra-example-driver.git
cd dra-example-driver
./demo/build-driver.sh
./demo/clusters/kind/create-cluster.sh

Then install the driver:

helm upgrade -i --create-namespace --namespace dra-example-driver \
  dra-example-driver deployments/helm/dra-example-driver

Check the inventory:

kubectl get resourceslices

NAME                                          NODE                              DRIVER            POOL                              AGE
dra-example-driver-cluster-worker-gpu.exa...  dra-example-driver-cluster-worker gpu.example.com   dra-example-driver-cluster-worker  30s

And look inside one:

kubectl get resourceslice -o yaml | head -40

You will see devices gpu-0 through gpu-7, each with a model attribute and memory: 80Gi of capacity. That is the whole point of DRA — the cluster now knows what those devices are.

If you are building your own kind cluster instead, the only thing you genuinely need is CDI enabled in containerd:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri"]
    enable_cdi = true
nodes:
- role: control-plane
- role: worker

CDI is on by default in containerd v2.0+ and CRI-O v1.27+. On containerd 1.x you must patch it in, and forgetting to is one of the more common ways a DRA setup fails silently.

Your first DeviceClass and ResourceClaim

A DeviceClass is the admin-facing half. It says “these devices, from this driver, are a thing users may ask for”:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: example-device-class
spec:
  selectors:
  - cel:
      expression: |-
        device.driver == "driver.example.com"

A ResourceClaim is the user-facing half:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: example-resource-claim
spec:
  devices:
    requests:
    - name: single-gpu-claim
      exactly:
        deviceClassName: example-device-class
        allocationMode: All
        selectors:
        - cel:
            expression: |-
              device.attributes["driver.example.com"].type == "gpu" &&
              device.capacity["driver.example.com"].memory == quantity("64Gi")

Note exactly. That is the “give me a specific count of one kind of device” form. Its siblings are allocationMode: ExactCount with a count, allocationMode: All, and firstAvailable for prioritized lists (below).

A pod consumes the claim in two places — spec.resourceClaims names it, and each container opts in via resources.claims:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: workload
    image: ubuntu:24.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; sleep 9999"]
    resources:
      claims:
      - name: gpu
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: single-gpu

That two-step is deliberate. It is what makes sharing expressible: two containers can list the same claim name and get the same physical device, which the device plugin model simply could not represent.

ResourceClaim vs ResourceClaimTemplate

This trips up nearly everyone the first time.

• A ResourceClaim is a single, concrete object. Every pod that references it by resourceClaimName gets the same devices. Use it for sharing, and for claims that should outlive an individual pod.
• A ResourceClaimTemplate is a stamp. Kubernetes creates a fresh claim per pod from it. Use it for the normal “each replica gets its own GPU” case.

Templates have a slightly odd shape — spec.spec, because the outer spec wraps a claim spec:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: example-resource-claim-template
spec:
  spec:
    devices:
      requests:
      - name: gpu-claim
        exactly:
          deviceClassName: example-device-class
          selectors:
          - cel:
              expression: |-
                device.attributes["driver.example.com"].type == "gpu"

Here is a Job that uses both at once — one per-pod device, one device shared between two containers:

apiVersion: batch/v1
kind: Job
metadata:
  name: example-dra-job
spec:
  completions: 10
  parallelism: 2
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: container0
        image: ubuntu:24.04
        command: ["sleep", "9999"]
        resources:
          claims:
          - name: separate-gpu-claim
      - name: container1
        image: ubuntu:24.04
        command: ["sleep", "9999"]
        resources:
          claims:
          - name: shared-gpu-claim
      - name: container2
        image: ubuntu:24.04
        command: ["sleep", "9999"]
        resources:
          claims:
          - name: shared-gpu-claim
      resourceClaims:
      - name: separate-gpu-claim
        resourceClaimTemplateName: example-resource-claim-template
      - name: shared-gpu-claim
        resourceClaimName: example-resource-claim

Watch it allocate:

kubectl get resourceclaims

NAME                          STATE                AGE
example-resource-claim        allocated,reserved   12s
example-dra-job-xk29p-gpu-c   allocated,reserved   12s

The STATE column is computed from the object: pending when status.allocation is nil, allocated once the scheduler has picked devices, and reserved once status.reservedFor is non-empty.

Real hardware: the NVIDIA DRA driver

Two things have changed that will break older blog posts you find:

1. The driver moved from NVIDIA/k8s-dra-driver-gpu to kubernetes-sigs/dra-driver-nvidia-gpu, with docs at dra-driver-nvidia-gpu.sigs.k8s.io.
2. The chart is served from an OCI registry under registry.k8s.io.

helm install dra-driver-nvidia-gpu \
  oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
  --version 0.5.0 \
  --create-namespace \
  --namespace dra-driver-nvidia-gpu \
  --set gpuResourcesEnabledOverride=true

gpuResourcesEnabledOverride=true is not optional. It defaults to false, and without it you get the ComputeDomain half of the driver and no GPU DeviceClasses at all. This is the single most common install mistake.

Prerequisites worth checking before you start: Kubernetes v1.34.2+, NVIDIA driver v565+, NVIDIA Container Toolkit v1.18.0+, a CDI-capable runtime, and Node Feature Discovery v0.18.2+.

Once installed:

kubectl get deviceclass

NAME                                        AGE
compute-domain-daemon.nvidia.com            1m
compute-domain-default-channel.nvidia.com   1m
gpu.nvidia.com                              1m
mig.nvidia.com                              1m
vfio.gpu.nvidia.com                         1m

Full GPUs, MIG slices and VFIO devices all live under one driver (gpu.nvidia.com), one pool per node, distinguished by a type attribute.

Selecting GPUs with CEL

Attributes are addressed through the driver domain. Pick A100s by name:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: gpu-example
  name: a100-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          selectors:
          - cel:
              expression: |
                device.attributes['gpu.nvidia.com'].productName.lowerAscii().matches('^.*a100.*$')

Or by memory, which is usually the better instinct — it survives a hardware refresh:

          selectors:
          - cel:
              expression: |
                device.capacity['gpu.nvidia.com'].memory.isGreaterThan(quantity("40Gi"))

Attributes you can select on for a full GPU include architecture (Ampere, Hopper, Blackwell), productName, brand, cudaComputeCapability, driverVersion, uuid, type, plus the standardized resource.kubernetes.io/pciBusID, resource.kubernetes.io/pcieRoot and resource.kubernetes.io/numaNode. MIG devices add parentUUID and profile (e.g. 1g.5gb).

Constraints: co-locating devices

This is the capability that has no device-plugin equivalent at all. Two GPUs, same NUMA node:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: same-numa-gpus
spec:
  spec:
    devices:
      requests:
      - name: gpus
        exactly:
          deviceClassName: gpu.nvidia.com
          allocationMode: ExactCount
          count: 2
      constraints:
      - requests:
        - gpus
        matchAttribute: resource.kubernetes.io/numaNode

matchAttribute says “all devices in these requests must share this attribute value.” Its counterpart distinctAttribute says the opposite — useful for spreading replicas across failure domains.

Prioritized lists: “an H100, or failing that, two L4s”

With DRAPrioritizedList (GA in 1.36), a request can carry ordered alternatives:

      requests:
      - name: req-0
        firstAvailable:
        - name: large-black
          deviceClassName: resource.example.com
          selectors:
          - cel:
              expression: |-
                device.attributes["resource-driver.example.com"].size == "large"
        - name: small-white
          deviceClassName: resource.example.com
          count: 2
          selectors:
          - cel:
              expression: |-
                device.attributes["resource-driver.example.com"].size == "small"

The scheduler tries each subrequest in order. In a mixed fleet this is the difference between a job starting now on second-choice hardware and a job sitting Pending overnight.

ComputeDomains, briefly

On GB200/GB300 NVL72 systems, GPUs across different nodes are on the same NVLink fabric. Exploiting that needs NVIDIA’s IMEX (Internode Memory Exchange) daemons, domains and channels wired up correctly — and torn down again when the job ends.

The NVIDIA DRA driver models this as a ComputeDomain: an ephemeral, workload-scoped fabric that follows the workload and dies with it.

apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: imex-channel-injection
spec:
  numNodes: 0
  channel:
    resourceClaimTemplate:
      name: imex-channel-0

Pods then claim imex-channel-0 and pin to a clique via nvidia.com/gpu.clique node affinity. If you are not on MNNVL hardware, you can ignore this entirely — but it is a good illustration of what DRA makes possible that the old model could not express.

Troubleshooting

error: the server doesn't have a resource type "deviceclasses" — the resource.k8s.io API group is disabled on your apiserver. A healthy-but-empty cluster says No resources found instead.

Pod stuck Pending. Check the scheduler’s message on the pod. The real strings to look for:

• cannot allocate all claims — no node has devices matching every claim. This is the “no available devices” case.
• resourceclaim not available on the node
• resourceclaim in use
• request <x>: device class <y> does not exist

ResourceClaim never allocated, no events anywhere. Almost always one of two things: the claim you referenced by resourceClaimName does not exist in the same namespace (this fails silently — the pod just waits forever), or the driver is not publishing slices. Check kubectl get resourceslices first.

No ResourceSlices at all. The kubelet plugin is not registering. It must hostPath-mount both /var/lib/kubelet/plugins_registry (registrar socket) and /var/lib/kubelet/plugins/<driver-name> (plugin data). If your kubelet root is not /var/lib/kubelet, registration silently never completes.

Devices transiently unallocatable. The scheduler only allocates from a pool once it sees all of that pool’s slices at the same generation. A driver mid-republish will look empty for a moment.

Preemption does not work for DRA. This one deserves a note in your runbook: the Kubernetes scheduler does not preempt for DRA resources. A high-priority pod will stay Pending until a device is genuinely freed. If you need priority-based reclaim of GPUs, you need a queueing layer above the scheduler — Kueue, for example.

Pod rejected with must specify one of: resourceClaimName, resourceClaimTemplateName. Each entry in pod.spec.resourceClaims must set exactly one. If yours does, suspect a mutating admission webhook built against pre-1.32 APIs that is stripping the field.

Where to go next

DRA is a genuine architectural change, not a new flag. The mental shift is from “how many GPUs do I want” to “what properties must my devices have, and how must they relate to each other.” Once you internalize that, a lot of the ugly workarounds in existing GPU manifests — node labels, taints, per-node MIG pre-configuration, vendor annotations — collapse into a handful of CEL expressions.

Start on kind with the example driver. Get comfortable with claims and templates on mock devices where nothing costs money. Then bring the NVIDIA driver up on one real node before you touch the fleet.

The things to keep an eye on next: DRAPartitionableDevices and DynamicMIG (MIG profiles decided at allocation time rather than baked into the node), DRAConsumableCapacity (splitting one device across independent claims), and DRAExtendedResource (a bridge that lets old nvidia.com/gpu manifests be satisfied by DRA under the hood).