Kueue Tutorial: Quota, Queueing and Gang Admission for GPU Jobs on Kubernetes

The Kubernetes scheduler was designed for services: long-running pods that should start immediately and stay up. Batch AI work is the opposite. A training run wants eight GPUs at the same time, is happy to wait its turn, and would rather sit in a queue for two hours than start with six GPUs and deadlock.

Run enough of those on a shared cluster and you discover the failure modes fast. Two teams each grab four of the eight free GPUs and both jobs hang forever. A research notebook parks on an H100 for a week. The expensive production run queues behind forty experiments because nothing understands priority. And because pods either schedule or they don’t, there is no queue to inspect — just a growing pile of Pending pods with no ordering and no story.

Kueue is the CNCF/SIG-Scheduling answer to this. It sits above the scheduler as a job-level queueing and quota system: it decides whether and when a job may run, and lets kube-scheduler decide where the pods go. This tutorial gets you to a working multi-tenant GPU quota setup and explains the parts that the quickstart glosses over.

The one idea that makes Kueue click

Kueue does not schedule pods. It suspends and resumes Jobs.

When you create a Job carrying a queue label, a Kueue admission webhook sets .spec.suspend = true before the job controller ever creates a pod. Kueue then creates a parallel object — a Workload — that describes the job’s pod sets and their total resource requests. That Workload goes into a queue.

When quota is available, Kueue flips .spec.suspend back to false and, at the same moment, injects the chosen resource flavor’s nodeLabels into the pod template’s nodeSelector and its tolerations into the pod template. Only then do pods get created, and kube-scheduler places them normally.

You can watch the whole handshake in the job’s events:

Normal  Suspended         job-controller        Job suspended
Normal  CreatedWorkload   kueue-job-controller  Created Workload: default/sample-job-sl4bm
Normal  Started           kueue-job-controller  Admitted by clusterQueue cluster-queue
Normal  Resumed           job-controller        Job resumed

One immediate practical consequence: you do not write suspend: true in your Job manifests. The webhook does it. Plenty of older tutorials tell you to set it by hand; upstream explicitly says you don’t need to, and no official example does it.

Install

Kueue’s current release is v0.19.1, and it wants Kubernetes 1.34 or newer.

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.1/manifests.yaml

The --server-side flag is required — the CRDs are too large for client-side apply annotations.

kubectl wait deploy/kueue-controller-manager -n kueue-system --for=condition=available --timeout=5m

Helm, from the OCI registry:

helm install kueue oci://registry.k8s.io/kueue/charts/kueue \
  --version=0.19.1 \
  --namespace kueue-system \
  --create-namespace \
  --wait --timeout 300s

The API version gotcha

Every Kueue CRD currently serves both v1beta1 and v1beta2, with v1beta2 as the storage version since v0.16. Nothing has graduated to v1 yet — if you see kueue.x-k8s.io/v1 anywhere, it’s wrong.

Two renames landed with v1beta2 that will bite you when copying older manifests:

ClusterQueue.spec.cohort → ClusterQueue.spec.cohortName
ClusterQueue.spec.admissionChecks → removed; use admissionChecksStrategy

Upstream prose docs are still stale on both. If you’re upgrading from v0.15 or earlier, run hack/migrate-to-v1beta2.sh from the repo, which forces a no-op update on every Kueue object so the conversion webhook rewrites them into v1beta2 storage.

Lab 1: the smallest thing that works

Three objects. A ResourceFlavor describes a kind of node. A ClusterQueue holds quota and policy and is cluster-scoped. A LocalQueue is the namespaced handle that users actually reference.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all namespaces
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
      - name: "memory"
        nominalQuota: 9Gi
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"

An empty ResourceFlavor spec means “any node.” A namespaceSelector: {} means “any namespace may use this ClusterQueue.”

Now a job. The only Kueue-specific thing about it is one label:

apiVersion: batch/v1
kind: Job
metadata:
  name: sample-job
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  parallelism: 3
  completions: 3
  template:
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: ["/bin/sh"]
        args: ["-c", "sleep 30"]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
      restartPolicy: Never

Apply it a few times in a row and watch the queue do its job:

for i in $(seq 1 6); do kubectl create -f sample-job.yaml --dry-run=client -o yaml \
  | sed "s/name: sample-job/name: sample-job-$i/" | kubectl apply -f -; done

kubectl get workloads

NAME                   QUEUE        RESERVED IN     ADMITTED   AGE
job-sample-job-1-2b4c  user-queue   cluster-queue   True       5s
job-sample-job-2-9f1a  user-queue   cluster-queue   True       5s
job-sample-job-3-77de  user-queue   cluster-queue   True       5s
job-sample-job-4-c2a0  user-queue                              5s

That fourth row is the whole value proposition. It’s not a failure — it’s a queue entry, with an explanation attached, that will start on its own when capacity frees up.

Lab 2: GPU quota, cohorts, and borrowing

The real setup has multiple teams and expensive hardware. Here’s a two-flavor configuration with a shared pool.

# --- Flavors -------------------------------------------------------------
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: default-flavor
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gpu-a100
spec:
  nodeLabels:
    cloud.google.com/gke-accelerator: nvidia-tesla-a100
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"
---
# --- Cohort: the shared pool ---------------------------------------------
apiVersion: kueue.x-k8s.io/v1beta2
kind: Cohort
metadata:
  name: research-cohort
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: team-a-cq
spec:
  cohortName: research-cohort
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: team-a
  queueingStrategy: BestEffortFIFO
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
    borrowWithinCohort:
      policy: LowerPriority
      maxPriorityThreshold: 100
  fairSharing:
    weight: 1
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: cpu
        nominalQuota: 96
      - name: memory
        nominalQuota: 768Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 8
        borrowingLimit: 8   # may reach 16 GPUs when the cohort is idle
        lendingLimit: 4     # keeps 4 GPUs exclusively for team A
    - name: default-flavor
      resources:
      - name: cpu
        nominalQuota: 32
      - name: memory
        nominalQuota: 128Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 0
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: team-a
  name: team-a-queue
spec:
  clusterQueue: team-a-cq

Duplicate the ClusterQueue and LocalQueue for team B with the same cohortName, and the two now share.

How borrowing actually works

A pod set’s resource request fits a flavor if:

it’s within this ClusterQueue’s unused nominalQuota for that flavor; or
it’s within the cohort’s total unused nominalQuota for that flavor, and
it’s within this queue’s unused nominalQuota + borrowingLimit.

Conditions 2 and 3 without 1 is what borrowing means. Two constraints catch people out: a ClusterQueue can only borrow for flavors it explicitly declares (which is why nvidia.com/gpu: nominalQuota: 0 appears under default-flavor above), and for a given pod-set resource it can only borrow within one flavor.

lendingLimit is the mirror image: nominalQuota - lendingLimit is what this queue keeps for itself no matter how idle it is. Leave it null and you lend everything.

Both borrowingLimit and lendingLimit must be null if cohortName is empty — the API will reject them otherwise.

Priorities

apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: prod-training
value: 10000
description: "Production training runs"

metadata:
  labels:
    kueue.x-k8s.io/queue-name: team-a-queue
    kueue.x-k8s.io/priority-class: prod-training

WorkloadPriorityClass is deliberately separate from the standard pod PriorityClass. It affects queue ordering and preemption inside Kueue, not kube-scheduler’s pod preemption. Keeping them distinct means you can order your queue without touching cluster-wide pod priority.

Preemption

Field	Values	Default
`withinClusterQueue`	`Never`, `LowerPriority`, `LowerOrNewerEqualPriority`	`Never`
`reclaimWithinCohort`	`Never`, `LowerPriority`, `Any`	`Never`
`borrowWithinCohort.policy`	`Never`, `LowerPriority`	`Never`

reclaimWithinCohort only touches workloads that are running above their own nominal quota — i.e. borrowers. That’s the setting you want when a team’s own hardware is busy serving someone else’s overflow and they need it back. borrowWithinCohort is the aggressive one: it lets a workload preempt while itself borrowing. Note it is only valid with Classic Preemption, not Fair Sharing.

A preempted workload gets clear conditions:

status:
  conditions:
  - type: Evicted
    reason: Preempted
    message: 'Preempted to accommodate a workload (UID: ..., JobUID: ...) due to prioritization in the ClusterQueue'
  - type: Preempted
    reason: InClusterQueue

To find who did it:

kubectl get workloads.kueue.x-k8s.io --selector=kueue.x-k8s.io/job-uid=<JobUID> -A

Queueing strategy

BestEffortFIFO (default): ordered by priority then creation time, but a head-of-line workload that can’t be admitted does not block smaller ones behind it. Better utilization.
StrictFIFO: the head blocks. Fairer, and prevents large jobs from being starved by a stream of small ones. Side effect worth knowing when debugging: with StrictFIFO Kueue only attempts the head of each queue, so a queued Workload may have no status conditions at all.

All-or-nothing: the part everyone gets wrong

There is no single gangScheduling: true switch in Kueue. Instead there are four mechanisms that stack, and understanding which one you actually need matters.

1. Quota-based admission is the foundation and is genuinely atomic — Kueue reserves quota for all pod sets or none. It’s on by default. But quota is an accounting abstraction, not a placement guarantee.

2. Topology-Aware Scheduling (TAS) is what closes that gap, and for AI/ML it should be your default. Aggregate quota is blind to node layout: eight free GPUs spread across two four-GPU nodes satisfies quota for an eight-GPU pod, which then sits Pending in kube-scheduler forever. TAS tracks per-domain free capacity and assigns concrete topology domains at admission time.

You declare your fabric with a Topology object and then annotate pod templates:

metadata:
  annotations:
    kueue.x-k8s.io/podset-required-topology: "kubernetes.io/hostname"

podset-required-topology is a hard constraint; podset-preferred-topology degrades gracefully to a wider domain.

3. ProvisioningRequest admission checks delay admission until an autoscaler confirms capacity — check-capacity.autoscaling.x-k8s.io verifies existing nodes, best-effort-atomic-scale-up.autoscaling.x-k8s.io will try to scale up.

4. waitForPodsReady is the safety net, configured cluster-wide in the Kueue Configuration ConfigMap:

waitForPodsReady:
  timeout: 30m
  recoveryTimeout: 30m
  blockAdmission: false
  requeuingStrategy:
    timestamp: Eviction
    backoffLimitCount: 5
    backoffBaseSeconds: 60
    backoffMaxSeconds: 3600

Be clear-eyed about what this does: it does not prevent partial scheduling. It bounds how long a half-scheduled job is allowed to hold resources before Kueue evicts it and requeues with exponential backoff. Setting blockAdmission: true makes admissions sequential, which prevents two jobs from deadlocking each other but costs throughput.

The opposite knob exists too. If your job can usefully run smaller, annotate it:

metadata:
  annotations:
    kueue.x-k8s.io/job-min-parallelism: "5"

Kueue only shrinks parallelism after both borrowing and preemption have been ruled out.

Beyond batch/v1 Job

Kueue integrates with most of the AI/ML workload types you’d actually use: JobSet, RayJob and RayCluster, Kubeflow’s PyTorchJob / TFJob / MPIJob, AppWrapper, LeaderWorkerSet, plain Pods, and Deployment / StatefulSet.

Each is enabled through the integrations.frameworks list in the Kueue Configuration. Two options are worth knowing:

manageJobsWithoutQueueName: true makes Kueue manage everything, not only labelled jobs. Powerful and easy to regret — it will suspend workloads whose owners have never heard of Kueue.
managedJobsNamespaceSelector scopes that behaviour to specific namespaces, which is usually what you actually wanted.

The kubectl plugin

Install kueuectl and life gets easier:

kubectl kueue list localqueue
kubectl kueue list workload --status=pending
kubectl kueue describe workload <name>
kubectl kueue stop workload <name>
kubectl kueue resume workload <name>

kubectl kueue list workload --status=pending is the single most useful command in this whole tutorial. It answers “why hasn’t my job started” faster than anything else.

Troubleshooting

Job stays suspended forever, no Workload created. The queue label is missing or misspelled — it’s kueue.x-k8s.io/queue-name, and it goes on the Job’s metadata.labels, not the pod template. Also confirm the LocalQueue exists in the same namespace as the Job.

Workload exists but is never admitted. Read its conditions:

kubectl describe workload <workload-name>

Common messages and what they mean:

couldn't assign flavors to pod set <name>: insufficient quota for nvidia.com/gpu — self-explanatory; check kubectl get clusterqueue -o wide for current usage.
couldn't assign flavors to pod set <name> with no quota complaint — usually a flavor mismatch. Your job requests a resource that no declared flavor covers, or the flavor’s nodeLabels match no node.
No conditions at all — you’re on StrictFIFO and something older is ahead of you.

Quota looks free but nothing admits. Check whether another ClusterQueue in the cohort is borrowing it, and whether your lendingLimit is doing what you intended.

Pods land on the wrong nodes. Remember that Kueue injects the flavor’s nodeLabels as a nodeSelector at resume time. If a flavor’s labels don’t match reality, jobs admit and then hang in kube-scheduler. kubectl get job <name> -o jsonpath='{.spec.template.spec.nodeSelector}' after admission shows you exactly what got injected.

Workload deactivated. After exceeding backoffLimitCount requeues, or a permanently-rejected admission check, Kueue sets the Workload inactive. Set .spec.active: true to bring it back.

Kueue vs Volcano vs YuniKorn

Short version:

Kueue is quota-and-queue management that delegates placement to kube-scheduler. It doesn’t replace anything, integrates cleanly with cluster autoscaling, and is the SIG-Scheduling-blessed path. Pick it when your problem is fair sharing, quota and admission order.
Volcano is a full replacement scheduler with genuine gang scheduling, job dependencies and topology-aware placement built into the scheduling cycle. Pick it when your problem is placement — tightly-coupled MPI, complex co-scheduling.
YuniKorn is also a replacement scheduler, with a hierarchical queue model that will feel familiar if you’re coming from YARN. Strong on multi-tenant hierarchies.

They aren’t strictly exclusive — Kueue can front Volcano — but running two schedulers is real operational cost. For most teams whose actual pain is “we can’t tell whose job should run next,” Kueue alone is the right amount of machinery.

Where to start

Install Kueue on a dev cluster. Create one flavor, one ClusterQueue, one LocalQueue. Label one job. Watch it suspend and resume. That takes fifteen minutes and it’s the part that builds intuition.

Then, before you roll it out: turn on Topology-Aware Scheduling. Most of the “Kueue admitted my job and it still didn’t run” stories trace back to quota being satisfied by capacity that was never actually co-located.