GPU Sharing on Kubernetes: What HAMi Showed at KubeCon China 2026

KubeCon + CloudNativeCon China 2026 took place on September 8 and 9 at the Shanghai International Convention Center. One project showed up again and again on the schedule: HAMi, the Heterogeneous AI Computing Virtualization Middleware. It was HAMi’s first KubeCon as a CNCF Incubating project, and it had two keynote slots, a lightning talk and a full production session.

This post walks through what was shown, why it matters, and how you can try HAMi yourself.

The problem: one GPU, one pod

By default, Kubernetes treats a GPU as a whole device. If a pod asks for nvidia.com/gpu: 1, it gets the entire card, even if it only needs 3 GB of memory and 20% of the compute.

This is fine for large training jobs. It is wasteful for inference services, notebooks, CI jobs and small models, which is most of what runs on GPUs in a typical company. The upstream discussion on sharing GPUs across containers (Kubernetes issue #52757) has been open for years, and HAMi maintainer Xiao Zhang covered it in his KubeCon EU 2026 talk earlier this year.

What was shown in Shanghai

Keynote: Operating Frontier Intelligence at Scale. Chris Aniszczyk (CTO, Linux Foundation) and Xiao Zhang (CEO, Dynamia) opened day one. The main idea was that cloud native is now the operating system layer for AI. Part of that is making every GPU earn its cost: one card split and shared across several LLM workloads, with one scheduling plane for different accelerator types.

Keynote: PD Disaggregation vLLM Deployment. Jifei Wang and Mengxuan Li showed vLLM running on the llm-d stack, with HAMi handling sharing and scheduling across different accelerators. We cover this one in its own post on prefill and decode disaggregation.

Lightning talk: Dynamic MIG with HAMi. Jifei Wang argued that GPU partitioning should follow scheduling, not come before it. With static MIG you carve up cards in advance and hope the shapes match demand. HAMi’s approach is “schedule first, partition second”: the scheduler decides where a workload goes, then the MIG layout on that card is changed to fit it.

Production session: Intsig. Mengxuan Li and Walter Duan from Intsig (the company behind document scanning apps used by millions) described running HAMi on a cluster of around 10,000 GPUs across their own data centers and several clouds. They run more than 1,000 online inference services and more than 1,000 offline training jobs, with GPU utilization above 90% across a full day.

The numbers people shared

The HAMi team’s recap lists results from several adopters:

• Intsig: about 50% higher GPU utilization, about 30% lower overall cost, and less than 10% inference overhead after moving to HAMi.
• China Merchants Bank: single-card inference throughput up 46.7% (ResNet-50, batch 32), and GPU utilization up from 20% to 80%. Their platform (Kubernetes, Kueue, KEDA, Fluid, Prometheus and HAMi) won the Cloud Native China 2026 case study award.
• SF Express: went from 1,400 GPUs to 1,000 with no business impact.
• ICBC: GPU utilization up from 20% to 70%.
• NIO: 10x faster CI.

There was also a useful warning. Slicing a card into pieces smaller than about one sixth gave diminishing returns. More slices is not always better, because each workload still needs enough memory bandwidth and compute to do useful work.

How HAMi works, in short

HAMi runs as a scheduler extender and a device plugin. It lets pods ask for a share of a GPU instead of the whole card:

• nvidia.com/gpu: how many physical GPUs the pod spans
• nvidia.com/gpumem: how much GPU memory (in MiB) the pod can use
• nvidia.com/gpucores: what percentage of the GPU’s compute the pod can use

Limits are enforced inside the container, so a pod that asks for 3000 MiB cannot quietly take the whole card. HAMi supports NVIDIA MIG and MPS modes, works with NVIDIA Ampere, Hopper and Blackwell cards as well as Huawei Ascend and other accelerators, and plays well with batch schedulers like Volcano, Kueue and KAI Scheduler.

Try it on your cluster

You need a node with an NVIDIA GPU, NVIDIA driver 440 or newer, the NVIDIA container toolkit set as the default runtime for containerd (or Docker or CRI-O), Kubernetes 1.23 or newer, and Helm 3.

First, label the GPU nodes that HAMi should manage:

kubectl label nodes <node-name> gpu=on

Add the Helm repo and install HAMi into kube-system:

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
helm install hami hami-charts/hami -n kube-system

Check that the scheduler and device plugin pods are running:

kubectl get pods -n kube-system | grep hami

Now run a pod that asks for part of a GPU. This one asks for one GPU, limited to 3000 MiB of memory and 30% of the compute:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-share-demo
spec:
  containers:
  - name: cuda
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["bash", "-c", "nvidia-smi && sleep 3600"]
    resources:
      limits:
        nvidia.com/gpu: 1
        nvidia.com/gpumem: 3000
        nvidia.com/gpucores: 30

kubectl apply -f gpu-share-demo.yaml
kubectl logs gpu-share-demo

In the nvidia-smi output inside the pod, the total memory shown should match the 3000 MiB limit, not the full card. Create a second copy of the pod with a different name and both will land on the same GPU.

Where this fits with DRA

Dynamic Resource Allocation (DRA) is now the standard Kubernetes way to describe and claim devices like GPUs, and NVIDIA donated its DRA GPU driver to the CNCF this year. DRA gives Kubernetes a common language for devices. Projects like HAMi focus on the part DRA does not solve by itself: splitting one physical card safely between many workloads and packing those workloads well. Expect the two to move closer over time.

Takeaways

• GPU sharing is no longer an experiment. Banks, logistics companies and large consumer apps run it in production.
• Utilization of 20% on whole-GPU allocation is common. Getting to 70% or 80% is realistic with sharing and good scheduling.
• Do not over-slice. Below one sixth of a card the gains drop off.
• Start small: one labeled node, one Helm install, two pods on one card.

Watch the talks

All four HAMi sessions from KubeCon China 2026 are on the event schedule, and recordings are being published on the CNCF YouTube channel.

Sources: HAMi KubeCon China 2026 recap, HAMi at KubeCon China 2026, HAMi KubeCon EU 2026 recap, HAMi on GitHub