Prefill and Decode Disaggregation with llm-d: A KubeCon China 2026 Keynote Explained

At KubeCon + CloudNativeCon China 2026 in Shanghai, one of the short keynotes on September 8 was called “PD Disaggregation vLLM Deployment”. Jifei Wang and Mengxuan Li from Dynamia showed vLLM running on llm-d, with HAMi sharing and scheduling GPUs underneath.

Five minutes on stage is not much time, so this post fills in the background: what “PD disaggregation” means, why it helps, how llm-d does it on Kubernetes, and where GPU sharing comes in.

Two very different jobs inside one request

Every LLM request has two phases:

1. Prefill. The model reads the whole prompt in one forward pass and builds the KV cache. This is heavy on compute. A long prompt means a lot of math done at once.
2. Decode. The model produces the answer one token at a time, reading the KV cache on every step. This is light on compute but heavy on memory bandwidth.

When both phases run in the same vLLM pod, they get in each other’s way. A big prefill for a new request can hold the GPU for a while, and every user who is in the middle of a streamed answer sees a pause. In metrics terms, time per output token (TPOT) gets jumpy.

Splitting them apart

PD disaggregation runs prefill and decode in separate pods:

• Prefill pods take new prompts, build the KV cache, then hand it over.
• Decode pods receive the KV cache and stream tokens back to the user.

The KV cache moves between them over a fast connector (llm-d uses NIXL, with NCCL as another option). Now a burst of new requests no longer slows down the answers already being streamed. As Mengxuan Li put it in a related talk at the vLLM Shanghai meetup, prefill stops interfering with in-flight decode, which keeps TPOT stable.

You can also size each side on its own. If your traffic has long prompts and short answers, add prefill pods. If it has short prompts and long answers, add decode pods.

How llm-d does it on Kubernetes

llm-d joined the CNCF as a Sandbox project at KubeCon EU 2026. It was donated jointly by IBM Research, Red Hat and Google Cloud, with backing from NVIDIA, AMD, CoreWeave, Hugging Face, Intel, Lambda and Mistral AI. It uses vLLM as the inference engine and adds the Kubernetes parts around it:

• Gateway API Inference Extension. Requests come in through a normal Gateway API gateway. An InferencePool resource groups the model server pods.
• Endpoint Picker (EPP). Instead of round robin, the EPP scores each pod and sends the request to the best one. Scorers look at whether the pod already has the prompt prefix cached, how full its KV cache is, and how long its queue is.
• Prefill and decode roles. Pods are tagged as prefill or decode, and the router sends each phase to the right group.
• KV cache offload. With LMCache, the KV cache can spill from GPU memory to CPU memory and then to disk or shared storage, so long chats can resume without recomputing everything.
• LeaderWorkerSet. Models too big for one node run as multi-node groups managed by LeaderWorkerSet.

The project calls its tested recipes “well-lit paths”: documented setups for things like PD disaggregation or prefix-cache-aware routing, with benchmarks, instead of one default config for everyone.

Once llm-d is installed, the routing setup is visible as normal Kubernetes objects:

kubectl get gateway -A
kubectl get inferencepool -A
kubectl get pods -l llm-d.ai/role=prefill
kubectl get pods -l llm-d.ai/role=decode

The exact labels depend on the chart and version you install, so check the llm-d guides for the release you use.

Where GPU sharing comes in

This was the part of the keynote that was new. Decode is limited by memory bandwidth, not compute. That means a decode pod that has a whole GPU to itself usually leaves a lot of that GPU’s compute unused.

The HAMi team’s answer is to let several decode instances share one card using GPU virtualization. In their vLLM meetup recap, they describe each card hosting roughly twice as many decode instances, with a small drop in per-card speed but about 20% more throughput across the cluster.

They describe the path most teams follow in stages:

1. Single node. vLLM as a process or container on one GPU server. Good for testing.
2. Kubernetes Deployment. vLLM as a standard workload with GPU requests. Easier to run, but prefill and decode still fight for the same card.
3. PD disaggregation. Split prefill and decode, using vLLM’s native connector, Mooncake, or llm-d on Kubernetes.
4. GPU virtualization. Pack several decode instances per card with HAMi.

The keynote also showed this across different accelerator types, not just NVIDIA. HAMi provides the sharing and scheduling layer, so the same llm-d setup can place prefill and decode on different hardware.

Should you do this today?

PD disaggregation adds moving parts: a router, two pod groups, and a KV cache transfer path that needs a fast network. It pays off when:

• you serve one model at real scale, with many concurrent users,
• your users notice pauses in streamed answers, and
• you care about time to first token and TPOT targets.

If you run a few small models with light traffic, a plain vLLM Deployment behind a gateway is still the simpler choice. Prefix-cache-aware routing through the Gateway API Inference Extension is a good first step either way, since it helps even without splitting prefill and decode.

Takeaways

• Prefill is compute bound. Decode is memory bandwidth bound. Running them together causes pauses.
• llm-d splits them into separate pod groups and routes requests with a cache-aware Endpoint Picker built on Gateway API.
• Decode pods leave compute on the table, so sharing decode GPUs with HAMi can add about 20% cluster throughput.
• Start with cache-aware routing, then split prefill and decode when traffic justifies it.

Sources: HAMi at KubeCon China 2026, Three Stages of vLLM Inference Cluster Optimization, llm-d: Kubernetes-Native Distributed LLM Inference, llm-d project