KServe 0.20 Tutorial: Serving LLMs on Kubernetes with Autoscaling and Scale-to-Zero

You can serve a model on Kubernetes with a Deployment and a Service. Plenty of teams do, and for a single model behind steady traffic it’s fine.

The trouble starts around model number five. Now you need a consistent way to pull weights from S3, GCS and Hugging Face. You need autoscaling driven by something more meaningful than CPU. You need canary rollouts, because swapping a model version is exactly as risky as swapping application code. You need a request/response schema that isn’t different for every framework. And you need somebody other than the platform team to be able to ship a model without writing a Deployment from scratch.

KServe is the CNCF project that packages all of that behind one CRD. It became a CNCF incubating project in November 2025, and the current release is v0.20.0, which requires Kubernetes 1.32+.

This tutorial covers the current API — which matters, because a lot of what’s written about KServe online describes names that were changed.

What changed in 2026 (read this before copying any older tutorial)

KServe renamed its deployment modes. This is the single biggest source of confusion right now:

Old name	Current name (0.20)	Status
Serverless	Knative	old value deprecated, auto-normalized
RawDeployment	Standard	old value deprecated, auto-normalized
ModelMesh	ModelMesh	legacy multi-model, not recommended for new work

And critically: Standard is now the default mode, not Knative.

That’s a real change in philosophy. KServe’s original pitch was serverless model serving on Knative, with scale-to-zero as the headline feature. For LLMs, scale-to-zero is often a liability — a cold start means pulling 16 GB of weights and warming a GPU, which is a minute or more of latency, not a few hundred milliseconds. The docs now explicitly recommend Standard mode for production LLM workloads: no cold start, predictable behaviour, direct control over pod lifecycle.

Use Knative mode when you genuinely want scale-to-zero — bursty internal tools, dozens of rarely-used small models, dev environments.

Install

The quickstart script is the fastest path, but note the docs’ own warning: “KServe Quickstart Environments are for experimentation use only.”

# Standard mode
curl -sL "https://github.com/kserve/kserve/releases/download/v0.20.0/kserve-standard-mode-full-install-with-manifests.sh" | bash

# Knative (serverless) mode
curl -sL "https://github.com/kserve/kserve/releases/download/v0.20.0/kserve-knative-mode-full-install-with-manifests.sh" | bash

For anything real, use Helm. Be aware that the chart layout changed in v0.17: the single chart was split into ten, and helm upgrade from v0.16 or earlier will fail. The resource chart is called kserve-resources, not kserve.

helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd --version v0.20.0

helm install kserve oci://ghcr.io/kserve/charts/kserve-resources --version v0.20.0 \
  --set kserve.controller.deploymentMode=Standard \
  --set kserve.controller.gateway.ingressGateway.enableGatewayApi=true \
  --set kserve.controller.gateway.ingressGateway.kserveGateway=kserve/kserve-ingress-gateway

kubectl apply --server-side -f https://github.com/kserve/kserve/releases/download/v0.20.0/kserve-cluster-resources.yaml

--server-side is required — the InferenceService CRD is too large for client-side apply.

Verify:

kubectl get pods -n kserve
kubectl get crd | grep serving.kserve.io

Prerequisites by mode

Component	Standard	Knative
Kubernetes	1.32+	1.32+
cert-manager	1.15.0+	1.15.0+
Knative Serving	—	1.19 or 1.20
Istio	optional (ingress class)	1.27 or 1.28
Gateway API CRDs	recommended	—
Envoy Gateway	optional	—
KEDA	optional	n/a

One honest caveat: Gateway API versions are inconsistent across the official docs — the admin guide pins v1.2.1, the LLM pages require 1.3.0+, and the v0.20.0 changelog bumps to v1.5.1. Treat hack/setup/kserve-deps.env on the v0.20.0 tag as ground truth for what was actually tested together.

Lab 1: your first InferenceService (no GPU needed)

Start with a classical model so you can learn the shape of the API without waiting on GPU nodes.

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
  namespace: kserve-test
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
      resources:
        requests:
          cpu: "100m"
          memory: "512Mi"
        limits:
          cpu: "1"
          memory: "1Gi"

That’s the entire manifest. modelFormat: sklearn selects a ClusterServingRuntime automatically — KServe ships nine of them: kserve-sklearnserver, kserve-xgbserver, kserve-lgbserver, kserve-pmmlserver, kserve-paddleserver, kserve-mlserver, kserve-tensorflow-serving, kserve-tritonserver and kserve-huggingfaceserver.

storageUri accepts s3://, gs://, http(s)://, git://, pvc://, hf:// and oci://. A storage-initializer init container downloads the model to /mnt/models before the serving container starts, which is why so much KServe debugging starts with that init container’s logs.

kubectl get inferenceservice sklearn-iris -n kserve-test

NAME           URL                                                 READY   AGE
sklearn-iris   http://sklearn-iris.kserve-test.example.com          True    45s

Calling it

KServe routes by hostname, not path, so you need the service hostname as a Host header when you’re hitting an IP directly:

export INGRESS_HOST=localhost
export INGRESS_PORT=8080

SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve-test \
  -o jsonpath='{.status.url}' | cut -d "/" -f 3)

V1 protocol:

curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" \
  http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict \
  -d @./iris-input.json

V2 — the Open Inference Protocol, which is the one to standardise on for new work because it’s shared with Triton and MLServer:

curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" \
  -d @./iris-input.json \
  http://${INGRESS_HOST}:${INGRESS_PORT}/v2/models/sklearn-iris/infer

Protocol	Predict	Metadata	Ready
V1	POST /v1/models/<name>:predict	GET /v1/models	GET /v1/models/<name>
V2	POST /v2/models/<name>/infer	GET /v2/models/<name>	GET /v2/models/<name>/ready

Opt into V2 explicitly with protocolVersion: v2 on the model spec.

Lab 2: serving an LLM

The huggingface model format routes to kserve-huggingfaceserver, whose --backend defaults to auto — it uses vLLM when the model is supported and falls back to HF transformers otherwise.

Small model first, so you can prove the path works on one GPU:

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "qwen-llm"
  namespace: kserve-test
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=qwen
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      resources:
        limits:
          cpu: "2"
          memory: 6Gi
          nvidia.com/gpu: "1"
        requests:
          cpu: "1"
          memory: 4Gi
          nvidia.com/gpu: "1"

Scaling up to a real model is the same manifest with bigger numbers:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-8b
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=llama3
      storageUri: hf://meta-llama/meta-llama-3-8b-instruct
      resources:
        limits:
          cpu: "6"
          memory: 24Gi
          nvidia.com/gpu: "1"
        requests:
          cpu: "6"
          memory: 24Gi
          nvidia.com/gpu: "1"

No GPU handy? vLLM has a CPU path that’s slow but functional for learning:

      args:
        - --model_name=qwen2
        - --dtype=bfloat16
        - --max-model-len=2048
      storageUri: hf://Qwen/Qwen2-0.5B-Instruct
      env:
        - name: VLLM_CPU_KVCACHE_SPACE
          value: "4"
        - name: VLLM_CPU_OMP_THREADS_BIND
          value: "auto"
        - name: VLLM_ENABLE_V1_MULTIPROCESSING
          value: "0"

The OpenAI-compatible endpoints

This is where KServe earns its keep for LLM work — your existing OpenAI SDK code points at it with a base URL change.

Note the /openai prefix. For an InferenceService using the huggingface runtime, the working paths are /openai/v1/completions and /openai/v1/chat/completions. This catches almost everyone.

MODEL_NAME=llama3
SERVICE_HOSTNAME=$(kubectl get inferenceservice llama3-8b -o jsonpath='{.status.url}' | cut -d "/" -f 3)

curl -v http://${INGRESS_HOST}:${INGRESS_PORT}/openai/v1/chat/completions \
  -H "Host: ${SERVICE_HOSTNAME}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"${MODEL_NAME}"'",
    "messages": [
      {"role": "user", "content": "Explain Kubernetes DRA in two sentences."}
    ],
    "max_tokens": 100,
    "stream": false
  }'

The "model" value must match your --model_name argument (llama3), not the Hugging Face repo id. Set "stream": true for SSE streaming.

Gated models and HF tokens

There’s a known trap here. Setting HF_TOKEN as an env var on the model container does not give it to the storage-initializer init container — which is the thing that actually needs to download the weights. The reliable pattern is a ClusterStorageContainer:

apiVersion: v1
kind: Secret
metadata:
  name: hf-secret
type: Opaque
stringData:
  HF_TOKEN: <your-hugging-face-token>
---
apiVersion: serving.kserve.io/v1alpha1
kind: ClusterStorageContainer
metadata:
  name: hf-hub
spec:
  container:
    name: storage-initializer
    image: 'kserve/storage-initializer:latest'
    env:
      - name: HF_TOKEN
        valueFrom:
          secretKeyRef:
            name: hf-secret
            key: HF_TOKEN
            optional: false
    resources:
      requests:
        memory: 2Gi
        cpu: '1'
      limits:
        memory: 4Gi
        cpu: '1'
  supportedUriFormats:
    - prefix: 'hf://'

Autoscaling

KServe gives you three autoscalers, selected with the serving.kserve.io/autoscalerClass annotation: hpa (default), kpa, keda, plus external and none.

HPA in Standard mode

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris-hpa"
  annotations:
    serving.kserve.io/deploymentMode: Standard
    serving.kserve.io/autoscalerClass: hpa
spec:
  predictor:
    scaleTarget: 80
    scaleMetric: cpu
    minReplicas: 1
    maxReplicas: 5
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"

Fine for classical models. Useless for LLMs — GPU inference is not CPU-bound, and CPU utilisation tells you almost nothing about whether a vLLM server is saturated.

KEDA on vLLM queue depth — the one you actually want

The right signal for an LLM server is how many requests are in flight or queued. vLLM exports exactly that as vllm:num_requests_running.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-qwen
  annotations:
    serving.kserve.io/deploymentMode: "Standard"
    serving.kserve.io/autoscalerClass: "keda"
    serving.kserve.io/enable-prometheus-scraping: "true"
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"
    prometheus.io/scheme: "http"
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=qwen
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
      resources:
        limits:
          cpu: "2"
          memory: 6Gi
          nvidia.com/gpu: "1"
    minReplicas: 1
    maxReplicas: 5
    autoScaling:
      metrics:
        - type: External
          external:
            metric:
              backend: "prometheus"
              serverAddress: "http://prometheus.istio-system.svc.cluster.local:9090"
              query: vllm:num_requests_running
              authModes: "bearer"
            authenticationRef:
              name: keda-prom-creds
            target:
              type: Value
              value: "2"

KEDA autoscaling is only supported in Standard mode. autoScaling.metrics[].type accepts Resource, External and PodMetric; target.type accepts Utilization, Value and AverageValue.

Worth flagging honestly: every KEDA example in the current docs uses minReplicas: 1, and scale-to-zero via KEDA on a plain InferenceService is not documented. Don’t assume it works without testing.

Scale-to-zero, properly

The documented path is Knative mode:

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "flowers-sample"
  annotations:
    serving.kserve.io/deploymentMode: Knative
spec:
  predictor:
    minReplicas: 0
    scaleTarget: 1
    scaleMetric: concurrency
    model:
      modelFormat:
        name: tensorflow
      storageUri: "gs://kfserving-examples/models/tensorflow/flowers"

scaleMetric accepts concurrency, rps, qps, cpu and memory. containerConcurrency sets a hard per-pod limit.

Watch it work:

hey -z 30s -c 5 -m POST -host ${SERVICE_HOSTNAME} -D input.json \
  http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/flowers-sample:predict

Sit on your hands for a minute afterwards and the pods disappear.

Canary rollouts

Shifting traffic to a new model version is one field:

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
  namespace: kserve-test
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kserve-examples/models/sklearn/1.0/model-2"
    canaryTrafficPercent: 10

PREV   LATEST
90     10

Promote by deleting canaryTrafficPercent and re-applying. Roll back by setting it to 0, which returns 100% of traffic to the previous revision — a nice property, since rollback is a one-character edit rather than finding the old manifest.

Caveat: the canary docs page still says this is Knative-only, but the v0.20.0 changelog includes canary support for Standard mode. The docs haven’t caught up. Test it in Standard mode before relying on it.

Troubleshooting

InferenceService not READY. Start with kubectl get inferenceservice <name> -o yaml and read .status.conditions.

• IngressNotConfigured — Istio ingress gateway probes are failing. kubectl logs -l app=networking-istio -n knative-serving; HTTP 403s there mean an AuthorizationPolicy is blocking Knative’s probe traffic.
• RevisionMissing — pods aren’t ready, and it’s usually the storage-initializer.

Predictor pod Pending. kubectl describe pod <pod> and read Events. For GPU LLMs it’s nearly always one of: no node with allocatable nvidia.com/gpu (device plugin or GPU Operator not installed), a GPU count larger than any single node has, a taint on GPU nodes with no matching toleration, or an unsatisfiable node selector.

Model download failures. This is the most common failure by a wide margin:

kubectl logs -l model=sklearn-iris -c storage-initializer

Typical output: Failed to fetch model. The path or model gs://... does not exist.

For S3, credentials go on a Secret with KServe-specific annotations, referenced through a ServiceAccount:

apiVersion: v1
kind: Secret
metadata:
  name: s3creds
  annotations:
     serving.kserve.io/s3-endpoint: s3.amazonaws.com
     serving.kserve.io/s3-usehttps: "1"
     serving.kserve.io/s3-region: "us-east-2"
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: XXXX
  AWS_SECRET_ACCESS_KEY: XXXXXXXX
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3access

Then predictor.serviceAccountName: sa.

GPU OOM on model load. Usually --max-model-len left at the model’s maximum. Cap it, or reduce gpu_memory_utilization. Remember vLLM preallocates KV cache aggressively — the container will use far more GPU memory than the weights alone.

404 on your LLM requests. You almost certainly dropped the /openai prefix, or your "model" value doesn’t match --model_name.

When to reach for LLMInferenceService instead

v0.20 ships an alpha CRD, LLMInferenceService, for advanced LLM serving: prefix-cache-aware routing, disaggregated prefill/decode, multi-node tensor parallelism via LeaderWorkerSet, and the Workload Variant Autoscaler which scales on KV-cache utilisation rather than request counts.

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: llama-70b-production
spec:
  model:
    uri: hf://meta-llama/Llama-2-70b-hf
    name: meta-llama/Llama-2-70b-hf
  parallelism:
    tensor: 4
  replicas: 2
  router:
    gateway: {}
    route: {}
    scheduler: {}

Two warnings. It is genuinely alpha and churning fast — v0.20 alone added traffic splitting, KV-cache offloading, model-name-based routing and DRA support. And its dependency chain is heavy: Gateway API 1.3+, Envoy Gateway, Envoy AI Gateway, the Gateway API Inference Extension, and LeaderWorkerSet.

The docs’ own guidance is the right default: start with InferenceService in Standard mode. It handles both classical ML and standard LLM serving. Move to LLMInferenceService only when you have a specific need — a 70B model that won’t fit one node, or prefix-cache routing that would meaningfully change your economics.

So should you use KServe at all?

If you serve one model and it never changes, a Deployment is fine and KServe is overhead.

The moment you have multiple models, multiple people shipping them, or a rollout process that needs to be safe, the calculus flips. What you’re really buying is a contract: model authors write twenty lines of YAML, and the platform handles storage credentials, runtime selection, autoscaling, ingress, protocol and rollout. That contract is worth considerably more than the sum of the features.

Start with the sklearn example on any cluster you have. It takes five minutes and gives you the whole mental model. Then swap in modelFormat: huggingface and a GPU.