Load Balance LLM Requests with AI Gateway Patterns

The burgeoning field of Artificial Intelligence, particularly Large Language Models (LLMs), presents unprecedented opportunities. However, deploying and scaling these powerful models in production environments comes with unique challenges. LLMs are resource-intensive, often requiring specialized hardware like GPUs, and their inference requests can vary wildly in complexity and latency. Managing traffic to these models, ensuring high availability, optimal resource utilization, and predictable performance, is crucial for any AI-driven application.

This is where AI Gateway patterns shine. An AI Gateway acts as a sophisticated traffic manager, sitting in front of your LLM inference services. It provides a centralized point for request routing, load balancing, authentication, rate limiting, and observability, specifically tailored for the demands of AI workloads. By implementing robust load balancing strategies within your Kubernetes cluster, you can efficiently distribute incoming LLM requests across multiple model instances, preventing bottlenecks, improving response times, and maximizing the utilization of your expensive GPU resources.

In this comprehensive guide, we’ll dive deep into building and configuring AI Gateway patterns on Kubernetes to effectively load balance LLM requests. We’ll explore practical implementations using popular tools like NGINX Ingress Controller and the Kubernetes Gateway API, demonstrating how to set up resilient and scalable infrastructure for your AI applications. Whether you’re dealing with a single LLM or a fleet of diverse models, mastering these patterns is essential for delivering performant and reliable AI services.

# Install NGINX Ingress Controller
helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace

# Deploy a sample LLM service (e.g., a simple API mimicking LLM inference)
kubectl apply -f https://raw.githubusercontent.com/kubezilla/ai-gateway-examples/main/llm-app.yaml

# Create an Ingress resource for basic load balancing
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: llm.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: llm-service
            port:
              number: 80
EOF

# Install Gateway API CRDs (if not already present)
kubectl get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || \
  kubectl apply -k "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.0.0"

# Deploy a Gateway and HTTPRoute for advanced LLM routing
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: llm-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-httproute
spec:
  parentRefs:
  - name: llm-gateway
  hostnames:
  - "llm.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/models/gpt-3
    backendRefs:
    - name: llm-service-gpt3
      port: 80
  - matches:
    - path:
        type: PathPrefix
        value: /v1/models/llama2
    backendRefs:
    - name: llm-service-llama2
      port: 80
EOF

Prerequisites

Before we embark on configuring our AI Gateway, ensure you have the following tools and knowledge:

• Kubernetes Cluster: A running Kubernetes cluster (v1.22+ is recommended for Gateway API compatibility). This can be a local cluster like Minikube or Kind, or a managed service like EKS, GKE, or AKS.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster.
• Helm: The Kubernetes package manager, version 3+. We’ll use Helm to install ingress controllers.
• Basic Kubernetes Knowledge: Familiarity with Deployments, Services, Ingress, and basic networking concepts in Kubernetes.
• Domain Name (Optional but Recommended): For testing external access, a domain name configured with DNS records pointing to your Ingress controller’s external IP or Load Balancer.
• Container Registry: Access to a container registry (Docker Hub, ECR, GCR, etc.) if you plan to build and push custom LLM inference images.

Step-by-Step Guide

Step 1: Deploy a Sample LLM Inference Service

To demonstrate load balancing, we first need a backend service that mimics an LLM inference endpoint. For simplicity, we’ll deploy a basic Python Flask application that returns its pod name and a timestamp. In a real-world scenario, this would be your actual LLM serving application, potentially using frameworks like vLLM, NVIDIA Triton Inference Server, or Hugging Face Transformers.

We’ll create a Deployment with multiple replicas to showcase load balancing and a Service to expose it within the cluster. This allows the gateway to target a stable endpoint, abstracting away the individual pods.

# llm-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  labels:
    app: llm-inference
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
      - name: llm-mock-server
        image: python:3.9-slim-buster
        ports:
        - containerPort: 80
        command: ["/bin/bash", "-c"]
        args:
          - |
            pip install Flask gunicorn
            cat <<EOF > app.py
            from flask import Flask, request, jsonify
            import os
            import time

            app = Flask(__name__)

            @app.route('/v1/chat/completions', methods=['POST'])
            @app.route('/v1/models//completions', methods=['POST'])
            def completions(model_name=None):
                pod_name = os.getenv('HOSTNAME', 'unknown-pod')
                timestamp = time.time()
                request_data = request.json
                print(f"[{timestamp}] Request received by {pod_name}: {request_data}")
                
                # Simulate LLM processing time
                time.sleep(0.5 + (hash(pod_name) % 100) / 200.0) # Vary processing slightly

                response_content = f"Response from {pod_name} at {timestamp}. Model: {model_name or 'default'}. Input: {request_data.get('prompt', request_data.get('messages', ''))[:50]}..."
                
                return jsonify({
                    "id": f"chatcmpl-{int(timestamp)}",
                    "object": "chat.completion",
                    "created": int(timestamp),
                    "model": model_name or "gpt-3.5-turbo",
                    "choices": [
                        {
                            "index": 0,
                            "message": {
                                "role": "assistant",
                                "content": response_content
                            },
                            "logprobs": None,
                            "finish_reason": "stop"
                        }
                    ],
                    "usage": {
                        "prompt_tokens": 10,
                        "completion_tokens": 20,
                        "total_tokens": 30
                    }
                })

            @app.route('/health')
            def health():
                return "OK", 200

            if __name__ == '__main__':
                gunicorn --bind 0.0.0.0:80 app:app
            EOF
            gunicorn --bind 0.0.0.0:80 app:app
---
apiVersion: v1
kind: Service
metadata:
  name: llm-service
spec:
  selector:
    app: llm-inference
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

Apply this manifest to your cluster:

kubectl apply -f llm-app.yaml

Verify

Check if the pods and service are running:

kubectl get pods -l app=llm-inference
kubectl get svc llm-service

Expected Output:

NAME                             READY   STATUS    RESTARTS   AGE
llm-inference-74f4f5f5f4-abcde   1/1     Running   0          2m
llm-inference-74f4f5f5f4-fghij   1/1     Running   0          2m
llm-inference-74f4f5f5f4-klmno   1/1     Running   0          2m

NAME          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
llm-service   ClusterIP   10.96.X.Y     <none>        80/TCP    2m

You can also test the service internally:

kubectl run -it --rm --restart=Never curl-test --image=curlimages/curl -- \
  curl -X POST -H "Content-Type: application/json" \
  -d '{"prompt": "Hello, LLM!"}' http://llm-service/v1/chat/completions

You should get a JSON response indicating which pod served the request.

Step 2: Implement AI Gateway with NGINX Ingress Controller

The NGINX Ingress Controller is a widely used and robust solution for exposing HTTP/S services in Kubernetes. It offers advanced features like path-based routing, SSL termination, and various load balancing algorithms. For our AI Gateway, NGINX provides a solid foundation for distributing LLM requests.

First, we need to install the NGINX Ingress Controller into our cluster. We’ll use Helm for a straightforward installation.

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer

The --set controller.service.type=LoadBalancer flag ensures that your Ingress Controller gets an external IP address (if your cloud provider supports it) or exposes a NodePort for local clusters. For more details on Ingress Controller configurations, refer to the official NGINX Ingress documentation.

Next, we’ll create an Ingress resource to route traffic to our llm-service. This Ingress will act as our basic AI Gateway, directing requests for a specific host (e.g., llm.example.com) to our LLM backend.

# llm-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-ingress
  annotations:
    # Use round-robin by default, or specify other algorithms if needed
    # nginx.ingress.kubernetes.io/load-balance: "round_robin"
    # Enable session affinity for sticky sessions (useful if state is maintained per-client)
    # nginx.ingress.kubernetes.io/affinity: "cookie"
    # nginx.ingress.kubernetes.io/session-cookie-name: "llm_affinity"
    # nginx.ingress.kubernetes.io/session-cookie-path: "/"
spec:
  ingressClassName: nginx
  rules:
  - host: llm.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: llm-service
            port:
              number: 80

Apply this Ingress resource:

kubectl apply -f llm-ingress.yaml

Verify

Check if the Ingress controller is running and has an external IP:

kubectl get svc -n ingress-nginx ingress-nginx-controller

Expected Output:

NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
ingress-nginx-controller        LoadBalancer   10.X.Y.Z       XX.YY.ZZ.AA    80:30080/TCP,443:30443/TCP   5m

Note down the EXTERNAL-IP. If you’re on a local cluster (Minikube/Kind), you might need to use minikube ip or port-forward the service.

Now, test the Ingress. If you have a domain name, configure DNS to point llm.example.com to the EXTERNAL-IP. Otherwise, you can modify your /etc/hosts file (or equivalent on Windows) for testing:

# Add this line to /etc/hosts (replace XX.YY.ZZ.AA with your EXTERNAL-IP)
XX.YY.ZZ.AA llm.example.com

Send several requests to observe load balancing:

for i in {1..6}; do
  curl -s -X POST -H "Content-Type: application/json" \
    -d '{"prompt": "Generate a short story about a brave knight."}' \
    http://llm.example.com/v1/chat/completions | grep -o "Response from [a-zA-Z0-9-]*"
done

Expected Output: (You should see responses from different pods)

Response from llm-inference-74f4f5f5f4-abcde
Response from llm-inference-74f4f5f5f4-fghij
Response from llm-inference-74f4f5f5f4-klmno
Response from llm-inference-74f4f5f5f4-abcde
Response from llm-inference-74f4f5f5f4-fghij
Response from llm-inference-74f4f5f5f4-klmno

This confirms that the NGINX Ingress is successfully distributing requests across your LLM inference pods.

Step 3: Advanced Routing with Kubernetes Gateway API

While Ingress is powerful, the Kubernetes Gateway API offers a more expressive, extensible, and role-oriented approach to API gateway management. It’s designed for more complex routing scenarios, multi-cluster deployments, and advanced traffic manipulation, making it ideal for sophisticated AI Gateway patterns. For a deeper dive into migrating from Ingress to Gateway API, check out our Kubernetes Gateway API vs Ingress: The Complete Migration Guide.

First, ensure the Gateway API CRDs are installed in your cluster. Many Kubernetes distributions now include it by default, but you might need to install it manually:

kubectl get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || \
  kubectl apply -k "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.0.0"

Next, we need a Gateway API controller. NGINX Ingress Controller has experimental support for Gateway API, but for a more robust experience, we can use a dedicated implementation like NGINX Gateway Fabric or Contour. For simplicity, we’ll demonstrate with the NGINX Gateway Fabric. Install it:

helm upgrade --install nginx-gateway-fabric oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric \
  --version 1.0.0 \
  --namespace nginx-gateway \
  --create-namespace \
  --set service.type=LoadBalancer

Now, we define a GatewayClass, a Gateway, and an HTTPRoute. The GatewayClass defines a template for Gateways, the Gateway resource requests a load balancer, and HTTPRoute defines the actual routing rules.

Let’s imagine we have two different LLM models, gpt-3 and llama2, served by different services. We’ll simulate this by creating two services pointing to our existing deployment but with different names, and then routing based on the URL path.

# llm-gateway-api.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: gateway.nginx.org/controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: llm-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
# Simulate a separate service for GPT-3 model
apiVersion: v1
kind: Service
metadata:
  name: llm-service-gpt3
spec:
  selector:
    app: llm-inference # Still points to the same pods for this example
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
# Simulate a separate service for Llama2 model
apiVersion: v1
kind: Service
metadata:
  name: llm-service-llama2
spec:
  selector:
    app: llm-inference # Still points to the same pods for this example
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-httproute
spec:
  parentRefs:
  - name: llm-gateway
  hostnames:
  - "llm-api.example.com" # Use a different hostname for this example
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/models/gpt-3
    backendRefs:
    - name: llm-service-gpt3
      port: 80
  - matches:
    - path:
        type: PathPrefix
        value: /v1/models/llama2
    backendRefs:
    - name: llm-service-llama2
      port: 80
  - matches: # Default route for other paths
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: llm-service
      port: 80

Apply these resources:

kubectl apply -f llm-gateway-api.yaml

Verify

Check the status of your Gateway:

kubectl get gateway llm-gateway

Expected Output:

NAME          CLASS   ADDRESS        PROGRAMMED   AGE
llm-gateway   nginx   XX.YY.ZZ.AA    True         2m

Note the ADDRESS. Update your /etc/hosts file (or DNS) for llm-api.example.com to point to this IP.

Now, test the path-based routing:

for i in {1..3}; do
  curl -s -X POST -H "Content-Type: application/json" \
    -d '{"prompt": "Explain quantum entanglement."}' \
    http://llm-api.example.com/v1/models/gpt-3/completions | grep -o "Response from [a-zA-Z0-9-]*"
done

for i in {1..3}; do
  curl -s -X POST -H "Content-Type: application/json" \
    -d '{"prompt": "Write a haiku about Kubernetes."}' \
    http://llm-api.example.com/v1/models/llama2/completions | grep -o "Response from [a-zA-Z0-9-]*"
done

# Test default path
curl -s -X POST -H "Content-Type: application/json" \
  -d '{"prompt": "What is the capital of France?"}' \
  http://llm-api.example.com/v1/chat/completions | grep -o "Response from [a-zA-Z0-9-]*"

Expected Output: (You should see responses from different pods, correctly routed)

Response from llm-inference-74f4f5f5f4-abcde
Response from llm-inference-74f4f5f5f4-fghij
Response from llm-inference-74f4f5f5f4-klmno
Response from llm-inference-74f4f5f5f4-abcde
Response from llm-inference-74f4f5f5f4-fghij
Response from llm-inference-74f4f5f5f4-klmno
Response from llm-inference-74f4f5f5f4-abcde

This demonstrates how the Gateway API allows for much finer-grained control over routing, enabling you to direct requests to specific LLM versions or models based on URL paths, headers, or query parameters. This is crucial for A/B testing, blue/green deployments, and multi-model serving strategies.

Production Considerations

Deploying AI Gateways for LLMs in production requires careful planning beyond basic routing:

1. Resource Management for LLMs: LLMs are resource-hungry, especially for GPUs. Ensure your Kubernetes cluster has adequate GPU nodes. Tools like LLM GPU Scheduling Best Practices can help you efficiently schedule GPU-bound workloads. Consider using node autoscalers like Karpenter for Cost Optimization to dynamically scale your GPU infrastructure based on demand.
2. Scalability and Autoscaling:
   • Horizontal Pod Autoscaler (HPA): Scale your LLM inference pods based on CPU, memory, or custom metrics like GPU utilization or request queue depth.
   • Vertical Pod Autoscaler (VPA): Optimize CPU/memory requests and limits for your LLM pods.
   • Cluster Autoscaler/Karpenter: Ensure your cluster can scale nodes up and down to meet demand for LLM pods.
3. Observability and Monitoring:
   • Metrics: Collect metrics from your gateway (request rates, latencies, error rates) and your LLM pods (GPU utilization, inference latency, token generation rates). Prometheus and Grafana are standard tools.
   • Logging: Centralize logs from your gateway and LLM pods (e.g., using Fluentd/Fluent Bit to Elasticsearch/Loki).
   • Tracing: Implement distributed tracing (e.g., with OpenTelemetry) to understand request flow through the gateway and into the LLM service, crucial for debugging performance issues. For advanced network observability, consider eBPF Observability with Hubble.
4. Security:
   • Authentication & Authorization: Integrate your AI Gateway with identity providers (e.g., OAuth2/OIDC) for API key validation or user authentication.
   • Rate Limiting: Protect your LLM services from abuse and ensure fair usage by implementing rate limiting at the gateway level.
   • Network Policies: Restrict traffic flow to and from your LLM services and gateway. Our Kubernetes Network Policies: Complete Security Hardening Guide provides detailed instructions.
   • Encryption: Ensure all traffic is encrypted in transit (TLS at the gateway, mTLS between services, potentially Cilium WireGuard Encryption for pod-to-pod traffic).
   • Container Security: Use signed images (Sigstore and Kyverno) and regularly scan for vulnerabilities.
5. Load Balancing Algorithms:
   • Round Robin: Simple, evenly distributes requests. Good default.
   • Least Connections: Sends requests to the backend with the fewest active connections. Good for varying request processing times.
   • Weighted Round Robin/Least Connections: Assigns weights to backends based on capacity (e.g., node GPU memory, model version).
   • Session Affinity (Sticky Sessions): If your LLM application requires maintaining state for a client across multiple requests, use cookie-based session affinity.
6. Blue/Green Deployments & Canary Releases: Use gateway routing capabilities to gradually roll out new LLM versions, minimizing risk and allowing for A/B testing.
7. Service Mesh Integration: For advanced traffic management, resiliency patterns (retries, circuit breakers), and enhanced observability, consider integrating a service mesh like Istio (e.g., Istio Ambient Mesh Production Guide) or Linkerd. While gateways handle north-south traffic, service meshes excel at east-west traffic.
8. Health Checks: Configure robust health checks for your LLM pods. Beyond simple HTTP 200, consider checks that validate the model is actually loaded and ready to serve inferences. This prevents the gateway from sending traffic to unhealthy instances.

Troubleshooting

Here are common issues you might encounter and their solutions:

1. Issue: Ingress/Gateway is not getting an External IP.

   Solution:

   • Cloud Provider: Ensure your Kubernetes cluster is running on a cloud provider that supports LoadBalancer services (AWS, GCP, Azure). If on-premises, you might need a bare-metal load balancer solution like MetalLB.
   • Local Cluster: For Minikube, use minikube tunnel in a separate terminal. For Kind, you might need to port-forward or use a tool like kind get kubeconfig --name kind-cluster --internal and adjust your local routing.
   • Service Status: Check kubectl get svc -n ingress-nginx ingress-nginx-controller (or nginx-gateway-fabric-controller). If EXTERNAL-IP is <pending>, your cloud provider might be slow or encountering issues provisioning the load balancer.
2. Issue: Requests to the Ingress/Gateway return 404 or “default backend – 404”.

   Solution:

   • Hostname Mismatch: Ensure the host in your Ingress/HTTPRoute exactly matches the hostname you are using in your curl command or browser.
   • Path Mismatch: Verify the path and pathType in your Ingress/HTTPRoute. A PathPrefix of / is usually safe for root paths.
   • Service Not Found: Check that the service.name and service.port.number in your Ingress/HTTPRoute correctly point to your backend service. Run kubectl get svc <service-name>.
   • Ingress Class: Ensure your Ingress resource specifies the correct ingressClassName (e.g., nginx).
   • HTTPRoute ParentRef: For Gateway API, ensure the parentRefs in your HTTPRoute correctly point to your Gateway.
3. Issue: Requests time out or are very slow.

   Solution:

   • LLM Pod Health: Check if your LLM inference pods are healthy and not overloaded. Use kubectl get pods and check logs with kubectl logs <pod-name>.
   • Resource Limits: Ensure your LLM pods have sufficient CPU, memory, and GPU resources. Over-committing resources can lead to throttling.
   • Ingress/Gateway Logs: Check the logs of your Ingress Controller or Gateway Fabric pods for errors (kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx or -n nginx-gateway -l app.kubernetes.io/name=nginx-gateway-fabric).
   • Network Connectivity: Verify network connectivity between the Ingress/Gateway pods and your LLM service pods.
4. Issue: Load balancing doesn’t seem to be working (all requests go to one pod).

   Solution:

   • Replicas: Ensure your LLM Deployment has more than one replica (e.g., replicas: 3).
   • Session Affinity: Check if session affinity (sticky sessions) is accidentally enabled on your Ingress/HTTPRoute. For NGINX Ingress, look for annotations like nginx.ingress.kubernetes.io/affinity: "cookie". Disable it if you want round-robin distribution.
   • Backend Weighting: If using weighted load balancing, ensure weights are configured as expected.
5. Issue: Gateway API HTTPRoute is not taking effect.

   Solution:

   • GatewayClass Status: Check if your GatewayClass is Accepted: True.
   • Gateway Status: Check if your Gateway is Programmed: True and has an address.
   • HTTPRoute Status: Check the status of your HTTPRoute. It should show ParentRefs as Programmed: True. Look for any conditions indicating errors.
   • Controller Logs: Examine the logs of the Gateway API controller (e.g., NGINX Gateway Fabric) for any configuration errors or warnings.
6. Issue: Specific NGINX Ingress annotations are not working.

   Solution:

   • Annotation Syntax: Double-check the