Migrate to Kubernetes Gateway API Now

Kubernetes Gateway API: Complete Migration from Ingress

For years, Kubernetes Ingress has been the de-facto standard for exposing HTTP/S applications to the outside world. It provided a simple, albeit limited, way to manage external access to services within a cluster. However, as Kubernetes environments grew in complexity and demands for more advanced traffic management features increased, Ingress began to show its age. Limitations around protocol support, traffic routing capabilities, and integration with modern service mesh patterns became increasingly apparent, leading to a fragmented ecosystem of Ingress controllers each offering proprietary extensions.

Enter the Kubernetes Gateway API. Designed as a successor to Ingress, the Gateway API offers a more expressive, extensible, and role-oriented approach to traffic management. It provides a structured set of resources (GatewayClass, Gateway, HTTPRoute, TCPRoute, UDPRoute, TLSRoute) that empower cluster operators to define networking infrastructure, while application developers can self-serve their routing needs. This separation of concerns, combined with advanced features like weighted traffic splitting, header-based routing, and robust policy attachment, makes the Gateway API an indispensable tool for modern Kubernetes deployments. Migrating from Ingress to Gateway API is not just an upgrade; it’s a strategic move towards a more flexible, future-proof, and powerful traffic management architecture.

kubectl get crd gatewayclasses.gateway.networking.k8s.io > /dev/null 2>&1 || \
    kubectl apply -k "https://github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v1.1.0"

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: example-http-route
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: my-api-service
      port: 80

Prerequisites

Before embarking on your migration journey, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.25+ recommended for full Gateway API v1.0 support). You can use Minikube, Kind, or a cloud-managed cluster like EKS, GKE, or AKS.
• kubectl: Command-line tool configured to connect to your cluster.
• Helm (Optional but Recommended): For easier installation of Gateway API controllers.
• Basic Kubernetes Networking Knowledge: Familiarity with Services, Deployments, Pods, and existing Ingress concepts. For a deeper dive into networking, consider our Network Policies Security Guide.
• Administrative Privileges: Sufficient permissions to install CRDs and cluster-scoped resources.
• Chosen Gateway API Controller: Decide which controller you’ll use. Popular choices include:
  • AWS Load Balancer Controller (for AWS EKS)
  • Istio (for advanced service mesh capabilities) – See our Istio Ambient Mesh Guide for more.
  • Cilium Gateway API (leveraging eBPF) – Explore eBPF Observability with Hubble for related topics.
  • NGINX Gateway Fabric (a dedicated NGINX implementation)
  • Kong Gateway

Step-by-Step Guide: Migration from Ingress to Gateway API

Step 1: Install Gateway API CRDs

The Gateway API is implemented as a set of Custom Resource Definitions (CRDs). These CRDs define the new API objects like GatewayClass, Gateway, and HTTPRoute. Before you can use any Gateway API resources, you must install these CRDs into your cluster. It’s crucial to install the correct version that matches your chosen Gateway controller and desired feature set. The experimental channel often contains the latest features, while the standard channel offers more stability.

For this guide, we’ll use the experimental channel (v1.1.0) which includes the most recent updates and features, but for production, you might consider the stable channel if your controller supports it and you don’t need the latest experimental features.

# Install Gateway API CRDs (experimental channel for v1.1.0)
# Check if CRDs are already installed to avoid errors
kubectl get crd gatewayclasses.gateway.networking.k8s.io > /dev/null 2>&1 || \
    kubectl apply -k "https://github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v1.1.0"

echo "Waiting for Gateway API CRDs to be ready..."
sleep 10
kubectl get crd gatewayclasses.gateway.networking.k8s.io

Verify: You should see output confirming the gatewayclasses.gateway.networking.k8s.io CRD (and others) are present.

NAME                                         CREATED AT
gatewayclasses.gateway.networking.k8s.io     2023-10-27T10:00:00Z
gateways.gateway.networking.k8s.io           2023-10-27T10:00:00Z
httproutes.gateway.networking.k8s.io         2023-10-27T10:00:00Z
...

Step 2: Deploy a Gateway Controller

Unlike Ingress, where the controller is often a separate installation that watches Ingress resources, the Gateway API uses a GatewayClass resource to bind a specific implementation (controller) to a Gateway. For this example, we will deploy NGINX Gateway Fabric, a dedicated open-source implementation for the Gateway API. NGINX Gateway Fabric provides a robust and performant data plane for handling traffic. Other controllers like Istio or Cilium offer even more advanced functionalities, such as those discussed in Cilium WireGuard Encryption for network security.

# Add NGINX Gateway Fabric Helm repository
helm repo add nginx-gateway-fabric https://nginx.github.io/gateway-api
helm repo update

# Install NGINX Gateway Fabric
# This creates the Deployment, Service, and necessary RBAC for the controller
helm install nginx-gateway-fabric nginx-gateway-fabric/nginx-gateway-fabric \
    --namespace nginx-gateway \
    --create-namespace \
    --set service.type=LoadBalancer # Use LoadBalancer for external access, NodePort or ClusterIP+Ingress for internal testing

echo "Waiting for NGINX Gateway Fabric controller to be ready..."
kubectl rollout status deployment/nginx-gateway-fabric -n nginx-gateway

Verify: Check that the controller Pods are running and the LoadBalancer Service has an external IP/Hostname.

kubectl get pods -n nginx-gateway
kubectl get svc -n nginx-gateway

NAME                                         READY   STATUS    RESTARTS   AGE
nginx-gateway-fabric-7b8c7b8c7-abcde         1/1     Running   0          2m

NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
nginx-gateway-fabric     LoadBalancer   10.43.123.45    EXTERNAL_IP_OR_HOSTNAME   80:30080/TCP,443:30443/TCP   2m

Note down the EXTERNAL-IP_OR_HOSTNAME from the nginx-gateway-fabric service. This will be your entry point.

Step 3: Define a GatewayClass

The GatewayClass resource defines a class of Gateways that share common behavior and are implemented by a specific controller. It acts as a template or blueprint. When you create a Gateway resource, you reference a GatewayClass to indicate which controller should manage it. This separation allows cluster operators to define multiple types of gateways (e.g., “internet-facing-nginx”, “internal-istio”) and delegate their management to different teams or controllers.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: nginx.org/gateway-controller # This must match the controller's identifier

Apply this GatewayClass to your cluster:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: nginx.org/gateway-controller
EOF

Verify: Check the status of the GatewayClass.

kubectl get gatewayclass nginx

NAME    CONTROLLER                    ACCEPTED   AGE
nginx   nginx.org/gateway-controller   True       30s

Step 4: Create a Gateway

The Gateway resource represents a specific instance of a load balancer or proxy that processes network traffic. It defines the entry points (listeners) where traffic comes into the cluster, such as specific ports and protocols (HTTP, HTTPS). A Gateway references a GatewayClass, linking it to the chosen controller (e.g., NGINX Gateway Fabric). This resource is typically managed by cluster operators, defining the underlying infrastructure that application developers can then attach their routes to.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default # Gateways can be namespace-scoped
spec:
  gatewayClassName: nginx # Reference the GatewayClass created in Step 3
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All # Allow HTTPRoutes from all namespaces to attach to this listener
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: example-tls-secret # Replace with your TLS secret name
    allowedRoutes:
      namespaces:
        from: All

Before applying, ensure you have a TLS secret. If not, create a dummy one for testing:

kubectl create secret tls example-tls-secret --cert="$(openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 365 -subj "/CN=example.com" &>/dev/null && cat cert.pem)" --key=key.pem -n default

Now, apply the Gateway resource:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: example-tls-secret
    allowedRoutes:
      namespaces:
        from: All
EOF

Verify: Check the status of your Gateway. It should show an address (the external IP/hostname from the NGINX Gateway Fabric service) and listeners should be ready.

kubectl get gateway my-gateway -n default

NAME         CLASS   ADDRESS                 READY   HTTP   HTTPS   AGE
my-gateway   nginx   EXTERNAL_IP_OR_HOSTNAME   True    80     443     1m

The EXTERNAL_IP_OR_HOSTNAME should match the one from Step 2.

Step 5: Deploy Sample Application

To demonstrate routing, let’s deploy a simple “hello-world” application. This will be our backend service that the Gateway API routes traffic to.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment
  labels:
    app: hello-world
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: nginxdemos/hello:plain-text
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world-service
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Apply these resources:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment
  labels:
    app: hello-world
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: nginxdemos/hello:plain-text
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world-service
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

Verify: Ensure the deployment and service are up.

kubectl get deployment hello-world-deployment
kubectl get service hello-world-service

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
hello-world-deployment   2/2     2            2           30s

NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
hello-world-service ClusterIP   10.43.0.10   <none>        80/TCP    30s

Step 6: Migrate Ingress Rules to HTTPRoute

The HTTPRoute resource is the application-developer facing component of the Gateway API, replacing the functionality of Ingress rules. It defines how HTTP/S requests are routed from a Gateway to backend services. HTTPRoute offers significant improvements over Ingress, including:

• Multiple BackendRefs: Route to multiple services with weighted traffic splitting.
• Advanced Matching: Match based on headers, query parameters, and more, not just host and path.
• Request/Response Modification: Add/remove headers, rewrite paths.
• Policy Attachment: Attach policies (e.g., authentication, rate limiting) to routes.

Let’s consider a common Ingress example and its HTTPRoute equivalent.

Original Ingress Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
spec:
  ingressClassName: nginx # Or another controller
  rules:
  - host: example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: my-api-service
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-frontend-service
            port:
              number: 80
  tls:
  - hosts:
    - example.com
    secretName: example-tls-secret

Equivalent HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: example-http-route
  namespace: default
spec:
  parentRefs:
  - name: my-gateway # Reference the Gateway created earlier
    namespace: default
  hostnames:
  - "example.com" # Matches the Ingress host
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api # Matches Ingress /api pathType: Prefix
    backendRefs:
    - name: hello-world-service # Our sample service
      port: 80
  - matches:
    - path:
        type: PathPrefix
        value: / # Matches Ingress / pathType: Prefix
    backendRefs:
    - name: hello-world-service # Our sample service
      port: 80

Apply this HTTPRoute:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: example-http-route
  namespace: default
spec:
  parentRefs:
  - name: my-gateway
    namespace: default
  hostnames:
  - "example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: hello-world-service
      port: 80
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: hello-world-service
      port: 80
EOF

Verify: Check the status of your HTTPRoute. It should show that it’s attached to the Gateway.

kubectl get httproute example-http-route -n default

NAME                   HOSTNAMES      PARENTREFS         AGE
example-http-route     ["example.com"]   ["my-gateway"]     1m

You can also describe it for more details:

kubectl describe httproute example-http-route -n default

Look for conditions like Accepted: True and Programmed: True.

Step 7: Test and Validate Traffic

Now that everything is set up, it’s time to test if traffic is being routed correctly through your Gateway API resources. You’ll need the EXTERNAL_IP_OR_HOSTNAME you noted earlier from the nginx-gateway-fabric LoadBalancer service.

# Get the external IP/hostname of your Gateway
GATEWAY_IP=$(kubectl get svc -n nginx-gateway nginx-gateway-fabric -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
if [ -z "$GATEWAY_IP" ]; then
  GATEWAY_IP=$(kubectl get svc -n nginx-gateway nginx-gateway-fabric -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
fi
echo "Gateway IP/Hostname: $GATEWAY_IP"

# Test HTTP access to /
curl -H "Host: example.com" http://$GATEWAY_IP/
echo ""
curl -H "Host: example.com" http://$GATEWAY_IP/api
echo ""

# Test HTTPS access (using -k for self-signed cert)
curl -k -H "Host: example.com" https://$GATEWAY_IP/
echo ""
curl -k -H "Host: example.com" https://$GATEWAY_IP/api
echo ""

Verify: You should receive “Hello NGINX!” from the hello-world-deployment for both HTTP and HTTPS requests, and for both / and /api paths, confirming successful routing.

Gateway IP/Hostname: 192.0.2.123 # or a hostname like a12345.elb.us-east-1.amazonaws.com
Hello NGINX!

Hello NGINX!

Hello NGINX!

Hello NGINX!

This confirms that your HTTPRoute is successfully routing traffic for example.com to your hello-world-service via the my-gateway.

Production Considerations

Migrating to Gateway API in a production environment requires careful planning and consideration beyond the basic setup:

• High Availability: Ensure your Gateway controller deployment is highly available, typically with multiple replicas spread across availability zones. The underlying load balancer (e.g., AWS NLB/ALB, GCP L7 LB) configured by your Gateway controller should also be highly available.
• Observability: Integrate your Gateway controller with your existing observability stack. This includes:
  • Metrics: Export Prometheus metrics for traffic, error rates, latency. Consider eBPF Observability with Hubble if using Cilium.
  • Logs: Centralize access logs and error logs for debugging and auditing.
  • Tracing: Implement distributed tracing (e.g., OpenTelemetry) to follow requests through the Gateway and into your services, especially important for complex microservices architectures.
• Security:
  • TLS Management: Automate TLS certificate provisioning and renewal using tools like cert-manager integrated with your Gateway for ACME support or internal CAs.
  • Network Policies: Use Kubernetes Network Policies to restrict traffic flow to and from your Gateway controller Pods, ensuring only expected traffic reaches your services.
  • WAF/DDoS Protection: For internet-facing Gateways, consider integrating with cloud provider WAFs or dedicated DDoS protection services.
  • Authentication/Authorization: Implement authentication and authorization policies directly on HTTPRoute or through an external authorization service (e.g., OPA Gatekeeper).
• Traffic Management Advanced Features:
  • Weighted Round Robin: For canary deployments or A/B testing, use multiple backendRefs with weights in your HTTPRoute.
  • Header/Query Parameter Matching: Leverage advanced matching for fine-grained routing.
  • Traffic Mirroring: Duplicate traffic to a shadow service for testing without impacting production.
  • Rate Limiting & Circuit Breaking: Implement these at the Gateway layer to protect your backend services from overload.
• Cost Optimization: Monitor the cost of the underlying load balancers provisioned by your Gateway controller. Tools like Karpenter Cost Optimization can help manage node costs for your controller Pods if they require dedicated nodes.
• GitOps Workflows: Manage all Gateway API resources (GatewayClass, Gateway, HTTPRoute, etc.) using GitOps principles (e.g., Argo CD, Flux CD) for version control, auditing, and automated deployments.
• Migration Strategy:
  • Phased Rollout: Instead of a big bang, route a small percentage of traffic through the new Gateway API setup, gradually increasing it.
  • DNS Cutover: Once confident, update DNS records to point to the new Gateway’s external IP/hostname.
  • Rollback Plan: Have a clear plan to revert to the Ingress setup if issues arise.

Troubleshooting

1. Gateway or HTTPRoute Not Ready/Programmed

Issue: Your Gateway or HTTPRoute resource shows Programmed: False or Accepted: False in its status conditions.

Solution:

1. Check Controller Logs: The most common reason is an issue with the Gateway controller. Check its logs for errors.

   kubectl logs -n nginx-gateway -l app.kubernetes.io/name=nginx-gateway-fabric
   

2. Verify GatewayClass: Ensure the gatewayClassName in your Gateway matches an existing and accepted GatewayClass.

   kubectl get gatewayclass
   kubectl describe gatewayclass nginx
   

3. Verify ParentRef: For HTTPRoute, ensure the parentRefs correctly point to an existing and ready Gateway. Check namespace and name.

   kubectl describe httproute example-http-route -n default
   

4. RBAC Issues: The controller might lack necessary permissions to create underlying cloud resources (Load Balancers, DNS records). Check controller Pod events and RBAC permissions.

2. External IP/Hostname Not Assigned to Gateway

Issue: The Gateway resource’s status lacks an address in the .status.addresses field.

Solution:

1. Check Controller Service: Verify the Gateway controller’s service (e.g., nginx-gateway-fabric in nginx-gateway namespace) is of type LoadBalancer and has an external IP/hostname.

   kubectl get svc -n nginx-gateway
   

2. Cloud Provider Quotas/Permissions: If running on a cloud provider, ensure you haven’t hit Load Balancer quotas or that the Kubernetes cloud controller manager (or specific Gateway controller) has permissions to provision Load Balancers.
3. Controller Status: Ensure the Gateway controller Pods are running and healthy.

3. Traffic Not Reaching Backend Service

Issue: Requests to the Gateway’s external IP/hostname result in 404, 503, or timeouts.

Solution:

1. DNS Resolution: If using a custom domain, ensure your DNS records correctly point to the Gateway’s external IP/hostname.
2. HTTPRoute Hostnames: Verify the hostnames in your HTTPRoute match the Host header you are sending in your requests.
3. HTTPRoute Matches: Double-check path, header, or queryParam matches in your HTTPRoute rules. Ensure they are correctly configured to direct traffic to the right backendRefs.
4. Service/Pod Health: Ensure your backend service (e.g., hello-world-service) is healthy and its Pods are running and ready.

   kubectl get pods -l app=hello-world
   kubectl get svc hello-world-service
   

5. Firewall/Security Groups: Confirm that network firewalls or cloud security groups allow traffic to reach your Gateway’s external IP/hostname and from the Gateway controller Pods to your backend services. Kubernetes Network Policies could also be blocking traffic.

4. TLS/HTTPS Issues

Issue: HTTPS requests fail, or browsers report certificate errors.

Solution:

1. TLS Secret Name: Verify the certificateRefs.name in your Gateway listener points to an existing and valid Kubernetes TLS secret in the correct namespace.

   kubectl get secret example-tls-secret -n default
   

2. Certificate Contents: Ensure the TLS secret contains valid tls.crt and tls.key entries.
3. SNI Mismatch: If you’re using multiple hostnames on the same Gateway, ensure the client is sending the correct Server Name Indication (SNI) header.
4. Controller TLS Capabilities: Some controllers might have specific requirements or limitations regarding TLS termination. Refer to your chosen controller’s documentation.

5. Incompatible API Version

Issue: You encounter errors like “no matches for kind ‘Gateway’ in version ‘gateway.networking.k8s.io/v1beta1′” when applying resources.

Solution:

1. CRD Installation: Ensure the correct Gateway API CRDs are installed for the API version you are trying to use (e.g., v1, v1beta1).

   kubectl get crd | grep gateway.networking.k8s.io
   

2. API Version in YAML: Double-check the apiVersion field in your Gateway API YAML files. It must match the installed CRD versions.

   apiVersion: gateway.networking.k8s.io/v1 # This should match your installed CRDs
   kind: Gateway
   # ...
   

3. Controller Compatibility: Verify that your Gateway controller supports the specific Gateway API version you are using. Older controllers might only support v1beta1, while newer ones support v1.

FAQ Section

Q1: What are the main advantages of Gateway API over Ingress?

A1: The Gateway API offers several key advantages: a more expressive and extensible API, a clear separation of concerns (infrastructure vs. application routing), support for advanced traffic management features (weighted routing, header matching, traffic mirroring), broader protocol support (TCP, UDP, TLS passthrough), and a more robust policy attachment model. It’s designed to be more role-oriented, allowing platform teams to manage Gateways and application teams to manage Routes, enhancing collaboration and self-service.

Q2: Can I run Ingress and Gateway API simultaneously in the same cluster?

<