Orchestration

Envoy Gateway: Production Deployment Guide

July 5, 2026 Kubezilla Team 17 min read

Introduction

In the complex landscape of modern cloud-native applications, managing external traffic into your Kubernetes clusters is a critical, yet often challenging, endeavor. Traditional Ingress resources, while functional, have shown their limitations in expressing advanced routing, policy, and cross-team collaboration requirements. This is where the Kubernetes Gateway API steps in, offering a more expressive, role-oriented, and extensible approach to traffic management. Designed to overcome the shortcomings of Ingress, Gateway API provides a structured way to define how traffic enters your cluster, from basic HTTP routing to complex L4/L7 policies.

Among the various implementations of the Gateway API, Envoy Gateway stands out as a robust, open-source solution built on the power of Envoy Proxy. Envoy Gateway provides a production-ready, highly performant data plane for your Gateway API resources, offering features like advanced load balancing, traffic splitting, TLS termination, and more. This guide will walk you through a comprehensive production deployment of the Gateway API using Envoy Gateway, empowering you to effectively manage north-south traffic in your Kubernetes environment with confidence and precision. For a deeper dive into the transition from Ingress to Gateway API, check out our Kubernetes Gateway API vs Ingress: The Complete Migration Guide.

TL;DR: Envoy Gateway Production Deployment

Deploying Gateway API with Envoy Gateway involves installing the Gateway API CRDs, then deploying Envoy Gateway itself. You’ll then define GatewayClass, Gateway, and HTTPRoute resources to expose your applications. This setup provides advanced traffic management capabilities, acting as a modern ingress solution.

Key Commands:

# Install Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

# Install Envoy Gateway
helm install envoy-gateway oci://docker.io/envoyproxy/envoy-gateway-charts/envoy-gateway --version v1.0.0 -n envoy-gateway-system --create-namespace

# Deploy a sample application
kubectl create deployment my-app --image=nginx --replicas=3
kubectl expose deployment my-app --port=80 --type=ClusterIP

# Create GatewayClass (usually handled by Envoy Gateway install)
# Create Gateway
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "my-app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-app
      port: 80
EOF

# Get external IP of the Gateway
kubectl get svc -n envoy-gateway-system envoy-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Prerequisites

Before embarking on your Envoy Gateway production deployment, ensure you have the following:

  • A Running Kubernetes Cluster: Version 1.25 or higher is recommended to ensure full compatibility with the latest Gateway API features. This can be a local cluster (e.g., Minikube, Kind) for testing, or a cloud-managed cluster (EKS, GKE, AKS) for production.
  • kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
  • helm: The Kubernetes package manager, version 3.x or higher. Envoy Gateway is primarily deployed via Helm charts. Install Helm from its official website.
  • Basic Kubernetes Knowledge: Familiarity with core Kubernetes concepts such as Deployments, Services, Pods, and Namespaces is essential.
  • Domain Name (Optional but Recommended): For production deployments, having a registered domain name and access to its DNS configuration will allow you to properly configure hostnames and TLS certificates.

Step-by-Step Guide: Envoy Gateway Production Deployment

Step 1: Install Gateway API CRDs

The Kubernetes Gateway API defines a set of Custom Resource Definitions (CRDs) that extend Kubernetes to understand new traffic management concepts. These CRDs must be installed in your cluster before you can deploy any Gateway API resources or an implementation like Envoy Gateway. Installing them separately ensures that your cluster has the foundational definitions, allowing for independent upgrades of the API itself.

It’s crucial to install the correct version of the CRDs to match the Gateway API version supported by your chosen Envoy Gateway release. The official Kubernetes SIGs Gateway API repository provides these standard installation manifests.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

Verify

Confirm that the Gateway API CRDs have been installed successfully by listing them. You should see resources like gateways.gateway.networking.k8s.io, httproutes.gateway.networking.k8s.io, etc.

kubectl get crd | grep gateway.networking.k8s.io

Expected Output:

gatewayclasses.gateway.networking.k8s.io      2023-11-20T12:00:00Z
gateways.gateway.networking.k8s.io            2023-11-20T12:00:00Z
httproutes.gateway.networking.k8s.io          2023-11-20T12:00:00Z
referencegrants.gateway.networking.k8s.io     2023-11-20T12:00:00Z
tcproutes.gateway.networking.k8s.io           2023-11-20T12:00:00Z
tlsroutes.gateway.networking.k8s.io           2023-11-20T12:00:00Z
udproutes.gateway.networking.k8s.io           2023-11-20T12:00:00Z

Step 2: Install Envoy Gateway

Envoy Gateway is a production-grade implementation of the Gateway API that uses Envoy Proxy as its data plane. Deploying it involves adding the Envoy Gateway Helm repository and then installing the chart. This will create a dedicated namespace (envoy-gateway-system by default) and deploy the necessary controller components, along with an Envoy Proxy deployment and service.

The Helm chart simplifies the deployment process, handling the creation of Deployments, Services, RBAC roles, and the default GatewayClass resource that Envoy Gateway manages. For advanced networking scenarios, consider integrating with solutions like Cilium WireGuard Encryption to secure traffic within your cluster.

helm install envoy-gateway oci://docker.io/envoyproxy/envoy-gateway-charts/envoy-gateway --version v1.0.0 -n envoy-gateway-system --create-namespace

Verify

Check the status of the Envoy Gateway deployment. It may take a few minutes for all pods to become ready and for the LoadBalancer service to provision an external IP.

kubectl get all -n envoy-gateway-system

Expected Output (truncated):

NAME                                             READY   STATUS    RESTARTS   AGE
pod/envoy-gateway-65b79c5c7d-abcde                1/1     Running   0          2m
pod/envoy-gateway-envoy-6789abcd-efgh             1/1     Running   0          1m

NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
service/envoy-gateway                            LoadBalancer   10.43.123.45     34.123.45.67    80:30080/TCP,443:30443/TCP   2m
service/envoy-gateway-metrics                    ClusterIP      10.43.54.32      <none>          8080/TCP                     2m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/envoy-gateway               1/1     1            1           2m
deployment.apps/envoy-gateway-envoy         1/1     1            1           1m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/envoy-gateway-65b79c5c7d               1         1         1       2m
replicaset.apps/envoy-gateway-envoy-6789abcd           1         1         1       1m

Note the EXTERNAL-IP for the envoy-gateway service. This is your entry point to the cluster.

Step 3: Deploy a Sample Application

To demonstrate traffic routing, we’ll deploy a simple Nginx application. This application will serve as the backend for our Gateway API configuration. We’ll create a Deployment and expose it via a ClusterIP Service.

This mimics a typical microservice that you want to expose to external users. For more advanced application deployments, consider practices like GPU scheduling for LLMs if your application has specific hardware requirements.

kubectl create deployment my-app --image=nginx --replicas=3
kubectl expose deployment my-app --port=80 --type=ClusterIP

Verify

Ensure the deployment and service are up and running in the default namespace.

kubectl get deploy,svc my-app

Expected Output:

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-app     3/3     3            3           30s

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/my-app     ClusterIP   10.43.200.100   <none>        80/TCP    20s

Step 4: Create a Gateway Resource

The Gateway resource is the entry point for traffic into your cluster. It defines the listeners (ports, protocols) and references a GatewayClass, which specifies the controller responsible for implementing the Gateway (in our case, Envoy Gateway). This resource is typically managed by infrastructure providers or cluster operators.

Here, we define an HTTP listener on port 80. The allowedRoutes section specifies which namespaces are permitted to attach routes to this Gateway. Setting from: All is common for initial setups but should be refined for production environments using specific namespace selectors for better isolation, aligning with principles discussed in our Network Policies Security Guide.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
spec:
  gatewayClassName: envoy-gateway
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All # In production, restrict this to specific namespaces

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
spec:
  gatewayClassName: envoy-gateway
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All # In production, restrict this to specific namespaces
EOF

Verify

Check the status of your Gateway. It should show as “Programmed” once Envoy Gateway has reconciled it and created the corresponding Envoy Proxy configuration and service.

kubectl get gateway my-gateway -n default

Expected Output (ADDRESS will be the same as the Envoy Gateway service’s external IP):

NAME         CLASS             ADDRESS            PROGRAMMED   AGE
my-gateway   envoy-gateway     34.123.45.67       True         30s

Step 5: Create an HTTPRoute Resource

The HTTPRoute resource defines how HTTP traffic arriving at a Gateway should be routed to backend services. This is where application developers or application operators primarily interact with the Gateway API. It allows for advanced routing rules based on hostnames, paths, headers, and more.

In this step, we create an HTTPRoute that attaches to my-gateway, listens for traffic to my-app.example.com, and routes all requests (PathPrefix: /) to our my-app service on port 80. This separation of concerns (Gateway managed by infra vs. HTTPRoute by app dev) is a core benefit of the Gateway API.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
  namespace: default
spec:
  parentRefs:
  - name: my-gateway
    namespace: default # Important: specify namespace if Gateway is not in the same namespace
  hostnames:
  - "my-app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-app
      port: 80

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
  namespace: default
spec:
  parentRefs:
  - name: my-gateway
    namespace: default
  hostnames:
  - "my-app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-app
      port: 80
EOF

Verify

Check the status of your HTTPRoute. It should show as “Accepted” and “Programmed” once it’s successfully attached to the Gateway and the routing rules are active.

kubectl get httproute my-app-route -n default

Expected Output:

NAME           HOSTNAMES              PARENTREFS        AGE   STATUS
my-app-route   ["my-app.example.com"]   ["my-gateway"]    30s   Accepted

Step 6: Test the Deployment

Now that everything is configured, it’s time to test if traffic is correctly routed through Envoy Gateway to your application. You’ll need the external IP of your Envoy Gateway LoadBalancer service and a way to resolve the hostname my-app.example.com to that IP. For local testing, you can modify your /etc/hosts file. For production, you would configure your DNS provider.

First, get the external IP of the Gateway:

GATEWAY_IP=$(kubectl get svc -n envoy-gateway-system envoy-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "Gateway External IP: $GATEWAY_IP"

Expected Output:

Gateway External IP: 34.123.45.67

Next, use curl to send a request, specifying the hostname via the Host header. If you’ve updated your /etc/hosts file to map my-app.example.com to $GATEWAY_IP, you can simply use curl http://my-app.example.com.

curl -H "Host: my-app.example.com" http://$GATEWAY_IP

Verify

You should receive the default Nginx welcome page as output, indicating that the request was successfully routed through Envoy Gateway to your my-app service.

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Production Considerations

Deploying Envoy Gateway in production requires more than just basic installation. Consider these crucial aspects to ensure reliability, security, and scalability:

  1. TLS Configuration: For any production workload, TLS encryption is non-negotiable. Envoy Gateway supports TLS termination on the Gateway resource. You can use Kubernetes Secrets to store certificates, often provisioned by cert-manager. 
    # Example Gateway with HTTPS listener
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-secure-gateway
  namespace: default
spec:
  gatewayClassName: envoy-gateway
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: my-tls-secret # Secret containing 'tls.crt' and 'tls.key'
    allowedRoutes:
      namespaces:
        from: All
  2. Custom Domains and DNS: Ensure your domain’s A/CNAME records point to the external IP of your Envoy Gateway service. Automate this process if possible, especially in cloud environments, using external DNS controllers.
  3. Resource Management: Configure appropriate CPU and memory limits/requests for the Envoy Gateway controller and Envoy proxy pods. Monitor resource usage closely and scale horizontally as needed. For cost optimization, consider tools like Karpenter to manage underlying nodes efficiently.
  4. Observability: Integrate Envoy Gateway with your existing monitoring and logging stacks. Envoy Proxy provides rich metrics (Prometheus format) and access logs. Use tools like Prometheus, Grafana, and Loki/Fluentd to gain insights. For advanced eBPF-based observability, explore solutions like eBPF Observability with Hubble.
  5. High Availability: Deploy Envoy Gateway with multiple replicas for the controller and Envoy proxy pods. Ensure the underlying infrastructure (e.g., cloud LoadBalancer) is highly available.
  6. Security Policies: Implement strict allowedRoutes policies on your Gateway resources to control which namespaces can attach routes. Leverage Kubernetes Network Policies to restrict traffic to and from the Envoy Gateway pods. Consult our Network Policies Security Guide for best practices.
  7. GitOps Workflow: Manage all Gateway API resources (Gateway, HTTPRoute, TLSRoute, etc.) using a GitOps approach. Tools like Argo CD or Flux CD can automate the deployment and synchronization of your configurations from a Git repository.
  8. Rate Limiting and WAF: For advanced traffic control and security, explore integrating with Envoy’s native rate limiting capabilities or deploying a Web Application Firewall (WAF) in front of or alongside Envoy Gateway.
  9. Canary Deployments and A/B Testing: Leverage Envoy Gateway’s advanced routing capabilities to implement traffic splitting for canary releases or A/B testing. 
    # Example HTTPRoute for Canary Deployment
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-canary-route
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "canary.example.com"
  rules:
  - backendRefs:
    - name: my-app-v1 # 90% traffic to stable
      port: 80
      weight: 90
    - name: my-app-v2 # 10% traffic to canary
      port: 80
      weight: 10
  10. Service Mesh Integration: While Envoy Gateway handles north-south traffic, a service mesh like Istio Ambient Mesh can manage east-west (inter-service) traffic within your cluster, providing advanced features like mTLS, circuit breaking, and more. Envoy Gateway and a service mesh can complement each other, with the Gateway handling inbound traffic and the mesh managing internal service communication.

Troubleshooting

  1. Issue: Gateway Stuck in “Pending” State or No External IP

    Explanation: This usually means the underlying cloud provider (AWS, GCP, Azure) is failing to provision a LoadBalancer for the envoy-gateway service. This can be due to insufficient permissions, quota limits, or network configuration issues in your cloud environment.

    Solution:

    • Check the events of the envoy-gateway service: 
      kubectl describe svc envoy-gateway -n envoy-gateway-system

      Look for errors related to LoadBalancer provisioning.

    • Verify your cluster’s cloud provider integration and permissions. For example, on AWS EKS, ensure the IAM role associated with your worker nodes has permissions to create ELBs/NLBs.
    • If running on bare-metal or a local cluster, ensure you have a LoadBalancer implementation (e.g., MetalLB) installed and configured.
  2. Issue: HTTPRoute Status is “Not Accepted” or “Invalid”

    Explanation: The Gateway API controller (Envoy Gateway) is unable to process your HTTPRoute resource. This often happens due to misconfigurations in the YAML, incorrect parentRefs, or issues with backend service references.

    Solution:

    • Examine the status field of the HTTPRoute for detailed error messages: 
      kubectl get httproute my-app-route -n default -o yaml

      Look under status.parents for conditions like Accepted: false and the associated reasons.

    • Ensure the parentRefs correctly points to an existing Gateway resource by name and namespace.
    • Verify that the backendRefs (service name and port) are correct and the service exists in the specified namespace.
    • Check for conflicting hostnames or path prefixes across multiple HTTPRoute resources attached to the same Gateway.
  3. Issue: “404 Not Found” when accessing the application via Gateway

    Explanation: The Gateway is reachable, but it’s not correctly routing traffic to your application. This points to an issue with the HTTPRoute configuration or the backend service.

    Solution:

    • Double-check the hostnames in your HTTPRoute. Ensure the Host header in your curl command (or browser request) exactly matches one of the defined hostnames.
    • Verify the path matching rules (PathPrefix, Exact, RegularExpression).
    • Confirm the backendRefs service name and port are correct and that the service is running and has healthy endpoints. 
      kubectl get endpoints my-app -n default

      The ENDPOINTS column should show active pod IPs.

    • Check the Envoy Gateway controller logs for any routing errors: 
      kubectl logs -f -n envoy-gateway-system deployment/envoy-gateway
  4. Issue: TLS Handshake Errors or Certificate Issues

    Explanation: If you’ve configured HTTPS and are encountering errors like “SSL_ERROR_RX_RECORD_TOO_LONG” or certificate warnings, it indicates a problem with your TLS setup on the Gateway.

    Solution:

    • Ensure the Secret referenced in tls.certificateRefs of your Gateway resource exists in the correct namespace and contains valid tls.crt and tls.key entries. 
      kubectl get secret my-tls-secret -n default -o yaml
    • Verify the certificate and key are in PEM format.
    • Confirm that the domain name in your certificate matches the hostname configured in your HTTPRoute.
    • Check that the listener on the Gateway is using protocol: HTTPS and port: 443.
    • If using cert-manager, check its logs and events for certificate issuance failures.
  5. Issue: High CPU/Memory Usage by Envoy Gateway Pods

    Explanation: Excessive resource consumption by the Envoy Gateway controller or Envoy proxy pods can indicate heavy traffic, misconfiguration leading to inefficient routing, or a potential memory leak.

    Solution:

    • Monitor resource usage using kubectl top pods -n envoy-gateway-system or your cluster’s monitoring solution.
    • Increase resource limits/requests in the Helm chart values if necessary (e.g., controller.resources, envoy.resources).
    • Scale out the Envoy proxy replicas if traffic is high: envoy.replicas: 2 in Helm values.
    • Review your HTTPRoute configurations for overly complex regex matching or a large number of routes, which can increase Envoy’s configuration load.
    • Check for continuous configuration updates (e.g., from a constantly changing backend) that might churn Envoy’s configuration.
  6. Issue: Envoy Proxy Pods Crashing or Restarting Frequently

    Explanation: Frequent restarts (CrashLoopBackOff) of Envoy proxy pods indicate a critical issue, often related to invalid Envoy configuration generated by the Envoy Gateway controller or resource constraints.

    Solution:

    • Check the logs of the crashing Envoy proxy pod: 
      kubectl logs -f <envoy-proxy-pod-name> -n envoy-gateway-system

      Look for errors related to configuration loading or startup.

    • Examine the Envoy Gateway controller logs for errors during configuration generation.
    • Ensure the Envoy proxy pods have sufficient memory and CPU resources. Increase limits if necessary.
    • Rollback any recent changes to Gateway API resources (Gateway, HTTPRoute, etc.) that might have introduced an invalid configuration.

FAQ Section

  1. Q: What is the main difference between Gateway API and Ingress?

    A: The Gateway API is a more expressive, extensible, and role-oriented API compared to Ingress. Ingress is primarily focused on HTTP routing, often limited in features, and less extensible. Gateway API introduces resources like GatewayClass, Gateway, and various Route types (HTTPRoute, TLSRoute, etc.) to separate concerns between infrastructure providers, cluster operators, and application developers, offering advanced traffic management features out-of-the-box. For a detailed comparison, see our Gateway API vs Ingress Migration Guide.

  2. Q: Can I use Envoy Gateway with a service mesh like Istio?

    A: Yes, Envoy Gateway can complement a service mesh. Envoy Gateway typically handles “north-south” traffic (traffic entering and exiting the cluster), while a service mesh like Istio manages “east-west” traffic (inter-service communication within the cluster). You can deploy Envoy Gateway at the edge of your cluster and have it route traffic to services that are part of your service mesh. For more on Istio, check out our Istio Ambient Mesh Production Guide.

  3. Q: How do I handle advanced routing features like traffic splitting or header-based routing with Envoy Gateway?

    A: The Gateway API’s HTTPRoute resource provides rich capabilities for advanced routing. You can define multiple rules with matches based on headers, query parameters, or methods. Traffic splitting for canary deployments or A/B testing is achieved using backendRefs with weight fields, as shown in the Production Considerations section.

  4. Q: Is Envoy Gateway suitable for multi-cluster deployments?

    A: While Envoy Gateway itself is deployed per cluster, its design aligns well with multi-cluster strategies. You can deploy Envoy Gateway in each cluster and use a global load balancer or DNS-based routing to distribute traffic across your clusters. The Gateway API standard itself is being extended to support multi-cluster concepts, and various tools are emerging to facilitate this.

  5. Q: How do I secure my container images and supply chain when using Envoy Gateway?

    A: Securing your container supply chain is crucial. This involves using trusted base images, scanning for vulnerabilities, and signing your images. Tools like Sigstore can provide cryptographic guarantees about the origin and integrity of your software. You can enforce policies using Kubernetes admission controllers like Kyverno to ensure only signed images are deployed. For more on this, refer to Securing Container Supply Chains with Sigstore and Kyverno.

Cleanup Commands

To remove all resources created during this tutorial:

# Delete the HTTPRoute and Gateway resources

kubectl delete httproute my-app-route -n default

kubectl delete gateway my-gateway -n default


# Delete the sample application

kubectl delete deployment my-app -n default

kubectl delete service my-app -n default


# Uninstall Envoy Gateway via Helm

helm uninstall envoy-gateway -n envoy-gateway-system

kubectl delete namespace envoy-gateway-system


# Uninstall Gateway API CRDs (optional, only if no other Gateway API implementation is in use)

kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

Leave a comment