Orchestration

Kyverno: Secure Kubernetes with Policy

August 4, 2026 Kubezilla Team 16 min read

Introduction

In the dynamic world of Kubernetes, maintaining governance, security, and operational best practices across a sprawling cluster can be a daunting task. As your infrastructure scales, ensuring that every deployed resource adheres to organizational standards becomes increasingly complex. Traditional approaches often involve manual checks, pre-deployment scripts, or custom admission controllers, each with its own set of limitations and maintenance overhead. This is where policy engines like Kyverno step in, offering a declarative, Kubernetes-native solution to enforce policies directly within your cluster.

Kyverno, a CNCF graduated project, acts as a powerful admission controller that can validate, mutate, and generate Kubernetes resources based on defined policies. Unlike other policy engines that require learning a new language (like Rego for OPA/Gatekeeper), Kyverno uses standard Kubernetes YAML, making it incredibly accessible for anyone familiar with Kubernetes. From ensuring proper labels and annotations to restricting image registries, enforcing resource limits, or even auto-generating network policies, Kyverno provides a flexible and robust framework for policy enforcement, significantly enhancing your cluster’s security posture and operational consistency.

TL;DR

Kyverno is a Kubernetes-native policy engine that validates, mutates, and generates resources using declarative policies. Install it with Helm, then define ClusterPolicy or Policy resources to enforce rules like image registry restrictions, label requirements, or resource limits.

# Add Kyverno Helm repository
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

# Install Kyverno
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

# Example: Apply a policy to restrict image registries
kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-image-registries
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Images must come from approved registries (e.g., myregistry.com)."
      pattern:
        spec:
          containers:
            - image: "myregistry.com/*"
          initContainers:
            - image: "myregistry.com/*"
EOF

# Test the policy (this should be blocked)
kubectl run nginx --image=nginx:latest

# Cleanup
helm uninstall kyverno --namespace kyverno
kubectl delete clusterpolicy restrict-image-registries

Prerequisites

Before diving into Kyverno, ensure you have the following:

  • Kubernetes Cluster: A running Kubernetes cluster (version 1.16+). This guide was tested on Kubernetes 1.27. You can use Kind, Minikube, or any cloud provider’s Kubernetes service (EKS, GKE, AKS).
  • kubectl: The Kubernetes command-line tool, configured to connect to your cluster. You can find installation instructions on the official Kubernetes documentation.
  • Helm: The Kubernetes package manager, version 3.x or higher. If you don’t have it, follow the Helm installation guide.
  • Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Namespaces, and Custom Resource Definitions (CRDs).

Step-by-Step Guide

Step 1: Install Kyverno

Kyverno is best installed using Helm, which simplifies the deployment and management of its components, including the admission controller, CRDs, and necessary RBAC rules. We’ll add the Kyverno Helm repository and then install the chart into its own namespace. This ensures a clean separation of Kyverno’s components from other applications in your cluster.

The installation will deploy several components: a Kyverno Deployment, a Service, various ClusterRoles and ClusterRoleBindings for necessary permissions, and the Custom Resource Definitions (CRDs) for Policies and ClusterPolicies. The admission controller webhook will be registered with the Kubernetes API server, allowing Kyverno to intercept API requests.

# Add the Kyverno Helm repository
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

# Install Kyverno into its own namespace
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

# Verify the installation
kubectl get pods --namespace kyverno

Verify Installation

You should see Kyverno pods running in the `kyverno` namespace.

NAME                      READY   STATUS    RESTARTS   AGE
kyverno-698f797d5-abcde   1/1     Running   0          2m

Step 2: Create a Basic Validation Policy (Restrict Image Registries)

One of the most common security requirements is to restrict where container images can originate. This helps prevent the use of untrusted or vulnerable images. Kyverno allows you to define policies that validate incoming resources, rejecting those that don’t meet the specified criteria. In this example, we’ll create a `ClusterPolicy` to ensure all Pods (and their controllers) use images from a specific registry, such as `myregistry.com`.

The `validationFailureAction: Enforce` setting means that any resource violating this policy will be rejected by the Kubernetes API server. If set to `Audit`, violations would only be logged, allowing the resource to be created. The `match` block specifies which resources the policy applies to, and the `validate` block contains the actual rules, using a pattern-matching approach. For more advanced security policies, consider how Kyverno can integrate with tools like Sigstore for image signing.

# policy-restrict-registry.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-image-registries
    match:
      any:
      - resources:
          kinds:
          - Pod
          - Deployment
          - StatefulSet
          - DaemonSet
    validate:
      message: "Images must come from approved registries (e.g., myregistry.com). Please use a trusted image source."
      pattern:
        spec:
          containers:
            - image: "myregistry.com/*"
          initContainers:
            - image: "myregistry.com/*"

Apply this policy:

kubectl apply -f policy-restrict-registry.yaml

Verify Policy Enforcement

Now, try to create a Pod with an unapproved image.

kubectl run untrusted-nginx --image=nginx:latest

Expected Output

This command should fail, showing that Kyverno has blocked the creation.

Error from server (Images must come from approved registries (e.g., myregistry.com). Please use a trusted image source.): admission webhook "validate.kyverno.svc-fail" denied the request: validation failure: Images must come from approved registries (e.g., myregistry.com). Please use a trusted image source.

Now, try with an approved image (this will still fail as `myregistry.com` doesn’t exist, but it demonstrates the policy pattern).

kubectl run trusted-nginx --image=myregistry.com/nginx:latest

This should succeed (from Kyverno’s perspective, though the Pod will likely fail to pull the image).

pod/trusted-nginx created

Step 3: Create a Mutation Policy (Add Labels)

Mutation policies automatically modify resources as they are created or updated, ensuring consistency and adherence to standards without requiring manual intervention. A common use case is to ensure all resources within a certain namespace or with a specific label automatically get additional, required labels or annotations. Here, we’ll create a policy to automatically add a `team: devops` label and an `owner: platform` annotation to any new Pod created in the `default` namespace.

Mutation policies are incredibly useful for enforcing organizational metadata standards, which can be critical for cost allocation (e.g., with tools like Karpenter for cost optimization), observability, and security. The `patchesJson6902` field allows you to define JSON Patch operations to add, remove, or replace fields in the resource.

# policy-add-labels.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-required-labels
spec:
  validationFailureAction: Audit # Audit mode for mutation policies is often safer initially
  rules:
  - name: add-team-label-and-owner-annotation
    match:
      any:
      - resources:
          kinds:
          - Pod
          namespaces:
          - default # Apply only to the default namespace for this example
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            team: devops
          annotations:
            owner: platform

Apply this policy:

kubectl apply -f policy-add-labels.yaml

Verify Policy Mutation

Create a simple Pod in the `default` namespace.

kubectl run test-pod --image=busybox --command sleep 3600

Check the labels and annotations of the newly created Pod.

kubectl get pod test-pod -o yaml | grep -E 'labels:|annotations:|team:|owner:'

Expected Output

You should see the `team` label and `owner` annotation automatically added.

  annotations:
    owner: platform
  labels:
    team: devops
    run: test-pod

Step 4: Create a Generation Policy (Auto-generate NetworkPolicy)

Generation policies are a powerful feature that allows Kyverno to create new resources based on the creation or update of other resources. This is particularly useful for automating the creation of boilerplate configurations. Here, we’ll configure Kyverno to automatically generate a basic NetworkPolicy for every new Namespace created, ensuring that all new namespaces start with a baseline level of network isolation.

This policy will create a `NetworkPolicy` that denies all ingress and egress traffic by default for any new namespace. This “deny-all” approach is a strong security baseline, forcing applications to explicitly define their communication needs. This can be combined with advanced CNI solutions like Cilium for enhanced network security.

# policy-generate-networkpolicy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-deny-networkpolicy
spec:
  validationFailureAction: Audit # Audit for generation policies
  rules:
  - name: generate-network-policy
    match:
      any:
      - resources:
          kinds:
          - Namespace
    generate:
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      name: default-deny
      namespace: "{{request.object.metadata.name}}" # Target the new namespace
      data:
        spec:
          podSelector: {}
          policyTypes:
          - Ingress
          - Egress

Apply this policy:

kubectl apply -f policy-generate-networkpolicy.yaml

Verify Policy Generation

Create a new namespace.

kubectl create namespace my-app-namespace

Check if the `NetworkPolicy` was created in the new namespace.

kubectl get networkpolicy -n my-app-namespace

Expected Output

You should see the `default-deny` NetworkPolicy.

NAME           POD-SELECTOR   AGE
default-deny   <none>         10s

Step 5: Using Resource Limits and Requests Policy

Enforcing resource limits and requests is crucial for cluster stability, performance, and cost management. Without them, a runaway application could consume all available resources, impacting other workloads. This policy ensures that all containers within Pods specify CPU and memory limits and requests. This is especially important for large-scale deployments or when running resource-intensive workloads like those for LLMs with GPU scheduling.

This `ClusterPolicy` uses a `validate` rule with a `pattern` to check for the presence of `resources.limits` and `resources.requests` for both `cpu` and `memory`. The `validationFailureAction: Enforce` ensures that any Pods violating this policy are rejected.

# policy-resource-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-resources
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Containers must have CPU and memory requests and limits defined."
      pattern:
        spec:
          containers:
          - resources:
              requests:
                cpu: "?*" # Requires cpu request to be present
                memory: "?*" # Requires memory request to be present
              limits:
                cpu: "?*" # Requires cpu limit to be present
                memory: "?*" # Requires memory limit to be present

Apply this policy:

kubectl apply -f policy-resource-limits.yaml

Verify Policy Enforcement

Try to create a Pod without resource limits.

kubectl run no-limits-pod --image=busybox --command sleep 3600

Expected Output

This should be blocked by Kyverno.

Error from server (Containers must have CPU and memory requests and limits defined.): admission webhook "validate.kyverno.svc-fail" denied the request: validation failure: Containers must have CPU and memory requests and limits defined.

Now, create a Pod with proper limits.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: with-limits-pod
spec:
  containers:
  - name: busybox
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"
EOF

This should succeed.

pod/with-limits-pod created

Production Considerations

Deploying Kyverno in a production environment requires careful planning to ensure stability, performance, and security.

  • High Availability: Deploy Kyverno with multiple replicas to ensure that policy enforcement continues even if one instance fails. The Helm chart defaults to 2 replicas, which is a good starting point.
  • Resource Limits: Set appropriate CPU and memory requests/limits for the Kyverno pods themselves. Monitor their resource usage using tools like eBPF Observability with Hubble to fine-tune these values.
  • Monitoring and Alerting: Integrate Kyverno metrics into your monitoring system (Prometheus, Grafana). Kyverno exposes Prometheus metrics that provide insights into policy evaluation times, rule hits, and API call latency. Set up alerts for policy failures or high latency.
  • Policy Testing: Always test new policies thoroughly in a staging environment before applying them to production. Kyverno offers a `dryRun` mode for policies, allowing you to see the effects without actually enforcing them. The Kyverno CLI also allows testing policies locally.
  • `validationFailureAction`: Start with `Audit` for new policies, especially mutation and generation policies, to observe their impact without blocking workloads. Once confident, switch to `Enforce`.
  • Policy Granularity: Design policies to be as granular as possible. Avoid overly broad `match` rules that could unintentionally impact critical applications. Use `namespaces`, `labelSelectors`, and `resourceSelectors` to target policies precisely.
  • Exemptions: Be prepared to create exemptions for critical system namespaces (e.g., `kube-system`, `kyverno`, `istio-system` if you’re using Istio Ambient Mesh). Kyverno policies can include `exclude` blocks to prevent policies from applying to specific resources or namespaces.
  • Policy Management: Store your Kyverno policies in a Git repository and manage them using GitOps practices. This provides version control, auditability, and easier deployment.
  • Logging: Ensure Kyverno’s logs are collected and sent to a centralized logging system. This is crucial for debugging and auditing policy enforcement.
  • Performance: Large numbers of complex policies can introduce latency to API requests. Monitor the `kyverno_admission_review_duration_seconds` metric to identify potential bottlenecks. Optimize policies by making `match` rules as specific as possible.

Troubleshooting

Here are some common issues you might encounter with Kyverno and their solutions.

1. Kyverno Pods Not Running

Issue: Kyverno pods are stuck in Pending or CrashLoopBackOff state.

Solution:
Check the pod logs and events. Common causes include insufficient resources, incorrect RBAC permissions, or issues with the webhook configuration.

kubectl get pods -n kyverno
kubectl describe pod -n kyverno <kyverno-pod-name>
kubectl logs -n kyverno <kyverno-pod-name>

Ensure your cluster has enough resources. If it’s an RBAC issue, verify the `ClusterRole` and `ClusterRoleBinding` created by the Helm chart are correct and have sufficient permissions.

2. Policies Not Being Enforced (Audit Mode)

Issue: You apply a policy, but resources violating it are still created.

Solution:
Check the `validationFailureAction` field in your policy. If it’s set to `Audit`, the policy will only log violations without blocking the resource. Change it to `Enforce` to block violating resources.

spec:
  validationFailureAction: Enforce # Change from Audit to Enforce

Also, verify that the `match` and `exclude` blocks in your policy correctly target the resources you intend.

3. Admission Webhook Errors

Issue: API server logs show errors related to Kyverno’s admission webhook (e.g., “connection refused,” “timeout”).

Solution:
This often indicates network connectivity issues between the API server and Kyverno service, or that Kyverno is overloaded.

  1. Check Kyverno pod health and logs (see issue 1).
  2. Verify the Kyverno service is reachable: 
    kubectl get svc -n kyverno kyverno
kubectl describe svc -n kyverno kyverno
  3. Check for network policies that might be blocking communication. Refer to the Kubernetes Network Policies Guide.
  4. Increase Kyverno’s resource limits if it’s under heavy load.

4. Policy Does Not Apply to Existing Resources

Issue: A new policy is applied, but existing resources that violate it are not affected.

Solution:
Kyverno’s admission controller only intercepts new resource creations and updates. It does not retroactively apply policies to existing resources by default. To audit or enforce policies on existing resources, you can use:

  • Kyverno CLI: Use `kyverno apply –cluster-policies … –resource …` to test policies against existing resources.
  • Kyverno Reports: Enable policy reports in Kyverno (they are on by default in recent versions) and check `PolicyReport` or `ClusterPolicyReport` resources for violations on existing resources. 
    kubectl get clusterpolicyreport
  • Re-apply Resources: For critical policies, you might need to re-apply or restart existing deployments after the policy is in `Enforce` mode.

5. Unexpected Mutations or Generations

Issue: Kyverno is mutating or generating resources in ways you didn’t intend.

Solution:
Mutation and generation policies can be powerful but also dangerous if misconfigured.

  1. Set `validationFailureAction: Audit` for mutation/generation policies initially to observe their effects without making permanent changes.
  2. Carefully review the `match` and `generate` / `mutate` blocks of your policy. Use `kubectl describe clusterpolicy <policy-name>` to review the full YAML.
  3. Check Kyverno’s logs for details on which policies are triggering and what changes are being made.
  4. Use the Kyverno CLI’s `test` command to simulate policy application on specific resources.

6. Policy Error: “JSONPath expression could not be found”

Issue: A validation or mutation policy fails with an error indicating a JSONPath expression could not be found.

Solution:
This usually means that the path you’re trying to access in the resource (e.g., `spec.containers[0].image`) does not exist in the resource being evaluated.

  1. Ensure the `match` block correctly targets resources that are expected to have the field. For example, if you’re checking `spec.containers`, ensure the policy matches `Pod` or controller kinds.
  2. Use optional fields (`?`) in your patterns if a field might not always be present. For example, `spec.containers[?].image` will match if containers exist, even if some don’t have an image field.
  3. Check the exact structure of the resource using `kubectl get <kind> <name> -o yaml`.

FAQ Section

Q1: What is the difference between Kyverno and OPA/Gatekeeper?

A1: The primary difference lies in the policy language. Kyverno uses standard Kubernetes YAML and JSON Patch for policies, making it very accessible to Kubernetes users. OPA (Open Policy Agent) with Gatekeeper uses Rego, a purpose-built policy language. While Rego is very powerful and flexible, it has a steeper learning curve. Kyverno is designed to be Kubernetes-native from the ground up, offering specific features like image verification and auto-generation that might require more complex Rego policies or external tools with OPA. For image signing and verification, Kyverno integrates well with tools like Sigstore.

Q2: Can Kyverno enforce policies on existing resources?

A2: By default, Kyverno’s admission controller only acts on new resource creation or updates. However, Kyverno does offer reporting capabilities (via `PolicyReport` and `ClusterPolicyReport` CRDs) that can audit existing resources against policies. You can also use the Kyverno CLI to scan existing resources against your policies locally. For active enforcement on existing resources, you would typically need to re-apply or restart those resources after the policy is in place.

Q3: How do I exclude certain namespaces or resources from a policy?

A3: Kyverno policies support `exclude` blocks within rules. You can exclude resources based on their kind, name, namespace, or labels. For example, to exclude the `kube-system` namespace:

rules:
- name: my-rule
  match:
    any:
    - resources:
        kinds:
        - Pod
  exclude:
    any:
    - resources:
        namespaces:
        - kube-system
  validate:
    ...

Q4: Does Kyverno support policy versioning and GitOps?

A4: Yes! Kyverno policies are Kubernetes Custom Resources (CRDs), which means they can be stored in Git repositories just like any other Kubernetes manifest. This makes them perfectly suited for GitOps workflows. You can manage, version, and deploy your policies using tools like Argo CD or Flux CD, ensuring a single source of truth and an auditable change log. This approach aligns with best practices for managing Kubernetes configurations, including those for Gateway API configurations.

Q5: How does Kyverno handle performance and scalability in large clusters?

A5: Kyverno is designed for performance. It uses efficient in-memory caching of policies and can be horizontally scaled by increasing the number of Kyverno replicas. Complex policies or a very large number of policies can introduce latency to API requests, but Kyverno provides Prometheus metrics (e.g., `kyverno_admission_review_duration_seconds`) to monitor this. You can optimize performance by making `match` rules as specific as possible to avoid unnecessary policy evaluations. For advanced network traffic management, consider how it interacts with service mesh solutions like Istio.

Cleanup Commands

To remove Kyverno and all the policies created during this tutorial, execute the following commands:

# Delete the Kyverno policies
kubectl delete -f policy-restrict-registry.yaml
kubectl delete -f policy-add-labels.yaml
kubectl delete -f policy-generate-networkpolicy.yaml
kubectl delete -f policy-resource-limits.yaml

# Delete test pods and namespaces
kubectl delete pod untrusted-nginx trusted-nginx test-pod with-limits-pod
kubectl delete namespace my-app-namespace

# Uninstall Kyverno Helm chart
helm uninstall kyverno --namespace kyverno

# Delete the kyverno namespace
kubectl delete namespace kyverno

Next Steps / Further Reading

You’ve successfully deployed Kyverno and explored its core capabilities. Here are some resources to deepen your understanding and expand your use of Kyverno:

Conclusion

Kyverno stands out as a powerful, Kubernetes-native policy engine that simplifies the enforcement of governance, security, and operational best practices. By leveraging standard Kubernetes YAML, it lowers the barrier to entry for policy definition, allowing platform teams and security engineers to quickly implement validation, mutation, and generation rules. From ensuring image integrity and resource limits to automating network policies, Kyverno provides a comprehensive toolkit for maintaining a secure and compliant Kubernetes environment. Its declarative nature, combined with robust features and a growing community, makes it an indispensable tool for any organization running Kubernetes at scale. Embrace Kyverno to bring order and control to your dynamic cloud-native infrastructure.

Leave a comment