Orchestration

Secure Kubernetes: Audit Logs for Compliance

September 1, 2026 Kubezilla Team 16 min read

Kubernetes Audit Logging and Compliance: A Comprehensive Guide

In the dynamic world of cloud-native applications, Kubernetes has become the de facto operating system for the data center. However, this power and flexibility come with significant responsibilities, particularly concerning security, accountability, and compliance. As organizations increasingly deploy sensitive workloads on Kubernetes, the need for robust audit logging becomes paramount. Without a clear, immutable record of who did what, when, and where, achieving compliance with industry standards like SOC 2, HIPAA, GDPR, or PCI DSS is an uphill battle, and investigating security incidents becomes nearly impossible.

Kubernetes audit logging provides a chronological, tamper-resistant record of API server requests. Every interaction with your cluster – from a `kubectl get pods` to a `helm install` or an automated deployment from a CI/CD pipeline – generates an audit event. These logs are critical for security monitoring, forensic analysis, and, most importantly, demonstrating compliance. They offer unparalleled visibility into the operational state and security posture of your cluster, turning opaque actions into transparent, auditable events.

This guide will walk you through the essential concepts of Kubernetes audit logging, from configuring audit policies to integrating with external logging solutions and ensuring compliance. We’ll explore how to set up, manage, and leverage these logs to enhance your cluster’s security and meet stringent regulatory requirements. By the end, you’ll have a solid understanding of how to transform your Kubernetes clusters into auditable, compliant environments.

TL;DR: Quick Audit Logging Setup

Quickly enable basic Kubernetes audit logging by configuring the API server with an audit policy and log backend. This example creates a simple policy that logs metadata for all requests and writes them to a file.

  • Create Audit Policy: Define rules for what to log.
  • Configure API Server: Point the API server to the policy and log file.
  • Restart API Server: Apply changes (often via kubeadm config or cloud provider settings).
  • Verify Logs: Check the audit log file for events.
# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log metadata for all requests
  - level: Metadata
    verbs: ["get", "list", "watch"]
  # Don't log requests to certain endpoints
  - level: None
    resources:
      - group: ""
        resources: ["events"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
      - group: "authorization.k8s.io"
        resources: ["subjectaccessreviews"]
  # Log all other requests at Request level (request body included)
  - level: Request
    omitStages:
      - "RequestReceived"

# Example for kube-apiserver static pod manifests (e.g., /etc/kubernetes/manifests/kube-apiserver.yaml)
# Add these flags to the --kube-apiserver command:
# --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
# --audit-log-path=/var/log/kubernetes/audit/audit.log
# --audit-log-maxage=30
# --audit-log-maxbackup=10
# --audit-log-maxsize=100

# Create the directory for the audit policy and logs
sudo mkdir -p /etc/kubernetes/audit /var/log/kubernetes/audit

# Copy the audit policy file
sudo cp audit-policy.yaml /etc/kubernetes/audit/audit-policy.yaml

# Ensure the kube-apiserver manifest has the volume mounts:
# - mountPath: /etc/kubernetes/audit
#   name: audit-policy
#   readOnly: true
# - mountPath: /var/log/kubernetes/audit
#   name: audit-log-dir
# Add corresponding hostPath volumes:
# - hostPath:
#     path: /etc/kubernetes/audit
#     type: DirectoryOrCreate
#   name: audit-policy
# - hostPath:
#     path: /var/log/kubernetes/audit
#     type: DirectoryOrCreate
#   name: audit-log-dir

# After modifying the kube-apiserver manifest, the static pod will restart automatically.
# Verify logs:
sudo tail -f /var/log/kubernetes/audit/audit.log

Prerequisites

To follow this guide effectively, you’ll need:

  • A running Kubernetes cluster: This could be a local cluster (Minikube, Kind), a self-managed cluster, or a cloud-managed service (EKS, GKE, AKS). The exact steps for configuring the API server may vary slightly depending on your cluster’s setup.
  • kubectl installed and configured: To interact with your cluster.
  • Basic understanding of Kubernetes concepts: Pods, Deployments, Services, and how the API server functions.
  • Administrative access to your Kubernetes control plane: You’ll need to modify the API server configuration, which typically requires root access or appropriate cloud provider permissions.
  • Text editor: To create and modify YAML files.
  • Basic Linux command-line knowledge: For navigating directories and viewing logs.

Step-by-Step Guide: Configuring Kubernetes Audit Logging

1. Understanding Kubernetes Audit Policy Levels

Kubernetes audit logging is controlled by an audit policy, a YAML file that defines rules for what events should be recorded and at what level of detail. The policy consists of a list of rules, each specifying a `level` and criteria for matching requests. Understanding these levels is crucial for balancing verbosity and performance.

There are four audit levels:

  • None: Don’t log events that match this rule. This is useful for filtering out noisy or irrelevant events.
  • Metadata: Log request metadata (e.g., user, timestamp, resource, verb) but not the request or response body. This is a good balance for general monitoring and compliance.
  • Request: Log event metadata and the request body (for non-read requests). This provides more detail, especially for `create`, `update`, and `delete` operations.
  • RequestResponse: Log event metadata, the request body, and the response body. This is the most verbose level and can generate a significant amount of data, potentially impacting performance. Use it sparingly for specific debugging or high-security scenarios.

The rules are evaluated in order. The first rule that matches a request determines its audit level. If no rule matches, the request is not audited. You can also specify `omitStages` to exclude certain audit stages (e.g., `RequestReceived`, `ResponseStarted`, `ResponseComplete`, `Panic`). For more details, refer to the official Kubernetes Audit documentation.

2. Creating an Audit Policy File

Let’s create a comprehensive audit policy that captures essential information for compliance while filtering out common noise. This policy will log metadata for read-only operations, request bodies for modifications, and explicitly ignore high-volume, low-value events like `events` or authentication checks.

We’ll name this file `audit-policy.yaml`. Remember that the rules are evaluated sequentially, so more specific rules should generally come before more general ones.

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived" # Avoid logging requests twice if ResponseComplete is also enabled

rules:
  # Rule 1: Don't log requests to certain noisy resources
  - level: None
    resources:
      - group: ""
        resources: ["events"] # Events are often logged elsewhere and can be very noisy
      - group: "authentication.k8s.io"
        resources: ["tokenreviews", "subjectaccessreviews"] # Internal authentication checks
      - group: "authorization.k8s.io"
        resources: ["selfsubjectaccessreviews", "selfsubjectrulesreviews"] # Internal authorization checks
    users: ["system:kube-proxy"] # Kube-proxy's constant watches can be noisy
    verbs: ["get", "list", "watch"] # Only apply to read operations for these resources

  # Rule 2: Log metadata for all 'get', 'list', 'watch' requests (read-only operations)
  - level: Metadata
    verbs: ["get", "list", "watch"]

  # Rule 3: Log metadata and request body for operations on secrets, configmaps, and serviceaccounts
  # These are sensitive resources, so capturing the request body is important.
  - level: Request
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts"]
    omitStages:
      - "RequestReceived" # Ensure we only log the final stage

  # Rule 4: Log metadata and request and response bodies for modifications to critical resources
  # Use RequestResponse for extremely sensitive operations like role/rolebinding modifications.
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["create", "update", "delete", "patch"]
    omitStages:
      - "RequestReceived" # Ensure we only log the final stage

  # Rule 5: Log metadata and request body for all other write operations
  - level: Request
    verbs: ["create", "update", "delete", "patch"]
    omitStages:
      - "RequestReceived" # Ensure we only log the final stage

  # Rule 6: Default catch-all rule - log metadata for anything else not caught above
  - level: Metadata
    omitStages:
      - "RequestReceived" # Ensure we only log the final stage

Explanation: This policy prioritizes security and compliance. It explicitly ignores very noisy internal requests. It then logs metadata for all read operations, which is often sufficient for basic monitoring. For sensitive resources like `secrets` and `configmaps`, and for all write operations (`create`, `update`, `delete`, `patch`), it captures the request body. Crucially, for RBAC resources, it captures both request and response bodies, as changes to these can have significant security implications. The `omitStages` ensures that we don’t duplicate logs for a single request if multiple stages are enabled, reducing log volume slightly.

3. Configuring the Kubernetes API Server for Audit Logging

The Kubernetes API server is responsible for processing audit logs. You enable audit logging by passing specific flags to the `kube-apiserver` process. The exact method for doing this depends on how your cluster is set up:

  • kubeadm-provisioned clusters: Modify the static Pod manifest located at `/etc/kubernetes/manifests/kube-apiserver.yaml`.
  • Cloud-managed clusters (EKS, GKE, AKS): Use the cloud provider’s console or CLI to enable and configure audit logging. This often involves selecting an audit policy and specifying a logging destination (e.g., CloudWatch Logs, Stackdriver, Azure Monitor).
  • Self-managed clusters (systemd service): Modify the systemd service file for `kube-apiserver`.

For a `kubeadm` cluster, you’ll need to update the `kube-apiserver.yaml` file to include the audit policy and log path. First, ensure your audit policy file is accessible to the API server container.

# Create directories for audit policy and logs
sudo mkdir -p /etc/kubernetes/audit /var/log/kubernetes/audit

# Copy the audit policy file to the designated directory
sudo cp audit-policy.yaml /etc/kubernetes/audit/audit-policy.yaml

# Open the kube-apiserver static pod manifest for editing
# (Path might vary, but this is common for kubeadm)
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Inside `kube-apiserver.yaml`, you need to:

  1. Add volume mounts for the audit policy file and the audit log directory.
  2. Add hostPath volumes that map these directories from the host to the Pod.
  3. Add the audit-related flags to the `command` section of the `kube-apiserver` container.
# --- Excerpt from /etc/kubernetes/manifests/kube-apiserver.yaml ---
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.config: |
      {"advertiseAddress":"192.168.1.10","bindPort":6443}
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.10
    - --allow-privileged=true
    # ... other existing flags ...
    # Add these audit logging flags:
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30 # Retain logs for 30 days
    - --audit-log-maxbackup=10 # Keep up to 10 rotated log files
    - --audit-log-maxsize=100 # Rotate log file when it reaches 100 MB
    # ... other existing flags ...
    image: k8s.gcr.io/kube-apiserver:v1.28.0 # Adjust image version as needed
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    # ... existing volume mounts ...
    - mountPath: /etc/kubernetes/audit # Mount for the audit policy file
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit # Mount for the audit log directory
      name: audit-log-dir
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  # ... existing volumes ...
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log-dir
# --- End excerpt ---

Explanation:

  • `–audit-policy-file`: Specifies the path to your audit policy YAML.
  • `–audit-log-path`: Defines where the audit logs will be written on the host. This should be a path within the container that is mapped to a host path via a volume mount.
  • `–audit-log-maxage`, `–audit-log-maxbackup`, `–audit-log-maxsize`: These flags control log rotation, preventing the log file from growing indefinitely and consuming all disk space. They are crucial for maintaining log hygiene.
  • `volumeMounts` and `volumes`: These entries ensure that the `kube-apiserver` container can access the audit policy file and write to the audit log directory on the host machine.

Once you save the `kube-apiserver.yaml` file, the `kubelet` on the control plane node will detect the change and automatically restart the `kube-apiserver` static Pod. This may cause a brief interruption in API server availability.

4. Verifying Audit Logs

After the API server restarts, it should begin writing audit events to the specified log file. You can verify this by tailing the log file and performing some Kubernetes operations.

# Tail the audit log file on the control plane node
sudo tail -f /var/log/kubernetes/audit/audit.log

Now, from your local machine with `kubectl`, perform some actions:

kubectl get pods --all-namespaces
kubectl create deployment nginx --image=nginx --replicas=1
kubectl get deployments
kubectl delete deployment nginx

Expected Output (in `sudo tail -f /var/log/kubernetes/audit/audit.log`):

You should see JSON-formatted audit events streaming into your terminal. Each event will correspond to an action you performed via `kubectl` or internal cluster operations.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "...",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/pods?limit=500",
  "verb": "list",
  "user": {
    "username": "kubernetes-admin",
    "groups": ["system:masters", "system:authenticated"]
  },
  "sourceIPs": ["..."],
  "userAgent": "kubectl/v1.28.0 (linux/amd64) kubernetes/...",
  "objectRef": {
    "resource": "pods",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "...",
  "stageTimestamp": "...",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"cluster-admin\" of ClusterRole \"cluster-admin\" to User \"kubernetes-admin\""
  }
}
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "...",
  "level": "Request",
  "stage": "ResponseComplete",
  "requestURI": "/apis/apps/v1/namespaces/default/deployments",
  "verb": "create",
  "user": {
    "username": "kubernetes-admin",
    "groups": ["system:masters", "system:authenticated"]
  },
  "sourceIPs": ["..."],
  "userAgent": "kubectl/v1.28.0 (linux/amd64) kubernetes/...",
  "objectRef": {
    "resource": "deployments",
    "namespace": "default",
    "name": "nginx",
    "apiVersion": "apps/v1"
  },
  "requestObject": {
    "kind": "Deployment",
    "apiVersion": "apps/v1",
    "metadata": {
      "name": "nginx",
      "namespace": "default",
      "creationTimestamp": null
    },
    "spec": {
      "replicas": 1,
      "selector": {
        "matchLabels": {
          "app": "nginx"
        }
      },
      "template": {
        "metadata": {
          "creationTimestamp": null,
          "labels": {
            "app": "nginx"
          }
        },
        "spec": {
          "containers": [
            {
              "name": "nginx",
              "image": "nginx",
              "resources": {}
            }
          ]
        }
      },
      "strategy": {}
    },
    "status": {}
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestReceivedTimestamp": "...",
  "stageTimestamp": "...",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"cluster-admin\" of ClusterRole \"cluster-admin\" to User \"kubernetes-admin\""
  }
}

You’ll observe that `kubectl get pods` generates a `Metadata` level event, while `kubectl create deployment` generates a `Request` level event, including the `requestObject` (the deployment manifest). This demonstrates the policy working as intended.

5. Integrating with External Logging Solutions (Optional but Recommended)

While writing audit logs to a local file is a good start, it’s not suitable for production. Local files can be lost if a node fails, are hard to centralize, and difficult to query. For robust compliance and operational visibility, you need to ship these logs to a centralized logging solution.

Common solutions include:

  • ELK Stack (Elasticsearch, Logstash, Kibana): A popular open-source choice.
  • Splunk: Enterprise-grade SIEM solution.
  • Cloud Provider Services:
    • AWS CloudWatch Logs / S3
    • Google Cloud Logging (Stackdriver)
    • Azure Monitor / Log Analytics
  • Fluentd/Fluent Bit: Lightweight log processors often deployed as DaemonSets to collect logs and forward them.

For this example, we’ll outline how to use Fluent Bit as a DaemonSet to collect logs from the host path `/var/log/kubernetes/audit` and send them to a simple `stdout` for demonstration. In a real-world scenario, `stdout` would be replaced with an output plugin for your chosen logging backend.

First, create a `ConfigMap` for Fluent Bit’s configuration:

# fluent-bit-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system # Or your logging namespace
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Daemon        Off
        Log_Level     info
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name          tail
        Path          /var/log/kubernetes/audit/audit.log
        Tag           kube_audit
        Parser        json # Parse JSON formatted audit logs
        Refresh_Interval 5
        Mem_Buf_Limit 5MB

    [OUTPUT]
        Name          stdout
        Match         kube_audit
        Format        json

kubectl apply -f fluent-bit-config.yaml

Next, create the `DaemonSet` for Fluent Bit. This will ensure Fluent Bit runs on every node, including your control plane node where audit logs are generated.

# fluent-bit-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system # Or your logging namespace
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fluent-bit
      annotations:
        prometheus.io/port: "2020"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.2.0 # Use a stable Fluent Bit image
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config

You’ll also need a `ServiceAccount` and `ClusterRole`/`ClusterRoleBinding` for Fluent Bit to read node information:

# fluent-bit-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-reader
rules:
- apiGroups: [""]
  resources:
  - pods
  - namespaces
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-reader-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-reader
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: kube-system

kubectl apply -f fluent-bit-rbac.yaml
kubectl apply -f fluent-bit-daemonset.yaml

Verify Fluent Bit logs:

# Get the name of a Fluent Bit pod running on your control plane node
kubectl get pods -n kube-system -l app.kubernetes.io/name=fluent-bit -o wide

# Assuming the pod name is fluent-bit-abcde, tail its logs
kubectl logs -f fluent-bit-abcde -n kube-system

Expected Output: You should see the Kubernetes audit events, now processed by Fluent Bit, appearing in the Fluent Bit pod’s logs (stdout). These logs would then be forwarded to your chosen backend.

{
    "kubernetes": {
        "pod_name": "fluent-bit-...",
        "namespace_name": "kube-system",
        "container_name": "fluent-bit",
        "container_id": "...",
        "host": "kube-control-plane"
    },
    "log": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\", ... }"
}

This setup centralizes your logs, making them searchable, analyzable, and durable, which is essential for compliance and security operations. For more advanced observability, consider integrating with tools like eBPF Observability with Hubble, which provides network-level visibility alongside API server logs.

Production Considerations

Deploying audit logging in production requires careful planning to ensure performance, security, and reliability.

  • Storage and Retention: Audit logs can be voluminous. Plan for adequate storage and a robust retention policy. Compliance standards often dictate how long logs must be kept (e.g., 1-7 years). Use tiered storage solutions (e.g., S3 infrequent access) to manage costs.
  • Performance Impact: `RequestResponse` logging can significantly impact API server performance and network traffic. Use it judiciously. Profile your API server if you notice degradation.
  • Log Security: Audit logs contain sensitive information. They must be protected from unauthorized access and tampering.
    • Encryption: Encrypt logs at rest and in transit.
    • Access Control: Implement strict RBAC for your logging solution. Only authorized personnel or systems should have access.
    • Integrity: Ensure log integrity (e.g., using hashing or digital signatures) to detect tampering. Sigstore and Kyverno offer solutions for supply chain integrity, which can be extended to log integrity if integrated.
  • Centralized Logging: Always ship logs to a centralized, durable, and searchable logging platform (e.g., Splunk, ELK, cloud-native services). This is non-negotiable for production.
  • Alerting and Monitoring: Configure alerts based on critical audit events (e.g., unauthorized access attempts, deletion of security-critical resources, changes to RBAC). Integrate with your existing monitoring solutions.
  • Policy Management: Regularly review and update your audit policy. As your cluster evolves, new resources or use cases might require adjustments to ensure relevant events are captured.
  • Cost Management: Be mindful of the costs associated with log storage and processing, especially in cloud environments. Optimize your audit policy to capture only necessary events. Consider tools like Karpenter Cost Optimization for overall cluster cost reduction, but remember log costs are separate.
  • Network Policies: Ensure your log-shipping agents (like Fluent Bit) have the necessary network access to reach your logging backend. This might involve configuring Kubernetes Network Policies to allow egress traffic.
  • Compliance Reporting: Understand how your logging solution can generate reports required for specific compliance frameworks (e.g., PCI DSS, HIPAA, GDPR, SOC 2).

Troubleshooting

  1. Issue: Audit logs are not being generated.

    Solution:

    • Check API Server Flags: Double-check the `kube-apiserver` manifest (`/etc/kubernetes/manifests/kube-apiserver.yaml` for kubeadm) to ensure `–audit-policy-file` and `–audit-log-path` flags are correctly set and the API server restarted.
    • Verify Policy File Path: Ensure the `audit-policy.yaml` file exists at the path specified by `–audit-policy-file` and that the `kube-apiserver` container has read access to it (via volume mounts).
    • Check Log Directory Permissions: Ensure the audit log directory (`/var/log/kubernetes/audit` in our example) exists on the host and the `kube-apiserver` container has write permissions (via volume mounts and host directory permissions).
    • API Server Logs: Check the `kube-apiserver` pod logs for any errors related to audit logging. 
      kubectl logs -f kube-apiserver-<your-node-name> -n kube-system
  2. Issue: Audit logs are too verbose or not verbose enough.

    Solution:

    • Review Audit Policy: The problem is almost certainly in your `audit-policy.yaml`. Review the rules carefully. Remember, rules are evaluated sequentially. A broad `Metadata` rule early in the policy might prevent more specific `Request` or `RequestResponse` rules from being applied.
    • Adjust Levels: Change `level` for specific resources or verbs. For example, if you need more detail for `secrets`, change its rule to `Request` or `RequestResponse`.
    • Use `omitStages`: If logs are duplicated or contain unnecessary intermediate stages, use `omitStages` in your policy.
  3. Issue: Audit log file is growing too large, filling disk space.

    Solution:

    • Configure Log Rotation: Ensure `–audit-log-maxsize`, `–audit-log-maxbackup`, and `–audit-log-maxage` flags are set in your `kube-apiserver` configuration. These flags manage log rotation and retention.
    • Optimize Audit Policy: Reduce the verbosity of your audit policy. Use `Metadata` level for common read operations and `None` for extremely noisy, irrelevant events.
    • Ship Logs Off-Node: Implement a log shipping solution (Fluent Bit, Filebeat) to move logs to a centralized store as soon as possible, then clear local files.
  4. Issue: Fluent Bit is not collecting audit logs.

    Solution:

    • Check Fluent Bit Pod Status: Ensure the Fluent Bit pods are `Running` in the `kube-system` namespace. 
      kubectl get pods -n kube-system -l app.kubernetes.io/name=fluent-bit
    • Fluent Bit Pod Logs: Check the logs of the Fluent Bit pod running on the control plane node for errors. 
      kubectl logs -f fluent-bit-<pod-name> -n kube-system
    • Volume Mounts in DaemonSet: Verify that the `hostPath` volume

Leave a comment