Secure Kubernetes: Audit Logs for Compliance

Introduction

In the dynamic world of cloud-native applications, Kubernetes has become the de facto standard for orchestrating containers. However, with great power comes great responsibility, especially when it comes to security and compliance. Understanding who did what, when, and where within your Kubernetes clusters is not just good practice; it’s a fundamental requirement for maintaining a secure and auditable environment. Without proper audit logging, investigating security incidents becomes a forensic nightmare, and meeting regulatory compliance standards like SOC 2, HIPAA, or GDPR becomes virtually impossible.

Kubernetes audit logging provides a chronological record of requests made to the Kubernetes API server. These logs are invaluable for security monitoring, post-mortem analysis, and ensuring adherence to internal policies and external regulations. They capture every authenticated request, detailing the user, the resource accessed, the action performed, and the outcome. This guide will walk you through the essential steps to configure, manage, and leverage Kubernetes audit logs effectively, transforming raw data into actionable insights for compliance and security.

# Example Audit Policy (audit-policy.yaml)
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse # Log request and response bodies for configmaps
    resources:
      - group: ""
        resources: ["configmaps"]
    omitStages:
      - "RequestReceived"
  - level: Metadata # Log metadata for all other resources
    resources:
      - group: ""
        resources: ["pods", "deployments", "services"]
  - level: None # Don't log watch requests
    verbs: ["watch"]
    resources:
      - group: "*"
        resources: ["*"]
  - level: Request # Log request body for all other write operations
    verbs: ["create", "update", "patch", "delete"]
  - level: Metadata # Log metadata for everything else
    omitStages:
      - "RequestReceived"
    users: ["system:serviceaccount:kube-system:replicaset-controller"]
    resources:
      - group: "*"
        resources: ["*"]
  - level: None # Default to not logging
    users: ["system:kube-proxy", "system:unsecured"]
    resources:
      - group: "*"
        resources: ["*"]
  - level: Metadata # Default for everything else
    omitStages:
      - "RequestReceived"
    resources:
      - group: "*"
        resources: ["*"]

# API Server flags (example for kubeadm)
# Edit /etc/kubernetes/manifests/kube-apiserver.yaml
# Add these to the 'command' section:
# - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
# - --audit-log-path=/var/log/kubernetes/audit/audit.log
# - --audit-log-maxbackup=10
# - --audit-log-maxsize=100
# - --audit-log-maxage=30

# View audit logs (after configuration)
tail -f /var/log/kubernetes/audit/audit.log

Prerequisites

Before diving into the configuration of Kubernetes audit logging, ensure you have the following:

• A running Kubernetes Cluster: This guide assumes you have an operational Kubernetes cluster. This could be a local setup (Minikube, Kind), a self-managed cluster (kubeadm), or a managed service (EKS, GKE, AKS). Note that managed services often have their own mechanisms for enabling and consuming audit logs, which we will touch upon.
• Administrative Access: You need administrative access to your Kubernetes cluster, specifically to modify the API server configuration. This typically means SSH access to the control plane nodes if you’re running a self-managed cluster.
• kubectl configured: Ensure your kubectl command-line tool is installed and configured to communicate with your cluster.
• Basic understanding of YAML: Kubernetes configurations are primarily defined in YAML.
• Understanding of Kubernetes API resources: Familiarity with common Kubernetes resources like Pods, Deployments, Services, and Namespaces will help you define effective audit policies.

Step-by-Step Guide

1. Understanding Kubernetes Audit Policy Levels

The core of Kubernetes audit logging is the Audit Policy. This policy defines what events should be recorded and at what level of detail. There are four audit levels:

• None: Don’t log events that match this rule. This is useful for filtering out noisy or irrelevant events.
• Metadata: Log request metadata (e.g., user, timestamp, resource, verb, IP), but not the request or response body. This is a good balance for general monitoring.
• Request: Log event metadata and the request body (for non-read requests). This is useful for understanding what data was sent to the API server.
• RequestResponse: Log event metadata, request, and response bodies. This is the most verbose level and provides the most detail, but it can generate a large volume of logs. Use this judiciously for sensitive resources or critical operations.

Each rule in the audit policy has a level and can specify criteria like users, verbs, resources, and namespaces to match specific API requests. The first rule that matches an event determines its audit level.

2. Creating an Audit Policy Manifest

Let’s create a comprehensive audit policy that balances security needs with log volume. We’ll log sensitive actions at RequestResponse, most operations at Request or Metadata, and filter out noisy events like watch operations or health checks.

This policy prioritizes logging for sensitive resources like Secrets and ConfigMaps (which often contain sensitive data), while reducing noise from common system operations. For more advanced security, consider integrating this with tools like Sigstore and Kyverno for policy enforcement and supply chain security.

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Rule 1: Don't log requests from the kube-proxy or unsecured users.
  # This significantly reduces noise from internal Kubernetes components.
  - level: None
    users: ["system:kube-proxy", "system:unsecured"]
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services", "services/status"]
  
  # Rule 2: Don't log watch requests for all resources.
  # Watch requests are continuous and can generate excessive log volume.
  - level: None
    verbs: ["watch"]
    resources:
      - group: "*"
        resources: ["*"]

  # Rule 3: Log RequestResponse for sensitive resources like Secrets and ConfigMaps.
  # This captures both the request and response bodies, crucial for forensic analysis.
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
    omitStages:
      - "RequestReceived" # Omit the initial request received stage to reduce redundancy

  # Rule 4: Log Request for all write operations (create, update, patch, delete).
  # This captures the request body for modifications to any resource.
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: "*"
        resources: ["*"]
    omitStages:
      - "RequestReceived"

  # Rule 5: Log Metadata for common read operations on core resources.
  # Provides context without logging full request/response bodies.
  - level: Metadata
    verbs: ["get", "list"]
    resources:
      - group: ""
        resources: ["pods", "deployments", "services", "namespaces", "nodes"]
    omitStages:
      - "RequestReceived"

  # Rule 6: Log Metadata for API server health and discovery requests.
  # These are frequent but contain little sensitive information.
  - level: Metadata
    resources:
      - group: ""
        resources: ["events", "nodes", "nodes/status", "pods", "pods/status"]
    users: ["system:kube-scheduler", "system:node-controller", "system:controller-manager"]
    omitStages:
      - "RequestReceived"

  # Rule 7: Default rule: Log Metadata for everything else.
  # This ensures that no API requests are completely missed, providing a safety net.
  - level: Metadata
    omitStages:
      - "RequestReceived"
    resources:
      - group: "*"
        resources: ["*"]

Save this content to a file named audit-policy.yaml.

Verify:

Ensure the YAML is syntactically correct. While there’s no direct command to “verify” an audit policy without applying it, a linter can help:

# Basic YAML linting
yamllint audit-policy.yaml

If you don’t have yamllint, you can install it via pip install yamllint.

3. Enabling Audit Logging on the Kubernetes API Server

This is the most critical step. You need to configure the Kubernetes API server to use your audit policy and specify where to store the logs. The method for doing this depends on how your Kubernetes cluster was set up.

For self-managed clusters (e.g., kubeadm):

If you used kubeadm, the API server runs as a static Pod. You’ll need to modify its manifest file, usually located at /etc/kubernetes/manifests/kube-apiserver.yaml on your control plane node(s).

1. Copy the audit policy: Place your audit-policy.yaml file in a directory accessible by the API server, for example, /etc/kubernetes/audit/.
2. Mount the directory: Add a volume mount to the API server manifest so it can access the policy file.
3. Add API server flags: Append the necessary --audit-policy-file and --audit-log-path flags to the API server command.

Here’s an example of modifications to /etc/kubernetes/manifests/kube-apiserver.yaml:

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.config: |
      {"bindAddress":"192.168.1.100","bindPort":6443,"advertiseAddress":"192.168.1.100"}
  creationTimestamp: null
  labels:
    app.kubernetes.io/component: apiserver
    app.kubernetes.io/name: kubernetes
    k8s-app: kube-apiserver
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.100
    - --allow-privileged=true
    # ... other existing flags ...
    # Add these audit logging flags:
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30      # Retain logs for 30 days
    - --audit-log-maxbackup=10   # Keep 10 old log files
    - --audit-log-maxsize=100    # Rotate log after 100 MB
    # ... other existing flags ...
    image: k8s.gcr.io/kube-apiserver:v1.28.0 # Adjust version as needed
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.100
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.100
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki/apiserver.crt
      name: apiserver-cert
      readOnly: true
    - mountPath: /etc/kubernetes/pki/apiserver-key.pem
      name: apiserver-key
      readOnly: true
    # Add this volume mount for the audit policy file
    - mountPath: /etc/kubernetes/audit
      name: audit-policy-volume
      readOnly: true
    - mountPath: /var/log/kubernetes/audit # Ensure this path is writable
      name: audit-log-volume
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: pki
  - hostPath:
      path: /etc/kubernetes/pki/apiserver.crt
      type: FileOrCreate
    name: apiserver-cert
  - hostPath:
      path: /etc/kubernetes/pki/apiserver-key.pem
      type: FileOrCreate
    name: apiserver-key
  # Add this volume for the audit policy file
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy-volume
  # Add this volume for the audit logs
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log-volume
status: {}

After saving the changes, the kubelet will automatically restart the API server Pod. This might take a minute or two.

For managed Kubernetes services (EKS, GKE, AKS):

Managed services abstract away direct access to the API server manifest. Instead, they provide their own mechanisms for enabling and configuring audit logging, usually integrated with their native logging services (e.g., CloudWatch Logs for EKS, Cloud Logging for GKE, Azure Monitor for AKS).

• AWS EKS: You can enable audit logging directly from the EKS console or via the AWS CLI/SDK. EKS audit logs are sent to Amazon CloudWatch Logs. Refer to the EKS control plane logging documentation.
• Google GKE: GKE audit logs are automatically enabled and sent to Google Cloud Audit Logs, which integrates with Cloud Logging. You can configure log sinks to export them. See the GKE audit logging documentation.
• Azure AKS: AKS integrates with Azure Monitor for audit logs. You can enable diagnostic settings for your cluster to send logs to a Log Analytics workspace or storage account. Check the AKS audit log documentation.

While the exact configuration differs, the underlying principles of what gets logged are similar, often allowing for custom audit policies via their respective cloud interfaces.

Verify:

Once the API server has restarted, you should see audit log files appearing in the specified path (e.g., /var/log/kubernetes/audit/).

# On your control plane node
sudo ls -l /var/log/kubernetes/audit/

Expected output (you might see a .log.tmp file initially, then a .log file):

total 4
-rw-r----- 1 root root 1234 Oct 26 10:30 audit.log

Now, let’s generate some activity and check the logs:

# From your local machine with kubectl configured
kubectl get pods -n kube-system
kubectl create configmap my-test-configmap --from-literal=key1=value1 -n default
kubectl get configmap my-test-configmap -n default
kubectl delete configmap my-test-configmap -n default

Then, back on the control plane node:

sudo tail -f /var/log/kubernetes/audit/audit.log | grep "my-test-configmap"

You should see entries related to the creation, getting, and deletion of my-test-configmap. Due to our RequestResponse policy for configmaps, these entries will be quite verbose.

# Example (truncated) output for configmap creation
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"...","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/configmaps","verb":"create","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["192.168.1.1"],"userAgent":"kubectl/v1.28.2 (linux/amd64) kubernetes/b0a48ae","objectRef":{"resource":"configmaps","namespace":"default","name":"my-test-configmap","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestObject":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-test-configmap","namespace":"default"},"data":{"key1":"value1"}},"responseObject":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"my-test-configmap","namespace":"default","uid":"...","resourceVersion":"...","creationTimestamp":"..."},"data":{"key1":"value1"}},"stageTimestamp":"2023-10-26T10:35:00Z","annotations":{"action":"create"}}

4. Centralized Logging and Analysis

Storing audit logs directly on the control plane node is a good start, but for production environments, you need a centralized logging solution. This allows for easier searching, analysis, alerting, and long-term retention, which are crucial for compliance.

Common approaches include:

• Fluentd/Fluent Bit: Deploy a DaemonSet that collects logs from /var/log/kubernetes/audit/audit.log and forwards them to a centralized logging system like Elasticsearch, Splunk, Loki, or a cloud-native solution.
• Logstash: Similar to Fluentd, Logstash can collect, parse, and forward logs.
• Cloud-Native Integrations: As mentioned, managed Kubernetes services integrate directly with their cloud provider’s logging solutions.

Here’s a basic example of a Fluent Bit configuration as a DaemonSet to send logs to Elasticsearch. For more advanced observability, consider how eBPF Observability with Hubble can provide deeper network insights alongside API audit logs.

# fluent-bit-audit-logs.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-reader
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-reader
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit-audit
  namespace: kube-system
  labels:
    app.kubernetes.io/name: fluent-bit
    app.kubernetes.io/instance: fluent-bit-audit
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
      app.kubernetes.io/instance: fluent-bit-audit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fluent-bit
        app.kubernetes.io/instance: fluent-bit-audit
    spec:
      serviceAccountName: fluent-bit
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.1.10 # Use a stable Fluent Bit image
        env:
          - name: KUBERNETES_HOST
            value: "kubernetes.default.svc.cluster.local"
          - name: KUBERNETES_PORT
            value: "443"
          - name: FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch.default.svc.cluster.local" # Replace with your Elasticsearch host
          - name: FLUENT_ELASTICSEARCH_PORT
            value: "9200" # Replace with your Elasticsearch port
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http" # or https
          - name: FLUENT_ELASTICSEARCH_USER
            valueFrom:
              secretKeyRef:
                name: elasticsearch-credentials
                key: username
          - name: FLUENT_ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                name: elasticsearch-credentials
                key: password
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluentbit-config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluentbit-config
        configMap:
          name: fluent-bit-audit-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-audit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
        Daemon       off
        Parsers_File parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

    [INPUT]
        Name             tail
        Path             /var/log/kubernetes/audit/audit.log # Path to your audit log
        Parser           json
        Tag              kube_audit
        Mem_Buf_Limit    5MB
        Skip_Long_Lines  On

    [OUTPUT]
        Name            es
        Match           kube_audit
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Index           kubernetes_audit_log-%Y.%m.%d
        Type            _doc
        tls             ${FLUENT_ELASTICSEARCH_SCHEME}
        tls.verify      Off
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Log_Level       info
        Suppress_Type_Name On

Note: This Fluent Bit configuration is a basic example. You would need to replace elasticsearch.default.svc.cluster.local with your actual Elasticsearch endpoint and create a Kubernetes Secret named elasticsearch-credentials for authentication. Ensure your Elasticsearch cluster is properly secured.

Verify:

Apply the DaemonSet and check its status:

kubectl apply -f fluent-bit-audit-logs.yaml
kubectl get ds -n kube-system fluent-bit-audit
kubectl logs -n kube-system -l app.kubernetes.io/instance=fluent-bit-audit

You should see logs indicating Fluent Bit starting and potentially connecting to Elasticsearch. If logs are flowing correctly, you can then search for them in your Elasticsearch instance or other configured logging backend.

Production Considerations

Deploying audit logging in production requires more than just enabling a flag. Here are critical considerations:

1. Log Volume and Storage: Audit logs can be extremely verbose. Plan for significant storage capacity and implement proper log rotation and retention policies. The --audit-log-maxage, --audit-log-maxbackup, and --audit-log-maxsize flags are essential for managing local log files. For centralized solutions, ensure your chosen backend can handle the ingestion rate and storage requirements.
2. Performance Impact: While audit logging is generally lightweight, the RequestResponse level can introduce a measurable overhead, especially for high-traffic clusters. Monitor API server latency and resource utilization after enabling verbose logging. Optimize your audit policy to only log RequestResponse for truly critical resources.
3. Security of Logs: Audit logs contain sensitive information. Ensure they are stored securely, encrypted at rest and in transit, and accessible only to authorized personnel. Implement strong access controls on your logging backend. Consider Cilium WireGuard Encryption for securing network traffic between your log forwarders and central logging system.
4. Centralized Logging and SIEM Integration: For compliance and effective security monitoring, integrate Kubernetes audit logs with a Security Information and Event Management (SIEM) system (e.g., Splunk, ELK Stack, Sumo Logic, Sentinel). This enables correlation with other security events, real-time alerting, and long-term archival.
5. Alerting and Monitoring: Define specific alerts for critical audit events, such as:
   • Attempts to access or modify sensitive resources (Secrets, RBAC roles) by unauthorized users.
   • API server authentication failures.
   • Deletions of critical resources (Namespaces, Deployments).
   • Modifications to audit policies themselves.
6. Compliance Requirements: Understand the specific compliance standards (GDPR, HIPAA, SOC 2, PCI DSS) that apply to your organization. These standards often dictate log retention periods, immutability requirements, and access controls for audit trails. Regularly review your audit policy to ensure it meets these requirements.
7. Audit Policy Management: As your cluster evolves, your audit policy may need updates. Treat your audit policy as code, store it in version control, and integrate its deployment into your CI/CD pipeline.
8. User and Service Account Identification: Ensure that all API requests are properly authenticated and attributed to specific users or service accounts. This is crucial for accountability. Combine audit logs with Kubernetes Network Policies for a comprehensive security posture, controlling both API access and pod-to-pod communication.
9. Managed Kubernetes Services: If you’re using a managed service, leverage their built-in audit logging features and integrations. While this simplifies setup, ensure you understand their log retention, access, and export capabilities for compliance.

Troubleshooting

1. No Audit Logs are Being Generated

Issue: After configuring the API server, no audit.log file appears or it remains empty.

Solution:

1. Check API Server Pod Status: Ensure the kube-apiserver Pod is running and healthy.

   kubectl get pod -n kube-system -l k8s-app=kube-apiserver

2. Review API Server Pod Logs: Look for errors related to audit logging.

   kubectl logs -n kube-system -l k8s-app=kube-apiserver

   Common errors include:

   • error reading audit policy file: Indicates a syntax error in your audit-policy.yaml or an incorrect path.
   • permission denied: The API server might not have write permissions to the --audit-log-path or read permissions to the --audit-policy-file.
3. Verify File Paths and Permissions: Double-check that the paths specified in --audit-policy-file and --audit-log-path exist and have correct permissions on the control plane node.

   # Example checks on control plane
   sudo ls -l /etc/kubernetes/audit/audit-policy.yaml
   sudo ls -ld /var/log/kubernetes/audit/
   sudo stat /var/log/kubernetes/audit/audit.log # Check if it exists and permissions

   The audit log directory usually needs to be writable by the user running the API server (often root or a specific system user).

4. Ensure Volume Mounts (kubeadm): If using kubeadm, confirm the kube-apiserver.yaml has the correct volumeMounts and volumes for both the policy file and the log output directory.
5. Restart API Server: If you manually edited kube-apiserver.yaml, the kubelet should automatically restart the Pod. If not, you might need to force a restart (e.g., by moving the file out and back into /etc/kubernetes/manifests/, or restarting kubelet).

2. Audit Logs are Too Voluminous / Too Sparse

Issue: Logs are either overwhelming or missing crucial information.

Solution:

1. Review and Refine Audit Policy: This is an iterative process.
   • Too Voluminous:
     • Change RequestResponse to Request or Metadata for less critical resources.
     • Add level: None rules for very noisy operations (e.g., get/list/watch on endpoints, events, nodes/status by system users).
     • Use omitStages to remove redundant log entries for a single API call.
   • Too Sparse:
     • Increase the level for critical resources (e.g., from Metadata to Request or RequestResponse for Secrets, Roles, RoleBindings).
     • Ensure your default rule (the last catch-all rule) is set to at least Metadata.
2. Adjust Log Rotation: Use --audit-log-maxage, --audit-log-maxbackup, and --audit-log-maxsize flags to manage local file size.

3. Audit Logs are Not Being Forwarded to Centralized System

Issue: Fluent Bit (or similar) is running, but logs aren’t reaching Elasticsearch/Splunk.

Solution:

1. Check Fluent Bit Pod Logs:

   kubectl logs -n kube-system -l app.kubernetes.io/instance=fluent-bit-audit

   Look for connection errors, authentication failures, or parsing issues.

2. Verify Fluent Bit Configuration:
   • Input Path: Ensure Path in the [INPUT] section matches the actual audit log path on the host (e.g., /var