Secure Kubernetes Runtime: Get Started with Falco

Introduction

In the dynamic and often hostile landscape of cloud-native environments, simply deploying your applications isn’t enough; you need robust security measures that can detect and respond to threats in real-time. Traditional perimeter defenses fall short in the highly distributed and ephemeral world of Kubernetes, where containers are constantly being created, destroyed, and moved. This is where runtime security becomes paramount. Imagine a scenario where an attacker manages to bypass your admission controllers and network policies, perhaps exploiting a zero-day vulnerability in an application dependency. Without runtime security, such an intrusion could go undetected until significant damage is done.

Falco, an open-source project and a Cloud Native Computing Foundation (CNCF) sandbox project, steps in to fill this critical gap. It acts as a behavioral activity monitor designed to detect anomalous activity within your Kubernetes clusters, hosts, and containers. By leveraging system calls, Falco provides deep visibility into the kernel, allowing it to identify suspicious activities like unexpected file access, privilege escalation attempts, unusual network connections, or unauthorized process execution. Integrating Falco into your Kubernetes security posture empowers you to move beyond static analysis and proactive prevention, adding a crucial layer of real-time threat detection that can alert you to potential breaches as they happen, ensuring the integrity and security of your workloads.

Prerequisites

Before diving into Falco, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (v1.18+ recommended). You can use Minikube, Kind, or a cloud-managed service like EKS, AKS, or GKE.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Helm (v3+): The package manager for Kubernetes. Install Helm by following the instructions on the Helm website.
• Basic understanding of Kubernetes concepts: Pods, Deployments, DaemonSets, Services, and Namespaces.
• Basic Linux command-line knowledge: Familiarity with commands like ls, cat, grep, and echo.

Step-by-Step Guide: Deploying and Configuring Falco

Step 1: Add the Falco Helm Repository

The easiest and most recommended way to deploy Falco into your Kubernetes cluster is by using its official Helm chart. Helm simplifies the installation and management of Kubernetes applications, allowing you to deploy complex software with a single command. First, you need to add the Falco Helm repository to your local Helm configuration. This makes the Falco chart available for installation.

After adding the repository, it’s good practice to update your Helm repositories to ensure you have the latest chart versions available. This step fetches the most recent chart metadata, including any new releases or updates to the Falco chart.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Verify:

You should see output similar to this, confirming the repository has been added and updated successfully:

"falcosecurity" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
Update Complete. ⎈Happy Helming!⎈

Step 2: Install Falco using Helm

Now that the Falco Helm repository is added, you can proceed with the installation. We will install Falco into its own dedicated namespace, falco, to keep our cluster organized. The Helm chart will deploy Falco as a DaemonSet, ensuring that a Falco agent runs on every node in your Kubernetes cluster. This is crucial for comprehensive runtime security, as each agent monitors the system calls on its respective node, providing full coverage.

The Falco DaemonSet typically includes a kernel module or an eBPF probe (for newer kernels) to intercept system calls. This deep integration allows Falco to observe low-level system activity and apply its rule engine to detect policy violations. For those interested in advanced eBPF usage, our guide on eBPF Observability with Hubble provides further insights into this powerful technology.

helm install falco falcosecurity/falco --namespace falco --create-namespace

Verify:

Check the deployed pods in the falco namespace. You should see Falco pods running, one for each node in your cluster. It might take a minute or two for the pods to transition to the Running state.

kubectl get pods -n falco

Expected output:

NAME                   READY   STATUS    RESTARTS   AGE
falco-f9p8r            1/1     Running   0          2m
falco-gh7j2            1/1     Running   0          2m
falco-jx6m5            1/1     Running   0          2m

Also, check the Falco service and daemonset:

kubectl get svc,daemonset -n falco

Expected output (truncated):

NAME          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/falco ClusterIP   None         <none>        8765/TCP   2m

NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/falco   3         3         3       3            3           <none>          2m

Step 3: Monitor Falco Logs

Once Falco is running, it immediately starts monitoring your cluster. You can observe its output directly from the logs of the Falco pods. This is invaluable for understanding what Falco is detecting out-of-the-box and for debugging custom rules later on. Falco’s default rules cover a wide range of common security threats, such as privilege escalation, sensitive file access, and unexpected network connections.

By tailing the logs, you’ll see alerts generated in real-time. These alerts typically include details about the event, the rule that was triggered, the process involved, and the container context. This information is critical for incident response and for fine-tuning your security policies.

kubectl logs -f -l app.kubernetes.io/name=falco -n falco

Verify:

You should see a continuous stream of logs. Initially, these might be informational messages, but if any suspicious activity occurs, Falco will log an alert like the one below:

17:03:04.999602052: Warning Sensitive file opened for reading by non-trusted program (user=root command=cat /etc/shadow container_id=2a7b8c9d0e1f container_name=my-app-pod-12345-abcde)
17:03:05.123456789: Notice A shell was spawned in a container with an attached terminal (user=root shell=bash container_id=2a7b8c9d0e1f container_name=my-app-pod-12345-abcde)

Step 4: Test a Default Falco Rule

Let’s trigger a common Falco rule to see it in action. A good way to do this is to attempt to open a sensitive file, like /etc/shadow, from within a container. This action is typically restricted and will trigger Falco’s “Read sensitive file” rule. We’ll create a simple busybox pod and then execute a command inside it.

This exercise demonstrates Falco’s ability to monitor process execution and file system access within containers, even for privileged operations. It highlights the importance of runtime security in identifying actions that could indicate a compromise, regardless of network policies or admission controls. For further reading on securing your cluster, consider our guide on Kubernetes Network Policies: Complete Security Hardening Guide.

1. Create a test pod:

kubectl run -it --rm --restart=Never busybox --image=busybox:1.36 -- /bin/sh

2. Inside the busybox pod, try to read /etc/shadow:

cat /etc/shadow

(You will likely get a “Permission denied” error, but the attempt is what Falco detects.)

exit

Verify:

In the Falco logs (from Step 3), you should see an alert related to the cat /etc/shadow command. This confirms Falco is actively monitoring and detecting suspicious activities.

kubectl logs -f -l app.kubernetes.io/name=falco -n falco

Expected output (look for entries similar to this):

17:05:30.456789012: Warning Read sensitive file /etc/shadow (user=root command=cat /etc/shadow container_id=... container_name=busybox)

Step 5: Define a Custom Falco Rule

While Falco’s default rules are extensive, you’ll often need to define custom rules to fit your specific application behavior and security requirements. This allows you to detect anomalies unique to your environment or enforce specific compliance policies. Custom rules are defined in YAML format and are loaded by Falco.

For this example, let’s create a rule that triggers an alert whenever the curl command is executed within any container, except for a specific namespace (e.g., monitoring). This could be useful for environments where outbound connections from application containers are generally restricted, or where you want to specifically monitor such activity.

1. Create a custom rules file:

Create a file named my-custom-rules.yaml:

# my-custom-rules.yaml
- rule: Disallow Curl in Default Namespace
  desc: Detects when 'curl' is executed in any container outside of the monitoring namespace.
  condition: >
    spawned_process and proc.name = "curl" and
    not container.image.repository startswith "gcr.io/google-containers/k8s-dns" and
    not k8s.ns.name = "monitoring"
  output: >
    Curl command executed in container (user=%user.name command=%proc.cmdline container_id=%container.id
    container_name=%container.name k8s.pod.name=%k8s.pod.name k8s.ns.name=%k8s.ns.name)
  priority: WARNING
  tags: [network, process, custom]

2. Update Falco Helm release with custom rules:

To apply your custom rules, you need to provide them to Falco via its Helm chart. The --set-file option is perfect for this, allowing you to pass the content of your YAML file directly to the Helm release. This restarts the Falco pods, loading your new rules.

helm upgrade falco falcosecurity/falco --namespace falco --reuse-values -f my-custom-rules.yaml

Note: The -f flag directly loads the YAML file as values, which is suitable for simple rule additions. For more complex configurations, you might embed rules under a specific key in a values file, e.g., customRules: | ....

Verify:

Check the Falco pod logs to ensure the new rules were loaded without errors. You can also exec into a Falco pod and list the rules to confirm your custom rule is present.

# Get the name of one of your Falco pods
FALCO_POD_NAME=$(kubectl get pods -n falco -l app.kubernetes.io/name=falco -o jsonpath="{.items[0].metadata.name}")

# Exec into the Falco pod and list rules
kubectl exec -it $FALCO_POD_NAME -n falco -- falco --list-rules | grep "Disallow Curl in Default Namespace"

Expected output:

Falco rules:
  - Disallow Curl in Default Namespace (enabled: yes)

Step 6: Test the Custom Falco Rule

Now, let’s trigger our newly created custom rule. We’ll run a curl command from a busybox container in the default namespace, which should trigger the “Disallow Curl in Default Namespace” rule. This verifies that Falco is correctly parsing and applying your custom logic.

Understanding how to define and test custom rules is a powerful aspect of Falco. It allows security teams to adapt Falco’s detection capabilities to evolving threats and specific application requirements, providing a flexible and robust runtime security solution.

1. Create a test pod and execute curl:

kubectl run -it --rm --restart=Never curl-test --image=busybox:1.36 -- /bin/sh

Inside the busybox pod:

curl google.com

exit

2. (Optional) Create a monitoring namespace and run curl there to ensure it’s ignored:

kubectl create namespace monitoring
kubectl run -it --rm --restart=Never curl-test-monitoring --namespace monitoring --image=busybox:1.36 -- /bin/sh

Inside the busybox pod in the monitoring namespace:

curl google.com

exit

Verify:

Check the Falco logs again. You should see an alert for the curl command executed in the default namespace, but not for the one in the monitoring namespace.

kubectl logs -f -l app.kubernetes.io/name=falco -n falco

Expected output (for the default namespace curl):

17:10:15.678901234: Warning Curl command executed in container (user=root command=curl google.com container_id=... container_name=curl-test k8s.pod.name=curl-test k8s.ns.name=default)

You should NOT see an alert for the curl command executed in the monitoring namespace.

Step 7: Configure Falco Alert Outputs

Falco’s real power comes from its ability to send alerts to various destinations, allowing for immediate action and integration with existing security workflows. While logging to stdout is useful for monitoring, in a production environment, you’ll want to integrate Falco with a SIEM, a messaging service like Slack, or an alerting system like PagerDuty.

The Falco Helm chart allows you to configure these outputs easily. For this example, let’s configure Falco to send alerts to a local file and also to a webhook, simulating a Slack or PagerDuty integration.

1. Create a custom values file for outputs:

Create a file named falco-outputs.yaml:

# falco-outputs.yaml
falco:
  outputs:
    # Enable file output
    fileOutput:
      enabled: true
      keepAlive: false
      filename: /var/log/falco/alerts.json
    
    # Enable webhook output (replace with your actual webhook URL)
    webhook:
      enabled: true
      url: "https://your-webhook-url.com/falco-alerts" # Replace with a real webhook URL for testing, e.g., a Slack incoming webhook
      insecure: false
      verifyHostname: true
      headers:
        Content-Type: "application/json"
      # You can customize the JSON output format for webhooks
      # For Slack, you might want to adjust this for better formatting.
      # For example:
      # customHeaders:
      #   - "Content-Type: application/json"
      # customBody: '{ "text": "Falco Alert: %output" }'
    
    # Disable stdout output if you only want alerts to go to specific destinations
    stdout:
      enabled: false

# Mount a volume for the log file (requires PVC or hostPath for persistence)
# For demonstration, we'll just let it log inside the container.
# In production, consider persistent storage.

Note: For a real webhook, you’d generate one from a service like Slack Incoming Webhooks or PagerDuty Events API. For local testing, you can use a service like webhook.site to get a temporary URL that logs incoming requests.

2. Upgrade Falco with the new output configuration:

helm upgrade falco falcosecurity/falco --namespace falco --reuse-values -f falco-outputs.yaml

Verify:

After the upgrade, Falco pods will restart. Now, trigger another alert (e.g., by running cat /etc/shadow in a busybox pod).
If you configured a real webhook, check the destination (e.g., Slack channel or webhook.site) for incoming alerts.
To check the file output, you’d need to exec into a Falco pod and check the file:

FALCO_POD_NAME=$(kubectl get pods -n falco -l app.kubernetes.io/name=falco -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $FALCO_POD_NAME -n falco -- cat /var/log/falco/alerts.json

Expected output (example JSON alert in the file):

{"output":"17:15:20.123456789: Warning Read sensitive file /etc/shadow (user=root command=cat /etc/shadow container_id=... container_name=busybox)","priority":"Warning","rule":"Read sensitive file /etc/shadow","time":"2023-10-27T17:15:20.123456789Z", ...}

Production Considerations

When deploying Falco in a production Kubernetes environment, several factors need careful consideration to ensure optimal performance, reliability, and security.

1. Resource Management:
   • CPU/Memory: While Falco is generally lightweight, monitoring a high-volume system call environment can consume resources. Monitor Falco pod resource usage and adjust requests/limits in the Helm chart values.
   • NodeSelector/Tolerations: If you have specialized nodes (e.g., GPU nodes for LLM GPU Scheduling), ensure Falco agents are deployed correctly or excluded if not needed, using nodeSelector or tolerations.
2. Rule Management:
   • Custom Rules Version Control: Store your custom Falco rules in a version control system (e.g., Git). This allows for tracking changes, auditing, and easy deployment via CI/CD pipelines.
   • Testing Rules: Thoroughly test new and modified rules in a staging environment before deploying to production to avoid false positives or missed detections.
   • Baselining: Understand your application’s normal behavior to create effective rules and reduce alert fatigue.
   • Community Rules: Leverage the Falco community rules, but review and adapt them to your specific needs.
3. Alerting and Integration:
   • SIEM Integration: For centralized logging and analysis, integrate Falco alerts with your Security Information and Event Management (SIEM) system (e.g., Splunk, ELK Stack, QRadar). Falco supports various output formats (JSON, gRPC).
   • Incident Response Workflow: Define clear incident response procedures for Falco alerts. Who gets alerted? How are alerts escalated? What steps are taken to investigate and remediate?
   • Alert Fatigue: Fine-tune rules and suppress known benign events to prevent alert fatigue, which can lead to important alerts being overlooked.
4. Persistence and Logging:
   • Persistent Storage for Logs: If using file output for alerts, ensure the volume is persistent (e.g., using a PersistentVolumeClaim) so logs are not lost if a Falco pod restarts.
   • Centralized Logging: Forward Falco logs to a centralized logging system (e.g., Elasticsearch, Loki, Datadog) for long-term storage, correlation, and analysis.
5. Kernel Module vs. eBPF Probe:
   • Falco can use either a kernel module or an eBPF probe to capture system calls. For modern kernels (v5.4+), the eBPF probe is generally preferred for its stability, performance, and reduced risk of kernel panics. Ensure your kernel version supports eBPF and configure Falco accordingly in the Helm chart.
   • driver.kind: eBPF in your Helm values can explicitly set this.
6. Security Context:
   • Falco requires privileged access to the host kernel to function. Understand the implications of running privileged containers and ensure your security policies allow this for Falco’s DaemonSet.
7. Upgrade Strategy:
   • Plan for Falco upgrades, especially when kernel versions change or new Falco features are released. Test upgrades in a staging environment.
8. High Availability:
   • Falco runs as a DaemonSet, providing a degree of high availability as it runs on every node. However, ensure the underlying nodes themselves are highly available.

Troubleshooting

Here are some common issues you might encounter with Falco and their solutions:

1. Issue: Falco pods are stuck in Init or CrashLoopBackOff state.

   Explanation: This often indicates a problem with the Falco driver (kernel module or eBPF probe) failing to load or compile. This is common if the kernel headers on your nodes don’t match the Falco driver requirements or if there are insufficient permissions.

   Solution:

   • Check Falco Init Container Logs:

     kubectl logs -n falco <falco-pod-name> -c falco-driver-loader

     Look for errors related to dkms, kernel headers, or eBPF.

   • Verify Kernel Headers: Ensure your nodes have the correct kernel headers installed. For cloud providers, this might involve specific AMI/OS versions. For example, on Ubuntu, you might need apt install linux-headers-$(uname -r).
   • Try eBPF (if kernel v5.4+): If your kernel supports it, try forcing the eBPF driver in your Helm values:

     driver:
       kind: eBPF

     Then upgrade Falco:

     helm upgrade falco falcosecurity/falco --namespace falco --reuse-values -f your-values.yaml

   • Check Node Permissions: Ensure the Falco DaemonSet has the necessary privileges (privileged: true, host paths mounted) to load the driver. The Helm chart usually handles this, but custom configurations might interfere.
2. Issue: Falco is running, but no alerts are being generated for suspicious activity.

   Explanation: This could be due to incorrect rule conditions, misconfigured outputs, or Falco not actually monitoring the expected events.

   Solution:

   • Check Falco Logs:

     kubectl logs -f -l app.kubernetes.io/name=falco -n falco

     Look for any errors during rule loading or messages indicating that events are being processed.

   • Verify Rules:

     kubectl exec -it <falco-pod-name> -n falco -- falco --list-rules

     Ensure your custom rules are listed and marked as enabled. Double-check the rule conditions and fields.

   • Test Rule Specificity: Temporarily broaden a rule’s condition to see if it triggers, then narrow it down.
   • Check Outputs: If you’re expecting alerts in a webhook or file, verify the output configuration in your Helm values and check the destination (e.g., webhook.site).
3. Issue: Excessive false positives or alert fatigue.

   Explanation: Default rules or overly broad custom rules can trigger alerts for legitimate application behavior.

   Solution:

   • Baseline Normal Behavior: Understand what’s normal for your applications.
   • Refine Rules: Add exceptions (and not ...) to your rules to exclude specific processes, users, or container images that are known to perform the “suspicious” action legitimately. For example:

     - rule: My Custom Rule
       condition: >
         ... and not (k8s.ns.name = "my-trusted-namespace" and proc.name = "my-trusted-process")
       ...

   • Adjust Priorities: Change the priority of less critical rules from ERROR to WARNING or NOTICE to filter what gets sent to high-priority channels.
   • Disable Unnecessary Rules: Review the default rules and disable any that are not relevant to your environment.
4. Issue: Falco performance overhead on nodes.

   Explanation: In very busy clusters with high system call rates, Falco can consume significant CPU or memory.

   Solution:

   • Optimize Rules: Complex rules or rules that match on very frequent events can increase overhead. Simplify conditions where possible.
   • Use eBPF (if possible): The eBPF driver generally has better performance characteristics than the kernel module.
   • Adjust Resource Limits: Increase CPU and memory limits for the Falco DaemonSet pods if they are being throttled or OOMKilled.

     falco:
       resources:
         limits:
           cpu: 500m
           memory: 512Mi
         requests:
           cpu: 100m
           memory: 256Mi

   • Filter Events at Source: In advanced configurations, Falco can be configured to filter some events at the driver level to reduce the amount of data processed by the userspace Falco engine.
5. Issue: Falco alerts are not reaching external systems (e.g., Slack, SIEM).

   Explanation: This is typically a network connectivity issue, incorrect API key/URL, or misconfigured output format.

   Solution:

   • Check Network Connectivity:

     kubectl exec -it <falco-pod-name> -n falco -- curl -v <your-webhook-url>

     Ensure the Falco pod can reach the external webhook URL. Check network policies (e.g., Kubernetes Network Policies) that might be blocking egress traffic from the Falco namespace.

   • Verify Webhook URL/API Key: Double-check the URL, tokens, or API keys configured in falco-outputs.yaml.
   • Inspect Webhook Server Logs: If you control the webhook server, check its logs for incoming requests and any errors.
   • Check Falco Output Configuration: Ensure webhook.enabled: true and any custom headers or body formats are correct for your target system.
6. Issue: Falco driver fails to load after a kernel upgrade on the nodes.

   Explanation: When a node’s kernel is upgraded, the previously compiled Falco kernel module becomes incompatible. The Falco driver loader needs to recompile it for the new kernel version.

   Solution:

   • Restart Falco Pods: The simplest solution is often to delete and recreate the Falco pods to force the driver loader to run again:

     kubectl delete pods -l app.kubernetes.io/name=falco -n falco

     The DaemonSet will recreate them.

   • Ensure Kernel Headers are Present: Confirm that the new kernel’s headers are installed on the nodes.
   • Consider Live Kernel Patching (Advanced): For environments with strict uptime requirements, explore solutions like live kernel patching for Falco, though this is more complex.

FAQ Section

1. What is the difference between Falco and Kubernetes Admission Controllers?

Kubernetes Admission Controllers operate at the API server level, enforcing policies before objects are persisted in etcd. They are preventive measures, stopping undesirable configurations from ever entering the cluster. Falco, on the other hand, provides runtime security by monitoring system calls at the kernel level after pods are deployed and running. It detects anomalous behavior and policy violations in real-time, acting as a detective control. Both