Secure Kubernetes Runtime with Falco

Introduction

In the dynamic and often hostile landscape of cloud-native environments, securing your Kubernetes clusters is paramount. While traditional security measures like static analysis, vulnerability scanning, and network segmentation are crucial, they often fall short when it comes to detecting threats at runtime. What happens when an attacker bypasses your perimeter defenses, or a legitimate application is exploited to perform malicious actions? This is where runtime security steps in, providing a critical layer of defense that monitors system behavior, identifies anomalies, and alerts you to potential breaches in real-time.

Falco, an open-source project incubated at the Cloud Native Computing Foundation (CNCF), is the de facto standard for Kubernetes runtime security. It acts as a behavioral activity monitor designed to detect unexpected application behavior, unauthorized access to sensitive files, suspicious network connections, and other malicious activities within your containers and hosts. By leveraging system calls, Falco provides deep visibility into your cluster’s operations, allowing you to define rules that trigger alerts when predefined security policies are violated. In this comprehensive guide, we’ll walk you through deploying, configuring, and leveraging Falco to fortify your Kubernetes deployments against runtime threats.

# Add Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco
helm install falco falcosecurity/falco --namespace falco --create-namespace

# View Falco logs (replace with your Falco pod name)
kubectl logs -f -l app.kubernetes.io/name=falco -n falco

# Test a rule (e.g., shell in container)
kubectl exec -it <your-pod-name> -- bash

Prerequisites

Before diving into Falco, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (v1.18+ recommended). You can use Minikube, Kind, or a cloud-managed service like EKS, GKE, or AKS.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• helm: The Kubernetes package manager, version 3+. Install it by following the Helm installation guide.
• Basic Linux and Kubernetes Knowledge: Familiarity with Linux commands, Docker/container concepts, and Kubernetes primitives (Pods, Deployments, DaemonSets, etc.) will be beneficial.
• Sufficient Permissions: Your kubeconfig context should have administrative privileges to install cluster-wide components like Falco.

Step-by-Step Guide: Deploying and Configuring Falco

1. Add the Falco Helm Repository

To easily deploy Falco, we’ll use its official Helm chart. Helm simplifies the installation and management of Kubernetes applications. The first step is to add the Falco Helm repository to your local Helm client. This allows you to discover and install Falco’s chart.

This command fetches the repository information, making the Falco chart available for installation. It’s good practice to update your Helm repositories regularly to ensure you have access to the latest versions of charts.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Verify:

You should see output similar to this, indicating the repository has been added and updated:

"falcosecurity" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
Update Complete. ⎈Happy Helming!⎈

2. Install Falco using Helm

Now that the repository is added, we can install Falco. We’ll deploy Falco into its own namespace for better resource isolation and management. The Falco Helm chart deploys Falco as a DaemonSet, ensuring that a Falco agent runs on every node in your Kubernetes cluster. This is crucial for comprehensive runtime security, as each agent monitors the system calls originating from pods and processes on its host node.

The installation will create a new namespace called `falco`, deploy the Falco DaemonSet, a ServiceAccount, and other necessary Kubernetes objects. Falco typically uses a kernel module or an eBPF probe to capture system calls, with the eBPF probe being the modern and often preferred method due to its performance and stability. For more on eBPF, consider exploring eBPF Observability: Building Custom Metrics with Hubble.

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf # Use eBPF probe (recommended)

Verify:

Check the status of the Falco pods. They should be running. It might take a minute or two for the pods to transition from `ContainerCreating` to `Running` as the driver is loaded.

kubectl get pods -n falco

Expected output:

NAME                   READY   STATUS    RESTARTS   AGE
falco-XXXXX            1/1     Running   0          2m
falco-YYYYY            1/1     Running   0          2m
# (one pod per node in your cluster)

3. Access Falco Logs and Alerts

Once Falco is running, it immediately starts monitoring your cluster for suspicious activities based on its default rules. These rules cover a wide range of common attack patterns, such as a shell being run in a container, sensitive files being accessed, or outbound network connections to unusual ports. The primary way to see Falco’s alerts is by tailing its logs.

You can view the logs of any Falco pod to see the alerts it’s generating. Remember that Falco runs as a DaemonSet, so you’ll have multiple pods (one per node). You can either tail a specific pod’s logs or use a label selector to aggregate logs from all Falco pods, though the latter might be overwhelming in a busy cluster.

# Get the name of a Falco pod
FALCO_POD=$(kubectl get pods -n falco -l app.kubernetes.io/name=falco -o jsonpath='{.items[0].metadata.name}')

# Tail the logs of one Falco pod
kubectl logs -f $FALCO_POD -n falco

Verify:

You should see a stream of logs. Initially, these might be informational messages about Falco starting up. To generate an alert, let’s try to get a shell inside a running container. First, deploy a simple Nginx application:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

kubectl apply -f nginx-deployment.yaml
kubectl get pods -l app=nginx

Once the Nginx pod is running, get a shell inside it. This action is covered by a default Falco rule (`Run shell in a container`).

NGINX_POD=$(kubectl get pods -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $NGINX_POD -- bash

As soon as you execute the `kubectl exec` command, switch back to your terminal where you’re tailing Falco logs. You should see an alert similar to this:

{"output":"16:34:01.077271891: Warning A shell was spawned in a container with an attached terminal (user=root shell=bash parent=runc cmdline=bash container_id=...)","priority":"Warning","rule":"Run shell in a container","time":"2023-10-27T16:34:01.077271891Z", ...}

4. Explore Falco Rules

Falco’s power lies in its flexible rule engine. Rules are defined in YAML files and typically consist of conditions, output messages, and priority levels. Falco comes with a rich set of default rules that cover many common attack vectors. However, you’ll often need to customize or add rules specific to your applications and threat model.

You can inspect the default rules directly from the Falco configuration within the Falco pod. Understanding these rules is key to both tuning Falco and developing your own custom detections.

# Copy default rules from a Falco pod to your local machine
FALCO_POD=$(kubectl get pods -n falco -l app.kubernetes.io/name=falco -o jsonpath='{.items[0].metadata.name}')
kubectl cp $FALCO_POD:/etc/falco/falco_rules.local.yaml falco_rules.local.yaml -n falco
kubectl cp $FALCO_POD:/etc/falco/falco_rules.yaml falco_rules.yaml -n falco
kubectl cp $FALCO_POD:/etc/falco/k8s_audit_rules.yaml k8s_audit_rules.yaml -n falco

# View the contents of a rule file
cat falco_rules.yaml | less

Verify:

Review the `falco_rules.yaml` file. You’ll find rules like `Run shell in a container`, `Write below /etc`, `Unexpected inbound network connection`, etc. Each rule has a `rule` name, `desc`ription, `condition` (using Falco’s expression language), `output` format, and `priority`.

- rule: Run shell in a container
  desc: a shell was spawned by a container
  condition: >
    spawned_process and container and shell_procs and proc.name=/bin/bash and proc.args contains "-i"
    and not user_expected_shell_in_container_start
    and not user_expected_shell_in_container
  output: >
    A shell was spawned in a container with an attached terminal (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
  priority: WARNING
  tags: [container, shell, process, mitre_execution]

This snippet shows the `Run shell in a container` rule. The `condition` uses Falco’s powerful filter syntax to detect when a process matching `shell_procs` (a list of common shell executables like `bash`, `sh`, `zsh`) is spawned within a container and has an attached terminal.

5. Create Custom Falco Rules

One of Falco’s greatest strengths is its extensibility. You can define your own rules to detect application-specific threats or address unique compliance requirements. Custom rules are typically placed in `falco_rules.local.yaml` to separate them from the default rules and simplify upgrades.

Let’s create a simple custom rule to detect when the `curl` command is executed inside a specific application container. This could be useful if your application should never initiate outbound connections using `curl` directly.

# custom_falco_rules.yaml
- rule: Curl in Nginx Container
  desc: Detects when curl is executed inside an Nginx container.
  condition: >
    container.name="nginx" and proc.name="curl"
  output: >
    Curl command executed in Nginx container (user=%user.name proc.cmdline=%proc.cmdline container.id=%container.id)
  priority: CRITICAL
  tags: [container, network, custom]

Apply this custom rule by creating a Kubernetes ConfigMap and then configuring the Falco Helm chart to use it. First, create the ConfigMap:

kubectl create configmap custom-falco-rules -n falco --from-file=custom_falco_rules.yaml

Now, upgrade your Falco installation to include this ConfigMap. We’ll mount it into the Falco pods as an additional rules file.

helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --reuse-values \
  --set customRules.custom_falco_rules\.yaml=custom-falco-rules

Verify:

After the upgrade, Falco pods will restart. Once they are `Running` again, get a shell into your Nginx container and execute `curl`. Tail the Falco logs to see the alert.

# Get Nginx pod name
NGINX_POD=$(kubectl get pods -l app=nginx -o jsonpath='{.items[0].metadata.name}')

# Execute curl inside the Nginx container
kubectl exec -it $NGINX_POD -- curl example.com

In your Falco logs, you should now see an alert with `priority: CRITICAL` for your custom rule:

{"output":"16:38:22.123456789: Critical Curl command executed in Nginx container (user=root proc.cmdline=curl example.com container.id=...)","priority":"Critical","rule":"Curl in Nginx Container","time":"2023-10-27T16:38:22.123456789Z", ...}

6. Integrate Falco with External Systems (Optional)

Falco is designed to be integrated with existing security and operational workflows. While viewing logs directly is useful for testing, in a production environment, you’ll want to forward these alerts to a centralized logging system (like Elasticsearch/Splunk), a SIEM, or a notification system (like Slack, PagerDuty, or Opsgenie). Falco can output alerts in various formats (JSON, text) and supports different output channels.

The Falco Helm chart provides configurations for various outputs. Let’s demonstrate how to configure Falco to send alerts to a Slack channel. You’ll need a Slack webhook URL for this.

# Replace with your actual Slack webhook URL
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"

helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --reuse-values \
  --set falco.jsonOutput=true \
  --set falco.jsonIncludeOutputProperty=true \
  --set falco.outputs.slack.enabled=true \
  --set falco.outputs.slack.webhookURL=$SLACK_WEBHOOK_URL \
  --set falco.outputs.slack.level=warning \
  --set falco.outputs.stdout.enabled=false # Optionally disable stdout if you only want external outputs

Verify:

After the upgrade, get a shell inside your Nginx container again:

kubectl exec -it $NGINX_POD -- bash

You should receive a notification in your configured Slack channel, detailing the `Run shell in a container` alert. This demonstrates how Falco can become an integral part of your incident response pipeline. For more advanced networking and integration, consider solutions like Cilium WireGuard Encryption or Istio Ambient Mesh Guide which can also play a role in securing data in transit.

Production Considerations

Deploying Falco in production requires careful planning to ensure it’s effective, efficient, and doesn’t introduce operational overhead.

• Rule Management:
  • Custom Rules: Maintain your custom rules in a version-controlled repository. Use Helm’s `customRules` feature or a ConfigMap to inject them.
  • Baselines: Establish a baseline of normal behavior for your applications. This helps in tuning rules to reduce false positives, which can lead to alert fatigue.
  • Rule Updates: Regularly review and update Falco’s default rules and your custom rules to keep pace with new threats and application changes.
• Alerting and Integration:
  • SIEM Integration: For production, forward Falco alerts to a Security Information and Event Management (SIEM) system (e.g., Splunk, ELK Stack, QRadar). This centralizes security events for analysis, correlation, and long-term storage.
  • Incident Response: Integrate Falco with your incident response workflows. High-priority alerts should trigger automated actions or escalate to on-call teams via PagerDuty, Opsgenie, or similar tools.
  • Cloud-Native Logging: Leverage cloud provider logging services (CloudWatch Logs, Stackdriver, Azure Monitor) for collecting and analyzing Falco logs.
• Performance and Resource Usage:
  • eBPF vs. Kernel Module: While the eBPF driver is generally recommended for performance and stability, ensure your kernel supports it. If not, the kernel module is an alternative, but it requires recompilation on kernel upgrades.
  • Resource Limits: Set appropriate CPU and memory limits for Falco pods to prevent them from consuming excessive resources, especially in dense clusters. Monitor their resource usage closely.
  • Filtering: Use Falco’s powerful filtering language to reduce the volume of events processed and alerts generated, focusing only on relevant security events.
• High Availability and Scalability:
  • DaemonSet: Falco runs as a DaemonSet, ensuring it’s present on every node. This provides inherent high availability at the agent level.
  • Centralized Logging: Ensure your centralized logging and alerting systems are highly available and scalable to handle the volume of alerts Falco can generate.
• Security Context and Privileges:
  • Falco requires privileged access to the host’s kernel to capture system calls. Ensure the DaemonSet runs with appropriate `securityContext` and `hostPath` mounts (e.g., `/dev`) for the driver to function correctly.
  • For more on hardening your cluster, consider our guide on Kubernetes Network Policies: Complete Security Hardening Guide.
• Continuous Monitoring:
  • Monitor the health of Falco pods and ensure they are always running and collecting events.
  • Monitor for Falco itself being tampered with or stopped, as this could indicate a sophisticated attack.

Troubleshooting

Here are common issues you might encounter when working with Falco and their solutions.

1. Falco Pods are in `CrashLoopBackOff` or `Pending` State

Issue: Falco pods repeatedly crash or fail to start.

Solution:

• Check Pod Logs: The first step is always to check the logs of the crashing pod.

  kubectl logs <falco-pod-name> -n falco

  Look for error messages related to driver loading, missing dependencies, or configuration issues.

• Verify Driver Installation: Falco needs a kernel module or eBPF probe.
  • If using eBPF, ensure your kernel version is compatible (usually 4.14+).
  • If using the kernel module, ensure `kernel-headers` are installed on the host nodes for your specific kernel version.
• Resource Constraints: Check if the nodes have enough resources. Sometimes, Falco might fail to start if the node is overloaded or `resourceLimits` are too low.
• Permissions: Ensure the Falco ServiceAccount has the necessary permissions (e.g., `privileged` security context) to interact with the host kernel.

2. Falco is Not Generating Any Alerts

Issue: You’ve performed actions that should trigger alerts (e.g., `kubectl exec — bash`), but no alerts appear in Falco logs.

Solution:

• Check Falco Pod Status: Ensure all Falco pods are `Running` and healthy.
• Verify Driver Status: Look in the Falco pod logs for messages indicating the driver (eBPF or kernel module) loaded successfully. If not, refer to the `CrashLoopBackOff` solution.
• Review Rules:
  • Are the rules you expect to fire actually enabled?
  • Are the conditions in the rules correct and matching the events you are generating?
  • Have you accidentally overridden or disabled default rules?

  You can verify the active rules by examining the Falco configuration files inside a running pod:

  kubectl exec -it <falco-pod-name> -n falco -- cat /etc/falco/falco.yaml
  kubectl exec -it <falco-pod-name> -n falco -- cat /etc/falco/falco_rules.yaml

• Container Runtime: Ensure Falco is compatible with your container runtime (e.g., containerd, CRI-O, Docker). Most are supported, but specific configurations might be needed.

3. Too Many False Positives / Alert Fatigue

Issue: Falco is generating a high volume of alerts for legitimate application behavior, overwhelming your monitoring systems.

Solution:

• Tune Existing Rules: Modify the conditions of the rules causing false positives. For example, add exceptions for specific container names, image names, or user IDs.

  # Example: Modify "Run shell in a container" to ignore specific service account
  - rule: Run shell in a container
    condition: >
      ...
      and not user.name in ("system:serviceaccount:kube-system:some-internal-sa")
    ...

• Create Whitelisting Rules: Implement specific rules that explicitly allow certain behaviors for known applications and set their priority to `DEBUG` or `INFO` (or disable their output entirely).
• Adjust Priorities: Change the `priority` of less critical rules to `INFO` or `DEBUG` so they don’t trigger high-priority alerts.
• Filter at Source: Use Falco’s `jsonOutput` and `output_fields` to send only relevant data to your SIEM, or filter alerts based on priority or tags before forwarding.
• Utilize `falco_rules.local.yaml`: Place your custom overrides and whitelisting rules in `falco_rules.local.yaml` (or via Helm `customRules`) to keep them separate from default rules.

4. Falco is Consuming Too Many Resources

Issue: Falco pods are using excessive CPU or memory on your nodes.

Solution:

• Reduce Event Volume:
  • Disable Unnecessary Rules: Turn off rules that are not relevant to your security posture.
  • Refine Conditions: Make rule conditions more specific to reduce the number of events that trigger them.
  • Exclude System Calls: If certain system calls are known to be noisy but irrelevant for security, you can exclude them from Falco’s processing (advanced configuration).
• Set Resource Limits: Ensure `requests` and `limits` are set appropriately for Falco pods in your Helm chart values. Start with reasonable values and adjust based on observation.

  # values.yaml snippet for Falco Helm chart
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

• eBPF vs. Kernel Module: If you are using the kernel module, consider switching to the eBPF probe if your kernel supports it, as it generally has better performance characteristics.

5. Falco Alerts to Slack/External System Not Working

Issue: You’ve configured external outputs (e.g., Slack), but alerts are not being sent.

Solution:

• Check Falco Pod Logs: The Falco pod logs will often show errors if it fails to send an alert to an external endpoint (e.g., HTTP 400/500 errors from the webhook).

  kubectl logs <falco-pod-name> -n falco

• Verify Webhook URL: Double-check that the webhook URL provided in the Helm values (e.g., `falco.outputs.slack.webhookURL`) is correct and active.
• Network Connectivity: Ensure that the Kubernetes cluster’s nodes or Falco pods have outbound network connectivity to the external service (e.g., Slack’s API). Check network policies if you have them enabled (e.g., Kubernetes Network Policies: Complete Security Hardening Guide).
• Firewall Rules: If running in a cloud environment, ensure any security groups or network ACLs allow outbound connections from your worker nodes to the external service.
• Output Level: Verify that the `level` configured for the external output (`falco.outputs.slack.level`) is low enough to include the alerts you expect to see (e.g., `warning` or `info` instead of `critical`).

FAQ Section

1. What is the difference between Falco and a traditional IDS/IPS?

Falco operates at the system call level, providing deep visibility into container and host behavior. Traditional Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) primarily focus on network traffic analysis and signature matching for known exploits. While both are crucial for security, Falco offers critical runtime visibility within the host and container that network-based systems cannot provide. It complements, rather than replaces, traditional IDS/IPS.

2. Can Falco prevent attacks?

By default, Falco is a detection and alerting tool, not a prevention tool. It observes system calls and alerts on suspicious activity. However, Falco can be integrated with external tools (like Kubernetes Admission Controllers or custom operators) to trigger automated responses, effectively moving towards prevention. For example, an alert from Falco could trigger a Kubernetes Network Policy update or cordon a compromised node, but this requires custom integration.

3. Does Falco support Kubernetes Audit Logs?

Yes, Falco can consume Kubernetes Audit Logs to detect suspicious API server activity. This allows you to monitor for actions like sensitive `kubectl` commands, unauthorized role creations, or modifications to critical resources. The Falco Helm chart includes options (`falco.k8sAudit.enabled=true`) to enable this feature, which deploys an additional component to tail audit logs. For deep dives into Kubernetes security, integrating audit logs with runtime security is highly recommended.

4. What is the performance overhead of running Falco?

The performance overhead of Falco is generally low, especially when using the eBPF driver. Falco is designed to be efficient, processing system calls in kernel space (with eBPF) or with minimal overhead via its kernel module. The actual impact depends on the workload’s system call intensity and the complexity of your Falco rules. In most production environments, the overhead is negligible, but it’s always recommended to monitor resource usage and perform load testing.

5. How does Falco compare to other runtime security tools?

Falco is unique in its focus on system call analysis and its strong integration with the Kubernetes ecosystem. Other tools might offer broader endpoint detection and response (EDR) capabilities or focus on specific aspects like vulnerability management. Falco’s strength lies in its real-time, behavioral detection within containers and hosts, making it a critical component of a layered security strategy in Kubernetes. Its CNCF incubation status also provides significant community support and development.

Cleanup Commands

To remove Falco and its associated resources from your cluster, use the following Helm command:

helm uninstall falco --namespace falco
kubectl delete namespace falco

If you deployed the Nginx application, remove it as well:

kubectl delete -f nginx-deployment.yaml

Next Steps / Further Reading

Congratulations! You’ve successfully deployed Falco and started securing your Kubernetes runtime. Here are some next steps to deepen your knowledge and enhance your security posture:

• Explore More Falco Rules: Dive deeper into the official Falco rules documentation and the Falco rules repository to understand the full range of detection capabilities.
• Integrate with Cloud-Native Security Tools: Explore integrations with other CNCF projects like Sigstore and Kyverno for supply chain security and policy enforcement, or tools like Prometheus for metrics and Grafana for visualization.
• Kubernetes Audit Logs with Falco: Configure Falco to consume Kubernetes Audit Logs for comprehensive control plane security. Refer to the Falco documentation on Kubernetes Audit Events.
• Falco Sidekick: For more advanced alerting and integration options, look into Falco Sidekick, a companion project that facilitates sending Falco events to a multitude of outputs.
• Threat Modeling: Perform a threat model for your applications to identify potential attack vectors and write specific Falco rules to detect them.
• Automated Response: Investigate how to automate responses to Falco alerts using tools like Kube-ripper or custom operators to prevent attacks in real-time.

Conclusion

Runtime security is an indispensable layer in your Kubernetes security strategy, and Falco stands out as the leading open-source solution for this critical domain. By monitoring system calls and applying a robust rule engine, Falco provides unparalleled visibility into the behavioral activities within your containers and hosts, allowing you to detect and respond to threats that bypass traditional defenses.

From initial deployment with Helm to creating custom rules and integrating with external alerting systems, this guide has equipped you with the knowledge to implement effective runtime security. Remember, security is an ongoing process. Continuously refine your Falco rules, stay updated with new threats, and integrate Falco deeply into your security operations to maintain a strong and resilient Kubernetes environment. Your journey towards a more secure cloud-native future is well underway.