Kubernetes Logging: Vector & ClickHouse

Introduction

In the dynamic world of Kubernetes, robust logging is not just a best practice; it’s a necessity. As applications scale, containers ephemeralize, and microservices proliferate, traditional logging approaches quickly fall short. Sifting through logs from hundreds or thousands of pods across multiple nodes manually becomes an impossible task. Without a centralized, efficient, and scalable logging solution, debugging production issues, monitoring application health, and maintaining security compliance turn into Sisyphean efforts. This is where a powerful combination like Vector and ClickHouse shines, transforming raw log data into actionable insights.

This guide will walk you through setting up a modern, high-performance logging stack for your Kubernetes clusters using Vector as a universal data router and ClickHouse as a lightning-fast analytical database. Vector, a lightweight and ultra-efficient observability data pipeline, excels at collecting, transforming, and routing logs from various sources to multiple destinations. Paired with ClickHouse, an open-source columnar database renowned for its incredible query performance on large datasets, you gain an unparalleled ability to store, search, and analyze your Kubernetes logs with speed and precision. Whether you’re a developer troubleshooting a microservice, an SRE monitoring system health, or a security analyst investigating an incident, this stack empowers you with the visibility you need to keep your Kubernetes environments running smoothly.

Prerequisites

Before embarking on this logging adventure, ensure you have the following tools and knowledge at your disposal:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.18+ recommended). This guide assumes you have kubectl configured and connected to your cluster. For setting up a cluster, refer to the official Kubernetes documentation.
• Helm: Version 3.x or higher, used for deploying Vector and other components. Install instructions can be found on the Helm website.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, DaemonSets, Services, and Namespaces.
• Basic Linux Command Line Skills: Comfort with using a terminal for commands.
• Resource Availability: Ensure your Kubernetes cluster has sufficient CPU, memory, and storage resources to run ClickHouse and Vector components. ClickHouse can be resource-intensive, especially for large log volumes.

Step-by-Step Guide

1. Deploy the ClickHouse Operator and Cluster

To manage ClickHouse deployments in Kubernetes, we’ll use the Altinity ClickHouse Operator. This operator simplifies the deployment and management of ClickHouse clusters, handling complex tasks like scaling, replication, and upgrades. We’ll start by installing the operator itself, then deploy a simple ClickHouse cluster.

First, apply the operator’s installation bundle. This will create the necessary Custom Resource Definitions (CRDs), RBAC roles, and the operator deployment. After the operator is running, we’ll deploy a basic ClickHouse cluster definition. For more advanced configurations, including sharding and replication, consult the Altinity ClickHouse Operator GitHub repository.

# Install the ClickHouse Operator
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle-metrics.yaml

# Wait for the operator to be ready
echo "Waiting for ClickHouse operator to be ready..."
kubectl wait --for=condition=available deployment/clickhouse-operator --timeout=300s -n clickhouse-operator

# Deploy a simple ClickHouse cluster
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/ch-cluster/chi-01-simple-layout.yaml

Verify Step 1

Confirm that the ClickHouse operator is running and that the ClickHouse cluster pods are being created and eventually reach a running state. This might take a few minutes as the pods pull images and initialize.

kubectl get pods -n clickhouse-operator
kubectl get pods -n default -l "clickhouse.altinity.com/chi=chi-01-simple-layout"

Expected Output (similar to):

NAME                                READY   STATUS    RESTARTS   AGE
clickhouse-operator-7b77d698f-abcde   1/1     Running   0          2m

NAME             READY   STATUS    RESTARTS   AGE
clickhouse-01-0-0   1/1     Running   0          1m

2. Configure ClickHouse Database and User

With the ClickHouse cluster up and running, we need to prepare a database for our Kubernetes logs and create a dedicated user for Vector to connect with appropriate permissions. This ensures that Vector has a secure and isolated way to write data to ClickHouse. We’ll create a database named `kubernetes_logs` and a user `vector` with a strong password. Remember to replace `vector_password` with a secure, unique password in a production environment.

Accessing the ClickHouse client via `kubectl exec` allows us to run SQL commands directly against the cluster. This is a common way to perform initial setup or administrative tasks.

# Get the name of a running ClickHouse pod
CH_POD=$(kubectl get pods -l "clickhouse.altinity.com/chi=chi-01-simple-layout" -o jsonpath="{.items[0].metadata.name}")

# Create the database
kubectl exec -it $CH_POD -- clickhouse client -mn -q "CREATE DATABASE IF NOT EXISTS kubernetes_logs;"

# Create a user for Vector with a strong password
kubectl exec -it $CH_POD -- clickhouse client -mn -q "CREATE USER vector IDENTIFIED WITH plaintext_password BY 'vector_password';"

# Grant permissions to the vector user on the kubernetes_logs database
kubectl exec -it $CH_POD -- clickhouse client -mn -q "GRANT ALL ON kubernetes_logs.* TO vector;"

Verify Step 2

You can verify the database and user creation by logging into ClickHouse as the `vector` user and listing databases, or by checking user grants.

# List databases as the new user (you'll be prompted for the password 'vector_password')
kubectl exec -it $CH_POD -- clickhouse client -u vector -mn -q "SHOW DATABASES;"

Expected Output (similar to):

kubernetes_logs
default
information_schema
system

3. Deploy Vector Agent (DaemonSet)

Vector will be deployed in two main components: an Agent and an Aggregator. The Vector Agent runs as a DaemonSet on each Kubernetes node. Its primary responsibility is to collect logs directly from the node’s filesystem (e.g., `/var/log/pods`) and forward them to the Vector Aggregator. This distributed collection model ensures that logs are captured close to their source, minimizing network latency and providing resilience.

We’ll use Helm to deploy the Vector Agent. The `vector-agent-values.yaml` file will configure the agent to tail container logs and send them to the aggregator. Make sure to create a dedicated namespace for Vector for better organization, adhering to good Kubernetes Network Policies practices.

First, create the `vector` namespace and add the Vector Helm repository.

kubectl create namespace vector
helm repo add vector https://helm.vector.dev
helm repo update

Now, create the `vector-agent-values.yaml` file:

# vector-agent-values.yaml
# Ensure the Vector agent has permissions to read /var/log/pods files
rbac:
  create: true

daemonset:
  enabled: true
  resources:
    limits:
      cpu: 200m
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi
  # Mount /var/log/pods to collect container logs
  extraVolumes:
    - name: var-log-pods
      hostPath:
        path: /var/log/pods
  extraVolumeMounts:
    - name: var-log-pods
      mountPath: /var/log/pods
      readOnly: true
  # Mount /var/lib/docker/containers for CRI-O or containerd, adjust if using Docker
    - name: var-lib-containers
      hostPath:
        path: /var/lib/docker/containers
  extraVolumeMounts:
    - name: var-lib-containers
      mountPath: /var/lib/docker/containers
      readOnly: true

# Vector configuration for the agent
# This configuration reads logs from Kubernetes, enriches them,
# and forwards them to the Vector aggregator service.
config:
  data_dir: /vector-data-dir
  sources:
    kubernetes_logs:
      type: "kubernetes_logs"
      # Specify the path where Kubernetes logs are stored
      # Adjust based on your container runtime (Docker, containerd, CRI-O)
      glob:
        - "/var/log/pods/*/*/*.log"
        - "/var/lib/docker/containers/*/*.log" # For Docker runtime
        - "/var/log/containers/*.log" # For containerd or CRI-O
      # Kubernetes metadata enrichment
      extra_field_selectors:
        - "metadata.labels"
        - "metadata.annotations"
  sinks:
    aggregator:
      type: "vector"
      inputs:
        - "kubernetes_logs"
      address: "vector-aggregator.vector.svc.cluster.local:8000" # Target the aggregator service
      encoding:
        codec: "json"

Deploy the Vector Agent using Helm:

helm install vector-agent vector/vector \
  --namespace vector \
  --create-namespace \
  -f vector-agent-values.yaml

Verify Step 3

Check if the Vector Agent DaemonSet is running and if pods are deployed on all your nodes.

kubectl get daemonset -n vector
kubectl get pods -n vector -l app.kubernetes.io/name=vector,app.kubernetes.io/component=agent

Expected Output (similar to):

NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vector-agent   3         3         3       3            3                     2m

NAME                 READY   STATUS    RESTARTS   AGE
vector-agent-abcde     1/1     Running   0          2m
vector-agent-fghij     1/1     Running   0          2m
vector-agent-klmno     1/1     Running   0          2m

4. Deploy Vector Aggregator (Deployment)

The Vector Aggregator acts as a central processing unit for logs collected by the agents. It runs as a Deployment and receives logs from all agents, applies further transformations (if needed), and then routes them to the final destination, which in our case is ClickHouse. This separation of concerns allows for efficient scaling of log processing and persistent storage.

The `vector-aggregator-values.yaml` will configure the aggregator to listen for incoming Vector connections, transform the data into a ClickHouse-compatible format, and then write it to our ClickHouse cluster. The `clickhouse` sink in Vector is highly optimized for performance and reliability.

Create the `vector-aggregator-values.yaml` file:

# vector-aggregator-values.yaml
# Enable the deployment for the aggregator
deployment:
  enabled: true
  replicas: 2 # You can scale this based on your log volume
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 250m
      memory: 512Mi

# Expose the aggregator via a service for agents to connect
service:
  enabled: true
  type: ClusterIP
  ports:
    - name: vector
      port: 8000
      targetPort: 8000

# Vector configuration for the aggregator
config:
  data_dir: /vector-data-dir
  sources:
    # Source to receive data from Vector agents
    vector_agent_sink:
      type: "vector"
      address: "0.0.0.0:8000"
  transforms:
    # Transform logs for ClickHouse compatibility
    # Flatten JSON, add timestamps, etc.
    process_logs:
      type: "remap"
      inputs:
        - "vector_agent_sink"
      source: |
        .timestamp = now()
        .message = .message ?? "" # Ensure message field exists
        # Flatten Kubernetes metadata for easier querying in ClickHouse
        .kubernetes.pod_name = .kubernetes.pod_name ?? "unknown"
        .kubernetes.namespace = .kubernetes.namespace ?? "unknown"
        .kubernetes.container_name = .kubernetes.container_name ?? "unknown"
        .kubernetes.host = .kubernetes.host ?? "unknown"
        # Add a _raw field for the original log line if needed
        ._raw = to_string(.)
  sinks:
    clickhouse_sink:
      type: "clickhouse"
      inputs:
        - "process_logs" # Use the transformed logs
      endpoint: "clickhouse-01-0-0.clickhouse-01.default.svc.cluster.local:8123" # ClickHouse service endpoint
      database: "kubernetes_logs"
      table: "logs"
      user: "vector"
      password: "vector_password" # Use the password created earlier
      auth_method: "plaintext"
      compression: "lz4"
      healthcheck: true
      skip_unknown_fields: true # Ignore fields not in ClickHouse schema
      # Define the ClickHouse table schema for logs
      schema:
        - name: "timestamp"
          type: "DateTime64(9)"
          # default: "now()" # Vector can set this
        - name: "message"
          type: "String"
        - name: "level"
          type: "Nullable(String)"
        - name: "kubernetes_pod_name"
          type: "String"
        - name: "kubernetes_namespace"
          type: "String"
        - name: "kubernetes_container_name"
          type: "String"
        - name: "kubernetes_host"
          type: "String"
        # Add other fields you want to store from Kubernetes metadata or log content
        - name: "json_payload" # For unstructured JSON logs
          type: "Nullable(String)"
          # Use a remap transform to convert .message to json_payload if it's JSON

      # ClickHouse DDL to create the table if it doesn't exist
      # This is crucial for initial setup
      create_table_ddl: |
        CREATE TABLE IF NOT EXISTS kubernetes_logs.logs
        (
            `timestamp` DateTime64(9),
            `message` String,
            `level` Nullable(String),
            `kubernetes_pod_name` String,
            `kubernetes_namespace` String,
            `kubernetes_container_name` String,
            `kubernetes_host` String,
            `json_payload` Nullable(String)
        )
        ENGINE = MergeTree()
        ORDER BY (kubernetes_namespace, kubernetes_pod_name, timestamp)
        PARTITION BY toYYYYMM(timestamp);

Deploy the Vector Aggregator using Helm:

helm install vector-aggregator vector/vector \
  --namespace vector \
  --create-namespace \
  -f vector-aggregator-values.yaml

Verify Step 4

Check if the Vector Aggregator Deployment and Service are running correctly. Then, the ultimate verification is to check if logs are actually flowing into ClickHouse.

kubectl get deployment -n vector -l app.kubernetes.io/name=vector,app.kubernetes.io/component=aggregator
kubectl get service -n vector -l app.kubernetes.io/name=vector,app.kubernetes.io/component=aggregator

Expected Output (similar to):

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
vector-aggregator       2/2     2            2           1m

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
vector-aggregator      ClusterIP   10.96.123.45           8000/TCP   1m

Now, let’s query ClickHouse to see if logs are being ingested. It might take a minute or two for the first logs to appear.

# Get the name of a running ClickHouse pod
CH_POD=$(kubectl get pods -l "clickhouse.altinity.com/chi=chi-01-simple-layout" -o jsonpath="{.items[0].metadata.name}")

# Query ClickHouse for logs
kubectl exec -it $CH_POD -- clickhouse client -u vector --password 'vector_password' -mn -q "SELECT count() FROM kubernetes_logs.logs;"
kubectl exec -it $CH_POD -- clickhouse client -u vector --password 'vector_password' -mn -q "SELECT timestamp, kubernetes_pod_name, message FROM kubernetes_logs.logs LIMIT 10;"

Expected Output (similar to):

# For count()
12345

# For SELECT query
2023-10-27 10:30:01.123456789   nginx-deployment-abcde   10.244.0.1 - - [27/Oct/2023:10:30:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0..."
2023-10-27 10:30:02.987654321   vector-agent-fghij     {"level":"info","message":"Successfully sent 10 events."}
...

If you see a count greater than 0 and some log entries, congratulations! Your Kubernetes logging stack with Vector and ClickHouse is operational.

Production Considerations

Deploying a logging stack in production requires careful planning and optimization. Here are key considerations:

• Resource Allocation: ClickHouse can be very resource-intensive for high-volume logging. Monitor CPU, memory, and disk I/O meticulously. Adjust replica counts for both ClickHouse and Vector Aggregators based on your log ingest rate. For cost optimization, consider tools like Karpenter to dynamically provision nodes that meet your ClickHouse resource demands.
• ClickHouse Persistence: Ensure your ClickHouse deployment uses Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) for data storage. Losing log data is generally unacceptable in production. Consider using cloud-provider specific storage classes for better performance and reliability.
• High Availability and Replication: For production, a single ClickHouse instance is a single point of failure. The Altinity Operator supports complex sharded and replicated ClickHouse clusters. Implement these for data redundancy and query resilience.
• Security:
  • Network Policies: Implement Kubernetes Network Policies to restrict traffic between Vector components, ClickHouse, and other applications. Only allow necessary ports and protocols.
  • Secrets Management: Do not hardcode sensitive information like ClickHouse passwords in plain text. Use Kubernetes Secrets and integrate with a secret management solution like HashiCorp Vault or cloud provider secret services.
  • RBAC: Review and tighten RBAC permissions for Vector and ClickHouse components, adhering to the principle of least privilege.
  • Encryption: Consider enabling TLS encryption for communication between Vector agents and aggregators, and between ClickHouse nodes. For pod-to-pod encryption, solutions like Cilium WireGuard Encryption can provide an additional layer of security.
• Monitoring and Alerting: Integrate ClickHouse and Vector metrics into your existing monitoring system (Prometheus, Grafana). Monitor log ingest rates, error rates, disk usage, and query performance. Set up alerts for anomalies. For advanced observability, explore eBPF Observability with Hubble to gain deeper insights into network and application performance.
• Log Retention Policy: Define and implement a clear log retention policy in ClickHouse. Large datasets can become expensive and impact performance. Use ClickHouse’s Time-to-Live (TTL) features to automatically drop old data or move it to colder storage.
• Schema Evolution: As your applications evolve, so will your log formats. Plan for schema changes in ClickHouse. Vector’s `skip_unknown_fields` helps, but for structured log fields, you’ll need to update your ClickHouse table schema.
• Centralized Access: Integrate a visualization tool like Grafana with ClickHouse to provide dashboards and a user-friendly interface for querying and analyzing logs.
• Cost Management: Beyond Karpenter, regularly review your storage costs and ClickHouse resource usage. Optimize queries and data compression to reduce expenses.

Troubleshooting

Here are common issues you might encounter and their solutions:

1. Issue: ClickHouse pods are stuck in Pending or CrashLoopBackOff.

   Solution:

   • Pending: Check `kubectl describe pod ` for events related to insufficient resources (CPU, memory, storage) or node affinity issues. Ensure you have enough nodes and resources.
   • CrashLoopBackOff: Check pod logs with `kubectl logs `. Common causes include incorrect configuration, storage issues, or OOMKills. Verify ClickHouse configuration files and ensure PVs are correctly provisioned and mounted.
2. Issue: Vector Agent pods are not starting or reporting errors.

   Solution:

   • Logs: Use `kubectl logs -n vector ` to check for configuration errors or permission issues.
   • Permissions: Ensure the `extraVolumeMounts` for `/var/log/pods` and `/var/lib/docker/containers` are correct and that the host paths exist on your nodes and are readable by the Vector agent. Incorrect paths or permissions are a frequent cause of agents failing to collect logs.
   • Resource Limits: Check if the agent is being OOMKilled by increasing memory limits in `vector-agent-values.yaml`.
3. Issue: Logs are not appearing in ClickHouse, but Vector Aggregator logs show “successfully sent events.”

   Solution:

   • ClickHouse Connectivity: Verify the `endpoint` in `vector-aggregator-values.yaml` points to the correct ClickHouse service and port. Test connectivity from the aggregator pod using `telnet` or `nc`.
   • ClickHouse User/Password: Double-check the `user` and `password` in the Vector aggregator sink configuration.
   • ClickHouse Table Schema: Ensure the ClickHouse table `kubernetes_logs.logs` exists and its schema matches the fields Vector is trying to write. Mismatched types or missing columns will cause write failures. Check ClickHouse server logs for insertion errors.
   • Vector Transform Errors: If your `remap` transform has errors, logs might be dropped before reaching the sink. Check aggregator logs for `Vector Remap Language` errors.
4. Issue: ClickHouse queries are slow or ingest is lagging.

   Solution:

   • Resource Scaling: Scale up ClickHouse resources (CPU, memory, faster storage) or add more replicas/shards.
   • Vector Aggregator Replicas: Increase the `replicas` for the Vector Aggregator deployment to handle higher log volumes.
   • ClickHouse Configuration: Tune ClickHouse server parameters, such as `max_memory_usage` and `merge_tree_settings`.
   • Schema Optimization: Review your ClickHouse table schema. Ensure `ORDER BY` and `PARTITION BY` clauses are optimized for your most common queries. Avoid excessive string columns if possible.
5. Issue: Vector Aggregator logs show “connection refused” to ClickHouse.

   Solution:

   • ClickHouse Service: Verify the ClickHouse service name and port. The default service name for the simple layout is `clickhouse-01-0-0.clickhouse-01.default.svc.cluster.local`.
   • Firewall/Network Policies: Check if any Kubernetes Network Policies are blocking communication between the Vector namespace and the ClickHouse namespace/pods.
   • ClickHouse Status: Ensure the ClickHouse pods are `Running` and the ClickHouse server process is active within the pods.
6. Issue: Not all Kubernetes logs are being collected by Vector Agents.

   Solution:

   • `glob` Path: Verify the `glob` paths in `vector-agent-values.yaml` match your container runtime’s log locations. Different runtimes (Docker, containerd, CRI-O) store logs in different directories.
   • Permissions: Ensure the Vector agent has read permissions to all relevant log directories.
   • Vector Agent Health: Check logs of the Vector agent for any errors indicating it’s unable to process specific log files.

FAQ Section

1. Q: Why choose Vector over Fluentd/Fluent Bit?

   A: Vector is often preferred for its performance, memory efficiency, and unified approach to logs, metrics, and traces. It’s written in Rust, offering superior resource utilization compared to Ruby-based Fluentd. Its single binary design and rich set of transforms make it a powerful and flexible choice for modern observability pipelines. For more on data pipelines, see the Vector documentation.

2. Q: Why ClickHouse instead of Elasticsearch or Loki?

   A: ClickHouse excels at analytical queries over massive datasets, making it ideal for high-volume, structured log analysis where speed is paramount. While Elasticsearch is excellent for full-text search and Loki is optimized for cost-effective log storage based on labels, ClickHouse provides superior performance for aggregations, filtering, and complex analytics on structured log fields, often with lower resource requirements for the same query performance. It’s a great choice when you need to run SQL-like queries on your logs at scale.

3. Q: How can I visualize logs stored in ClickHouse?

   A: Grafana is an excellent choice for visualizing ClickHouse data. You can configure ClickHouse as a data source in Grafana and build dashboards to query and display your logs, metrics, and application performance. Many users also integrate with tools like Apache Superset for advanced BI capabilities.

4. Q: How do I handle multi-cluster logging with this setup?

   A: For multi-cluster logging, you would deploy Vector Agents and Aggregators in each Kubernetes cluster. The aggregators in each cluster would then forward their logs to a central ClickHouse cluster, which could be deployed in a dedicated “logging” cluster or outside of Kubernetes altogether (e.g., on VMs or a managed service). Ensure proper network connectivity and security between clusters, potentially leveraging a service mesh like Istio Ambient Mesh for secure communication.

5. Q: What if I have sensitive data in my logs? How can I redact or mask it?

   A: Vector’s powerful `remap` transform can be used to redact or mask sensitive information before logs are sent to ClickHouse. You can define regular expressions or custom logic within the `remap` transform to identify and replace sensitive patterns (e.g., credit card numbers, PII, API keys) with placeholders. This is a critical step for data privacy and compliance. Refer to the Vector Remap Language (VRL) documentation for advanced transformation examples.

Cleanup Commands

To remove all components deployed in this guide, execute the following commands:

# Uninstall Vector Aggregator
helm uninstall vector-aggregator -n vector

# Uninstall Vector Agent
helm uninstall vector-agent -n vector

# Delete the Vector namespace
kubectl delete namespace vector

# Delete the ClickHouse cluster
kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/ch-cluster/chi-01-simple-layout.yaml

# Delete the ClickHouse operator
kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle-metrics.yaml

# Delete the ClickHouse operator namespace
kubectl delete namespace clickhouse-operator

# Remove Helm repositories
helm repo remove vector

Next Steps / Further Reading

Congratulations on setting up your advanced Kubernetes logging solution! Here are some next steps and resources to deepen your knowledge:

• Grafana Integration: Learn how to connect Grafana to ClickHouse and build powerful dashboards for log visualization. The Grafana ClickHouse documentation is a great starting point.
• Advanced Vector Transforms: Explore more of Vector’s capabilities, including parsing different log formats, enriching logs with external data, and routing based on content. The Vector Transforms documentation is extensive.
• ClickHouse Optimization: Dive deeper into ClickHouse performance tuning, sharding, and replication for production-grade