Kubernetes ConfigMaps & Secrets: Best Practices

Managing configuration and sensitive data in distributed systems like Kubernetes can quickly become a tangled mess. Hardcoding values into container images or injecting them directly into Pod definitions is a recipe for disaster, leading to security vulnerabilities, deployment inconsistencies, and operational headaches. Imagine needing to update a database password across dozens of microservices, each requiring a new image build and redeployment – the thought alone is enough to send shivers down a developer’s spine.

Kubernetes offers elegant solutions to these challenges through ConfigMaps and Secrets. These resources provide a clean, decoupled way to inject configuration data and sensitive information into your applications at runtime, without modifying your application code or container images. By externalizing configuration, you enhance security, promote reusability, and streamline your CI/CD pipelines. This guide will dive deep into best practices for using ConfigMaps and Secrets, ensuring your Kubernetes applications are robust, secure, and easily manageable.

Whether you’re dealing with environment variables, configuration files, or crucial API keys, understanding how to effectively leverage ConfigMaps and Secrets is fundamental to building production-ready Kubernetes applications. We’ll explore various injection methods, discuss security considerations, and provide practical examples to help you master these essential Kubernetes primitives. Let’s transform your configuration management from a pain point into a powerful asset.

Prerequisites

To follow along with this guide, you’ll need:

• A working Kubernetes cluster (e.g., Minikube, Kind, or a cloud-managed cluster like GKE, EKS, AKS). For local development, Minikube is an excellent choice.
• kubectl command-line tool installed and configured to communicate with your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Basic understanding of Kubernetes concepts like Pods, Deployments, and Services.
• A text editor of your choice (e.g., VS Code, Sublime Text).

Step-by-Step Guide: ConfigMaps and Secrets Best Practices

1. Understanding ConfigMaps for Non-Sensitive Data

ConfigMaps are API objects used to store non-sensitive configuration data as key-value pairs. They allow you to decouple configuration from your application code, making your applications more portable and easier to manage. You can use ConfigMaps to store configuration files, command-line arguments, or other environment-specific settings. This separation is crucial for maintaining a clean and flexible deployment pipeline, as it means you don’t have to rebuild your container images just to change a configuration parameter.

ConfigMaps are typically used for data that isn’t sensitive, such as log levels, database connection strings (without credentials), API endpoints, or feature flags. They can be consumed by Pods in various ways: as environment variables, as command-line arguments, or as files mounted into a volume. Each method has its use cases, but mounting as files is often preferred for complex configurations or when applications expect configuration in specific file formats.

Creating a ConfigMap from a file

Let’s start by creating a ConfigMap from a simple configuration file. This is a common scenario where you have an existing application configuration file (e.g., application.properties for a Java app or nginx.conf for NGINX) that you want to inject into your Pods.

# Create a sample configuration file
echo "app.name=MyWebApp
app.version=1.0.0
database.host=mydb.example.com
log.level=INFO" > application.properties

# Create the ConfigMap from the file
kubectl create configmap my-app-config --from-file=application.properties

Verify: Check if the ConfigMap was created successfully and inspect its contents.

kubectl get configmap my-app-config -o yaml

apiVersion: v1
data:
  application.properties: |-
    app.name=MyWebApp
    app.version=1.0.0
    database.host=mydb.example.com
    log.level=INFO
kind: ConfigMap
metadata:
  creationTimestamp: "2023-10-27T10:00:00Z"
  name: my-app-config
  namespace: default
  resourceVersion: "12345"
  uid: a1b2c3d4-e5f6-7890-1234-567890abcdef

Consuming a ConfigMap as an Environment Variable

Environment variables are a straightforward way to inject individual configuration values into your application. This is ideal for simple key-value pairs that your application expects as environment variables. However, be mindful that environment variables are inherited by child processes and can be easily inspected, so avoid sensitive data here.

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod-env
spec:
  containers:
  - name: my-app-container
    image: busybox
    command: ["sh", "-c", "echo App Name: $APP_NAME && echo Log Level: $LOG_LEVEL && sleep 3600"]
    env:
    - name: APP_NAME
      valueFrom:
        configMapKeyRef:
          name: my-app-config
          key: app.name
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef:
          name: my-app-config
          key: log.level
  restartPolicy: Never

kubectl apply -f my-app-pod-env.yaml

Verify: Check the Pod logs to see if the environment variables were correctly injected.

kubectl logs my-app-pod-env

App Name: MyWebApp
Log Level: INFO

Consuming a ConfigMap as a Mounted Volume

Mounting ConfigMaps as files in a volume is generally the preferred method, especially for complex configurations or when your application expects a configuration file at a specific path. This method allows you to inject entire configuration files or multiple key-value pairs as individual files into a directory within the Pod. This is particularly useful for applications that read configuration from files at startup.

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod-volume
spec:
  containers:
  - name: my-app-container
    image: busybox
    command: ["sh", "-c", "echo '--- Config File Content ---' && cat /etc/config/application.properties && sleep 3600"]
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
  volumes:
  - name: config-volume
    configMap:
      name: my-app-config
  restartPolicy: Never

kubectl apply -f my-app-pod-volume.yaml

Verify: Check the Pod logs to confirm the configuration file content is accessible.

kubectl logs my-app-pod-volume

--- Config File Content ---
app.name=MyWebApp
app.version=1.0.0
database.host=mydb.example.com
log.level=INFO

2. Understanding Secrets for Sensitive Data

Secrets are similar to ConfigMaps but are specifically designed for storing sensitive information, such as passwords, OAuth tokens, and SSH keys. Kubernetes Secrets are base64 encoded, which provides a basic level of obfuscation but does not encrypt the data at rest by default. It’s crucial to understand that base64 encoding is not encryption; it’s merely a way to represent binary data in an ASCII string format. Anyone with access to the Secret can easily decode its contents.

Therefore, additional measures are required to truly secure Secrets, especially when stored on disk or transferred over networks. We’ll delve into these security considerations later. Like ConfigMaps, Secrets can be consumed as environment variables or mounted as files. For sensitive data, mounting Secrets as files is the recommended best practice, as it limits the exposure of the secret to the application’s filesystem and avoids common pitfalls associated with environment variables.

Creating a Secret from literal values

Creating a Secret from literal values is straightforward. Remember, these values will be base64 encoded by Kubernetes. You should never store plain text sensitive data directly in your Git repository. Always create Secrets dynamically or use external secret management systems.

kubectl create secret generic my-database-secret \
  --from-literal=username=dbuser \
  --from-literal=password=supersecurepassword123

Verify: Inspect the created Secret. Notice the data field contains base64 encoded values.

kubectl get secret my-database-secret -o yaml

apiVersion: v1
data:
  password: c3VwZXJzZWN1cmVwYXNzd29yZDEyMw==
  username: ZGJ1c2Vy
kind: Secret
metadata:
  creationTimestamp: "2023-10-27T10:05:00Z"
  name: my-database-secret
  namespace: default
  resourceVersion: "67890"
  uid: f1e2d3c4-b5a6-9876-5432-10fedcba9876
type: Opaque

To decode the password for verification (do not do this in production environments without extreme caution):

kubectl get secret my-database-secret -o jsonpath='{.data.password}' | base64 --decode

supersecurepassword123

Consuming a Secret as an Environment Variable (Discouraged)

While possible, exposing Secrets as environment variables is generally discouraged. Environment variables can be easily leaked through logs, crash dumps, or by unauthorized processes on the same node. Moreover, they are inherited by child processes, increasing their attack surface. If you absolutely must use them, ensure your application handles them with extreme care and minimizes their lifetime.

apiVersion: v1
kind: Pod
metadata:
  name: my-db-pod-env
spec:
  containers:
  - name: my-db-container
    image: busybox
    command: ["sh", "-c", "echo DB User: $DB_USER && echo DB Pass: $DB_PASSWORD && sleep 3600"]
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: my-database-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-database-secret
          key: password
  restartPolicy: Never

kubectl apply -f my-db-pod-env.yaml

Verify: Check Pod logs. Note that the password is now visible in the logs, which is undesirable for production.

kubectl logs my-db-pod-env

DB User: dbuser
DB Pass: supersecurepassword123

Consuming a Secret as a Mounted Volume (Recommended)

Mounting Secrets as files in a volume is the recommended and most secure way to consume them. When a Secret is mounted as a volume, each key in the Secret becomes a file in the specified mount path, with the key’s value as the file’s content. These files are typically mounted into an in-memory tmpfs filesystem, meaning they are not written to disk and are automatically removed when the Pod is deleted. This significantly reduces the risk of sensitive data persistence.

apiVersion: v1
kind: Pod
metadata:
  name: my-db-pod-volume
spec:
  containers:
  - name: my-db-container
    image: busybox
    command: ["sh", "-c", "echo '--- Secret Files ---' && cat /etc/secrets/username && cat /etc/secrets/password && sleep 3600"]
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true # Always mount secrets read-only
  volumes:
  - name: secret-volume
    secret:
      secretName: my-database-secret
  restartPolicy: Never

kubectl apply -f my-db-pod-volume.yaml

Verify: Check Pod logs to confirm the secrets are accessible as files.

kubectl logs my-db-pod-volume

--- Secret Files ---
dbuser
supersecurepassword123

3. Immutable ConfigMaps and Secrets

Once created, ConfigMaps and Secrets are generally mutable. This means their contents can be changed, and these changes are automatically propagated to running Pods (with a small delay). While convenient, this mutability can lead to unintended consequences, such as accidental modifications or difficulty in debugging due to changing configurations. For critical configurations that should not change after deployment, or to improve performance and reliability, Kubernetes introduced immutable ConfigMaps and Secrets.

By marking a ConfigMap or Secret as immutable, you prevent accidental or malicious updates to its data. If you need to change an immutable resource, you must create a new one with a different name and update your Pods to refer to the new resource. This approach aligns with GitOps principles, promoting a more predictable and auditable configuration management workflow. It also reduces the load on the Kubernetes API server, as the controller manager doesn’t need to watch for changes to these objects.

Creating an Immutable ConfigMap

To make a ConfigMap immutable, simply add immutable: true to its metadata section.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-immutable-config
  immutable: true # Mark as immutable
data:
  feature.enabled: "true"
  api.endpoint: "https://api.example.com/v1"

kubectl apply -f my-immutable-config.yaml

Verify: Attempt to update the immutable ConfigMap. It should fail.

# Attempt to update the immutable ConfigMap (this will fail)
kubectl patch configmap my-immutable-config -p '{"data":{"feature.enabled":"false"}}'

The ConfigMap "my-immutable-config" is immutable and cannot be updated.

Creating an Immutable Secret

Similarly, for Secrets, add immutable: true to its metadata. This prevents changes to sensitive data, enhancing security and stability.

apiVersion: v1
kind: Secret
metadata:
  name: my-immutable-secret
  immutable: true # Mark as immutable
type: Opaque
data:
  api_token: dGhpcyBpcyBhbiBpbW11dGFibGUgdG9rZW4= # base64 encoded "this is an immutable token"

kubectl apply -f my-immutable-secret.yaml

Verify: Attempt to update the immutable Secret. It should fail.

# Attempt to update the immutable Secret (this will fail)
kubectl patch secret my-immutable-secret -p '{"data":{"api_token":"bmV3IGltbXV0YWJsZSB0b2tlbg=="}}'

The Secret "my-immutable-secret" is immutable and cannot be updated.

4. Best Practices for Secret Management

Securing Secrets in Kubernetes goes beyond just using the Secret object. It involves a holistic approach to their lifecycle, from creation to consumption and eventual rotation. Here are key best practices:

a. Encryption at Rest

As mentioned, Kubernetes Secrets are base64 encoded, not encrypted, by default. To protect Secrets at rest, you must enable encryption at rest for your cluster’s etcd database. Most managed Kubernetes services (GKE, EKS, AKS) offer this as a feature, often integrating with their respective Key Management Systems (KMS). For self-managed clusters, you’ll need to configure KMS encryption providers.

Example (conceptual, configuration depends on cluster setup):

# This is a conceptual example for a KMS configuration in a self-managed cluster
# The actual configuration varies by cloud provider and KMS solution.
# For example, in AWS EKS, you would enable KMS encryption directly on the cluster.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          name: mykmskey
          endpoint: unix:///var/run/kmsplugin/grpc.sock
          timeout: 3s
      - aescbc:
          keys:
            - name: key1
              secret: 
      - identity: {}

Consult your cloud provider’s documentation for specific instructions on enabling etcd encryption:

• GKE Secrets Encryption
• EKS Secrets Encryption
• AKS Secrets CSI Driver Integration with KMS

b. External Secret Management Systems

For even greater security and to centralize secret management, integrate Kubernetes with external secret management systems like HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. These systems offer advanced features like secret rotation, fine-grained access control, auditing, and secret versioning. Tools like External Secrets Operator bridge the gap, allowing you to define Kubernetes Secrets that automatically synchronize with external providers.

Example: Using External Secrets Operator (conceptual)

First, install the External Secrets Operator. Then, you can define an ExternalSecret resource that references your secret in AWS Secrets Manager.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-external-db-secret
spec:
  refreshInterval: 1h # How often to re-fetch the secret
  secretStoreRef:
    name: aws-secrets-manager # Reference to a SecretStore configured for AWS
    kind: SecretStore
  target:
    name: my-db-secret-synced # The name of the Kubernetes Secret to create/update
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: my/database/credentials # Path to the secret in AWS Secrets Manager
      property: username
  - secretKey: password
    remoteRef:
      key: my/database/credentials
      property: password

This approach keeps sensitive data entirely out of your Kubernetes YAML files and Git repositories, making your deployments significantly more secure. For more advanced security topics, consider exploring Securing Container Supply Chains with Sigstore and Kyverno.

c. Restrict Access with RBAC

Limit who can read, create, or update Secrets using Kubernetes Role-Based Access Control (RBAC). Only service accounts and users that absolutely need access to Secrets should be granted the necessary permissions. This is a fundamental security principle – the principle of least privilege.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"] # Only allow reading secrets
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-secret-reader
subjects:
- kind: ServiceAccount
  name: my-app-service-account
  namespace: default
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f secret-rbac.yaml

d. Secret Rotation

Regularly rotate your secrets (passwords, API keys, certificates) to minimize the impact of a potential compromise. External secret management systems often automate this process. If managing manually, plan for a rotation strategy that involves updating the Secret and then gracefully rolling out new Pods that consume the updated Secret.

5. Production Considerations

a. Namespace Isolation

Always deploy ConfigMaps and Secrets within the same namespace as the Pods that consume them. This provides natural isolation and prevents accidental exposure across different applications or environments. Use Kubernetes Namespaces to logically separate your resources.

b. Use subPath for Specific Files

When mounting a ConfigMap or Secret as a volume, if you only need a specific file from it and want to avoid mounting the entire directory, use the subPath property in your volumeMount. This is useful for injecting a single configuration file into a specific path without creating a subdirectory. This also prevents other files in the ConfigMap/Secret from being overwritten if the mount path is already populated.

apiVersion: v1
kind: Pod
metadata:
  name: my-app-subpath-config
spec:
  containers:
  - name: my-app-container
    image: busybox
    command: ["sh", "-c", "cat /etc/app/config.properties && sleep 3600"]
    volumeMounts:
    - name: config-volume
      mountPath: /etc/app/config.properties # Mount as a specific file
      subPath: application.properties      # Reference the key in the ConfigMap
  volumes:
  - name: config-volume
    configMap:
      name: my-app-config
  restartPolicy: Never

kubectl apply -f my-app-subpath-config.yaml

Verify:

kubectl logs my-app-subpath-config

app.name=MyWebApp
app.version=1.0.0
database.host=mydb.example.com
log.level=INFO

c. Update Strategy for ConfigMaps/Secrets

When you update a ConfigMap or Secret that is mounted as a volume, Kubernetes automatically propagates these changes to the running Pods. However, this propagation is not instantaneous and can take some time (typically up to 60 seconds by default, configurable via kubelet‘s sync-frequency). For environment variables, Pods need to be restarted to pick up changes.

For critical configurations, consider using a rollout strategy that creates a new version of the ConfigMap/Secret and updates the Deployment to reference the new version. This ensures a controlled rollout and allows for easy rollback. Immutable ConfigMaps/Secrets naturally enforce this pattern.

d. Network Policies for Secret Access

While ConfigMaps and Secrets are Kubernetes internal resources, the applications consuming them still communicate over the network. Ensure your cluster’s Network Policies are properly configured to restrict unnecessary network access to and from Pods that handle sensitive data. This adds another layer of defense in depth.

e. Observability and Auditing

Monitor access to ConfigMaps and Secrets using Kubernetes audit logs. Tools like eBPF Observability with Hubble or other custom monitoring solutions can help detect suspicious activity. Ensure proper logging is in place for applications that consume secrets to track their usage, but be careful not to log the secret values themselves.

Troubleshooting

1. Issue: ConfigMap/Secret not found when Pod starts.

   Explanation: The Pod’s definition refers to a ConfigMap or Secret that does not exist in the same namespace, or there’s a typo in the name.

   Solution: Double-check the ConfigMap/Secret name and namespace in your Pod/Deployment YAML. Ensure the resource exists using kubectl get configmap <name> or kubectl get secret <name>.

   
   kubectl get configmap my-app-config -n default
   kubectl get secret my-database-secret -n default
           

2. Issue: ConfigMap/Secret data not appearing in Pod (environment variables).

   Explanation: This often happens if the key referenced in configMapKeyRef or secretKeyRef does not exist within the ConfigMap/Secret, or if the Pod was already running and hasn’t been restarted after an update to the ConfigMap/Secret.

   Solution: Verify the key name using kubectl get configmap <name> -o yaml or kubectl get secret <name> -o yaml. For environment variables, Pods must be restarted to pick up changes. For Deployments, trigger a rollout restart: kubectl rollout restart deployment <deployment-name>.

3. Issue: ConfigMap/Secret data not appearing in Pod (mounted volume).

   Explanation: The volume mount path might be incorrect, or the ConfigMap/Secret data hasn’t propagated yet. If subPath is used, the key might be misspelled.

   Solution: Check the mountPath and subPath in your Pod YAML. ConfigMap/Secret updates to mounted files usually propagate within 60 seconds. If not, check kubelet logs on the node for errors. Ensure the volume mount is not read-only if you expect to write to it (though for ConfigMaps/Secrets, read-only is usually desired).

4. Issue: Permissions error when accessing mounted Secret files.

   Explanation: By default, files mounted from Secrets have permissions rw-r--r-- (0644) and are owned by root. Your application might be running as a non-root user and lack read permissions.

   Solution: You can specify the file permissions for mounted Secret items using defaultMode in the volume definition. For example, defaultMode: 0400 would make them read-only for the owner.

   
     volumes:
     - name: secret-volume
       secret:
         secretName: my-database-secret
         defaultMode: 0400 # Read-only for owner (root by default), no access for others
           

   Alternatively, your application should run as a user with appropriate permissions, or you can use an initContainer to change permissions before the main container starts.

5. Issue: Sensitive data visible in Pod logs or descriptions.

   Explanation: This typically happens when Secrets are exposed as environment variables, and the application prints these variables to standard output/error, which then gets captured by logging systems. Or, someone might be inspecting the Pod definition directly.

   Solution: Always prefer mounting Secrets as files. Configure your application to read secrets from files and ensure it does not log sensitive information. Use RBAC to restrict who can get or describe Pods, and enable etcd encryption for Secrets at rest.

6. Issue: Cannot update an immutable ConfigMap or Secret.

   Explanation: You explicitly marked the resource as immutable: true, preventing any modifications.

   Solution: This is expected behavior. To update an immutable ConfigMap or Secret, you must create a new one with a different name (e.g., my-config-v2) and then update your Deployment or Pods to reference the new resource. This forces a controlled rollout of the new configuration.

FAQ Section

1. What’s the difference between ConfigMaps and Secrets?

   ConfigMaps store non-sensitive configuration data (e.g., log levels, API endpoints) as key-value pairs. Secrets store sensitive data (e.g., passwords, API keys, certificates) and provide basic base64 encoding for obfuscation, but require additional encryption at rest for true security. The primary difference is the intended use case and the security measures applied.

2. Is base64 encoding in Secrets considered encryption?

   No, base64 encoding is not encryption. It’s a simple encoding scheme that converts binary data into an ASCII string format. Anyone can easily decode a base64-encoded string. For true encryption at rest, you must configure etcd encryption for your Kubernetes cluster, often integrating with a Key Management System (KMS).

3. When should I use environment variables vs. mounted files for ConfigMaps/Secrets?

   For ConfigMaps: Environment variables are fine for simple, single-value configurations. Mounted files are better for complex configurations, multiple values, or when applications expect specific file formats (e.g., .properties, .json). For Secrets: Always prefer mounted files. Environment variables are easily leaked and are less secure for sensitive data.

4. How do I ensure my Secrets are truly secure in Kubernetes?

   A multi-layered approach is best:

   • Enable etcd encryption at rest for your cluster.
   • Mount Secrets as files in Pod