Kubernetes ConfigMaps & Secrets: Best Practices

Introduction

Managing configuration and sensitive data within containerized applications can be a significant challenge. Hardcoding values into Docker images leads to inflexible, non-portable, and insecure deployments, making updates a nightmare and increasing the risk of data exposure. Imagine rebuilding and redeploying an entire application just to change a database connection string or an API key – it’s inefficient, error-prone, and goes against the principles of cloud-native development.

Kubernetes addresses these pain points with two fundamental resources: ConfigMaps and Secrets. These objects provide a robust and standardized way to decouple configuration data and sensitive information from your application code and container images. By externalizing these values, you gain immense flexibility, improve security posture, and streamline your CI/CD pipelines. This guide will walk you through the best practices for leveraging ConfigMaps and Secrets, ensuring your applications are not only configurable but also secure and maintainable in a Kubernetes environment.

# Create a ConfigMap from literal values
kubectl create configmap my-config --from-literal=config.key=config.value --from-literal=another.key=another.value

# Create a ConfigMap from a file
kubectl create configmap my-app-config --from-file=./config/app.properties

# Create a Secret from literal values (base64 encoded)
echo -n 'mysecretpassword' | base64
kubectl create secret generic my-secret --from-literal=db_password=mysecretpassword

# Create a Secret from a file
kubectl create secret generic my-api-key --from-file=./secrets/api_key.txt

# Mount a ConfigMap as environment variables in a Pod
# (See Step 4 for YAML example)

# Mount a Secret as a volume in a Pod
# (See Step 5 for YAML example)

# Get ConfigMap/Secret details
kubectl get configmap my-config -o yaml
kubectl get secret my-secret -o yaml

Prerequisites

• A working Kubernetes cluster (e.g., Minikube, kind, or a cloud-managed cluster like GKE, EKS, AKS).
• kubectl command-line tool installed and configured to communicate with your cluster. You can find installation instructions on the official Kubernetes documentation.
• Basic understanding of Kubernetes Pods, Deployments, and YAML syntax.
• Familiarity with command-line operations.

Step-by-Step Guide: Kubernetes ConfigMaps and Secrets Best Practices

Step 1: Understanding ConfigMaps – Non-Sensitive Configuration

ConfigMaps are API objects used to store non-sensitive configuration data in key-value pairs. They allow you to inject configuration into your Pods in various ways: as environment variables, command-line arguments, or as files in a volume. The primary benefit is the decoupling of configuration from your application’s container image, making your applications more portable and easier to manage. For instance, you might store database hostnames, logging levels, or feature flags in a ConfigMap.

It’s crucial to remember that ConfigMaps are not designed for sensitive data. While the data is stored within the Kubernetes cluster, it’s not encrypted by default at the API level (though it might be encrypted at rest by the underlying storage provider). Treat ConfigMaps as plain text storage for configuration that doesn’t pose a security risk if exposed.

Creating a ConfigMap

You can create ConfigMaps from literal values, from files, or from directories. Let’s start with creating one from literal key-value pairs.

# Create a ConfigMap from literal values
kubectl create configmap app-settings \
  --from-literal=log_level=INFO \
  --from-literal=feature_flag_a=true \
  --from-literal=api_endpoint=http://api.example.com/v1

# Verify the ConfigMap was created
kubectl get configmap app-settings -o yaml

Expected Output:

apiVersion: v1
data:
  api_endpoint: http://api.example.com/v1
  feature_flag_a: "true"
  log_level: INFO
kind: ConfigMap
metadata:
  creationTimestamp: "2023-10-27T10:00:00Z" # Timestamp will vary
  name: app-settings
  namespace: default
  resourceVersion: "12345" # Will vary
  uid: a1b2c3d4-e5f6-7890-1234-567890abcdef # Will vary

Next, let’s create a ConfigMap from a file. This is particularly useful when you have configuration files like .properties, .json, or .xml.

# Create a dummy config file
mkdir -p config
echo "database.host=mydb.production.svc.cluster.local" > config/application.properties
echo "server.port=8080" >> config/application.properties

# Create a ConfigMap from the file
kubectl create configmap app-properties-config --from-file=./config/application.properties

# Verify the ConfigMap
kubectl get configmap app-properties-config -o yaml

Expected Output:

apiVersion: v1
data:
  application.properties: |
    database.host=mydb.production.svc.cluster.local
    server.port=8080
kind: ConfigMap
metadata:
  creationTimestamp: "2023-10-27T10:05:00Z" # Timestamp will vary
  name: app-properties-config
  namespace: default
  resourceVersion: "67890" # Will vary
  uid: b2c3d4e5-f6a7-8901-2345-67890abcdef0 # Will vary

Step 2: Understanding Secrets – Sensitive Configuration

Secrets are similar to ConfigMaps but are specifically designed for storing sensitive information, such as passwords, OAuth tokens, and SSH keys. Kubernetes Secrets provide a more secure way to handle this data compared to ConfigMaps, though it’s important to understand their limitations. By default, Secrets are base64 encoded, not encrypted. This encoding is primarily to ensure data integrity and safe transmission, not to provide confidentiality. Anyone with API access can decode a Secret.

For true security, you should ensure that Secrets are encrypted at rest by your cluster’s underlying storage. Most cloud providers offer this by default for managed Kubernetes services. For self-managed clusters, you might need to configure encryption at rest for etcd. For advanced secret management and rotation, consider integrating with external secret management systems like HashiCorp Vault or cloud-specific services like AWS Secrets Manager or Azure Key Vault. For a deeper dive into securing your supply chain, including sensitive data, check out our guide on Securing Container Supply Chains with Sigstore and Kyverno.

Creating a Secret

Similar to ConfigMaps, Secrets can be created from literal values or files. The key difference is that the values are automatically base64 encoded. Let’s create a Secret for a database password.

# Generate a base64 encoded password
# IMPORTANT: Never store sensitive data in plain text in your version control.
# This is for demonstration. In production, use a secret management system.
DB_PASS="mySuperSecretDbPassword123"
DB_USER="dbuser"

# Create a Secret from literal values
kubectl create secret generic db-credentials \
  --from-literal=username=$DB_USER \
  --from-literal=password=$DB_PASS

# Verify the Secret was created
kubectl get secret db-credentials -o yaml

Expected Output:

apiVersion: v1
data:
  password: bXlTdXBlclNlY3JldERiUGFzc3dvcmQxMjM= # base64 encoded 'mySuperSecretDbPassword123'
  username: ZGJ1c2Vy # base64 encoded 'dbuser'
kind: Secret
metadata:
  creationTimestamp: "2023-10-27T10:10:00Z" # Timestamp will vary
  name: db-credentials
  namespace: default
  resourceVersion: "98765" # Will vary
  uid: c3d4e5f6-a7b8-9012-3456-7890abcdef12 # Will vary
type: Opaque # Default type for generic secrets

To decode the password, you would use:

kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 --decode

Step 3: Injecting ConfigMaps as Environment Variables

One of the most common ways to consume ConfigMap data is by injecting it as environment variables into your application containers. This is ideal for simple, single-value configurations that your application expects in its environment.

Deployment YAML for ConfigMap as Env Vars

Let’s create a simple Nginx deployment that uses our app-settings ConfigMap.

# configmap-env-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-configmap-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-configmap-env
  template:
    metadata:
      labels:
        app: my-app-configmap-env
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["sh", "-c", "echo 'Log Level: $LOG_LEVEL'; echo 'Feature Flag A: $FEATURE_FLAG_A'; echo 'API Endpoint: $API_ENDPOINT'; sleep 3600"]
        env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-settings
              key: log_level
        - name: FEATURE_FLAG_A
          valueFrom:
            configMapKeyRef:
              name: app-settings
              key: feature_flag_a
        - name: API_ENDPOINT
          valueFrom:
            configMapKeyRef:
              name: app-settings
              key: api_endpoint

# Apply the deployment
kubectl apply -f configmap-env-deployment.yaml

# Check the logs of the pod to see the environment variables
kubectl logs deployment/my-app-configmap-env

Expected Output (from logs):

Log Level: INFO
Feature Flag A: true
API Endpoint: http://api.example.com/v1

Alternatively, you can inject all key-value pairs from a ConfigMap as environment variables using envFrom. This is convenient but can pollute the environment if the ConfigMap contains many irrelevant keys.

# configmap-envfrom-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-configmap-envfrom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-configmap-envfrom
  template:
    metadata:
      labels:
        app: my-app-configmap-envfrom
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["sh", "-c", "echo 'Log Level: $LOG_LEVEL'; echo 'Feature Flag A: $FEATURE_FLAG_A'; echo 'API Endpoint: $API_ENDPOINT'; sleep 3600"]
        envFrom:
        - configMapRef:
            name: app-settings # All keys from app-settings will be env vars

kubectl apply -f configmap-envfrom-deployment.yaml
kubectl logs deployment/my-app-configmap-envfrom

Expected Output (from logs):

Log Level: INFO
Feature Flag A: true
API Endpoint: http://api.example.com/v1

Step 4: Injecting ConfigMaps as Files (Volumes)

For more complex configurations, especially those involving structured files (JSON, YAML, XML) or multiple configuration files, mounting ConfigMaps as files in a volume is the preferred method. This allows your application to read configuration from a file path, mimicking traditional application setups.

Deployment YAML for ConfigMap as Volume

Let’s use our app-properties-config ConfigMap, which contains application.properties.

# configmap-volume-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-configmap-volume
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-configmap-volume
  template:
    metadata:
      labels:
        app: my-app-configmap-volume
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["sh", "-c", "echo 'Reading application.properties:'; cat /etc/config/application.properties; sleep 3600"]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
      - name: config-volume
        configMap:
          name: app-properties-config

# Apply the deployment
kubectl apply -f configmap-volume-deployment.yaml

# Check the logs of the pod
kubectl logs deployment/my-app-configmap-volume

Expected Output (from logs):

Reading application.properties:
database.host=mydb.production.svc.cluster.local
server.port=8080

When mounted as a volume, each key in the ConfigMap becomes a file in the specified mountPath directory. The filename will be the key name, and the content will be the key’s value.

Step 5: Injecting Secrets as Environment Variables

Similar to ConfigMaps, Secrets can be injected as environment variables. This is suitable for single, sensitive values like API keys or database credentials that your application expects in its environment. However, be cautious: environment variables are easily accessible from within the container and can sometimes be exposed in logs or debugging tools. For critical secrets, mounting as a file is generally safer.

Deployment YAML for Secret as Env Vars

Let’s use our db-credentials Secret.

# secret-env-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-secret-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-secret-env
  template:
    metadata:
      labels:
        app: my-app-secret-env
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["sh", "-c", "echo 'DB User: $DB_USERNAME'; echo 'DB Pass: $DB_PASSWORD'; sleep 3600"]
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

# Apply the deployment
kubectl apply -f secret-env-deployment.yaml

# Check the logs of the pod
kubectl logs deployment/my-app-secret-env

Expected Output (from logs):

DB User: dbuser
DB Pass: mySuperSecretDbPassword123

Just like ConfigMaps, you can also inject all Secret keys as environment variables using envFrom. Again, use with caution due to potential environment pollution.

# secret-envfrom-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-secret-envfrom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-secret-envfrom
  template:
    metadata:
      labels:
        app: my-app-secret-envfrom
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["sh", "-c", "echo 'DB User: $USERNAME'; echo 'DB Pass: $PASSWORD'; sleep 3600"]
        envFrom:
        - secretRef:
            name: db-credentials

kubectl apply -f secret-envfrom-deployment.yaml
kubectl logs deployment/my-app-secret-envfrom

Expected Output (from logs):

DB User: dbuser
DB Pass: mySuperSecretDbPassword123

Step 6: Injecting Secrets as Files (Volumes)

Mounting Secrets as files within a volume is often considered the most secure way to handle sensitive data in Kubernetes, especially for API keys, certificates, or multi-line private keys. When mounted this way, the files are typically given restrictive permissions (e.g., 0644 by default, but can be customized to 0400 or 0600) and are stored in a tmpfs (in-memory filesystem) volume, meaning they are not written to disk.

Deployment YAML for Secret as Volume

Let’s create a new secret for a mock API key and then mount it.

# Create a dummy API key file
mkdir -p secrets
echo "sk_live_abcdef1234567890" > secrets/api_key.txt

# Create a Secret from the file
kubectl create secret generic api-key-secret --from-file=./secrets/api_key.txt

# Verify the Secret
kubectl get secret api-key-secret -o yaml

Expected Output:

apiVersion: v1
data:
  api_key.txt: c2tfbGl2ZV9hYmNkZWYxMjM0NTY3ODkwCg== # base64 encoded 'sk_live_abcdef1234567890'
kind: Secret
metadata:
  creationTimestamp: "2023-10-27T10:20:00Z" # Timestamp will vary
  name: api-key-secret
  namespace: default
  resourceVersion: "11223" # Will vary
  uid: d4e5f6a7-b8c9-0123-4567-890abcdef123 # Will vary
type: Opaque

Now, let’s create a deployment to mount this secret.

# secret-volume-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-secret-volume
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-secret-volume
  template:
    metadata:
      labels:
        app: my-app-secret-volume
    spec:
      containers:
      - name: my-app
        image: busybox
        command: ["sh", "-c", "echo 'Reading API Key from file:'; cat /etc/secrets/api_key.txt; ls -l /etc/secrets/; sleep 3600"]
        volumeMounts:
        - name: secret-volume
          mountPath: /etc/secrets
          readOnly: true # Best practice for secrets
      volumes:
      - name: secret-volume
        secret:
          secretName: api-key-secret
          defaultMode: 0400 # Restrict permissions to owner read-only

# Apply the deployment
kubectl apply -f secret-volume-deployment.yaml

# Check the logs of the pod
kubectl logs deployment/my-app-secret-volume

Expected Output (from logs):

Reading API Key from file:
sk_live_abcdef1234567890
total 4
-r-------- 1 root root 23 Oct 27 10:25 api_key.txt

Notice the defaultMode: 0400 which sets very restrictive permissions, allowing only the owner (root, by default in the container) to read the file. This is a critical security measure. For more on securing your network traffic, consider reading our guide on Cilium WireGuard Encryption.

Production Considerations

Deploying ConfigMaps and Secrets in production requires careful planning beyond basic injection:

1. Secret Encryption at Rest: While Kubernetes Secrets are base64 encoded, they are not truly encrypted by default within etcd. For production, ensure etcd encryption is enabled. Managed Kubernetes services usually handle this, but verify for self-managed clusters.
2. External Secret Management: For the highest level of security, compliance, and secret rotation capabilities, integrate Kubernetes with external secret managers like HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Tools like the External Secrets Operator can bridge the gap, synchronizing external secrets into native Kubernetes Secrets.
3. RBAC for Secrets: Restrict access to Secrets using Kubernetes Role-Based Access Control (RBAC). Only specific service accounts or users should have permissions to get, list, or watch Secrets. A pod’s service account should only have access to the specific secrets it needs. For robust security, refer to the Kubernetes RBAC documentation.
4. Secret Rotation: Implement a strategy for regular secret rotation. External secret managers often provide this out-of-the-box. For native Kubernetes Secrets, you’ll need a mechanism (e.g., a CI/CD pipeline or a custom controller) to update the Secret and trigger a rolling update of the dependent Deployments to pick up the new values.
5. Immutability: For ConfigMaps and Secrets that are not frequently updated, consider setting immutable: true in their metadata. This prevents accidental updates and can improve performance by allowing controllers to skip watching these objects for changes.
6. Sensitive Data in Logs: Be extremely careful about sensitive data appearing in application logs. If you inject secrets as environment variables, ensure your application doesn’t log its entire environment. When mounting as files, prevent files from being logged.
7. Use Pod Security Standards: Leverage Pod Security Standards or admission controllers like Kyverno to enforce policies, such as requiring Secrets to be mounted with restrictive file permissions (e.g., defaultMode: 0400). For more on policy enforcement, check out our guide on Sigstore and Kyverno Security.
8. Configuration Reloads: Applications don’t automatically pick up changes to ConfigMaps or Secrets. You typically need to perform a rolling update of the Deployment to restart Pods and load the new configuration. Tools like Reloader can automate this by watching ConfigMaps/Secrets and triggering Deployment updates.
9. Namespace Isolation: Store ConfigMaps and Secrets in the same namespace as the applications that consume them. This provides a natural boundary for access control and limits the blast radius of a compromise.

Troubleshooting

1. Issue: ConfigMap/Secret not found.

   Problem: Your Pod fails to start or your application crashes because it cannot find the ConfigMap or Secret.

   Solution:

   1. Verify the ConfigMap/Secret exists in the correct namespace:

      
      kubectl get configmap my-config -n my-namespace
      kubectl get secret my-secret -n my-namespace
                      

   2. Ensure the name in your Pod/Deployment YAML matches exactly:

      
      # ...
              env:
              - name: MY_VAR
                valueFrom:
                  configMapKeyRef:
                    name: correct-configmap-name # Check this name
                    key: my_key
      # ...
                      

2. Issue: Incorrect value loaded from ConfigMap/Secret.

   Problem: Your application is getting an unexpected or empty value for a configuration item.

   Solution:

   1. Inspect the ConfigMap/Secret content:

      
      kubectl get configmap my-config -o yaml
      kubectl get secret my-secret -o yaml # Data will be base64 encoded
                      

   2. If it’s a Secret, decode the relevant field to check its actual value:

      
      kubectl get secret my-secret -o jsonpath='{.data.my_key}' | base64 --decode
                      

   3. Verify the key in your Pod/Deployment YAML matches the key in the ConfigMap/Secret:

      
      # ...
              env:
              - name: MY_VAR
                valueFrom:
                  configMapKeyRef:
                    name: my-config
                    key: correct_key_name # Check this key
      # ...
                      

3. Issue: ConfigMap/Secret changes not reflected in running Pods.

   Problem: You updated a ConfigMap or Secret, but your application is still using old values.

   Solution: Kubernetes Pods do not automatically reload ConfigMaps or Secrets. You need to trigger a rolling update of the Deployment to restart the Pods and pick up the new values.

   
   # For a Deployment
   kubectl rollout restart deployment/my-app
   
   # Or manually update a label to force recreation
   kubectl patch deployment my-app -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"$(date +'%s')\"}}}}}"
               

   Consider using tools like Reloader to automate this process.

4. Issue: Permissions denied when mounting Secret as a volume.

   Problem: Your application can’t read a file mounted from a Secret volume.

   Solution:

   1. Check the defaultMode in your Secret volume mount:

      
      # ...
            volumes:
            - name: secret-volume
              secret:
                secretName: my-secret
                defaultMode: 0400 # This sets read-only for owner. If your app runs as non-root, it might not have access.
      # ...
                      

   2. If your application runs as a non-root user (which is a good security practice), you might need to adjust defaultMode (e.g., 0444 for read-only by all) or configure securityContext for the Pod or container to ensure the user has appropriate permissions.
   3. Verify the file path in your application code matches the mountPath and the key name.
5. Issue: Secrets exposed in logs or debugging output.

   Problem: Sensitive data is appearing where it shouldn’t.

   Solution:

   1. Audit application logging: Ensure your application is not logging environment variables or contents of configuration files that might contain secrets.
   2. Avoid envFrom for secrets: If possible, use valueFrom with secretKeyRef for specific keys, rather than injecting all keys from a Secret with envFrom, to minimize potential exposure.
   3. Prefer volume mounts for critical secrets: Mounting secrets as files in tmpfs volumes with restrictive permissions (defaultMode: 0400) is generally more secure than environment variables.
   4. Review Network Policies: Ensure network policies are in place to restrict access to pods that handle sensitive data, limiting potential exfiltration.

FAQ Section

1. Q: What’s the main difference between a ConfigMap and a Secret?

   A: The primary difference is their intended use and default handling. ConfigMaps are for non-sensitive configuration data and are stored in plain text. Secrets are for sensitive data (passwords, tokens, keys) and are base64 encoded by default. While base64 encoding is not encryption, Secrets are often handled with more care by Kubernetes (e.g., mounted to tmpfs volumes) and are subject to stricter RBAC policies by convention. For more on securing data in transit, see our guide on Cilium WireGuard Encryption.

2. Q: Are Kubernetes Secrets truly secure?

   A: By default, Kubernetes Secrets are base64 encoded, not encrypted. This means anyone with API access to the cluster can easily decode them. For true security, you need to ensure encryption at rest for etcd (where Secrets are stored) is enabled, and ideally, integrate with an external secret management system like HashiCorp Vault. RBAC is also critical to restrict who can access secrets.

3. Q: When should I use environment variables versus volume mounts for ConfigMaps/Secrets?

   A: Use environment variables for simple, single-value configurations that your application expects in its environment (e.g., DB_HOST, LOG_LEVEL). Use volume mounts (as files) for more complex configurations, structured files (JSON, YAML), multiple files, or highly sensitive data (like TLS certificates or private keys). Volume mounts offer better control over file permissions and can prevent sensitive data from appearing in environment dumps or logs.

4. Q: How do I update a ConfigMap or Secret and make