Encrypt Kubernetes Secrets with Sealed Secrets

Introduction

Managing sensitive information like API keys, database credentials, and private certificates is a persistent challenge in any modern application deployment. In the Kubernetes ecosystem, the native Secret object provides a way to store and manage this data. However, a fundamental security concern arises: by default, Kubernetes Secrets are merely base64 encoded, not encrypted at rest. This means anyone with API access to your cluster (or direct access to the underlying etcd data store) can easily decode and view your sensitive information. This gaping security hole is unacceptable for production environments and compliance requirements.

This tutorial addresses this critical security gap by introducing Sealed Secrets, a popular open-source solution from Bitnami. Sealed Secrets allows you to encrypt your Kubernetes Secrets into a SealedSecret custom resource, which can then be safely stored in Git – enabling true GitOps workflows for sensitive data. Only the Sealed Secrets controller running in your cluster can decrypt these secrets, ensuring that your sensitive data remains encrypted until it reaches its intended destination within the cluster. This approach significantly enhances your cluster’s security posture, aligning with best practices for secret management in cloud-native environments.

# Install kubeseal CLI
brew install kubeseal # macOS
# Or download from GitHub releases: https://github.com/bitnami-labs/sealed-secrets/releases

# Install Sealed Secrets controller (example with Helm)
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system

# Get public key
kubeseal --fetch-cert > public-key.pem

# Create a Secret YAML
cat < my-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
stringData:
  api_key: super-secret-api-key
  db_password: super-secret-db-password
EOF

# Seal the Secret
kubeseal --scope cluster-wide --format yaml < my-secret.yaml > my-sealed-secret.yaml

# Apply the SealedSecret
kubectl apply -f my-sealed-secret.yaml

# Verify the native Secret is created
kubectl get secret my-app-secret -o yaml

Prerequisites

Before diving into Sealed Secrets, ensure you have the following:

• A Kubernetes Cluster: Any Kubernetes cluster (local like Minikube/Kind, or cloud-based like EKS, GKE, AKS) will work.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Helm (Optional but Recommended): A package manager for Kubernetes, useful for installing the Sealed Secrets controller. Install it from the official Helm website.
• Basic understanding of Kubernetes Secrets: Familiarity with how Secrets work and their limitations is beneficial.
• kubeseal CLI: The client-side tool used to encrypt (seal) your secrets. We will install this in the first step.

Step-by-Step Guide to Kubernetes Secrets Encryption with Sealed Secrets

Step 1: Install the kubeseal CLI Tool

The kubeseal command-line utility is essential for encrypting your Kubernetes Secrets into SealedSecret objects. It uses the public key from the Sealed Secrets controller to perform the encryption locally on your machine. This ensures that your sensitive data never leaves your workstation unencrypted, adhering to the principle of “encrypt early, encrypt often.”

Installation methods vary by operating system. For macOS users, Homebrew is the easiest option. For other systems, you can typically download the binary directly from the GitHub releases page or compile it from source.

# For macOS users with Homebrew:
brew install kubeseal

# For Linux users (example for AMD64, adjust for your architecture):
# Check for the latest release here: https://github.com/bitnami-labs/sealed-secrets/releases
RELEASE_VERSION=$(curl -sL https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest | grep '"tag_name":' | cut -d '"' -f 4)
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/${RELEASE_VERSION}/kubeseal-${RELEASE_VERSION}-linux-amd64 -O kubeseal
chmod +x kubeseal
sudo mv kubeseal /usr/local/bin/

# Verify installation
kubeseal --version

Verify:

You should see the version information of kubeseal, confirming a successful installation.

kubeseal --version

kubeseal version: v0.20.5

Step 2: Install the Sealed Secrets Controller in Your Cluster

The Sealed Secrets controller is a Kubernetes operator that runs within your cluster. Its primary responsibility is to watch for SealedSecret objects, decrypt them using its private key, and then create or update standard Kubernetes Secret objects based on the decrypted data. This controller is the only component that holds the private key, ensuring that only it can unlock your sensitive data within the cluster boundary. Installing it via Helm is generally the recommended approach for its simplicity and manageability.

The controller typically runs in the kube-system namespace, but you can choose another namespace if it aligns better with your organizational policies. It generates a new key pair on its first run, which is then used for encryption and decryption. This key pair is stored as a standard Kubernetes Secret within the cluster.

# Add the Bitnami Sealed Secrets Helm repository
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

# Update your Helm repositories to fetch the latest charts
helm repo update

# Install the Sealed Secrets controller into the kube-system namespace
helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system

# Verify the controller deployment
kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

Verify:

You should see the Sealed Secrets controller pod running in the kube-system namespace. Its status should be Running.

kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

NAME                                READY   STATUS    RESTARTS   AGE
sealed-secrets-7c87c947d6-abcdef     1/1     Running   0          2m

Step 3: Retrieve the Public Key from the Controller

Once the Sealed Secrets controller is running, it exposes its public key. This public key is what kubeseal uses to encrypt your secrets. It’s crucial to retrieve this public key directly from your cluster’s controller to ensure that the secrets you seal can only be decrypted by that specific controller instance. The public key can be safely shared and used in CI/CD pipelines, as it can only encrypt, not decrypt.

You can fetch the certificate directly using kubeseal, which communicates with the API server to get the controller’s public key. It’s a good practice to save this certificate to a file, especially if you plan to automate the sealing process or use it across different machines.

# Fetch the public certificate and save it to a file
kubeseal --fetch-cert > public-cert.pem

# Verify the content of the public certificate
cat public-cert.pem

Verify:

The public-cert.pem file should contain a PEM-encoded X.509 certificate. This is the public key used for encryption.

cat public-cert.pem

-----BEGIN CERTIFICATE-----
MIIC1jCCAj+gAwIBAgIRAPyKk8Z...
...
-----END CERTIFICATE-----

Step 4: Create a Standard Kubernetes Secret

Before you can seal a secret, you first need to define it as a standard Kubernetes Secret object. This YAML manifest will contain the actual sensitive data in its plaintext (or base64 encoded) form. It’s important to treat this file with extreme care, as it contains unencrypted sensitive information. This file is typically used as an intermediate step and should ideally not be committed to source control.

For this example, we’ll create a simple Opaque secret containing an API key and a database password. Remember that stringData is a convenient field that allows you to provide secret values as plain strings, and Kubernetes will automatically base64 encode them when the Secret is created. If you use the data field, you must manually base64 encode the values.

# my-app-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
stringData:
  api_key: super-secret-api-key-123
  db_password: my-strong-db-password

Verify:

You can inspect the content of the file. Do NOT apply this secret directly to your cluster if you intend to use Sealed Secrets, as it would expose your plaintext secrets.

cat my-app-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
stringData:
  api_key: super-secret-api-key-123
  db_password: my-strong-db-password

Step 5: Seal the Kubernetes Secret

This is the core step where you transform your plaintext Secret into a SealedSecret. Using the kubeseal CLI, you’ll pipe your plaintext secret YAML into kubeseal, providing the public certificate you fetched earlier. kubeseal encrypts the sensitive data fields and outputs a new YAML manifest for a SealedSecret custom resource.

The --scope flag is important:

• --scope cluster-wide: The SealedSecret can be decrypted by any controller in any namespace. The label selector for the controller is app.kubernetes.io/name=sealed-secrets.
• --scope namespace-wide: The SealedSecret can only be decrypted by a controller in the same namespace.
• --scope strict: The SealedSecret can only be decrypted by a controller in the same namespace and with the same name.

For most GitOps scenarios, cluster-wide or namespace-wide are common choices. We’ll use cluster-wide for simplicity here.

# Seal the secret using the public certificate
# Note: You can also pipe directly: kubectl create secret generic my-app-secret --dry-run=client -o yaml --from-literal=api_key=secretkey | kubeseal ...
kubeseal --cert public-cert.pem --scope cluster-wide --format yaml < my-app-secret.yaml > my-sealed-secret.yaml

# Inspect the generated SealedSecret
cat my-sealed-secret.yaml

Verify:

You should see a YAML manifest for a SealedSecret. Notice the encryptedData field, which contains the encrypted versions of your original secret values. This file is now safe to commit to Git.

cat my-sealed-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-app-secret
  namespace: default
spec:
  encryptedData:
    api_key: AgBh/Hk42... # long encrypted string
    db_password: AgA1+Hk42... # another long encrypted string
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: my-app-secret
      namespace: default
    type: Opaque

Step 6: Apply the Sealed Secret to Your Cluster

With the SealedSecret manifest created, you can now safely apply it to your Kubernetes cluster. The Sealed Secrets controller, which we installed in Step 2, will automatically detect this new SealedSecret object. It will then use its private key to decrypt the encryptedData and create a standard Kubernetes Secret with the same name and namespace as specified in the SealedSecret‘s template.

This is where the magic happens: you apply an encrypted resource, and the cluster automatically provisions the unencrypted secret, making it available to your applications. This process is idempotent, meaning you can apply the same SealedSecret multiple times, and the controller will ensure the underlying Secret is always up-to-date.

# Apply the SealedSecret manifest
kubectl apply -f my-sealed-secret.yaml

# Verify that the native Kubernetes Secret has been created
kubectl get secret my-app-secret -n default -o yaml

Verify:

You should see the standard Kubernetes Secret named my-app-secret in the default namespace. Inspecting its YAML will show the base64-encoded values of your original plaintext secrets, which the controller has decrypted and created.

kubectl get secret my-app-secret -n default -o yaml

apiVersion: v1
data:
  api_key: c3VwZXItc2VjcmV0LWFwaS1rZXktMTIz # base64 encoded 'super-secret-api-key-123'
  db_password: bXktc3Ryb25nLWRiLXBhc3N3b3Jk # base64 encoded 'my-strong-db-password'
kind: Secret
metadata:
  creationTimestamp: "2023-10-27T10:00:00Z"
  name: my-app-secret
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: my-app-secret
    uid: a1b2c3d4-e5f6-7890-1234-567890abcdef
  resourceVersion: "123456"
  uid: f1e2d3c4-b5a6-9876-5432-10fedcba9876
type: Opaque

Notice the ownerReferences field. This links the native Kubernetes Secret back to the SealedSecret, ensuring that if the SealedSecret is deleted, the native Secret is also garbage collected.

Step 7: Consume the Secret in an Application

Once the Sealed Secrets controller has successfully created the native Kubernetes Secret, your applications can consume it just like any other Kubernetes Secret. This means mounting it as a volume, injecting it as environment variables, or using it through various Kubernetes integrations. The application itself doesn’t need to be aware that Sealed Secrets were used; it simply interacts with a standard Kubernetes Secret.

Here’s an example of a simple Deployment that consumes the my-app-secret as environment variables:

# my-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest # Replace with your actual application image
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: my-app-secret
              key: api_key
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-app-secret
              key: db_password
        ports:
        - containerPort: 80

# Apply the deployment
kubectl apply -f my-app-deployment.yaml

# Verify the deployment
kubectl get deployment my-app -n default
kubectl get pods -n default -l app=my-app

# Inspect the environment variables of the running pod
POD_NAME=$(kubectl get pods -n default -l app=my-app -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n default -- printenv | grep -E "API_KEY|DB_PASSWORD"

Verify:

You should see the API_KEY and DB_PASSWORD environment variables populated with the correct (decrypted) values from your secret.

kubectl exec -it $POD_NAME -n default -- printenv | grep -E "API_KEY|DB_PASSWORD"

API_KEY=super-secret-api-key-123
DB_PASSWORD=my-strong-db-password

This confirms that your application can successfully access the sensitive data, which has been securely managed end-to-end.

Production Considerations

While Sealed Secrets significantly enhance Kubernetes secret management, deploying them in a production environment requires careful consideration beyond the basic setup:

1. Key Management and Rotation:

   The private key generated by the Sealed Secrets controller is critical. By default, it’s stored as a Kubernetes Secret. For higher security, consider:

   • Backup: Regularly back up the private key. Losing it means you can no longer decrypt existing SealedSecrets.
   • External Key Management: For extreme security or compliance, Sealed Secrets can be configured to use external Key Management Systems (KMS) like AWS KMS, Azure Key Vault, or Google Cloud KMS to protect the master key. Refer to the Sealed Secrets documentation on KMS integration.
   • Rotation: While Sealed Secrets supports key rotation, it’s a manual process that requires re-sealing all your secrets with the new public key. Plan this carefully.

   For more robust security practices, especially concerning infrastructure, you might also consider how you secure network traffic between components. Solutions like Cilium WireGuard Encryption can provide strong encryption for pod-to-pod communication, complementing secret encryption.

2. Role-Based Access Control (RBAC):

   Ensure that only authorized personnel or CI/CD systems have permission to apply SealedSecret objects. Similarly, restrict who can view the native Secret objects that the controller creates. Implement strict Kubernetes RBAC policies.

   Also, restrict access to the Sealed Secrets controller’s pod and logs, as these could potentially expose information about decryption attempts.

3. Monitoring and Alerting:

   Monitor the Sealed Secrets controller for any issues. Key metrics include:

   • Pod health and restarts.
   • Logs for decryption errors or warnings (e.g., failed to retrieve public key, invalid SealedSecret format).
   • Number of SealedSecret objects and corresponding Secret objects.

   Tools like Prometheus and Grafana can be used for this. For advanced observability, consider leveraging eBPF-based solutions like eBPF Observability with Hubble to gain deep insights into network and application behavior, which can indirectly help in identifying secret-related issues.

4. CI/CD Integration:

   Integrate kubeseal into your CI/CD pipeline. This allows developers to commit plaintext secret definitions (e.g., in a temporary local file or securely stored variable), and the pipeline automatically seals them using the fetched public key before committing the SealedSecret to your Git repository or applying it to the cluster.

   Example for a GitOps pipeline:

   1. Developer creates my-secret.yaml (local, not committed).
   2. CI/CD pipeline fetches public-cert.pem from the cluster.
   3. Pipeline runs kubeseal --cert public-cert.pem < my-secret.yaml > my-sealed-secret.yaml.
   4. Pipeline commits my-sealed-secret.yaml to Git.
   5. GitOps tool (Argo CD, Flux CD) syncs my-sealed-secret.yaml to the cluster.

   For securing your CI/CD pipeline and the container images it produces, tools like Sigstore and Kyverno can provide cryptographic signing and policy enforcement, ensuring the integrity and authenticity of your deployment artifacts.

5. Immutable Infrastructure and GitOps:

   Sealed Secrets perfectly aligns with the GitOps philosophy. All changes to secrets (and other Kubernetes resources) are version-controlled in Git. This provides an audit trail, allows for easy rollbacks, and ensures that the cluster state matches the desired state defined in Git. For a comprehensive guide on achieving GitOps with Kubernetes, refer to the CNCF GitOps whitepaper.

6. Namespace Scope vs. Cluster Scope:

   Carefully choose the --scope when sealing secrets. cluster-wide is convenient but means any controller in any namespace can decrypt. namespace-wide or strict offers tighter isolation, which might be preferred in multi-tenant environments or for sensitive secrets belonging to specific applications. If you are managing multiple applications in a shared cluster, robust Kubernetes Network Policies are also crucial for isolating traffic between tenants.

Troubleshooting

1. Sealed Secret not decrypting / Native Secret not appearing

   Issue: You’ve applied a SealedSecret, but the corresponding native Kubernetes Secret is not created or updated.

   Solution:

   1. Check Controller Logs: The most common reason is an issue with the Sealed Secrets controller. Check its logs for errors.

   kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets
   

   2. Verify Controller Status: Ensure the controller pod is running and healthy.

   kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets
   

   3. Public Key Mismatch: The SealedSecret might have been encrypted with a different public key than the one the current controller holds. If you reinstalled the controller, it generated a new key pair. You’ll need to re-seal your secrets with the new public key.
   4. Scope Mismatch: If you sealed with --scope namespace-wide or --scope strict, ensure the SealedSecret is in the correct namespace and that the controller’s label selector matches (for strict scope).

2. kubeseal fails to fetch certificate

   Issue: When running kubeseal --fetch-cert, you get an error like “Error: could not fetch certificate: secrets.bitnami.com is forbidden”.

   Solution: This usually indicates an RBAC issue. Your kubectl context’s user does not have permission to access the Sealed Secrets controller’s certificate.

   1. Ensure your kubectl context is pointing to the correct cluster.
   2. Verify your user has permissions to get the sealed-secrets-key secret or perform get sealedsecrets.bitnami.com. You might need to grant yourself a ClusterRole or Role with these permissions.

   # Example ClusterRole for fetching cert (if needed)
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: sealed-secrets-cert-fetcher
   rules:
   - apiGroups: ["bitnami.com"]
     resources: ["sealedsecrets"]
     verbs: ["get"]
   - apiGroups: [""] # "" indicates core API group
     resources: ["secrets"]
     resourceNames: ["sealed-secrets-key"] # Name of the secret containing the key
     verbs: ["get"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: sealed-secrets-cert-fetcher-binding
   subjects:
   - kind: User # Or Group, ServiceAccount
     name: your-user-name # Replace with your user/serviceaccount name
     apiGroup: rbac.authorization.k8s.io
   roleRef:
     kind: ClusterRole
     name: sealed-secrets-cert-fetcher
     apiGroup: rbac.authorization.k8s.io
   

3. Error: “No controller found” when sealing

   Issue: kubeseal throws an error that it cannot find the Sealed Secrets controller.

   Solution:

   1. Controller Not Running: Ensure the Sealed Secrets controller pod is up and running in the kube-system namespace (or wherever you installed it).

   kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets
   

   2. kubectl Context: Verify your kubectl is pointing to the correct cluster where the controller is installed.
   3. Network Issues: Check if there are any network policies or firewall rules (e.g., using Kubernetes Network Policies) preventing kubeseal from communicating with the Kubernetes API server, or preventing the API server from reaching the controller.

4. SealedSecret applied, but the native Secret has incorrect data.

   Issue: The native Secret is created, but the values are not what you expected (e.g., empty, malformed, or old values).

   Solution:

   1. Re-seal with correct public key: Ensure you are using the most current public key when sealing. If the controller was restarted or reinstalled, its key might have changed.
   2. Check input YAML: Double-check the original plaintext secret YAML (my-app-secret.yaml) for typos or incorrect data.
   3. stringData vs data: If you used the data field in your original secret, ensure the values were properly base64 encoded. stringData handles this automatically and is generally preferred.
   4. Reapply SealedSecret: Sometimes, reapplying the SealedSecret can resolve transient issues.

   kubectl apply -f my-sealed-secret.yaml
   

5. How to rotate the Sealed Secrets private key?

   Issue: You need to rotate the private key for security reasons or if it was compromised.

   Solution: Key rotation is a multi-step process in Sealed Secrets:

   1. Generate a new key: The controller automatically generates a new key periodically (default 30 days) or you can force it by deleting the old key secret (sealed-secrets-key).
   2. Update kubeseal: Fetch the new public key using kubeseal --fetch-cert > new-public-cert.pem.
   3. Re-seal all secrets: This is the most critical step. You must re-seal all your existing SealedSecrets using the new public key and apply them to the cluster. The controller supports multiple keys for decryption, but new secrets must be sealed with the latest key.
   4. Cleanup old key (optional): Once all secrets are re-sealed and confirmed working with the new key, you can manually delete old key secrets if desired (though the controller manages this up to a certain number).

   Refer to the official documentation on key rotation for detailed steps.

FAQ Section

1. What is the difference between Kubernetes Secrets and Sealed Secrets?

   Kubernetes Secrets are native Kubernetes objects used to store sensitive data. By default, they are only base64 encoded, meaning anyone with API access to the cluster can easily decode and view their contents. They are not encrypted at rest in etcd unless etcd encryption at rest is enabled and configured, which is a separate and often complex cluster-level configuration.

   Sealed Secrets are an additional layer of security. They are custom resources that encapsulate an encrypted version of a Kubernetes Secret. The sensitive data within a SealedSecret is encrypted using a public key and can only be decrypted by the Sealed Secrets controller running in the cluster with the corresponding private key. This allows SealedSecrets to be safely stored in public repositories like Git.

2. Can I use Sealed Secrets with cloud provider KMS (Key Management Systems)?

   Yes, Sealed Secrets supports integrating with cloud provider KMS solutions like AWS KMS, Azure Key Vault, and Google Cloud KMS. This allows you to protect the Sealed Secrets controller’s private key itself, adding an extra layer of security and often meeting stringent compliance requirements. The controller will use the KMS to encrypt and decrypt its own private key. This advanced configuration is detailed in the Sealed Secrets GitHub repository documentation.

3. Is it safe to commit SealedSecret YAML files to Git?

   Yes, that is the primary purpose and advantage of Sealed Secrets! The SealedSecret YAML contains only encrypted data (in the encryptedData field). The public key used for encryption can only encrypt, not decrypt. Only the Sealed Secrets controller, which holds the corresponding private key within your Kubernetes cluster, can decrypt the data. This makes SealedSecret manifests safe to commit to version control systems like Git, even public ones, enabling true Git