Secure Kubernetes Secrets: Sealed Secrets Encryption

Introduction

Managing sensitive information like API keys, database credentials, and private certificates within Kubernetes is a critical security challenge. While Kubernetes provides a native Secret resource, these secrets are, by default, merely base64 encoded, not encrypted at rest within the etcd key-value store, nor are they encrypted when stored in version control systems like Git. This presents a significant vulnerability: anyone with access to your cluster’s etcd or your Git repository can easily decode and access your sensitive data. This fundamental flaw necessitates a robust encryption solution to protect your secrets throughout their lifecycle.

Enter Bitnami’s Sealed Secrets. Sealed Secrets addresses this gaping security hole by providing a one-way encryption mechanism for Kubernetes Secrets. Instead of storing raw, base64-encoded secrets in your Git repository, you store a special SealedSecret custom resource. This resource can only be decrypted by a controller running within your Kubernetes cluster, which holds a unique, asymmetric key pair. This means you can safely commit your SealedSecret manifests to public or private Git repositories without exposing the actual sensitive data, enabling true GitOps workflows for your secrets. This guide will walk you through deploying, configuring, and using Sealed Secrets to secure your Kubernetes environments.

Prerequisites

Before you begin, ensure you have the following:

• A running Kubernetes cluster (version 1.16 or higher). You can use Minikube, Kind, or a cloud-managed cluster like AWS EKS, GCP GKE, or Azure AKS.
• kubectl configured to connect to your cluster. Refer to the official Kubernetes documentation for kubectl installation.
• The kubeseal command-line tool installed on your local machine. This tool is essential for encrypting secrets.
• Basic understanding of Kubernetes concepts like Deployments, Services, and Secrets.

Installing kubeseal CLI

The kubeseal CLI is used to encrypt your secrets before committing them to Git. Installation varies by operating system:

macOS (using Homebrew)

brew install kubeseal

Linux (using apt/deb)

# Download the latest release from the official GitHub releases page
# Replace v0.22.0 with the latest version if available
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/kubeseal-0.22.0-linux-amd64.tar.gz
tar -xvzf kubeseal-0.22.0-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
rm kubeseal kubeseal-0.22.0-linux-amd64.tar.gz

Windows (using Chocolatey or Scoop)

For Chocolatey (run as administrator):

choco install kubeseal

For Scoop:

scoop install kubeseal

Verify the installation:

kubeseal --version

Expected Output:

kubeseal version: v0.22.0

The version number might differ based on the latest release.

Step-by-Step Guide: Kubernetes Secrets Encryption with Sealed Secrets

Step 1: Deploy the Sealed Secrets Controller

The Sealed Secrets controller is a Kubernetes operator that watches for SealedSecret resources. When it detects one, it uses its private key to decrypt the data and create a standard Kubernetes Secret in the same namespace. This controller is the cornerstone of the Sealed Secrets ecosystem, acting as the sole entity capable of decrypting your sensitive information within the cluster.

The controller runs as a Deployment and exposes a service. It’s crucial that this controller is running and healthy for Sealed Secrets to function correctly. The deployment manifest includes the necessary RBAC permissions and the Deployment definition itself. For production environments, consider isolating the controller in a dedicated namespace, though for this tutorial, we’ll deploy it to the default namespace for simplicity.

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/controller.yaml

Verify Controller Deployment

Check if the controller is running successfully:

kubectl get po -n kube-system -l app.kubernetes.io/name=sealed-secrets-controller

Expected Output:

NAME                                     READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-5c9f5f674d-abcde   1/1     Running   0          2m

Also, verify the service:

kubectl get svc -n kube-system -l app.kubernetes.io/name=sealed-secrets-controller

Expected Output:

NAME                      TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
sealed-secrets-controller   ClusterIP   10.96.123.456   <none>         80/TCP    2m

Step 2: Fetch the Public Certificate

The kubeseal CLI tool needs the public key of the Sealed Secrets controller to encrypt your secrets. This public key is dynamically generated by the controller upon its first startup and is exposed via its service. You’ll download this certificate and use it locally to encrypt your secret manifests. It’s safe to store this public certificate in your Git repository alongside your SealedSecret manifests, as it can only be used for encryption, not decryption.

This separation of concerns—public key for encryption on your local machine, private key for decryption within the cluster—is what makes Sealed Secrets a powerful and secure solution for GitOps. For those interested in more advanced security practices, consider integrating this with tools like Sigstore and Kyverno for supply chain integrity.

kubeseal --fetch-cert > kubeseal-cert.pem

Verify Certificate Download

Check if the certificate file was created and contains valid PEM data:

cat kubeseal-cert.pem

Expected Output (truncated):

-----BEGIN CERTIFICATE-----
MIIDezCCAmOgAwIBAgIRAPzK...
...
-----END CERTIFICATE-----

Step 3: Create a Standard Kubernetes Secret (Temporary)

Before you can seal a secret, you first need to define it as a standard Kubernetes Secret. This step is temporary; you will never apply this raw secret to your cluster. Instead, you’ll use this manifest as input for the kubeseal tool. For this example, we’ll create a secret containing a username and password. Remember, the values in the data field must be base64 encoded.

This process highlights a key aspect of Sealed Secrets: it leverages the existing Kubernetes Secret structure, making it familiar and easy to integrate into existing workflows. For more complex networking setups where secrets might be distributed across namespaces, consider how this integrates with solutions like Cilium WireGuard Encryption or Kubernetes Network Policies to secure inter-pod communication.

# my-app-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-credentials
  namespace: default
type: Opaque
data:
  username: bXlVc2Vy
  password: c3VwZXJTZWNyZXQQ

To generate base64 encoded values:

echo -n "myUser" | base64
echo -n "superSecretPassword123!" | base64

Expected Output (example):

bXlVc2Vy
c3VwZXJTZWNyZXQQMTIzIQ==

Step 4: Seal the Secret

Now, use the kubeseal CLI tool with the public certificate you fetched earlier to encrypt your my-app-secret.yaml into a SealedSecret. The output will be a new YAML file containing the encrypted data. This is the file you’ll store in your Git repository and apply to your Kubernetes cluster.

The --cert flag specifies the public key for encryption. You can also specify --scope cluster-wide (default), --scope namespace-wide, or --scope strict depending on your security requirements for the generated Secret. cluster-wide allows the SealedSecret to be decrypted in any namespace by the controller, but the resulting Kubernetes Secret will only appear in the namespace specified in the SealedSecret manifest. namespace-wide ties the SealedSecret to a specific namespace, and strict further binds it to a specific name in that namespace.

kubeseal --cert kubeseal-cert.pem < my-app-secret.yaml > my-sealedsecret.yaml

Verify Sealed Secret Creation

Inspect the generated my-sealedsecret.yaml file:

cat my-sealedsecret.yaml

Expected Output (truncated):

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-app-credentials
  namespace: default
spec:
  encryptedData:
    password: AgB64/iU5e+... # Encrypted base64 data
    username: AgBOp+z...  # Encrypted base64 data
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: my-app-credentials
      namespace: default
    type: Opaque

Notice the encryptedData field, which now contains the ciphertext. The original data field is replaced or omitted, ensuring the plaintext is never stored.

Step 5: Apply the Sealed Secret to Your Cluster

Finally, apply the my-sealedsecret.yaml manifest to your Kubernetes cluster. The Sealed Secrets controller will detect this new resource, decrypt it using its private key, and create a corresponding standard Kubernetes Secret in the specified namespace.

This is the magic moment where your encrypted secret becomes a usable Kubernetes Secret. The application pods can then mount or consume this standard Secret as they normally would, completely unaware that it originated from an encrypted SealedSecret. This seamless integration allows for consistent application deployment strategies while bolstering security.

kubectl apply -f my-sealedsecret.yaml

Verify Standard Secret Creation

Check if the standard Kubernetes Secret has been created by the controller:

kubectl get secret my-app-credentials -o yaml

Expected Output (truncated):

apiVersion: v1
data:
  password: c3VwZXJTZWNyZXQQMTIzIQ==
  username: bXlVc2Vy
kind: Secret
metadata:
  creationTimestamp: "2023-10-27T10:00:00Z"
  name: my-app-credentials
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: my-app-credentials
    uid: abcdef12-3456-7890-abcd-ef1234567890
type: Opaque

You can see the data field now contains the base64-encoded plaintext values, just like a regular Kubernetes Secret. The ownerReferences field indicates that this Secret was created and is managed by the SealedSecret.

Step 6: Consume the Secret in an Application

To demonstrate that the decrypted secret is usable, let’s deploy a simple Pod that reads these credentials from the generated Kubernetes Secret. Applications can consume secrets by mounting them as files or by exposing them as environment variables.

# my-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: busybox:latest
        command: ["sh", "-c"]
        args:
        - echo "Username: $(cat /etc/secrets/username)";
          echo "Password: $(cat /etc/secrets/password)";
          sleep 3600;
        volumeMounts:
        - name: app-secrets
          mountPath: "/etc/secrets"
          readOnly: true
      volumes:
      - name: app-secrets
        secret:
          secretName: my-app-credentials

kubectl apply -f my-app-deployment.yaml

Verify Secret Consumption

Check the logs of the deployed pod to see if it successfully read the secrets:

kubectl logs -l app=my-app

Expected Output:

Username: myUser
Password: superSecretPassword123!

This confirms that your application can successfully access the secrets that were originally encrypted as a SealedSecret and then decrypted by the controller.

Production Considerations

While Sealed Secrets significantly enhances security, deploying it in production requires careful planning:

1. Key Management and Rotation: The private key used by the Sealed Secrets controller is critical. By default, it’s stored as a Kubernetes Secret. For higher security, consider integrating with external Key Management Systems (KMS) like AWS KMS, GCP KMS, or Azure Key Vault to protect the private key. Regularly rotate the key pair. Sealed Secrets supports key rotation; you can generate a new key pair, update the controller, and then re-seal your secrets with the new public key.
2. Controller Redundancy and High Availability: Ensure the Sealed Secrets controller is deployed with multiple replicas and appropriate resource limits/requests to guarantee high availability and prevent it from becoming a single point of failure. If the controller is down, new SealedSecret resources won’t be decrypted, and existing Secrets might not be updated.
3. RBAC for SealedSecret: Implement strict Role-Based Access Control (RBAC) for who can create, update, or delete SealedSecret resources. Only authorized personnel or CI/CD pipelines should have these permissions. For example, you might want to restrict developers from directly creating Secret resources but allow them to create SealedSecret resources.
4. Namespace Scope vs. Cluster Scope: Decide on the appropriate scope for your SealedSecret resources (cluster-wide, namespace-wide, or strict). For multi-tenant clusters, namespace-wide or strict scope is generally preferred to prevent accidental decryption across namespaces, aligning with principles of isolation similar to those enforced by Kubernetes Network Policies.
5. Auditing and Logging: Configure robust auditing and logging for your Kubernetes API server to track who is creating/modifying SealedSecret resources and who is accessing the resulting standard Kubernetes Secrets.
6. GitOps Integration: Sealed Secrets shines in GitOps environments. Integrate the kubeseal command into your CI/CD pipelines to automatically seal secrets before they are committed to Git. Tools like Argo CD or Flux CD can then seamlessly deploy these SealedSecret manifests.
7. Backup and Restore: Include SealedSecret resources in your cluster backup strategy. Since the actual secrets are encrypted, backing up the SealedSecret manifests and the controller’s private key (if not using external KMS) is crucial for disaster recovery.
8. Alternative Solutions: While Sealed Secrets is excellent for GitOps, consider other advanced secret management solutions like HashiCorp Vault or cloud provider secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) for more complex scenarios involving dynamic secrets, secret rotation, and fine-grained access policies. Sealed Secrets can sometimes be used in conjunction with these tools.

Troubleshooting

Here are some common issues you might encounter with Sealed Secrets and their solutions.

1. Sealed Secrets Controller Not Running

Issue: The sealed-secrets-controller pod is not in a Running state.

Solution: Check the logs of the controller pod for errors. It might be due to incorrect RBAC permissions, resource constraints, or issues fetching its private key.

kubectl get po -n kube-system -l app.kubernetes.io/name=sealed-secrets-controller
kubectl describe po -n kube-system -l app.kubernetes.io/name=sealed-secrets-controller
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets-controller

Ensure the service account has permissions to create Secrets and SealedSecret resources.

2. kubeseal --fetch-cert Fails

Issue: The command kubeseal --fetch-cert hangs or returns an error like “no controller found”.

Solution: This usually means the Sealed Secrets controller is not running or its service is not accessible.

1. Verify the controller pod is running (see troubleshooting #1).
2. Verify the sealed-secrets-controller service exists and is healthy:

   kubectl get svc -n kube-system sealed-secrets-controller

3. Ensure your kubectl context is pointing to the correct cluster.

3. Sealed Secret Not Decrypted

Issue: You’ve applied a SealedSecret, but the corresponding standard Kubernetes Secret is not created.

Solution:

1. Check the logs of the Sealed Secrets controller pod. It might indicate decryption errors, mismatching namespaces, or issues with the SealedSecret manifest itself.
2. Ensure the SealedSecret‘s metadata.namespace matches the target namespace where you expect the standard Secret to be created.
3. Verify the SealedSecret‘s status field:

   kubectl get sealedsecret my-app-credentials -o yaml

   Look for any error messages in the status.

4. Ensure the SealedSecret was sealed with the correct public certificate corresponding to the *current* controller in your cluster. If the controller was redeployed or its key rotated, you might need to fetch a new certificate and re-seal the secret.

4. “Bad Encrypted Data” Error During Sealing

Issue: When running kubeseal, you get an error like “could not read secret: bad encrypted data”.

Solution: This typically happens if the input YAML for kubeseal is not a valid Kubernetes Secret, or if the data fields are not correctly base64 encoded.

1. Double-check your my-app-secret.yaml for syntax errors.
2. Ensure all values under the data field are correctly base64 encoded. The stringData field can be used for plaintext values in the initial Secret, and kubeseal will handle the base64 encoding, but it’s often safer to base64 encode explicitly.

5. Secret Values Are Incorrect After Decryption

Issue: The decrypted Secret contains unexpected or incorrect values.

Solution:

1. Verify the original base64 encoding of your plaintext values (Step 3).
2. Ensure you are using the correct kubeseal-cert.pem. If you have multiple clusters or have rotated keys, you might be using an old or incorrect certificate.
3. Check for any manual modifications to the SealedSecret manifest after sealing. Any changes to the encryptedData fields will render the secret undecryptable.

6. Permissions Issues for Applications

Issue: Your application pod cannot access the decrypted Secret.

Solution: This is a standard Kubernetes RBAC issue, not specific to Sealed Secrets.

1. Ensure the Service Account used by your application’s Pod has permission to get and list Secrets in its namespace.
2. Verify the volumeMounts and secretName in your Pod/Deployment manifest are correct.
3. Check for Kubernetes Network Policies that might be blocking access to the kubelet API server or the Secret itself, although this is less common for Secret access.

FAQ Section

Q1: Is Sealed Secrets truly secure if the private key is in the cluster?

A: Yes, it is secure for its intended purpose: enabling GitOps for secrets. The private key is only accessible by the Sealed Secrets controller within the cluster. Your SealedSecret manifests committed to Git are encrypted and cannot be decrypted without this private key. This prevents exposure of sensitive data to anyone with Git access but no cluster access. For ultimate security, especially if your cluster itself might be compromised, consider integrating with an external KMS like Google Cloud KMS or AWS KMS to protect the controller’s private key.

Q2: How do I rotate the Sealed Secrets encryption key?

A: Key rotation involves generating a new key pair for the controller, updating the controller to use the new key, and then re-sealing all your existing SealedSecret resources with the new public key. The process typically involves:

1. Deleting the old controller secret (e.g., sealed-secrets-key) if it’s not managed externally.
2. Redeploying the Sealed Secrets controller, which will generate a new key pair.
3. Fetching the new public certificate (kubeseal --fetch-cert > new-cert.pem).
4. Re-sealing all your SealedSecret manifests using the new new-cert.pem.
5. Applying the re-sealed SealedSecret manifests to the cluster.

The controller can handle multiple keys for a transition period, allowing old secrets to still be decrypted while new ones are sealed with the new key.

Q3: Can I use Sealed Secrets with multiple Kubernetes clusters?

A: Yes, but each cluster will have its own unique Sealed Secrets controller and thus its own unique key pair. This means you must seal secrets specifically for each cluster using that cluster’s public certificate. You cannot use a SealedSecret sealed for Cluster A on Cluster B, unless both clusters happen to share the exact same controller key (which is not the default behavior and not recommended). For example, if you have a dev and a prod cluster, you’d have dev-sealed-secrets-cert.pem and prod-sealed-secrets-cert.pem.

Q4: What if I accidentally delete the original standard Secret manifest before sealing?

A: This is not an issue. The standard Kubernetes Secret manifest (my-app-secret.yaml in our example) is only used as input for the kubeseal tool. Once the SealedSecret (my-sealedsecret.yaml) is generated, the original raw secret manifest is no longer needed and can be safely deleted from your local machine. It should never be committed to Git or applied to the cluster.

Q5: How does Sealed Secrets compare to other secret management solutions like HashiCorp Vault?

A: Sealed Secrets is simpler and designed specifically for static secrets in a GitOps workflow. It encrypts secrets at rest in Git and decrypts them in the cluster. HashiCorp Vault is a more comprehensive and powerful secret management solution offering dynamic secrets, secret rotation, fine-grained access control, auditing, and integration with various authentication backends. Vault is typically used for more complex, enterprise-level secret management requirements, often requiring its own dedicated infrastructure. Sealed Secrets is an excellent choice when you need to safely store static secrets in Git without the overhead of a full-fledged secret management system. They can also be used together, where Vault manages the “master” secrets, and Sealed Secrets is used to get a subset of those into Kubernetes for GitOps deployment.

Cleanup Commands

To remove all resources created during this tutorial:

1. Delete the application deployment:

kubectl delete -f my-app-deployment.yaml

2. Delete the Sealed Secret:

kubectl delete -f my-sealedsecret.yaml

This will also delete the standard Kubernetes Secret my-app-credentials created by the controller, as the SealedSecret is its owner.

3. Delete the Sealed Secrets controller and its associated resources:

kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/controller.yaml

4. Remove local files:

rm my-app-secret.yaml my-sealedsecret.yaml kubeseal-cert.pem

Next Steps / Further Reading

You’ve successfully implemented Sealed Secrets. Here are some ideas for where to go next:

• Integrate with CI/CD: Automate the kubeseal process in your CI/CD pipeline. For instance, a GitHub Actions workflow could trigger kubeseal on sensitive files before committing them.
• Explore Key Rotation: Research and practice key rotation strategies for Sealed Secrets in a non-production environment.
• Advanced Scopes: Experiment with --scope namespace-wide and --scope strict to understand their implications for multi-tenant clusters.
• External KMS Integration: For highly sensitive environments, investigate how to configure Sealed Secrets to use an external Key Management System (KMS) for storing its private key. The Sealed Secrets documentation on external KMS is a good starting point.
• GitOps Workflows: Learn how Sealed Secrets fits into a complete GitOps strategy using tools like Argo CD or Flux CD.
• Service Mesh Integration: Understand how secrets are managed in service mesh environments like Istio Ambient Mesh, especially for certificates or external service credentials.
• Cost Optimization: While not directly related to secrets, consider how secure secret management impacts overall operational efficiency and cost. Tools like Karpenter Cost Optimization can help manage the underlying infrastructure costs.
• Observability: For an in-depth look at how secrets might be accessed or misused, explore advanced observability tools. Our guide on eBPF Observability with Hubble can provide insights into network traffic patterns.

Conclusion

Securing sensitive data is paramount in any Kubernetes environment. Sealed Secrets provides an elegant, GitOps-friendly solution to encrypt your Kubernetes Secrets, allowing them to be safely stored in version control systems. By following this guide, you’ve learned how to deploy the controller, encrypt a secret using the kubeseal CLI, and verify its successful decryption and consumption within your cluster. This approach significantly reduces the risk of secret exposure, enhancing the overall security posture of your applications and infrastructure. Embrace Sealed Secrets to bring a new level of confidence to your Kubernetes secret management.