Encrypt Kubernetes Secrets with Sealed Secrets

Introduction

Managing sensitive information like API keys, database credentials, and private certificates in Kubernetes is a critical security challenge. While Kubernetes provides Secrets as a first-class object, they are merely base64 encoded and stored unencrypted in etcd by default. This means anyone with read access to etcd or API access to retrieve Secrets can easily decode and view their contents. For production environments, this default behavior is simply not secure enough, leaving your sensitive data vulnerable to insider threats and potential compromise.

This is where Sealed Secrets come to the rescue. Developed by Bitnami, Sealed Secrets provide a robust solution for encrypting your sensitive data directly into a Kubernetes-native SealedSecret object. This encrypted Secret can then be safely stored in your Git repository alongside your other Kubernetes manifests, enabling GitOps workflows without exposing plaintext secrets. Only the Sealed Secrets controller, running within your cluster, possesses the private key necessary to decrypt these secrets into standard Kubernetes Secret objects, making them accessible to your applications. This approach significantly enhances the security posture of your cluster by ensuring that sensitive data is encrypted at rest and in transit, and only decrypted within the secure boundaries of your Kubernetes cluster.

In this comprehensive guide, we’ll walk you through the process of installing, configuring, and using Sealed Secrets to secure your applications. We’ll cover everything from generating your first Sealed Secret to integrating it into your CI/CD pipelines, ensuring your sensitive data remains protected. By the end of this tutorial, you’ll have a solid understanding of how to implement a secure, GitOps-friendly secret management solution in your Kubernetes clusters.

# Install kubeseal CLI (macOS example)
brew install kubeseal

# Install Sealed Secrets controller via Helm
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system --create-namespace

# Create a sample Secret manifest
cat < mysecret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
type: Opaque
data:
  api_key: $(echo -n "supersecretapikey" | base64)
  db_password: $(echo -n "db_strong_password" | base64)
EOF

# Encrypt the Secret into a SealedSecret
kubeseal --scope cluster-wide < mysecret.yaml > mysealedsecret.yaml

# Apply the SealedSecret to your cluster
kubectl apply -f mysealedsecret.yaml

# Verify the Secret is created
kubectl get secret my-app-secret
kubectl get sealedsecret my-app-secret

# Clean up
kubectl delete -f mysealedsecret.yaml
kubectl delete -f mysecret.yaml
helm uninstall sealed-secrets --namespace kube-system
kubectl delete namespace kube-system # If only used for sealed-secrets

Prerequisites

Before we begin, ensure you have the following tools and knowledge:

• Kubernetes Cluster: A running Kubernetes cluster (v1.16+ recommended). This can be a local cluster like MiniKube or Kind, or a cloud-managed service like GKE, EKS, or AKS.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Helm: The Kubernetes package manager (v3+). We’ll use Helm to install the Sealed Secrets controller. Install it from the official Helm website.
• kubeseal CLI: The client-side tool for encrypting secrets. We’ll install this during the guide.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, and Secrets.
• Git: For storing your encrypted secrets in a version control system.

Step-by-Step Guide

Step 1: Install the kubeseal CLI Tool

The kubeseal command-line interface (CLI) is essential for encrypting your secrets outside the cluster. It uses the public key from the Sealed Secrets controller to perform the encryption. This tool is client-side, meaning you install it on your local machine, not directly in your Kubernetes cluster.

Installation methods vary by operating system. For macOS, you can use Homebrew. For Linux, you can download the binary directly or use a package manager. Windows users can also use Chocolatey or download the binary.

macOS (Homebrew)

brew install kubeseal

Linux (apt-get for Debian/Ubuntu)

# Download the latest release from GitHub
# Replace '0.20.4' with the latest stable version from https://github.com/bitnami/sealed-secrets/releases
wget https://github.com/bitnami/sealed-secrets/releases/download/v0.20.4/kubeseal-0.20.4-linux-amd64.tar.gz

# Extract the binary
tar -xvzf kubeseal-0.20.4-linux-amd64.tar.gz

# Move it to your PATH
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

# Clean up
rm kubeseal kubeseal-0.20.4-linux-amd64.tar.gz

Verify Installation

After installation, verify that kubeseal is correctly installed and accessible in your PATH by checking its version. This command should return the version number of the installed CLI tool.

kubeseal --version

Expected Output:

kubeseal version: v0.20.4

Step 2: Install the Sealed Secrets Controller in Your Cluster

The Sealed Secrets controller is a Kubernetes operator that runs inside your cluster. Its primary role is to watch for SealedSecret objects, decrypt them using its private key, and then create or update standard Kubernetes Secret objects. This controller is the only component that ever sees the plaintext secret, ensuring that your sensitive data remains secure.

We’ll install the controller using Helm, which is the recommended and easiest way to deploy it. The controller will be installed into the kube-system namespace by default, but you can choose another namespace if you prefer.

Add Helm Repository

First, add the Bitnami Sealed Secrets Helm repository.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update

Install the Controller

Now, install the Sealed Secrets controller. We’ll install it into the kube-system namespace.

helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system --create-namespace

Verify Installation

Check if the controller pod is running and healthy. You should see a pod named similar to sealed-secrets-controller-... in the kube-system namespace in a Running state.

kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

Expected Output:

NAME                                          READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6d7c67c784-abcde   1/1     Running   0          2m

Also, verify that the SealedSecret Custom Resource Definition (CRD) has been installed:

kubectl get crd sealedsecrets.bitnami.com

Expected Output:

NAME                       CREATED AT
sealedsecrets.bitnami.com   2023-10-26T10:00:00Z

Step 3: Retrieve the Public Key

The kubeseal CLI needs the public key of the Sealed Secrets controller to encrypt your secrets. This public key is exposed by the controller as a certificate. You can retrieve it directly from the cluster using kubectl. This key is public, so it can be safely shared and used for encryption outside the cluster.

The kubeseal CLI can automatically fetch this public key for you, which simplifies the encryption process. However, understanding how to manually retrieve it is useful for troubleshooting or advanced scenarios.

Fetch Public Key (Automatically by kubeseal)

When you run kubeseal, if you don’t specify a public key explicitly, it will attempt to fetch it directly from the controller in your cluster.

Fetch Public Key (Manually)

To manually retrieve and store the public key:

# Fetch the certificate and save it to a file
kubectl get secret -n kube-system sealed-secrets-key -o jsonpath='{.data.tls\.crt}' | base64 --decode > sealed-secrets-cert.pem

# Verify the certificate content
cat sealed-secrets-cert.pem

Expected Output (snippet of a certificate):

-----BEGIN CERTIFICATE-----
MIIDezCCAmOgAwIBAgIJANr3J0Ym...
...
-----END CERTIFICATE-----

You would then use this file with the --cert flag when running kubeseal, e.g., kubeseal --cert sealed-secrets-cert.pem < mysecret.yaml.

Step 4: Create a Kubernetes Secret Manifest

Before you can encrypt a secret, you first need to define it as a standard Kubernetes Secret object. This is the plaintext version of your secret. We will then feed this manifest into kubeseal to get its encrypted counterpart. Remember, the data fields in a Kubernetes Secret must be base64 encoded.

Let's create a sample secret that contains an API key and a database password. We'll put this in a file named mysecret.yaml.

# mysecret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default # Or your application's namespace
type: Opaque
data:
  api_key: c3VwZXJzZWNyZXRhcGlrZXk= # base64 encoded "supersecretapikey"
  db_password: ZGJfc3Ryb25nX3Bhc3N3b3Jk # base64 encoded "db_strong_password"

You can generate base64 encoded strings using echo -n "your_secret_value" | base64.

echo -n "supersecretapikey" | base64
echo -n "db_strong_password" | base64

Expected Output:

c3VwZXJzZWNyZXRhcGlrZXk=
ZGJfc3Ryb25nX3Bhc3N3b3Jk

Now, create the mysecret.yaml file:

cat < mysecret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
data:
  api_key: $(echo -n "supersecretapikey" | base64)
  db_password: $(echo -n "db_strong_password" | base64)
EOF

# Verify content
cat mysecret.yaml

Expected Output (similar to):

apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
data:
  api_key: c3VwZXJzZWNyZXRhcGlrZXk=
  db_password: ZGJfc3Ryb25nX3Bhc3N3b3Jk

Step 5: Encrypt the Secret into a SealedSecret

This is the core step where we transform the plaintext Kubernetes Secret into a SealedSecret. The kubeseal CLI tool reads your mysecret.yaml, encrypts its contents using the public key from your cluster's Sealed Secrets controller, and outputs a new YAML manifest for a SealedSecret custom resource.

There are different scopes you can apply when encrypting a secret:

• --scope cluster-wide: The secret can be decrypted into any namespace by the controller. This is generally less secure but convenient.
• --scope namespace-wide: The secret can only be decrypted into the specified namespace.
• --scope strict (default): The secret can only be decrypted into the specified namespace AND with the exact same name. This is the most secure option.

For most production use cases, --scope strict or --scope namespace-wide are preferred. We'll use --scope strict for this example.

kubeseal --scope strict --format yaml < mysecret.yaml > mysealedsecret.yaml

If you prefer to manually specify the certificate (from Step 3):

kubeseal --cert sealed-secrets-cert.pem --scope strict --format yaml < mysecret.yaml > mysealedsecret.yaml

Inspect the SealedSecret

Now, examine the generated mysealedsecret.yaml file. You'll notice that the data and stringData fields are replaced with an encryptedData field containing opaque, encrypted blobs. This file is safe to commit to your Git repository.

cat mysealedsecret.yaml

Expected Output (similar to, encrypted data will vary):

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-app-secret
  namespace: default
spec:
  encryptedData:
    api_key: AgBhWw... # long encrypted string
    db_password: AgBQwS... # long encrypted string
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: my-app-secret
      namespace: default
    type: Opaque

Notice the encryptedData section. This is the encrypted version of your secrets. The template section contains metadata that the controller uses to recreate the original Secret, such as its name, namespace, and type.

Step 6: Apply the SealedSecret to Your Cluster

With the mysealedsecret.yaml file created, you can now apply it to your Kubernetes cluster just like any other manifest. The Sealed Secrets controller will detect this new SealedSecret object, decrypt its contents using its private key, and create a corresponding standard Kubernetes Secret object in the specified namespace.

kubectl apply -f mysealedsecret.yaml

Expected Output:

sealedsecret.bitnami.com/my-app-secret created

Verify the Secret Creation

After applying, verify that both the SealedSecret and the actual Kubernetes Secret have been created in the default namespace.

kubectl get sealedsecret my-app-secret -n default
kubectl get secret my-app-secret -n default

Expected Output:

NAME            AGE
my-app-secret   10s

NAME            TYPE     DATA   AGE
my-app-secret   Opaque   2      10s

Crucially, you can now inspect the plaintext contents of the standard Kubernetes Secret. This confirms that the Sealed Secrets controller successfully decrypted your data.

kubectl get secret my-app-secret -n default -o jsonpath='{.data}' | base64 --decode

Expected Output:

{"api_key":"supersecretapikey","db_password":"db_strong_password"}

This output clearly shows the original plaintext values, demonstrating that the decryption process was successful and your application can now access these secrets.

Step 7: Using the Secret in Your Application

Once the Sealed Secrets controller has decrypted the SealedSecret into a standard Kubernetes Secret, your applications can consume it just like any other Kubernetes Secret. This typically involves mounting the secret as files into a Pod or injecting its values as environment variables.

Let's create a simple Nginx Deployment that consumes our my-app-secret as environment variables.

Create a Deployment Manifest

Save this as nginx-deployment.yaml:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-secret-consumer
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-secret-consumer
  template:
    metadata:
      labels:
        app: nginx-secret-consumer
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        env:
        - name: MY_API_KEY
          valueFrom:
            secretKeyRef:
              name: my-app-secret
              key: api_key
        - name: MY_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-app-secret
              key: db_password
        # Command to keep the container running and show env vars
        command: ["/bin/bash", "-c", "env | grep MY_ && sleep infinity"]

Apply the Deployment

kubectl apply -f nginx-deployment.yaml

Expected Output:

deployment.apps/nginx-secret-consumer created

Verify Secret Consumption

Check the logs of the Nginx pod to ensure it's receiving the environment variables from the secret.

# Get the pod name
POD_NAME=$(kubectl get pods -n default -l app=nginx-secret-consumer -o jsonpath='{.items[0].metadata.name}')

# View logs
kubectl logs $POD_NAME -n default

Expected Output (similar to):

MY_API_KEY=supersecretapikey
MY_DB_PASSWORD=db_strong_password

This confirms that your application can successfully consume the secrets that were originally encrypted as a SealedSecret.

Production Considerations

While Sealed Secrets significantly enhance security for secrets in GitOps workflows, there are several key considerations for production deployments:

1. Key Rotation: The private key used by the Sealed Secrets controller is critical. Implement a strategy for key rotation. This typically involves generating a new key, re-encrypting all your SealedSecret objects with the new public key, and then deploying the new key to the controller. The controller supports multiple keys, allowing for a smooth transition. For more complex key management, consider integrating with external KMS solutions, though Sealed Secrets doesn't directly support this out-of-the-box for its encryption key.
2. Backup and Disaster Recovery: Back up the Sealed Secrets private key securely. Without it, you cannot decrypt existing SealedSecret objects if the controller is reinstalled or moves to a new cluster. Store it in an encrypted, off-cluster location.
3. RBAC for kubeseal: While kubeseal runs on your workstation, ensure that the users or CI/CD pipelines running kubeseal only have read access to the public key (e.g., via a Service Account with appropriate RBAC if fetching automatically) and not to the private key or any sensitive secrets in the cluster.
4. Scope of Secrets: Always prefer the strictest possible scope (strict or namespace-wide) for your SealedSecret objects. cluster-wide should be used sparingly as it allows decryption into any namespace, potentially increasing the attack surface.
5. CI/CD Integration: Integrate kubeseal into your CI/CD pipeline. When a new secret is needed or an existing one is updated, the pipeline should encrypt the plaintext secret into a SealedSecret and commit it to your Git repository. This ensures that plaintext secrets never touch your Git repo.
6. Alternatives and Complementary Tools: For more advanced secret management needs, especially those requiring dynamic secret generation, audit trails, and fine-grained access control, consider solutions like HashiCorp Vault. Sealed Secrets can still be used for static secrets, and Vault for dynamic ones. For securing other aspects of your Kubernetes cluster, explore tools like Sigstore and Kyverno for supply chain security, or Kubernetes Network Policies for traffic isolation.
7. Monitoring and Alerting: Monitor the Sealed Secrets controller for any errors or failures in decrypting secrets. Ensure it has sufficient resources and is healthy.
8. Audit Logs: Ensure your Kubernetes API server audit logs are configured to track access to Secret objects. While Sealed Secrets protects secrets at rest, API access to decrypted secrets still needs to be monitored.

Troubleshooting

1. kubeseal cannot connect to the cluster to fetch the public key.

Issue: When running kubeseal without --cert, it fails with an error like "Error: could not get secret from cluster: Get "https://kubernetes.default.svc/..." connection refused."

Solution:

This usually means your kubectl context is not correctly set, or there's a network issue preventing kubeseal from reaching your cluster's API server.

• Verify your kubectl context: kubectl config current-context
• Ensure your cluster is running and accessible.
• If you are behind a firewall or VPN, ensure it allows connections to your cluster's API server.
• Alternatively, manually fetch the public key (as shown in Step 3) and use kubeseal --cert sealed-secrets-cert.pem ....

2. Sealed Secret is created, but the corresponding Kubernetes Secret is not.

Issue: You apply a SealedSecret, but kubectl get secret shows no secret, or the controller logs show errors.

Solution:

• Check the Sealed Secrets controller logs:

  
  kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets
  

  Look for errors related to decryption or secret creation.

• Verify the SealedSecret manifest:
  • Ensure the namespace in the SealedSecret manifest matches the target namespace.
  • If using --scope strict, ensure the name and namespace in the SealedSecret's metadata and template.metadata sections are identical.
  • Ensure the public key used for encryption matches the key used by the controller in the cluster. If the key was rotated or the controller reinstalled with a new key, you might need to re-encrypt your SealedSecret.

3. "permission denied" or "forbidden" when applying SealedSecret.

Issue: You get an RBAC error when trying to kubectl apply -f mysealedsecret.yaml.

Solution:

Ensure the Kubernetes user or Service Account you are using has permissions to create SealedSecret objects in the target namespace. You need create, get, update, patch, and delete permissions on sealedsecrets.bitnami.com resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sealed-secret-creator
  namespace: default
rules:
- apiGroups: ["bitnami.com"]
  resources: ["sealedsecrets"]
  verbs: ["create", "get", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-sealed-secret-creator
  namespace: default
subjects:
- kind: User # or ServiceAccount
  name: your-user-name # or your-service-account-name
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: sealed-secret-creator
  apiGroup: rbac.authorization.k8s.io

4. Secrets are not updated after modifying the SealedSecret.

Issue: You modify mysealedsecret.yaml (e.g., change an encrypted value) and re-apply, but the corresponding Kubernetes Secret doesn't reflect the changes.

Solution:

The Sealed Secrets controller watches for changes to SealedSecret objects.

• Ensure you re-encrypted the original plaintext secret after making changes. Simply editing the encryptedData in the SealedSecret manifest won't work as it's an opaque encrypted blob. You must rerun kubeseal < myupdatedsecret.yaml > mysealedsecret.yaml.
• After re-encrypting, ensure you apply the updated mysealedsecret.yaml to the cluster.
• Check the controller logs for any errors during the update process.

5. Sealed Secrets controller pod is not starting.

Issue: The sealed-secrets-controller pod is stuck in Pending or CrashLoopBackOff state.

Solution:

• Pending: Check events for the pod: kubectl describe pod -n kube-system . This often indicates resource constraints (e.g., no available nodes or insufficient CPU/memory) or scheduling issues (e.g., taints/tolerations). Consider Karpenter for efficient node provisioning if you're facing scaling issues.
• CrashLoopBackOff: Check the pod logs: kubectl logs -n kube-system . This usually points to configuration errors or issues with the controller's internal state. Reinstalling the Helm chart might resolve transient issues.
• Ensure the necessary RBAC resources (ServiceAccount, ClusterRole, ClusterRoleBinding) created by the Helm chart are present and correctly configured.

6. Decryption fails after cluster migration or controller reinstallation.

Issue: After moving to a new cluster or reinstalling the Sealed Secrets controller, existing SealedSecret objects are not decrypted.

Solution:

This is almost certainly because the new controller instance has generated a new private/public key pair.

• You must use the exact same private key that was used to encrypt the SealedSecret objects.
• If you have a backup of the original private key (e.g., sealed-secrets-key Secret in kube-system), you can restore it to the new cluster before installing the controller, or configure the Helm chart to use your existing key.
• If the original key is lost, you will have to regenerate all your SealedSecret objects by encrypting the plaintext secrets again using the new controller's public key. This highlights the importance of key backup and rotation strategies.

FAQ Section

Q1: Can I use Sealed Secrets with GitOps tools like Argo CD or Flux CD?

A1: Absolutely! Sealed Secrets are designed for GitOps. You store the encrypted SealedSecret manifests in your Git repository. Argo CD or Flux CD will then sync these manifests to your cluster. The in-cluster Sealed Secrets controller will automatically decrypt them into standard Kubernetes Secrets, which your applications can then consume. This ensures that plaintext secrets never reside in your Git repository.

Q2: How do I rotate the encryption key for Sealed Secrets?

A2: Key rotation involves generating a new key pair for the Sealed Secrets controller. The process generally is:

1. Generate a new private key and certificate outside the cluster.
2. Create a new Kubernetes Secret (e.g., sealed-secrets-key-new) in kube-system with this new key.
3. Configure the Sealed Secrets controller to use both the old and new keys (it supports multiple keys for graceful transitions).
4. Use the new public key to re-encrypt all your existing SealedSecret objects.
5. Once all secrets are re-encrypted with the new key and verified, you can remove the old key from the controller.

Refer to the official Sealed Secrets documentation on key rotation for detailed steps.

Q3: Is it possible to decrypt a Sealed Secret without access to the Kubernetes cluster?

A3: No, that's the core security feature of Sealed Secrets. Decryption can only happen inside the Kubernetes cluster by the Sealed Secrets controller, which holds the necessary private key. The kubeseal CLI tool only performs encryption using the public key; it cannot decrypt. This ensures that your sensitive data remains encrypted in your Git repository and is only exposed as plaintext within the secure boundaries of your cluster.

Q4: What happens if the Sealed Secrets controller goes down?

A4: If the Sealed Secrets controller goes down, existing Kubernetes Secrets that were previously decrypted will remain available to your applications. However, no new SealedSecret objects will be decrypted, and no existing SealedSecret objects will be updated if their source changes. Your applications will continue to run with their current secrets, but any secret management operations will be paused until the controller recovers.

Q5: Can I use Sealed Secrets to manage secrets across multiple clusters?

A5: Yes, but with a nuance. Each Kubernetes cluster typically has its own Sealed Secrets controller with its own unique private/public key pair. This means a SealedSecret encrypted for Cluster A cannot be decrypted by the controller in Cluster B unless both controllers are configured to use the same key. For multi-cluster deployments, you generally have two options:

• Encrypt secrets separately for each cluster using their respective public keys.
• Maintain a single, shared private key across all your Sealed Secrets controllers in different clusters. This simplifies encryption but requires careful management of the shared key.

The choice depends on your security posture and operational complexity tolerance.

Cleanup Commands

To remove all resources created during this tutorial:

# Delete the Nginx deployment

kubectl delete -f nginx-deployment.yaml

# Delete the SealedSecret

kubectl delete -f mysealedsecret.yaml
# Delete the temporary plaintext secret file (if you created it)

rm mysecret.yaml
# Delete the temporary public key file (if you created it)

rm sealed-secrets-cert.pem
# Uninstall the Sealed Secrets controller via Helm

helm uninstall sealed-secrets --namespace kube-system
# If you created the namespace specifically for sealed-secrets and nothing else, you can delete it.

# Be cautious if other system components