Secure Kubernetes Secrets: Sealed Secrets Encryption

# Install Sealed Secrets Controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system --create-namespace

# Fetch the public key
kubeseal --fetch-cert > sealed-secrets-public-key.pem

# Create a Kubernetes Secret YAML (e.g., my-secret.yaml)
# Encrypt it into a SealedSecret
kubeseal --format yaml < my-secret.yaml > my-sealed-secret.yaml

# Apply the SealedSecret to your cluster
kubectl apply -f my-sealed-secret.yaml

# Verify the decrypted Secret
kubectl get secret my-secret -o yaml

Introduction

Kubernetes has revolutionized how we deploy and manage applications, offering unparalleled flexibility and scalability. However, managing sensitive information like API keys, database credentials, and private certificates within this ecosystem presents a unique challenge. While Kubernetes provides a built-in Secret resource, it’s crucial to understand a fundamental misconception: Kubernetes Secrets are not encrypted by default. They are merely base64 encoded, which is a reversible process and offers no protection against unauthorized access to your cluster’s etcd database or configuration files.

This lack of inherent encryption means that anyone with read access to your Kubernetes cluster or its underlying storage could potentially decode and expose your sensitive data. Storing these base64 encoded secrets directly in version control systems like Git is an absolute security no-go, as it would expose them to anyone with repository access. This dilemma forces developers and operators to seek secure methods for handling secrets, often leading to complex external solutions or insecure workarounds. Enter Sealed Secrets, a powerful tool that bridges this security gap by allowing you to encrypt your secrets and safely store them in Git, bringing them inline with GitOps principles.

Sealed Secrets, developed by Bitnami, provides a robust solution by allowing you to encrypt your Kubernetes Secrets into a new custom resource called SealedSecret. This encrypted resource can then be safely committed to your Git repository. A controller running in your Kubernetes cluster holds the decryption key. When you apply a SealedSecret to your cluster, the controller decrypts it on the fly and creates a standard Kubernetes Secret resource, accessible only within that specific cluster. This approach ensures that your sensitive data is encrypted at rest, in transit, and within your version control system, while still offering the convenience of declarative secret management. In this comprehensive guide, we’ll walk through the process of installing, configuring, and utilizing Sealed Secrets to bolster your Kubernetes security posture.

Prerequisites

Before diving into Sealed Secrets, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.16 or higher). You can use Minikube, Kind, or a cloud-managed cluster (EKS, GKE, AKS).
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• helm: The Kubernetes package manager, used for installing the Sealed Secrets controller. Follow the Helm installation guide.
• kubeseal: The client-side CLI tool for encrypting secrets. We’ll install this in the first step.
• Basic understanding of Kubernetes Secrets: Familiarity with how Kubernetes Secrets work and their limitations.
• Git: A basic understanding of Git for version control.

Step-by-Step Guide

Step 1: Install the kubeseal CLI Tool

The kubeseal command-line interface (CLI) is essential for encrypting your secrets into SealedSecret objects. This tool uses the public key from your Sealed Secrets controller to perform the encryption locally on your machine. It’s crucial to install it before you can start sealing secrets.

There are several ways to install kubeseal, depending on your operating system. We’ll cover installation for Linux/macOS using Homebrew or direct download, and for Windows using Chocolatey.

# For macOS or Linux (using Homebrew)
brew install kubeseal

# For Linux (direct download and install)
# Replace x.y.z with the latest release version from https://github.com/bitnami-labs/sealed-secrets/releases
OS=$(uname -s)
ARCH=$(uname -m)
case $ARCH in
    x86_64) ARCH=amd64 ;;
    arm64) ARCH=arm64 ;;
esac
RELEASE_VERSION=$(curl -s https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest | grep tag_name | cut -d : -f 2 | tr -d \"\,v\ )

wget "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${RELEASE_VERSION}/kubeseal-${RELEASE_VERSION}-${OS}-${ARCH}"
sudo install -m 755 "kubeseal-${RELEASE_VERSION}-${OS}-${ARCH}" /usr/local/bin/kubeseal
rm "kubeseal-${RELEASE_VERSION}-${OS}-${ARCH}"

# For Windows (using Chocolatey)
choco install kubeseal

Verify:

After installation, verify that kubeseal is correctly installed and accessible by checking its version. This command should output the version number of the client tool.

kubeseal --version

Expected Output:

kubeseal version: v0.24.1 # Version might vary

Step 2: Install the Sealed Secrets Controller in Your Cluster

The Sealed Secrets controller is the heart of the system. It runs within your Kubernetes cluster, watches for SealedSecret resources, decrypts them using its private key, and then creates standard Kubernetes Secret objects. Installing it is straightforward using Helm.

We’ll install the controller into the kube-system namespace, which is a common practice for system-level components. Ensure you have Helm installed and configured before proceeding.

# Add the Sealed Secrets Helm repository
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

# Update your Helm repositories
helm repo update

# Install the Sealed Secrets controller
# We install it in the 'kube-system' namespace for general availability
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system \
  --create-namespace

Verify:

After installation, confirm that the Sealed Secrets controller pod is running and healthy in your cluster. You should see a pod named similar to sealed-secrets-controller-... in the kube-system namespace.

kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

Expected Output:

NAME                                       READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-f7c89f55c-abcde   1/1     Running   0          2m

Step 3: Fetch the Sealed Secrets Public Key

To encrypt your secrets, the kubeseal CLI needs access to the public key of the Sealed Secrets controller running in your cluster. This public key is used for encryption, ensuring that only the corresponding private key (held by the controller) can decrypt the secrets. This is a fundamental principle of asymmetric encryption.

The kubeseal tool can directly fetch this public key from the controller, or you can retrieve it manually from a specific Kubernetes Secret created by the controller. It’s good practice to save this public key to a file, especially if you plan to use it offline or in automated CI/CD pipelines.

# Fetch the public key and save it to a file
# This command connects to your cluster to get the public key from the controller.
kubeseal --fetch-cert > sealed-secrets-public-key.pem

Verify:

Check if the sealed-secrets-public-key.pem file was created and contains a valid public key certificate. You can inspect its content or use openssl to verify it.

cat sealed-secrets-public-key.pem

Expected Output (truncated):

-----BEGIN CERTIFICATE-----
MIIDezCCAmOgAwIBAgIRAMd...
...
-----END CERTIFICATE-----

Step 4: Create a Kubernetes Secret Manifest

Before you can seal a secret, you first need to define it as a standard Kubernetes Secret manifest. This manifest specifies the secret’s name, namespace, and the actual data you want to protect. Remember, at this stage, the data in this manifest should be base64 encoded, just like any regular Kubernetes Secret. This manifest will serve as the input for the kubeseal tool.

For demonstration, let’s create a secret named my-app-secret containing a database password and an API key. We’ll define it in the default namespace.

# my-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
data:
  DB_PASSWORD: bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk # base64 encoded "my-super-secret-password"
  API_KEY: YXBpa2V5LTEyMzQ1Njc4OTA=          # base64 encoded "apikey-1234567890"

To generate the base64 encoded values, you can use the following commands:

echo -n "my-super-secret-password" | base64
echo -n "apikey-1234567890" | base64

Expected Output for base64 encoding:

bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk
YXBpa2V5LTEyMzQ1Njc4OTA=

Verify:

Do not apply this raw secret to your cluster for security reasons! The purpose of this step is just to create the input YAML file. You can verify the file content:

cat my-secret.yaml

Expected Output: (Matches the YAML above)

apiVersion: v1
kind: Secret
metadata:
  name: my-app-secret
  namespace: default
type: Opaque
data:
  DB_PASSWORD: bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk
  API_KEY: YXBpa2V5LTEyMzQ1Njc4OTA=

Step 5: Seal the Kubernetes Secret

Now that you have your raw Kubernetes Secret manifest and the Sealed Secrets public key, it’s time to encrypt, or “seal,” it. The kubeseal CLI tool takes your raw secret manifest as input and, using the public key, encrypts the sensitive data fields into a SealedSecret custom resource.

The resulting SealedSecret YAML will contain the encrypted data, which is safe to commit to version control. When applied to the cluster, the Sealed Secrets controller will decrypt it back into a standard Kubernetes Secret.

# Seal the secret using the previously fetched public key
# The --scope cluster-wide flag ensures the secret can be decrypted by any controller in the cluster.
# For more granular control, you can use namespace-scoped or strict-scoped secrets.
kubeseal --cert sealed-secrets-public-key.pem \
         --format yaml \
         --scope cluster-wide \
         < my-secret.yaml > my-sealed-secret.yaml

Verify:

Inspect the generated my-sealed-secret.yaml file. You will notice that the data fields are now replaced with an encryptedData field containing large, unreadable encrypted strings. This is your securely sealed secret.

cat my-sealed-secret.yaml

Expected Output (truncated, encrypted data will vary):

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-app-secret
  namespace: default
spec:
  encryptedData:
    API_KEY: AgC... # Long encrypted string
    DB_PASSWORD: AgC... # Long encrypted string
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: my-app-secret
      namespace: default
    type: Opaque

Important Note: The --scope flag is crucial. cluster-wide secrets can be decrypted by any Sealed Secrets controller in the cluster, regardless of namespace. namespace-wide secrets are bound to a specific namespace and can only be decrypted by a controller in that namespace. strict secrets are bound to both namespace and name, offering the highest level of specificity but requiring the secret name to be known at sealing time. For most GitOps scenarios, cluster-wide or namespace-wide are common. For more details on scoping, refer to the Sealed Secrets documentation.

Step 6: Apply the Sealed Secret to Your Cluster

With the my-sealed-secret.yaml file created, you can now safely commit it to your Git repository and apply it to your Kubernetes cluster. When you apply this SealedSecret, the Sealed Secrets controller will automatically detect it, decrypt its contents using its private key, and create a standard Kubernetes Secret with the original sensitive data.

# Apply the sealed secret to your Kubernetes cluster
kubectl apply -f my-sealed-secret.yaml

Verify:

First, verify that the SealedSecret custom resource itself exists. Then, and more importantly, verify that the controller has successfully decrypted it and created the corresponding standard Kubernetes Secret. You should be able to see the my-app-secret in the default namespace.

# Check if the SealedSecret resource exists
kubectl get sealedsecret my-app-secret -n default

# Check if the standard Secret resource has been created
kubectl get secret my-app-secret -n default

# Inspect the content of the decrypted Secret (CAUTION: this exposes the plaintext data!)
# Only do this in a secure environment for verification.
kubectl get secret my-app-secret -n default -o yaml

Expected Output for kubectl get sealedsecret:

NAME            AGE
my-app-secret   1m

Expected Output for kubectl get secret:

NAME            TYPE      DATA   AGE
my-app-secret   Opaque    2      1m

Expected Output for kubectl get secret -o yaml (data will be base64 decoded if you pipe to base64 -d):

apiVersion: v1
data:
  API_KEY: YXBpa2V5LTEyMzQ1Njc4OTA=
  DB_PASSWORD: bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk
kind: Secret
metadata:
  annotations:
    sealedsecrets.bitnami.com/generated-by: kubeseal
    sealedsecrets.bitnami.com/name: my-app-secret
    sealedsecrets.bitnami.com/namespace: default
  creationTimestamp: "2023-10-27T10:00:00Z"
  name: my-app-secret
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: my-app-secret
    uid: abcdef12-3456-7890-abcd-ef1234567890
resourceVersion: "123456"
uid: fedcba98-7654-3210-fedc-ba9876543210
type: Opaque

You can now use this my-app-secret in your Kubernetes deployments, just like any other secret, by mounting it as a volume or injecting it as environment variables. The sensitive data is now securely managed via GitOps!

Production Considerations

While Sealed Secrets significantly enhances the security of your Kubernetes secrets, there are several key considerations for production environments:

1. Key Management and Rotation:
   • By default, the Sealed Secrets controller generates a self-signed key pair. For production, consider backing up the private key. If the controller’s private key is lost, all existing SealedSecret objects become undecryptable, leading to application downtime.
   • Key rotation is important for long-term security. Sealed Secrets supports this by allowing you to generate new keys. However, rotating keys requires re-sealing all existing secrets with the new public key. Plan this carefully.
   • For enhanced security, consider integrating with external Key Management Systems (KMS) like AWS KMS, GCP KMS, or Azure Key Vault to protect the Sealed Secrets private key. While Sealed Secrets doesn’t directly integrate with KMS for sealing/unsealing, you can use KMS to encrypt the private key itself, storing it securely.
2. Scope of Secrets:
   • Understand the implications of cluster-wide, namespace-wide, and strict scopes. cluster-wide is convenient for shared secrets but means any controller in any namespace can decrypt it. namespace-wide and strict offer more granular control but require more careful management, especially if secrets need to be shared across namespaces.
3. Access Control (RBAC):
   • Ensure that only authorized personnel or CI/CD systems have access to the kubeseal CLI tool and the public key.
   • Implement strict Kubernetes RBAC policies to control who can create, update, or delete SealedSecret resources.
   • Similarly, restrict access to the actual Kubernetes Secret resources once they are decrypted. Use Kubernetes Network Policies to control which pods can access the Kube API server to retrieve secrets, and which applications can actually consume them.
4. Auditing and Logging:
   • Enable comprehensive auditing for your Kubernetes API server to track who is creating, updating, or deleting SealedSecret and Secret resources.
   • Monitor the logs of the Sealed Secrets controller for any decryption failures or suspicious activity.
5. CI/CD Integration:
   • Integrate kubeseal into your CI/CD pipelines to automate the sealing process. This ensures that secrets are always sealed before being committed to Git and deployed.
   • Avoid manually sealing secrets if possible, as this introduces human error. Automate the process as part of your GitOps workflow.
6. Backup and Restore:
   • Regularly back up your Kubernetes cluster’s etcd, which includes your SealedSecret resources.
   • Ensure you have a disaster recovery plan that includes restoring the Sealed Secrets controller and its private key.
7. Alternative Solutions:
   • While Sealed Secrets is excellent for GitOps, consider other solutions like HashiCorp Vault or cloud provider secret managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) if you need more advanced features like dynamic secrets, fine-grained access policies, or secret leasing. Sealed Secrets often complements these tools by providing a GitOps-friendly way to inject secrets into Kubernetes after they’ve been managed externally.

Troubleshooting

1. Sealed Secrets Controller Pod Not Running

Issue: The sealed-secrets-controller pod is stuck in a pending, crashloop, or error state.

Solution:

1. Check the pod’s events for common issues like insufficient resources or image pull errors.
2. Inspect the pod logs for specific error messages.
3. Ensure your cluster has enough resources (CPU, Memory) for the controller.
4. Verify network connectivity to the Helm chart’s image registry.

kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets
kubectl describe pod -n kube-system -l app.kubernetes.io/name=sealed-secrets
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

2. kubeseal --fetch-cert Fails

Issue: You get an error like “Error: could not fetch certificate: secrets ‘sealed-secrets-key’ not found” or “Error: could not fetch certificate: Get “https://kubernetes.default.svc/…” connection refused”.

Solution:

1. Ensure the Sealed Secrets controller is running and healthy (see issue 1).
2. Verify your kubectl context is correct and pointing to the intended cluster.
3. Check your network connectivity to the Kubernetes API server.
4. Confirm the sealed-secrets-key Secret actually exists in the kube-system namespace. This Secret holds the controller’s private key and public certificate.

kubectl config current-context
kubectl get secret sealed-secrets-key -n kube-system

3. SealedSecret Applied, But Kubernetes Secret Not Created

Issue: You’ve applied a SealedSecret, but the corresponding standard Kubernetes Secret is not appearing in the specified namespace.

Solution:

1. Check the logs of the Sealed Secrets controller for decryption errors. This is the most common cause.
2. Ensure the SealedSecret‘s namespace field matches the namespace where you expect the secret to be created.
3. Verify that the SealedSecret‘s metadata.name and template.metadata.name fields match.
4. If you used a specific certificate with kubeseal --cert, ensure it’s the correct public key for the controller in your cluster.
5. Check the status of the SealedSecret resource itself. It might contain clues.

kubectl get sealedsecret my-app-secret -n default -o yaml
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

4. Permissions Issues (RBAC)

Issue: You get “Error from server (Forbidden)” when trying to create/update SealedSecret or Secret resources.

Solution:

1. Your Kubernetes user or service account lacks the necessary RBAC permissions to manage SealedSecret or Secret resources in the target namespace.
2. Grant your user/service account appropriate create, update, patch, and delete permissions for sealedsecrets.bitnami.com resources and secrets resources.
3. For CI/CD pipelines, ensure the service account used by your automation has these permissions. Consider following RBAC best practices.

# Example of checking your user's permissions
kubectl auth can-i create sealedsecrets --namespace default
kubectl auth can-i create secrets --namespace default

5. Decryption Fails Due to Key Mismatch

Issue: The Sealed Secrets controller logs show errors like “failed to decrypt: ciphertext is not valid” or “key mismatch”.

Solution:

1. This almost always means the SealedSecret was encrypted with a different public key than the one the controller currently possesses.
2. Verify that the sealed-secrets-public-key.pem you used with kubeseal --cert is indeed the public key from the *current* running controller in your cluster.
3. If the controller was reinstalled or its key rotated, you must re-fetch the new public key and re-seal all your secrets.
4. Ensure the --scope flag used during sealing matches the intended behavior for the target cluster.

# Re-fetch the current public key and compare
kubeseal --fetch-cert > new-public-key.pem
diff sealed-secrets-public-key.pem new-public-key.pem

6. Updating a Sealed Secret

Issue: You need to change a secret’s value, but simply editing the SealedSecret YAML doesn’t update the underlying Secret.

Solution:

1. You cannot directly edit the encryptedData in a SealedSecret. You must re-seal the secret.
2. Edit your original my-secret.yaml (or whatever raw secret manifest you use) with the new plaintext values.
3. Re-run the kubeseal command to generate a new my-sealed-secret.yaml.
4. Apply the new my-sealed-secret.yaml to your cluster. The controller will detect the change and update the underlying Kubernetes Secret.

# 1. Edit my-secret.yaml with new values
# 2. Re-seal:
kubeseal --cert sealed-secrets-public-key.pem --format yaml < my-secret.yaml > my-sealed-secret.yaml
# 3. Apply the updated sealed secret:
kubectl apply -f my-sealed-secret.yaml

FAQ Section

Q1: Are Kubernetes Secrets truly secure after being decrypted by Sealed Secrets?

A: Once a SealedSecret is decrypted by the controller, it becomes a standard Kubernetes Secret. At this point, its security relies on the inherent security of your Kubernetes cluster. This means:

• The secret data is stored unencrypted in etcd, the Kubernetes backing store. Ensure etcd is encrypted at rest (e.g., using disk encryption, or Kubernetes API encryption for secrets).
• Access to these secrets should be controlled via Kubernetes RBAC. Only pods and users with appropriate permissions should be able to read them.
• Network traffic within the cluster can expose secrets if not secured. Consider solutions like Cilium WireGuard Encryption or Istio Ambient Mesh for encrypting pod-to-pod communication.

Sealed Secrets solves the “secret in Git” problem; it doesn’t replace overall cluster security best practices.

Q2: Can I use Sealed Secrets with multiple Kubernetes clusters?

A: Yes, but each cluster typically needs its own Sealed Secrets controller and thus its own unique public/private key pair. This means a SealedSecret encrypted for Cluster A cannot be decrypted by the controller in Cluster B. If you need to deploy the same secret to multiple clusters, you would either:

• Generate a separate SealedSecret for each cluster using their respective public keys.
• Use a shared private key across controllers (advanced, requires careful key management).

Q3: What happens if the Sealed Secrets controller is down?

A: If the Sealed Secrets controller is down:

• Existing decrypted Kubernetes Secret objects will remain available and functional for your applications.
• New SealedSecret resources applied to the cluster will not be decrypted, and no corresponding Kubernetes Secret will be created until the controller is back up.
• Updates to existing SealedSecret resources will not propagate to the Kubernetes Secret until the controller is healthy again.

It’s critical to ensure the controller runs with appropriate Kubernetes reliability features (e.g., Deployments with multiple replicas, anti-affinity rules).

Q4: How do I rotate the Sealed Secrets private key?

A: Rotating the private key involves these steps:

1. Generate a new key pair for the controller (often by deleting the existing sealed-secrets-key Secret and letting the controller regenerate, or by manually creating one).
2. Update the Sealed Secrets controller to use the new key.
3. Fetch the new public key using kubeseal --fetch-cert.
4. Re-seal all your existing SealedSecret resources using the new public key. This is the most critical and time-consuming step.
5. Apply the newly sealed secrets to the cluster.

This process requires careful planning and execution to avoid downtime. For more details, refer to the official documentation on key rotation.

Q5: Can I use Sealed Secrets with other GitOps tools like Argo CD or Flux?

A: Absolutely! Sealed Secrets is a perfect fit for GitOps workflows. Tools like Argo CD and Flux CD can directly synchronize SealedSecret resources from your Git repository to your cluster. The GitOps operator simply applies the SealedSecret YAML, and the Sealed Secrets controller handles the decryption within the cluster, creating the standard Kubernetes Secret. This integrates seamlessly, allowing you to keep all your application configurations, including sensitive ones, in Git.

Cleanup Commands

To remove the Sealed Secrets controller and the example secrets from your cluster, follow these steps:

# 1. Delete the example SealedSecret and the resulting Kubernetes Secret

kubectl delete -f my-sealed-secret.yaml

# Or if you want to delete the raw secret (not recommended to apply