Secure Kubernetes Secrets: Encrypt with Sealed Secrets

Introduction

Managing sensitive information like API keys, database passwords, and private certificates in Kubernetes is a perennial challenge for developers and operators alike. While Kubernetes provides a native Secret resource, it stores data as base64-encoded strings, which is merely obfuscation, not encryption. Anyone with API access to the cluster can easily decode these secrets, posing a significant security risk. This inherent vulnerability means that if your cluster is compromised, or even if an authorized user has excessive permissions, your sensitive data is exposed.

This is where Sealed Secrets by Bitnami comes to the rescue. Sealed Secrets provides a robust solution to encrypt your Kubernetes Secrets directly into a custom resource, SealedSecret, which can be safely stored in Git. This encrypted secret can then be committed to your version control system (VCS) without exposing its plaintext contents. Only the Sealed Secrets controller running within your Kubernetes cluster can decrypt it, transforming it back into a native Kubernetes Secret. This approach bridges the gap between the need for secure secret management and the benefits of GitOps, allowing you to treat secrets as part of your application’s declarative configuration, fully auditable and version-controlled.

In this comprehensive guide, we’ll walk you through the process of installing, configuring, and using Sealed Secrets to encrypt your sensitive data in Kubernetes. We’ll cover everything from initial setup to advanced usage, troubleshooting, and important production considerations, ensuring your secrets remain secure while maintaining a smooth GitOps workflow. Say goodbye to insecure base64-encoded secrets and embrace a truly encrypted, version-controlled approach to secret management.

Prerequisites

Before diving into Sealed Secrets, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.16+ recommended). You can use Minikube, Kind, or any cloud-managed cluster (EKS, GKE, AKS).
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Helm (Optional but Recommended): For easier installation and management of the Sealed Secrets controller. You can find installation guides on the Helm website.
• Basic Kubernetes Knowledge: Familiarity with Kubernetes concepts like Pods, Deployments, Services, and Secrets.
• Git: For version control of your SealedSecret manifests.

Step-by-Step Guide

1. Install the Sealed Secrets Controller

The Sealed Secrets controller is a critical component that runs inside your Kubernetes cluster. Its primary role is to watch for SealedSecret resources, decrypt them using its private key, and then create or update corresponding native Kubernetes Secret objects. This controller is the only entity that holds the decryption key, ensuring that your sensitive data remains encrypted everywhere else.

You can install the controller using a direct manifest or via Helm, which is often preferred for managing Kubernetes applications. For production environments, Helm offers better upgrade paths and configuration management.

Method A: Direct Manifest Installation

This is the simplest way to get started. The manifest includes the Deployment, ServiceAccount, ClusterRole, and ClusterRoleBinding necessary for the controller to function.

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/controller.yaml

Verify Installation (Direct Manifest)

After applying the manifest, verify that the controller Pod is running and healthy in the kube-system namespace (or whichever namespace it was deployed to).

kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

Expected Output:

NAME                                     READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6d47f8c9d4-abcde   1/1     Running   0          2m

Method B: Helm Installation (Recommended)

Helm simplifies the installation and lifecycle management of the controller. It also allows for easier configuration of controller options.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update
helm install sealed-secrets sealed-secrets/sealed-secrets --namespace kube-system --create-namespace

Verify Installation (Helm)

Check the Helm release status and the running Pods.

helm list -n kube-system
kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

Expected Output (Helm list):

NAME            NAMESPACE       REVISION    UPDATED                                 STATUS      CHART                       APP VERSION
sealed-secrets  kube-system     1           2023-10-27 10:30:00.123456789 +0000 UTC Deployed    sealed-secrets-2.1.0        v0.22.0

Expected Output (kubectl get pods):

NAME                                     READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6d47f8c9d4-abcde   1/1     Running   0          2m

2. Install the Kubeseal CLI Tool

The kubeseal CLI tool is an essential client-side utility that encrypts your plaintext Kubernetes Secrets into SealedSecret objects. It uses the public key provided by the controller in your cluster to perform this encryption. This means you do not need to expose your private key locally, enhancing security. The kubeseal tool is available for various operating systems.

Download the latest release for your operating system from the Bitnami Sealed Secrets GitHub releases page. Replace v0.22.0 and the architecture/OS with the appropriate values for your system.

# For Linux AMD64
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/kubeseal-0.22.0-linux-amd64 -O kubeseal
chmod +x kubeseal
sudo mv kubeseal /usr/local/bin/kubeseal

# For macOS AMD64 (Intel Macs)
# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/kubeseal-0.22.0-darwin-amd64 -O kubeseal
# chmod +x kubeseal
# sudo mv kubeseal /usr/local/bin/kubeseal

# For macOS ARM64 (Apple Silicon)
# wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/kubeseal-0.22.0-darwin-arm64 -O kubeseal
# chmod +x kubeseal
# sudo mv kubeseal /usr/local/bin/kubeseal

Verify Kubeseal Installation

Check the version of the installed kubeseal to ensure it’s correctly installed and accessible in your PATH.

kubeseal --version

Expected Output:

kubeseal version: v0.22.0

3. Create a Plaintext Kubernetes Secret

First, you need to define your sensitive data in a standard Kubernetes Secret manifest. This manifest will contain the actual plaintext values. Remember, this file should not be committed to Git. We’ll use it as input for kubeseal.

Let’s create a secret named my-app-credentials with a database password and an API key. Using --dry-run=client -o yaml is a safe way to generate the YAML without actually creating the Secret in the cluster.

kubectl create secret generic my-app-credentials \
  --from-literal=db-password=superSecureDBPass! \
  --from-literal=api-key=abc123def456ghi789jkl \
  --dry-run=client -o yaml > secret.yaml

Verify Plaintext Secret Content

Inspect the generated secret.yaml. Notice the data fields are base64 encoded. This is what we want to encrypt.

cat secret.yaml

Expected Output:

apiVersion: v1
data:
  api-key: YWJjMTIzZGVmNDU2Z2hpNzg5amts
  db-password: c3VwZXJTZWN1cmVEQlBhc3Mh
kind: Secret
metadata:
  creationTimestamp: null
  name: my-app-credentials

4. Seal the Secret

Now, use the kubeseal tool to encrypt the secret.yaml into a SealedSecret resource. kubeseal fetches the public key from the controller running in your cluster to perform the encryption. This encrypted SealedSecret is safe to commit to Git.

You have a few options for how the SealedSecret behaves:

• --scope cluster-wide: The secret can be decrypted by any controller in any namespace.
• --scope namespace-wide: The secret can only be decrypted by a controller in the same namespace. This is useful for multi-tenant clusters.
• --scope strict: The secret can only be decrypted by a controller in the same namespace and with the exact same name. This offers the highest level of isolation.

For most applications, namespace-wide or cluster-wide is sufficient. We’ll use cluster-wide for simplicity here.

kubeseal --scope cluster-wide --format yaml < secret.yaml > sealedsecret.yaml

Verify Sealed Secret Content

Examine the sealedsecret.yaml. You’ll see the encryptedData field, which contains the ciphertext. This file is now safe to commit to your Git repository.

cat sealedsecret.yaml

Expected Output (will vary due to encryption):

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-app-credentials
  namespace: default
spec:
  encryptedData:
    api-key: AgCgJ3F0Xk... # long encrypted string
    db-password: AgD0y3S9H... # long encrypted string
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: my-app-credentials
      namespace: default
    type: Opaque

5. Apply the Sealed Secret to Your Cluster

Once you have your sealedsecret.yaml, apply it to your Kubernetes cluster. The Sealed Secrets controller will detect this new resource, decrypt it using its private key, and create a standard Kubernetes Secret with the same name and namespace.

kubectl apply -f sealedsecret.yaml

Verify Decrypted Secret

Check if the native Kubernetes Secret has been created. You should see it listed. Then, inspect its contents to ensure the data has been correctly decrypted.

kubectl get secret my-app-credentials
kubectl get secret my-app-credentials -o jsonpath='{.data}' | base64 --decode

Expected Output (kubectl get secret):

NAME                 TYPE     DATA   AGE
my-app-credentials   Opaque   2      10s

Expected Output (jsonpath and base64 decode):

{"api-key":"abc123def456ghi789jkl","db-password":"superSecureDBPass!"}

Success! Your sensitive data is now stored securely as an encrypted SealedSecret in Git, and only the controller in your cluster can transform it into a usable native Secret.

6. Using the Secret in a Pod

Once the SealedSecret has been decrypted by the controller and the native Kubernetes Secret is available, you can consume it in your Pods just like any other Kubernetes Secret. This typically involves mounting it as a volume or injecting its data as environment variables. For more advanced secret management patterns, consider integrating with tools like Secret Store CSI Driver.

Let’s create a simple Pod that reads the db-password from our my-app-credentials secret.

# pod-consumer.yaml
apiVersion: v1
kind: Pod
metadata:
  name: secret-consumer-pod
spec:
  containers:
  - name: my-app
    image: busybox
    command: ["sh", "-c"]
    args:
    - echo "DB Password: $(cat /etc/secrets/db-password)";
    - sleep 3600
    volumeMounts:
    - name: app-secrets
      mountPath: "/etc/secrets"
      readOnly: true
  volumes:
  - name: app-secrets
    secret:
      secretName: my-app-credentials
  restartPolicy: Never

kubectl apply -f pod-consumer.yaml

Verify Pod Consumption

Check the logs of the Pod to ensure it successfully read the secret data.

kubectl logs secret-consumer-pod

Expected Output:

DB Password: superSecureDBPass!

This confirms that the entire workflow, from sealing to decryption and consumption, is working as expected. For more details on Kubernetes networking and security, you might want to explore our Network Policies Security Guide.

Production Considerations

While Sealed Secrets significantly enhances the security posture of your Kubernetes secrets, deploying it in a production environment requires careful planning and adherence to best practices:

1. Key Management and Rotation:
   • Backup the Private Key: The controller’s private key is crucial. If you lose it, you lose the ability to decrypt your existing SealedSecrets. Back up the private key (stored in a native Kubernetes Secret, usually sealed-secrets-key in the controller’s namespace) to a secure, off-cluster location.
   • Key Rotation: Regularly rotate the decryption key. Sealed Secrets supports this by creating a new key pair and allowing the controller to decrypt with older keys while sealing new secrets with the latest key. This is a manual process and requires re-sealing all your secrets with the new key. Refer to the official documentation on key rotation.
2. Controller High Availability:
   • For production, ensure the Sealed Secrets controller deployment has multiple replicas (e.g., 3) and appropriate anti-affinity rules to spread them across different nodes. This prevents a single point of failure.
   • Consider resource requests and limits for the controller Pods to ensure stable operation.
3. RBAC and Permissions:
   • The Sealed Secrets controller requires specific RBAC permissions to create and manage Secrets. Ensure these permissions are scoped appropriately and follow the principle of least privilege.
   • Restrict who can apply SealedSecret resources to your cluster. Only trusted CI/CD pipelines or administrators should have this capability.
   • Limit access to the controller’s private key Secret itself.
4. Auditing and Logging:
   • Ensure that your cluster’s audit logs are configured to capture events related to SealedSecret creation, updates, and the resulting Secret creations. This helps in tracking who did what and when.
   • Monitor the controller’s logs for any errors or unusual activity. Tools like eBPF Observability with Hubble can provide deep insights into network and application behavior.
5. CI/CD Integration:
   • Integrate kubeseal into your CI/CD pipelines. This allows automated encryption of secrets before they are committed to Git or applied to the cluster.
   • Ensure the CI/CD environment has access to the cluster’s public key (via kubeseal‘s default behavior or by explicitly passing it) but never the private key.
6. Namespace Isolation and Scope:
   • Leverage --scope namespace-wide or --scope strict for improved isolation in multi-tenant environments. This prevents secrets sealed for one namespace from being accidentally or maliciously decrypted in another.
   • Be mindful of the implications of each scope option on your deployment strategy.
7. Secret Store CSI Driver (Advanced):
   • For even higher security, consider integrating Sealed Secrets with the Kubernetes Secret Store CSI Driver. This allows Pods to fetch secrets directly from external secret management systems (like AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) at runtime, preventing secrets from ever residing in etcd in plaintext, even temporarily. Sealed Secrets can be used to store references or credentials for these external systems.
8. Network Policies:
   • Implement Kubernetes Network Policies to restrict network access to the Sealed Secrets controller and the Secrets it creates. This adds another layer of defense against unauthorized access.
   • For enhanced pod-to-pod encryption, consider solutions like Cilium WireGuard Encryption.

Troubleshooting

Here are some common issues you might encounter with Sealed Secrets and their solutions:

1. Kubeseal Cannot Find Controller Public Key

Issue: When running kubeseal, you get an error like Error: could not fetch public key: Get "https://kubernetes.default.svc/api/v1/namespaces/kube-system/secrets/sealed-secrets-key": dial tcp 10.96.0.1:443: connect: connection refused or similar network/authentication errors.

Solution:

• Controller Not Running: Ensure the Sealed Secrets controller is running in your cluster. Check its Pod status in the kube-system namespace.

  
  kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets
  

• Kubeconfig Context: Verify your kubectl context is pointing to the correct cluster where the controller is deployed.

  
  kubectl config current-context
  

• Network Connectivity: If running kubeseal from outside the cluster, ensure your local machine can reach the Kubernetes API server. If running inside a CI/CD pipeline, ensure the runner has network access to the cluster.
• RBAC Issues: The user or service account running kubeseal needs permission to read the public key from the controller. This is usually handled automatically if kubeseal can access the API server with a valid context, but check if custom RBAC is interfering.

2. SealedSecret is Applied, but Native Secret is Not Created

Issue: You apply a SealedSecret manifest, but no corresponding native Kubernetes Secret appears.

Solution:

• Controller Logs: Check the logs of the Sealed Secrets controller Pod. This is the most common place to find the root cause.

  
  kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets
  

• Incorrect Scope: If you sealed with --scope namespace-wide or --scope strict, ensure the SealedSecret is applied to the exact namespace and/or has the exact name as expected by the scope. If the controller is in a different namespace than the SealedSecret, this could be an issue for namespace-scoped secrets.
• Key Mismatch: If the controller’s private key has been rotated or replaced, an older SealedSecret might not be decryptable by the new key. You would need to re-seal the secret with the current public key.
• CRD Not Found: Ensure the SealedSecret Custom Resource Definition (CRD) is installed in your cluster.

  
  kubectl get crd sealedsecrets.bitnami.com
  

3. “Forbidden” Error When Applying SealedSecret

Issue: You get an RBAC “Forbidden” error when trying to apply a SealedSecret.

Solution:

• User Permissions: The Kubernetes user or service account you are using does not have permissions to create or update SealedSecret resources. Grant the necessary create/update permissions for sealedsecrets.bitnami.com resources.

4. Decrypted Secret Data is Incorrect or Missing

Issue: The native Kubernetes Secret is created, but the data within it is garbled, incomplete, or not what you expected.

Solution:

• Input Secret Incorrect: Double-check the original secret.yaml that you used as input for kubeseal. Ensure the data was correctly base64-encoded and the values were as intended.
• Re-seal and Re-apply: The safest approach is to delete the existing SealedSecret, regenerate the plaintext secret.yaml, re-seal it, and then re-apply. This ensures a clean slate.
• Controller Version: Ensure your kubeseal CLI version is compatible with your controller version. While generally backward compatible, significant version differences might cause issues.

5. Unable to Delete SealedSecret

Issue: You try to delete a SealedSecret, but it remains stuck in a terminating state, or the associated native Secret is not deleted.

Solution:

• Finalizers: The Sealed Secrets controller adds a finalizer to the SealedSecret to ensure the native Secret is cleaned up. If the controller is not running or is unhealthy, the finalizer might prevent deletion.
  • Check controller logs and status.
  • If the controller is truly gone and you need to force delete, you might have to manually remove the finalizer (use with caution!):

    
    kubectl edit sealedsecret my-app-credentials
    # Remove the finalizer entry under metadata.finalizers
    

• Native Secret Deletion: The native Secret should be deleted automatically when the SealedSecret is deleted. If it’s not, you might have to manually delete the native Secret.

6. Performance Issues with Many Secrets

Issue: In clusters with a very large number of SealedSecrets, the controller might consume significant resources or become slow.

Solution:

• Resource Limits: Adjust the resource requests and limits for the Sealed Secrets controller Deployment in the kube-system namespace.

  
  # Example for increasing resources
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 256Mi
  

• Horizontal Scaling: While the controller is generally single-instance, if you have a very high volume of updates or creations, consider if your cluster size necessitates more robust infrastructure for the controller itself.
• Monitoring: Use Prometheus and Grafana or other observability tools (like eBPF Observability with Hubble) to monitor the controller’s CPU, memory, and API request usage.

FAQ Section

1. Is it safe to commit sealedsecret.yaml to a public Git repository?

Yes, that is the primary purpose of Sealed Secrets. The sealedsecret.yaml contains only encrypted data (ciphertext) and the public key used for encryption. Without the corresponding private key, which resides solely within your Kubernetes cluster’s controller, the data cannot be decrypted. So, it’s safe to commit to public or private Git repositories.

2. What happens if I delete the Sealed Secrets controller?

If you delete the Sealed Secrets controller, existing native Kubernetes Secrets that were created by the controller will remain in your cluster. However, no new SealedSecret resources will be decrypted, and existing SealedSecrets will not be updated if their source Secret changes. If you were to lose the controller’s private key without a backup, any SealedSecrets that haven’t yet been decrypted (e.g., in a new cluster) would become permanently undecryptable.

3. How do I rotate the decryption key?

Key rotation in Sealed Secrets involves a multi-step process:

1. Generate a new key pair for the controller.
2. Update the controller to use the new key while retaining the old one for decryption of existing secrets.
3. Re-seal all your existing SealedSecrets using the new public key.

This is a manual process detailed in the official Sealed Secrets documentation. It’s crucial for long-term security.

4. Can I use Sealed Secrets with multiple Kubernetes clusters?

Yes, but each cluster will have its own unique Sealed Secrets controller and thus its own unique public/private key pair. This means a SealedSecret encrypted for Cluster A cannot be decrypted by Cluster B’s controller unless you manually transfer Cluster A’s private key to Cluster B (which is generally not recommended for security). For multi-cluster deployments, you would typically re-seal secrets for each target cluster, or use a secret management system that integrates with a Secret Store CSI driver.

5. How does Sealed Secrets compare to other secret management solutions like HashiCorp Vault or cloud provider secret managers?

Sealed Secrets is simpler and more opinionated, focused specifically on enabling GitOps for Kubernetes native Secrets. It encrypts secrets at rest in Git.

• HashiCorp Vault or cloud provider solutions (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) are full-fledged secret management systems. They offer features like dynamic secrets, fine-grained access control, secret leasing, and robust auditing.
• Integration: Sealed Secrets can complement these. For instance, you could store the credentials needed to access Vault in a Sealed Secret, or use the Secret Store CSI Driver to fetch secrets from Vault or cloud providers at runtime, with Sealed Secrets potentially managing the CSI driver’s configuration or credentials.

Sealed Secrets is excellent for teams prioritizing GitOps and simplicity for static secrets within Kubernetes, while external secret managers are better for complex, dynamic, and enterprise-scale secret needs.

Cleanup Commands

To remove all resources created during this tutorial, follow these steps:

1. Delete the Pod:

kubectl delete -f pod-consumer.yaml
    

2. Delete the Sealed Secret:

kubectl delete -f sealedsecret.yaml
    

3. Delete the Plaintext Secret (if it was accidentally applied):

kubectl delete -f secret.yaml # Or kubectl delete secret my-app-credentials
    

4. Uninstall the Sealed Secrets Controller:
   

   If installed with Helm:

   
   helm uninstall sealed-secrets -n kube-system
           

   If installed with direct manifest:

   
   kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.22.0/controller.yaml
           

5. Remove Kubeseal CLI:

sudo rm /usr/local/bin/kubeseal
    

Next Steps / Further Reading

Congratulations on mastering Sealed Secrets! Here are some next steps and resources to deepen your knowledge:

• Explore Advanced Kubeseal Options: Dive deeper into kubeseal‘s capabilities, including specifying a public key directly, different scopes, and more. Check the official Kubeseal usage documentation.
• Key Rotation Strategies: Understand and plan for key rotation in your production environment. This is a critical security practice.
• GitOps Integration: Integrate Sealed Secrets into your CI/CD pipeline using tools like Argo CD or Flux CD for a fully automated GitOps workflow.
• Secret Store CSI Driver: Investigate the Kubernetes Secret Store CSI Driver for dynamic secret injection from external secret management systems.
• Kubernetes Security Best Practices: Continue to harden your Kubernetes clusters. Our Sigstore and Kyverno Security guide offers insights into supply chain security, and Kubernetes Network Policies can further secure your workloads.
• Kubernetes Networking: For a deeper dive into how traffic flows and can be secured, explore topics like the
  Was this article helpful?

  Yes No