Kubernetes Secrets: External Secrets Operator

Kubernetes Secrets Management with External Secrets Operator

Managing sensitive information like API keys, database credentials, and certificates within Kubernetes is a critical aspect of application security. While Kubernetes provides a native Secret object, storing secrets directly in Git repositories, even if encrypted, poses significant risks. Furthermore, manually synchronizing secrets between external secret management systems (like AWS Secrets Manager, HashiCorp Vault, or Google Secret Manager) and Kubernetes can be a tedious, error-prone, and non-scalable process. This challenge often leads development teams to compromise on security or operational efficiency.

Enter the External Secrets Operator (ESO). This powerful Kubernetes operator bridges the gap between external secret management systems and Kubernetes native Secret objects. ESO automatically fetches secrets from a wide array of external providers and injects them into your cluster as standard Kubernetes Secrets. This not only enhances your security posture by keeping sensitive data out of your Git repositories but also streamlines secret rotation, access control, and compliance efforts, allowing developers to focus on building applications rather than managing infrastructure-level secrets.

This comprehensive guide will walk you through deploying and configuring the External Secrets Operator, demonstrating how to seamlessly integrate your Kubernetes workloads with external secret stores. We’ll cover everything from installation to practical examples, ensuring you can securely and efficiently manage your sensitive data within a Kubernetes environment. By the end, you’ll have a robust solution for secret management that aligns with modern security best practices and simplifies your operational overhead.

TL;DR: External Secrets Operator in a Nutshell

The External Secrets Operator (ESO) enables you to securely manage Kubernetes Secrets by fetching them from external secret stores (e.g., AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) and synchronizing them into your cluster as native Kubernetes Secret objects. This keeps sensitive data out of your Git repos and automates secret rotation.

Install ESO: helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
Create a SecretStore: Define how ESO connects to your external secret provider (e.g., AWS Secrets Manager).
Create an ExternalSecret: Specify which secret from the external store to fetch and how it should be named in Kubernetes.
Consume: Mount the resulting Kubernetes Secret into your Pods.

# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

# Example SecretStore for AWS Secrets Manager
kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          secretAccessKeySecretRef:
            name: aws-secret-creds
            key: secret-access-key
          accessKeyIDSecretRef:
            name: aws-secret-creds
            key: access-key-id
EOF

# Example ExternalSecret to fetch a secret named "my/database/creds"
kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-db-creds
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: my-app-db-secret # Name of the Kubernetes Secret to be created
  data:
  - secretKey: db_username # Key in the K8s Secret
    remoteRef:
      key: my/database/creds # Name of the secret in AWS Secrets Manager
      property: username # Key within the JSON secret in AWS
  - secretKey: db_password
    remoteRef:
      key: my/database/creds
      property: password
EOF

# Verify the Kubernetes Secret
kubectl get secret my-app-db-secret -o yaml

Prerequisites

Before diving into the installation and configuration of the External Secrets Operator, ensure you have the following:

Kubernetes Cluster: A running Kubernetes cluster (version 1.16+). You can use Minikube, Kind, or any cloud provider’s managed Kubernetes service (EKS, GKE, AKS).
kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
helm: The Kubernetes package manager, used for installing External Secrets Operator. Install it by following the Helm installation guide.
External Secret Store Access: Access to an external secret management system, such as AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or GCP Secret Manager. This guide will primarily use AWS Secrets Manager for examples, but the concepts apply broadly.
Permissions: Appropriate IAM roles or service accounts configured in your external secret store to allow read access to the secrets. For AWS, this means an IAM user or role with permissions like secretsmanager:GetSecretValue.

Step-by-Step Guide: Kubernetes Secrets Management with External Secrets Operator

1. Install the External Secrets Operator

The first step is to deploy the External Secrets Operator into your Kubernetes cluster. We’ll use Helm for a straightforward installation. ESO is typically installed in its own namespace for better isolation.

Add the External Secrets Helm repository and then install the operator. This will deploy the necessary Custom Resource Definitions (CRDs), deployments, and service accounts for ESO to function.

# Add the External Secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io

# Update your Helm repositories
helm repo update

# Install the External Secrets Operator into its own namespace
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Verify:
After installation, verify that the External Secrets Operator pods are running in the external-secrets namespace. You should see a deployment named external-secrets and a pod in a Running state.

kubectl get pods -n external-secrets

# Expected Output:
# NAME                                     READY   STATUS    RESTARTS   AGE
# external-secrets-7d6f5c87-abcde          1/1     Running   0          2m

kubectl get crds | grep secrets.external-secrets.io

# Expected Output:
# clustersecretstores.external-secrets.io      2023-10-27T10:00:00Z
# externalsecrets.external-secrets.io          2023-10-27T10:00:00Z
# secretstores.external-secrets.io             2023-10-27T10:00:00Z

2. Configure Credentials for Your External Secret Store

For ESO to access your external secret store, it needs appropriate credentials. The method for providing these credentials varies by provider. For AWS Secrets Manager, you typically use IAM roles or access keys. Here, we’ll demonstrate using Kubernetes Secrets to store AWS access keys, which ESO will then use.

First, create an IAM user or role in AWS with permissions to read secrets from AWS Secrets Manager. A policy like the following would suffice:


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:YOUR_SECRET_NAME-*"
        }
    ]
}

Note: Replace REGION, ACCOUNT_ID, and YOUR_SECRET_NAME with your specific values. For better security, always scope down permissions to only the necessary secrets. For more on securing cloud resources, consider how Kubernetes Network Policies can isolate traffic within your cluster.

Next, create a Kubernetes Secret containing your AWS Access Key ID and Secret Access Key. This secret will reside in the same namespace where your SecretStore (or ClusterSecretStore) will be defined.

# Replace with your actual AWS credentials
export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"

# Create a Kubernetes Secret to hold these credentials
kubectl create secret generic aws-secret-creds \
  --from-literal=access-key-id="${AWS_ACCESS_KEY_ID}" \
  --from-literal=secret-access-key="${AWS_SECRET_ACCESS_KEY}" \
  -n default # Or your application namespace

Verify:
Ensure the Kubernetes Secret has been created in the specified namespace.

kubectl get secret aws-secret-creds -n default -o yaml

# Expected Output (values will be base64 encoded):
# apiVersion: v1
# data:
#   access-key-id: YOUR_BASE64_ENCODED_ACCESS_KEY_ID
#   secret-access-key: YOUR_BASE64_ENCODED_SECRET_ACCESS_KEY
# kind: Secret
# metadata:
#   name: aws-secret-creds
#   namespace: default
# type: Opaque

3. Define a SecretStore or ClusterSecretStore

A SecretStore (namespaced) or ClusterSecretStore (cluster-scoped) tells ESO how to connect to a specific external secret provider. You’ll define the provider type (e.g., AWS, Vault, GCP), region, and how to authenticate. For this example, we’ll use a SecretStore for AWS Secrets Manager, referencing the Kubernetes Secret created in the previous step.

The SecretStore object acts as a blueprint for ESO to understand where to fetch secrets from. It specifies the provider and the authentication method. Using a namespaced SecretStore means it can only be used by ExternalSecret resources within the same namespace.

# secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: default # Must be in the same namespace as aws-secret-creds
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1 # Replace with your AWS region
      auth:
        secretRef:
          secretAccessKeySecretRef:
            name: aws-secret-creds
            key: secret-access-key
          accessKeyIDSecretRef:
            name: aws-secret-creds
            key: access-key-id

kubectl apply -f secretstore.yaml

Verify:
Check the status of your SecretStore. It should show a Ready condition.

kubectl get secretstore aws-secrets-manager -n default -o yaml

# Expected Output (truncated for brevity):
# apiVersion: external-secrets.io/v1beta1
# kind: SecretStore
# metadata:
#   name: aws-secrets-manager
#   namespace: default
# spec:
#   provider:
#     aws:
#       auth:
#         secretRef:
#           accessKeyIDSecretRef:
#             key: access-key-id
#             name: aws-secret-creds
#           secretAccessKeySecretRef:
#             key: secret-access-key
#             name: aws-secret-creds
#       region: us-east-1
#       service: SecretsManager
# status:
#   conditions:
#   - lastTransitionTime: "2023-10-27T10:00:00Z"
#     message: SecretStore was synced successfully
#     reason: SecretStoreReady
#     status: "True"
#     type: Ready

4. Create an External Secret in AWS Secrets Manager

Before we can fetch a secret, we need to have one available in our external store. For this example, let’s create a simple JSON secret in AWS Secrets Manager.

Navigate to the AWS Secrets Manager console, click “Store a new secret”, choose “Other type of secret”, and enter your key-value pairs. Name the secret my/database/creds.


{
  "username": "dbuser",
  "password": "supersecurepassword123",
  "connection_string": "jdbc:mysql://mydb.example.com:3306/appdb"
}

Note: In a real-world scenario, you would use strong, unique passwords and frequently rotate them. ESO helps automate this rotation within Kubernetes.

5. Define an ExternalSecret Resource

The ExternalSecret is the core resource that tells ESO which secret to fetch from the external store and how to transform it into a Kubernetes Secret. You specify the SecretStore to use, the remote secret’s name, and how to map its data to keys in the target Kubernetes Secret.

This resource defines the desired state: a Kubernetes Secret named my-app-db-secret, containing db_username and db_password, fetched from the my/database/creds secret in AWS Secrets Manager. The refreshInterval specifies how often ESO should check for updates to the remote secret.

# externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-db-creds
  namespace: default
spec:
  refreshInterval: "1h" # How often to refresh the secret (e.g., "1h", "15m")
  secretStoreRef:
    name: aws-secrets-manager # Reference to the SecretStore created earlier
    kind: SecretStore
  target:
    name: my-app-db-secret # Name of the Kubernetes Secret to be created
    creationPolicy: Owner # ESO will own and manage this secret
  data:
  - secretKey: db_username # Key in the K8s Secret
    remoteRef:
      key: my/database/creds # Name of the secret in AWS Secrets Manager
      property: username # Key within the JSON secret in AWS
  - secretKey: db_password
    remoteRef:
      key: my/database/creds
      property: password
  - secretKey: db_connection_string
    remoteRef:
      key: my/database/creds
      property: connection_string

kubectl apply -f externalsecret.yaml

Verify:
After applying the ExternalSecret, ESO will create a corresponding Kubernetes Secret. Check for its existence and content.

kubectl get externalsecret my-app-db-creds -n default

# Expected Output:
# NAME              STORE               REFRESH INTERVAL   STATUS    AGE
# my-app-db-creds   aws-secrets-manager 1h                 Ready     20s

kubectl get secret my-app-db-secret -n default -o yaml

# Expected Output (values will be base64 encoded):
# apiVersion: v1
# data:
#   db_connection_string: amRiYzpteXNxbDovL215ZGIuZXhhbXBsZS5jb206MzMwNi9hcHBkYg==
#   db_password: c3VwZXJzZWN1cmVwYXNzd29yZDEyMw==
#   db_username: ZGJ1c2Vy
# kind: Secret
# metadata:
#   creationTimestamp: "2023-10-27T10:05:00Z"
#   labels:
#     external-secrets.io/externalsecret: my-app-db-creds
#   name: my-app-db-secret
#   namespace: default
#   ownerReferences: # ESO owns this secret
#   - apiVersion: external-secrets.io/v1beta1
#     controller: true
#     kind: ExternalSecret
#     name: my-app-db-creds
#     uid: abcdefg-1234-5678-ijklmnop
# type: Opaque

# Decode a secret value to confirm
kubectl get secret my-app-db-secret -n default -o jsonpath='{.data.db_username}' | base64 --decode

# Expected Output:
# dbuser

6. Consume the Kubernetes Secret in Your Application

Now that the Kubernetes Secret (my-app-db-secret) is available, your applications can consume it just like any other native Kubernetes Secret. This typically involves mounting it as environment variables or as a volume.

Here’s an example of a simple Nginx deployment that uses these secrets as environment variables:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:latest # Replace with your actual application image
        env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: my-app-db-secret # Name of the K8s Secret
              key: db_username       # Key within the K8s Secret
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-app-db-secret
              key: db_password
        - name: DB_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: my-app-db-secret
              key: db_connection_string
        ports:
        - containerPort: 80

kubectl apply -f deployment.yaml

Verify:
Check the logs or exec into the pod to confirm that the environment variables are correctly populated.

kubectl get pods -n default -l app=my-app

# Expected Output:
# NAME                       READY   STATUS    RESTARTS   AGE
# my-app-7c8d9f-ghijk        1/1     Running   0          30s

POD_NAME=$(kubectl get pods -n default -l app=my-app -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it "${POD_NAME}" -n default -- printenv | grep DB_

# Expected Output:
# DB_USERNAME=dbuser
# DB_PASSWORD=supersecurepassword123
# DB_CONNECTION_STRING=jdbc:mysql://mydb.example.com:3306/appdb

Congratulations! You have successfully integrated External Secrets Operator to manage your sensitive data in Kubernetes. This pattern significantly enhances security by centralizing secret management and automating synchronization. For further security enhancements, consider integrating tools like Sigstore and Kyverno for supply chain security and policy enforcement.

Production Considerations

Deploying External Secrets Operator in a production environment requires careful planning and adherence to best practices:

Least Privilege: Ensure the IAM roles/service accounts used by ESO (or the K8s Secret for credentials) have the absolute minimum permissions required to access only the necessary secrets in your external store. Avoid wildcard permissions.
Secret Rotation: Leverage your external secret manager’s capabilities for automated secret rotation. ESO will automatically pick up rotated secrets based on the refreshInterval defined in your ExternalSecret. While ESO refreshes the Kubernetes Secret, consider how your application handles secret changes without a pod restart.
Redundancy and High Availability: Ensure your External Secrets Operator deployment is highly available, typically with multiple replicas across different nodes. This is handled by default with Helm, but verify your deployment strategy.
Monitoring and Alerting: Monitor ESO’s logs and metrics for failures in fetching secrets or issues connecting to the external store. Integrate with your existing observability stack. You can even build custom metrics using tools like eBPF Observability with Hubble for deep insights into your cluster’s behavior.
Audit Logging: Enable audit logging in your external secret store to track who accessed which secrets and when. This is crucial for compliance and security investigations.
Network Policies: Implement Kubernetes Network Policies to restrict network access for the External Secrets Operator pods, allowing them to communicate only with the necessary external secret store endpoints. Similarly, restrict application pods to only access the Kubernetes API for secrets if necessary, or specific services.
Namespacing Strategy: Decide whether to use SecretStore (namespaced) or ClusterSecretStore (cluster-scoped). SecretStore offers better isolation and multi-tenancy, while ClusterSecretStore is convenient for cluster-wide secrets or shared configurations.
Provider-Specific Authentication: For highly secure environments, consider using IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Azure AD Workload Identity on AKS. This eliminates the need to store static cloud credentials in Kubernetes Secrets.
Backup and Recovery: While ESO fetches secrets, the ExternalSecret definitions themselves are critical. Include them in your cluster backup strategy.
Performance: Be mindful of the refreshInterval. While a shorter interval provides quicker secret propagation, it also increases the load on your external secret store. Balance security requirements with performance needs.
Sidecar Injection for Secrets: For certain advanced use cases, especially with service meshes like Istio Ambient Mesh, you might consider injecting secrets directly into pods using init containers or mutating webhooks, rather than relying solely on environment variables or volume mounts, though ESO’s primary function is to create the K8s Secret.

Troubleshooting

Here are some common issues you might encounter with External Secrets Operator and their solutions:

ExternalSecret status is ‘Error’ or ‘NotFound’

Issue: The ExternalSecret resource shows an error status or fails to create the target Kubernetes Secret.

Solution:
- Check ESO Pod Logs: The most common source of information.
- Inspect ExternalSecret Events:
- Verify SecretStore: Ensure the referenced SecretStore is Ready and correctly configured.
- Remote Secret Name/Path: Double-check that the remoteRef.key in your ExternalSecret exactly matches the name/path of the secret in your external store. Case sensitivity matters.
- Permissions: The most frequent cause. Ensure the credentials used by the SecretStore have sufficient permissions to read the specific secret(s) in the external store.
Kubernetes Secret not created or not updated

Issue: The target Kubernetes Secret is not appearing, or changes in the external secret store are not reflected in Kubernetes.

Solution:
- Check ExternalSecret Status: Confirm it’s Ready. If not, see issue 1.
- refreshInterval: The ExternalSecret has a refreshInterval (defaulting to 1 hour if not specified). Changes in the external store will only be reflected after this interval. You can force a refresh by deleting and re-creating the ExternalSecret, or by patching it with a dummy change.
- Target Secret Name: Ensure spec.target.name in your ExternalSecret is not clashing with an existing Kubernetes Secret that was not created by ESO. If creationPolicy: Owner is set, ESO will manage it; otherwise, it might not overwrite.
- ESO Controller Logs: Check the logs of the External Secrets Operator pod for any errors related to reconciliation.
Application Pods not getting secret values

Issue: The Kubernetes Secret exists, but your application pod’s environment variables or mounted files are empty or incorrect.

Solution:
- Verify K8s Secret Content: Decode the base64 values of the Kubernetes Secret to ensure they are correct.
- Deployment YAML: Double-check the env or volumeMounts section in your Deployment/Pod specification. Ensure secretKeyRef.name matches the Kubernetes Secret name, and secretKeyRef.key matches the key within that Kubernetes Secret.
- Pod Restart: Kubernetes environment variables are typically set at pod creation time. If the Secret was updated, the pod might need to be restarted to pick up the new values. Volume mounts, however, are often dynamic.
- Pod Logs: Check your application’s logs for any errors related to missing or malformed configuration.
Error: “Failed to get secret value: AccessDenied” (AWS Specific)

Issue: The ESO logs show AccessDenied errors when trying to fetch secrets from AWS.

Solution:
- IAM Policy: Review the IAM policy attached to the AWS credentials (user or role) used by your SecretStore. Ensure it explicitly grants secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions for the specific secret ARN(s).
- Resource ARN: Verify the ARN in the IAM policy is correct, including region and account ID.
- Secret Key/ID: If using a Kubernetes Secret for credentials, ensure the access-key-id and secret-access-key are correct and not expired.
- Region Mismatch: Confirm the region specified in the SecretStore matches the region where your AWS Secrets Manager secret resides.
Can’t install External Secrets Operator via Helm

Issue: Helm installation fails, or the repository cannot be found.

Solution:
- Helm Repo Update: Always run helm repo update after adding a new repository.
- Network Connectivity: Ensure your machine or cluster has outbound internet access to reach charts.external-secrets.io.
- Helm Version: Ensure you are using a compatible Helm version (Helm 3 is required).
- Namespace Permissions: If installing into a new namespace, ensure your Kubernetes user has permissions to create namespaces and resources within them.

FAQ Section

What is the difference between SecretStore and ClusterSecretStore?

A SecretStore is a namespaced resource, meaning it can only be referenced by ExternalSecret objects within the same namespace. This is ideal for multi-tenant clusters or when different teams manage their own secret backends. A ClusterSecretStore is a cluster-scoped resource, allowing any ExternalSecret in any namespace to reference it. This is useful for cluster-wide secrets or shared secret backends (e.g., a single Vault instance for the entire cluster).
How does External Secrets Operator handle secret rotation?

External Secrets Operator does not perform secret rotation itself in the external secret store. It relies on your external secret manager (e.g., AWS Secrets Manager, Vault) to handle the actual rotation. Once the secret is rotated in the external store, ESO will detect the change during its next refreshInterval and update the corresponding Kubernetes Secret. Your applications then need to be designed to pick up these updated secrets, either by restarting or dynamically reloading configuration.
Is it secure to store cloud credentials in Kubernetes Secrets for ESO?

While Kubernetes Secrets are base64 encoded (not encrypted by default at rest), they are generally considered safe if your cluster is properly secured (e.g., etcd encryption enabled, strong RBAC). However, for cloud providers like AWS, GCP, and Azure, the most secure method is to use Workload Identity or IAM Roles for Service Accounts (IRSA). This allows the ESO service account to assume a cloud IAM role directly, eliminating the need to store static long-lived credentials in Kubernetes.
Can I use External Secrets Operator with HashiCorp Vault?

Absolutely! External Secrets Operator has extensive support for HashiCorp Vault. You would configure a SecretStore with a vault provider, specifying the Vault address, authentication method (e.g., Kubernetes Auth, AppRole, Token), and the path to your secrets. ESO integrates seamlessly with Vault’s dynamic secret capabilities.
What if my application needs secrets that are not JSON key-value pairs?

External Secrets Operator is highly flexible. While the example uses a JSON secret with property, you can fetch an entire secret as a single value (e.g., a certificate or a single string). If your remote secret is not JSON, you can omit the property field in remoteRef, and ESO will fetch the entire secret value. For example, for a plain text secret named my/certificate, you would use remoteRef: { key: "my/certificate" } without a property.

Cleanup Commands

To remove all resources created during this tutorial, execute the following commands:

# Delete the application deployment
kubectl delete deployment my-app -n default

# Delete the ExternalSecret
kubectl delete externalsecret my-app-db-creds -n default

# The Kubernetes Secret 'my-app-db-secret' will be garbage collected by ESO because of creationPolicy: Owner

# Delete the SecretStore
kubectl delete secretstore aws-secrets-manager -n default

# Delete the Kubernetes Secret containing AWS credentials
kubectl delete secret aws-secret-creds -n default

# Uninstall the External Secrets Operator via Helm
helm uninstall external-secrets -n external-secrets

# Delete the external-secrets namespace
kubectl delete namespace external-secrets

# (Optional) Remove the Helm repository
helm repo remove external-secrets

Next Steps / Further Reading

You’ve successfully mastered the basics of External Secrets Operator. To deepen your knowledge