Introduction

In the modern cloud-native landscape, the integrity and authenticity of container images are paramount. As organizations increasingly rely on containerized applications, the supply chain for these images becomes a critical attack vector. How can you be sure that the image you’re deploying hasn’t been tampered with? How do you verify that it originated from a trusted source? These questions highlight a significant security gap that traditional security measures often overlook.

This is where Sigstore, a Linux Foundation project, steps in. Sigstore provides a set of tools and services designed to improve software supply chain security, with a strong focus on cryptographic signing of software artifacts. At its core is Cosign, a simple, free, and open-source tool for signing, verifying, and storing signatures for container images and other artifacts. By integrating Cosign into your CI/CD pipelines, you can establish a robust chain of trust, ensuring that only verified and authorized images make it into your Kubernetes clusters. This guide will walk you through the process of signing and verifying container images using Cosign, empowering you to fortify your container supply chain against tampering and unauthorized modifications.

Prerequisites

Before diving into image signing with Cosign, ensure you have the following:

• A Kubernetes Cluster: While not strictly required for signing/verifying images, this guide is geared towards securing deployments within a Kubernetes environment. Any functional cluster will do, local (e.g., Minikube, Kind) or cloud-based.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• Docker or Podman: A container runtime for building and pushing images.
• A Container Registry: A registry where you can push and pull images. Examples include Docker Hub, Google Container Registry (GCR), Amazon Elastic Container Registry (ECR), or Azure Container Registry (ACR). Ensure you have push/pull permissions.
• Go Language Environment: Cosign is written in Go, and installing it via go install is the recommended method. Ensure you have Go 1.16+ installed. You can download it from the official Go website.
• Basic understanding of Public Key Cryptography: Familiarity with concepts like private keys, public keys, and digital signatures will be helpful.

Step-by-Step Guide: Sigstore Cosign for Container Image Signing

This section will guide you through installing Cosign, generating cryptographic keys, signing a container image, and finally verifying its signature.

1. Install Cosign

First, you need to install the Cosign CLI tool. The easiest way is to use go install, which fetches and builds the latest version directly from the source.

The go install command places the executable in your GOPATH/bin directory. Ensure that this directory is included in your system’s PATH environmental variable so you can run cosign from any directory. If you prefer, Cosign also offers releases via GitHub releases which include pre-compiled binaries for various platforms.

# Install Cosign
go install github.com/sigstore/cosign/cmd/cosign@latest

# Verify installation by checking the version
cosign version

Verify

You should see output similar to this, indicating Cosign is installed and its version:

$ cosign version
  Version:          v2.2.3
  Commit:           a831e5e
  Target:           linux/amd64
  ... (other details)

2. Generate a Key Pair

Cosign uses asymmetric cryptography for signing. This means you’ll have a private key to sign images and a public key to verify them. It’s crucial to keep your private key secure and never share it.

When you generate a key pair, Cosign will prompt you for a password. This password encrypts your private key (cosign.key) on disk, adding an extra layer of security. Remember this password, as you’ll need it every time you sign an image with this private key. The public key (cosign.pub) can be freely distributed for verification purposes. For production environments, consider using hardware security modules (HSMs) or cloud key management services (KMS) for private key protection, which Cosign also supports.

# Generate a key pair
cosign generate-key-pair

Verify

You will be prompted to enter a password for the private key:

$ cosign generate-key-pair
  Enter password for private key:
  Enter again:
  Private key written to cosign.key
  Public key written to cosign.pub

You should now have two files in your current directory: cosign.key (your private key) and cosign.pub (your public key).

3. Build and Push a Sample Container Image

To demonstrate signing, we’ll need a container image. Let’s create a simple Nginx image.

This step involves creating a basic Dockerfile for an Nginx web server. After building the image, you’ll tag it with your container registry’s address and then push it. This makes the image available for signing and subsequent deployment. Replace <YOUR_REGISTRY> with your actual registry (e.g., docker.io/yourusername, gcr.io/your-project).

# Create a Dockerfile
cat <<EOF > Dockerfile
FROM nginx:latest
COPY index.html /usr/share/nginx/html/index.html
EXPOSE 80
EOF

# Create a simple index.html
echo "
Hello from Kubezilla! This image is signed!
" > index.html

# Define your image name and tag
export REGISTRY_HOST="<YOUR_REGISTRY>" # e.g., docker.io/yourusername
export IMAGE_NAME="${REGISTRY_HOST}/kubezilla-signed-nginx:latest"

# Build the image
docker build -t "${IMAGE_NAME}" .

# Push the image to your registry
docker push "${IMAGE_NAME}"

Verify

You should see output similar to this, indicating the image was built and pushed successfully:

$ docker push "${IMAGE_NAME}"
  The push refers to repository [docker.io/yourusername/kubezilla-signed-nginx]
  ...
  latest: digest: sha256:a1b2c3d4e5f6... size: 1789

4. Sign the Container Image

Now, let’s sign the image you just pushed using your private key.

The cosign sign command takes your private key (--key cosign.key) and the image reference. You’ll be prompted for the password you set for your private key. Cosign then generates a signature and pushes it to your container registry alongside your image. By default, Cosign stores the signature in an OCI-compatible format, often as a separate artifact that references the original image’s digest. This means the original image remains untouched, and its integrity is verified against the signature.

# Sign the image with your private key
cosign sign --key cosign.key "${IMAGE_NAME}"

Verify

You will be prompted for your private key password:

$ cosign sign --key cosign.key "${IMAGE_NAME}"
  Enter password for private key:
  Pushing signature to: <YOUR_REGISTRY>/kubezilla-signed-nginx:latest.sig

A signature artifact has been pushed to your registry. You can confirm this by listing artifacts related to your image, though the exact command depends on your registry.

5. Verify the Container Image Signature

With the image signed, you can now verify its authenticity using the public key.

The cosign verify command uses the public key (--key cosign.pub) to check if the signature associated with the image is valid. It fetches the signature from the registry, reconstructs the digest of the image, and compares it against the digest embedded in the signature. If everything matches, Cosign confirms the image’s integrity and authenticity. This step is critical in your CI/CD pipeline or admission controller to prevent unsigned or tampered images from being deployed.

# Verify the image signature using the public key
cosign verify --key cosign.pub "${IMAGE_NAME}"

Verify

A successful verification will show output like this:

$ cosign verify --key cosign.pub "${IMAGE_NAME}"
  The following checks were performed:
    - The co-signed container image was found.
    - The signature was successfully verified by the specified public key.

If the image or signature were tampered with, or if the wrong public key was used, the verification would fail.

6. Using Keyless Signing with Fulcio and Rekor

While key pairs are useful for local testing, Sigstore offers a more robust “keyless” signing mechanism using Fulcio (a root certificate authority) and Rekor (a transparency log). This method binds your signature to an OpenID Connect (OIDC) identity (e.g., your GitHub or Google account), eliminating the need to manage private keys.

Keyless signing is the recommended approach for production environments. When you sign an image without specifying a key, Cosign interacts with Fulcio to obtain a short-lived certificate based on your OIDC identity. This certificate is then used to sign the image, and the signature, along with the certificate, is recorded in the Rekor transparency log. Verification then involves checking the signature against the certificate and confirming its presence in Rekor, ensuring non-repudiation and transparency. This significantly reduces the overhead of key management and enhances security by tying signatures to verifiable identities.

# Sign an image using keyless signing (Fulcio and Rekor)
# You'll be prompted to authenticate with your OIDC provider (e.g., GitHub, Google)
cosign sign "${IMAGE_NAME}"

Verify

You will be prompted to authenticate via your web browser:

$ cosign sign "${IMAGE_NAME}"
  Retrieving signed certificate from Fulcio...
  Your browser will now be opened to:
  https://oauth2.sigstore.dev/auth?scope=openid+email&...
  ... (after successful browser authentication)
  Successfully verified SCT.
  Pushing signature to: <YOUR_REGISTRY>/kubezilla-signed-nginx:latest.sig

To verify a keyless signature, you don’t need a public key file. Cosign retrieves the necessary information (certificate, transparency log entry) from Rekor.

# Verify a keyless signed image
cosign verify "${IMAGE_NAME}"

Verify

A successful keyless verification will show output like this, including details about the signer’s identity:

$ cosign verify "${IMAGE_NAME}"
  The following checks were performed:
    - The co-signed container image was found.
    - The transparency log entry was found and verified.
    - The signer's identity was verified to be "<YOUR_OIDC_EMAIL>" from issuer "https://accounts.google.com".

7. Integrating with a Policy Engine (Kyverno)

To enforce signature verification in your Kubernetes cluster, you’ll need an admission controller. Kyverno is an excellent choice for this, as it has native integration with Cosign and Sigstore. For a deep dive into this integration, refer to our comprehensive guide on Securing Container Supply Chains with Sigstore and Kyverno.

Kyverno can intercept admission requests (like creating or updating a Pod) and check if the container images specified in the Pod definition are signed and verified according to predefined policies. This provides a critical enforcement point, preventing unsigned or untrusted images from running in your cluster. Below is a simple Kyverno policy example that requires all images to be signed and verified by a specific public key.

# kyverno-install.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: kyverno
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kyverno
  namespace: kyverno
spec:
  interval: 1h
  chart:
    spec:
      chart: kyverno
      version: "3.0.x" # Use a recent stable version
      sourceRef:
        kind: HelmRepository
        name: kyverno
        namespace: kyverno
  values:
    admissionController:
      image:
        registry: ghcr.io
        repository: kyverno/kyverno
        tag: 3.0.x # Match chart version
    reportsController:
      image:
        registry: ghcr.io
        repository: kyverno/kyverno
        tag: 3.0.x # Match chart version

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: kyverno
  namespace: kyverno
spec:
  interval: 1h
  url: https://kyverno.github.io/kyverno/

Apply the Kyverno installation YAML (or follow official Kyverno installation instructions). Once Kyverno is running, you can create a ClusterPolicy to enforce Cosign signatures.

# cosign-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
  annotations:
    policies.kyverno.io/description: |
      Requires all images to be signed by a specific Cosign key.
      Replace `<YOUR_PUBLIC_KEY_BASE64>` with the base64 encoded content of your `cosign.pub` file.
spec:
  validationFailureAction: Enforce
  rules:
  - name: verify-image-signature
    match:
      any:
      - resources:
          kinds:
          - Pod
    verifyImages:
    - image: "*"
      key: |-
        -----BEGIN PUBLIC KEY-----
        <YOUR_PUBLIC_KEY_BASE64>
        -----END PUBLIC KEY-----
      # You can also specify an issuer and subject for keyless signing:
      # subject: "your-email@example.com"
      # issuer: "https://accounts.google.com"
      # This is for keyless signing; for key-pair signing, only `key` is needed.
      # For keyless, you might also need to specify `rekor: {}`

Before applying the policy, you need to base64 encode your cosign.pub file:

# Encode your public key
cat cosign.pub | base64 -w 0

Replace <YOUR_PUBLIC_KEY_BASE64> in the cosign-policy.yaml with the output of the above command. Then apply the policy:

# Apply the Kyverno policy
kubectl apply -f cosign-policy.yaml

Verify

Now, try to deploy a Pod with an unsigned image, or an image signed with a different key.

# unsigned-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-nginx
spec:
  containers:
  - name: nginx
    image: nginx:latest # This image is not signed by your key

# Attempt to deploy an unsigned image
kubectl apply -f unsigned-pod.yaml

You should see an error similar to this, indicating Kyverno blocked the deployment:

$ kubectl apply -f unsigned-pod.yaml
  Error from server (admission webhook "validate.kyverno.svc-fail" denied the request):
  resource Pod/unsigned-nginx was blocked due to the following policies
  require-signed-images:
    verify-image-signature: 'failed to verify signature for image nginx:latest:
      error verifying signature: no matching signatures:
      <...>'

If you deploy your previously signed image, it should succeed:

# signed-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: signed-nginx
spec:
  containers:
  - name: nginx
    image: <YOUR_REGISTRY>/kubezilla-signed-nginx:latest # This image is signed by your key

# Attempt to deploy the signed image
kubectl apply -f signed-pod.yaml

This time, the Pod should be created successfully, demonstrating Kyverno’s enforcement.

$ kubectl apply -f signed-pod.yaml
  pod/signed-nginx created

Production Considerations

Implementing Cosign in a production environment requires careful planning beyond simple key generation and signing.

• Key Management:
  • KMS Integration: For private keys, strongly consider using cloud Key Management Services (KMS) like AWS KMS, Google Cloud KMS, or Azure Key Vault. Cosign has native support for these. This centralizes key management, provides auditing, and prevents private keys from being exposed directly.
  • Hardware Security Modules (HSMs): For even higher security, integrate Cosign with HSMs.
  • Key Rotation: Establish a policy for rotating signing keys periodically to mitigate the risk of compromise.
• CI/CD Pipeline Integration:
  • Automate Signing: Integrate Cosign signing as a mandatory step in your CI/CD pipeline immediately after image building and before pushing to the registry. This ensures all deployed images are signed.
  • Environment Variables: Use environment variables to pass KMS key references or private key passwords securely (e.g., using CI/CD secrets management).
  • OIDC for Keyless Signing: For keyless signing, configure your CI/CD system to authenticate with an OIDC provider (e.g., GitHub Actions’ OIDC provider, GitLab CI’s OIDC support) to obtain the necessary identity for Fulcio.
• Policy Enforcement:
  • Admission Controllers: Deploy an admission controller like Kyverno or OPA Gatekeeper to enforce signature verification at the Kubernetes admission layer. This is non-negotiable for production.
  • Audit Mode: Start with policies in Audit or DryRun mode to observe their impact before switching to Enforce.
  • Granular Policies: Create granular policies based on namespaces, image registries, and specific signing identities (e.g., only images signed by the “dev-team” or “release-pipeline” are allowed in the “production” namespace).
• Transparency Log (Rekor):
  • Public Rekor: For most use cases, the public Rekor instance is sufficient and recommended for its transparency benefits.
  • Private Rekor: For highly sensitive environments with strict air-gapped requirements, consider deploying a private Rekor instance, though this adds operational complexity.
• Monitoring and Alerting:
  • Monitor admission controller logs for signature verification failures. Integrate these logs with your centralized logging and alerting systems.
  • Track changes to signing keys and policies.
• Supply Chain Security Strategy:
  • Cosign is one piece of the puzzle. Combine it with other supply chain security practices like vulnerability scanning, software bill of materials (SBOM) generation, and secure base images.
  • Consider Kubernetes Network Policies and Cilium WireGuard Encryption to secure the runtime environment of your signed images.

Troubleshooting

Here are common issues you might encounter when using Cosign and their solutions.

1. Issue: cosign: command not found

   Solution: This usually means the Go bin directory isn’t in your PATH, or Cosign wasn’t installed correctly.

   
   # Ensure Go bin is in your PATH (add to .bashrc or .zshrc)
   export PATH=$PATH:$(go env GOPATH)/bin
   
   # Re-install Cosign
   go install github.com/sigstore/cosign/cmd/cosign@latest
           

2. Issue: Error: no matching signatures during cosign verify

   Solution: This means Cosign couldn’t find a valid signature for the image.

   • Ensure the image was actually signed with the correct key/method.
   • Verify the image name and tag are correct and match what was signed.
   • If using --key, ensure the public key file (cosign.pub) is correct and corresponds to the private key used for signing.
   • If using keyless signing, ensure your OIDC identity matches what was used during signing.
3. Issue: Error: error verifying signature: x509: certificate has expired or is not yet valid with keyless signing.

   Solution: Fulcio certificates are short-lived. This error indicates the certificate used to sign the image has expired. You’ll need to re-sign the image. This is by design for enhanced security, as it limits the window of opportunity for a compromised certificate to be exploited.

   
   # Re-sign the image
   cosign sign <YOUR_REGISTRY>/<YOUR_IMAGE>:<TAG>
           

4. Issue: Kyverno policy blocks a signed image with no matching signatures or certificate has expired.

   Solution:

   • Key-pair signing: Double-check that the public key embedded in your Kyverno policy is correct and base64 encoded without extra whitespace.
   • Keyless signing: Ensure your Kyverno policy correctly specifies the subject (signer’s email) and issuer (OIDC provider URL) that match the identity used during signing. Also, ensure the image was recently signed, as Fulcio certificates are short-lived.
   • Check Kyverno logs for more detailed error messages:

     
     kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller
                 

5. Issue: Unable to push signatures to registry (e.g., unauthorized: authentication required).

   Solution: Ensure your container runtime (Docker, Podman) is logged into the registry with credentials that have push permissions for both the image and its associated signature artifact (often with a .sig suffix).

   
   docker login <YOUR_REGISTRY_HOST>
           

6. Issue: cosign generate-key-pair hangs or fails to generate keys.

   Solution: This can sometimes happen in environments with low entropy.

   • Try running it again.
   • On Linux, ensure you have a good entropy source (e.g., haveged package).

FAQ Section

1. What is Sigstore and how does Cosign fit into it?

   Sigstore is a set of open-source tools and services for signing, verifying, and protecting software. It aims to make software signing accessible and ubiquitous. Cosign is the primary tool within Sigstore for signing and verifying container images and other software artifacts. It integrates with Fulcio (a root CA) and Rekor (a transparency log) to provide a keyless signing experience.

2. Why should I sign my container images?

   Signing container images provides cryptographic proof of their origin and integrity. It helps mitigate supply chain attacks by ensuring that the images you deploy are exactly what your trusted source (e.g., your CI/CD pipeline) produced and haven’t been tampered with or replaced by malicious actors. This is a fundamental step in securing your Kubernetes deployments.

3. What’s the difference between key-pair signing and keyless signing?

   Key-pair signing involves generating a private/public key pair. You use the private key to sign, and others use the public key to verify. This requires secure management of your private key. Keyless signing (using Fulcio and Rekor) eliminates the need to manage a long-lived private key. Instead, you authenticate with an OIDC provider (like Google or GitHub), and Fulcio issues a short-lived certificate to sign the image. The signature and certificate are then recorded in the Rekor transparency log. This method enhances security and simplifies key management.

4. Can Cosign sign artifacts other than container images?

   Yes! While container images are a primary use case, Cosign can sign any OCI artifact, including Helm charts, WebAssembly modules, software bills of materials (SBOMs), and other arbitrary blobs. This makes it a versatile tool for securing various components of your software supply chain.

5. How does Cosign compare to Docker Content Trust (DCT)?

   Both Cosign and DCT aim to secure container images. However, Cosign offers several advantages: it’s simpler to use, integrates with the broader Sigstore ecosystem (Fulcio, Rekor), supports keyless signing, and uses standard OCI registries for signature storage. DCT, based on Notary and The Update Framework (TUF), was more complex to set up and manage, leading to lower adoption. Cosign is generally considered the more modern and user-friendly solution for container image signing.

Cleanup Commands

To remove the resources created during this tutorial:

# Delete the Kyverno policy
kubectl delete -f cosign-policy.yaml

# Delete the signed Pod
kubectl delete -f signed-pod.yaml

# (Optional) Uninstall Kyverno if you installed it via the HelmRelease
kubectl delete -f kyverno-install.yaml
kubectl delete ns kyverno

# Remove the generated keys
rm cosign.key cosign.pub

# Remove the Dockerfile and index.html
rm Dockerfile index.html

# (Optional) Remove the image from your local Docker cache and remote registry
# NOTE: Deleting from remote registry depends on your registry provider.
# For Docker Hub, you might need to use the web UI or API.
docker rmi "${IMAGE_NAME}"

Next Steps / Further Reading

Congratulations! You’ve successfully signed and verified your first container image with Sigstore Cosign and integrated it with Kyverno. This is a significant step towards securing your Kubernetes supply chain. Here are some resources for your continued journey:

• Explore Kyverno’s capabilities: Dive deeper into Kyverno policies to enforce more complex rules, generate SBOMs, or mutate resources. Check out the Securing Container Supply Chains with Sigstore and Kyverno guide for advanced examples.
• Implement KMS integration: For production, learn how to integrate Cosign with your cloud provider’s KMS (e.g., AWS KMS, GCP KMS, Azure Key Vault).
• Understand the Sigstore ecosystem: Learn more about Fulcio, Rekor, and how they contribute to the overall trust model. The official Sigstore documentation is an excellent starting point.
• Software Bill of Materials (SBOMs): Combine image signing with SBOM generation to provide transparency into your image’s contents. Cosign can sign SBOMs directly.
• Supply Chain Security Best Practices: Explore other aspects of supply chain security, including vulnerability scanning (e.g., Trivy), dependency management, and secure coding practices.
• Advanced Kubernetes Security: Beyond image integrity, consider other security aspects like Kubernetes Network Policies, runtime security with eBPF Observability with Hubble, and service mesh security with Istio Ambient Mesh.

Conclusion

Securing the software supply chain is no longer an optional extra but a fundamental requirement for any organization deploying applications to Kubernetes. Sigstore Cosign offers a powerful, user-friendly, and open-source solution to address the critical need for container image signing and verification. By following this guide, you’ve gained hands-on experience with generating keys, signing images, verifying their authenticity, and enforcing these checks directly within your Kubernetes cluster using Kyverno. Embracing tools like Cosign not only enhances your security posture but also builds a stronger, more trustworthy foundation for your cloud-native applications, protecting them from tampering and unauthorized deployments. Continue to build upon these principles to create a truly resilient and secure containerized environment.