Flux CD: Auto-Update Deployments with Images

Introduction

In the fast-paced world of cloud-native development, keeping your Kubernetes deployments up-to-date with the latest container images is a constant challenge. Manually updating image tags in YAML manifests, committing changes to Git, and pushing them through a CI/CD pipeline can be tedious, error-prone, and slow. This process often introduces friction, especially when dealing with frequent image builds, security patches, or hotfixes. Imagine a scenario where a critical vulnerability is discovered, and you need to roll out new images across dozens of services immediately – the manual approach quickly becomes a bottleneck.

Enter GitOps and tools like Flux CD. Flux CD brings the power of GitOps to Kubernetes, ensuring your cluster state always matches the configuration defined in your Git repository. While Flux excels at synchronizing manifest changes, its true power for continuous delivery is unlocked through its image automation capabilities. This feature allows Flux to monitor container registries, detect new image tags, automatically update your Kubernetes manifests in Git, and then apply those changes to your cluster – all without human intervention. This tutorial will guide you through setting up Flux CD image automation to achieve true continuous delivery, making your deployments self-updating and your operations more efficient.

TL;DR Box

Automate Kubernetes deployment image updates using Flux CD’s Image Automation controller. Flux monitors container registries, updates Git manifests with new image tags, and applies changes to your cluster, ensuring deployments are always running the latest versions without manual intervention.

• Install Flux CD: flux bootstrap git --url=<repo-url> --branch=<branch> --path=<path-to-k8s-manifests>
• Define ImageRepository: Tells Flux where to find images.
• Define ImagePolicy: Specifies how to select the latest image tag.
• Annotate Deployment: Instructs Flux to update the image tag in your manifest.
• Commit and Push: Flux will now monitor and update your deployments automatically.

Prerequisites

To follow this guide, you’ll need the following:

• A running Kubernetes cluster (v1.20 or later). You can use Minikube, Kind, or a cloud-managed cluster (EKS, GKE, AKS).
• kubectl installed and configured to connect to your cluster. Refer to the official Kubernetes documentation for installation instructions.
• git installed.
• A Git repository (GitHub, GitLab, Bitbucket, Azure DevOps, etc.) to store your Kubernetes manifests. This repository will be used by Flux.
• A container registry (Docker Hub, AWS ECR, GCP Container Registry, Azure Container Registry, etc.) with at least one image available that you can push new tags to.
• The Flux CLI installed. Follow the official Flux CD installation guide.
• Basic understanding of Kubernetes Deployments, Services, and GitOps principles.

Step-by-Step Guide to Flux CD Image Automation

Step 1: Bootstrap Flux CD to Your Cluster

First, we need to install Flux CD on your Kubernetes cluster and configure it to synchronize with your Git repository. This process is called bootstrapping. Flux will install its controllers (Source, Kustomize, Helm, Notification, and Image Automation) into your cluster and set up the necessary Git repository synchronization.

Choose an empty Git repository or a specific branch/directory within an existing one. Flux will manage everything in that path. For this guide, we’ll assume you have a GitHub repository named flux-image-automation-demo.

Before bootstrapping, ensure you’ve authenticated the Flux CLI with your Git provider. For GitHub, you might need a Personal Access Token (PAT) with repo scope. Set it as an environment variable or provide it directly.

# Replace with your GitHub username and repository name
export GITHUB_USER="your-github-username"
export GITHUB_REPO="flux-image-automation-demo"
export GITHUB_TOKEN="ghp_YOUR_GITHUB_PAT" # Or use flux bootstrap --token-auth

# Bootstrap Flux CD
flux bootstrap github \
  --owner="$GITHUB_USER" \
  --repository="$GITHUB_REPO" \
  --branch=main \
  --path=./clusters/my-cluster \
  --personal

This command will:

• Install Flux controllers into a flux-system namespace.
• Create a GitRepository custom resource pointing to your specified Git repository.
• Create a Kustomization custom resource to apply the manifests from the ./clusters/my-cluster path within your repository.
• Generate a deploy key and add it to your GitHub repository (if --personal is used and you have the PAT).

Verify

After a few minutes, all Flux controllers should be running, and your repository should be synchronized.

flux check --pre
flux check
kubectl get pods -n flux-system

Expected output:

# flux check --pre output will show all prerequisites are met
► checking prerequisites
✔ Kubernetes 1.27.3 >=1.20.0
✔ prerequisites checks passed

# flux check output will show all Flux components are healthy
► checking toolkit components
✔ all components are healthy

# kubectl get pods -n flux-system
NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-74c65f97b6-j2d8x           1/1     Running   0          5m
image-automation-controller-7b568779b-2lq7c 1/1     Running   0          5m
image-reflector-controller-58474d75d-6h82g 1/1     Running   0          5m
kustomize-controller-59d4f9b87b-h4bqs      1/1     Running   0          5m
notification-controller-584444585c-l6v9l   1/1     Running   0          5m
source-controller-596956799-l7b7d          1/1     Running   0          5m

Step 2: Create a Sample Deployment

Now, let’s create a simple NGINX deployment that we want Flux to automatically update. We’ll use a placeholder image tag initially. Create a new directory in your Git repository, for example, ./apps/nginx, and add the following files.

This deployment will use a generic NGINX image. We’ll specifically target the image field for automation. Note the # {"$imagepolicy": "flux-system:nginx-policy"} comment. This is a special marker that the Image Automation controller looks for to know where to inject the updated image tag. Without it, Flux won’t know which field to modify.

# ./apps/nginx/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-app
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
      - name: nginx
        image: nginx:1.21.6 # {"$imagepolicy": "flux-system:nginx-policy"}
        ports:
        - containerPort: 80

# ./apps/nginx/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

Commit these files to your Git repository and push them. Then, tell Flux to synchronize this new application by adding a Kustomization resource in your clusters/my-cluster path.

# ./clusters/my-cluster/nginx-app.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: nginx-app
  namespace: flux-system
spec:
  interval: 1m0s
  path: ./apps/nginx
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: default

git add .
git commit -m "Add nginx deployment and kustomization"
git push origin main

Verify

Flux should detect the new Kustomization and deploy the NGINX application. This might take a minute or two.

flux get kustomizations -n flux-system
kubectl get deployment nginx-app -n default

Expected output:

# flux get kustomizations -n flux-system
NAME          REVISION        SUSPENDED   READY   MESSAGE                                         LAST APPLY
flux-system   main/c0a1b2c3   False       True    Applied revision: main/c0a1b2c3                 1m
nginx-app     main/d4e5f6g7   False       True    Applied revision: main/d4e5f6g7                 30s

# kubectl get deployment nginx-app -n default
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
nginx-app   1/1     1            1           45s

Step 3: Define ImageRepository

The ImageRepository custom resource tells Flux where to find the container images. It specifies the registry and the image name Flux should monitor. Flux’s Image Reflector controller will periodically scan this repository for new tags.

Create this manifest in your Git repository, for example, in ./clusters/my-cluster/image-automation.yaml.

# ./clusters/my-cluster/image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: nginx
  namespace: flux-system
spec:
  image: nginx
  interval: 1m0s # How often to scan the registry

This configuration tells Flux to monitor the nginx image in Docker Hub (which is the default if no registry is specified). If you’re using a private registry, you’ll need to specify the full image path (e.g., myregistry.com/myorg/nginx) and create an ImagePullSecret for Flux to authenticate. For more advanced networking configurations or private registry access, you might explore solutions like Cilium WireGuard Encryption if your registry is in a separate network segment.

git add .
git commit -m "Add ImageRepository for nginx"
git push origin main

Verify

Check if the ImageRepository is created and has successfully scanned for images.

flux get imagerepositories -n flux-system

Expected output (tags will vary):

NAME    IMAGE   LAST SCAN               LAST SUCCESS            TAGS    READY   MESSAGE
nginx   nginx   2023-10-27T10:00:00Z    2023-10-27T10:00:00Z    1300    True    successful scan, found 1300 tags

Step 4: Define ImagePolicy

The ImagePolicy custom resource defines the rules for selecting the “latest” image tag from the tags discovered by the ImageRepository. This is crucial for determining which new image version Flux should apply. You can specify various policies, such as semantic versioning, alphabetical sorting, or even regular expressions.

Continue adding to ./clusters/my-cluster/image-automation.yaml:

# ./clusters/my-cluster/image-automation.yaml (append to existing file)
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: nginx-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: nginx
  filterTags:
    pattern: '^1\.21\.(?P<version>[0-9]+)$' # Match tags like 1.21.x
  policy:
    semver:
      range: '1.21.x' # Select the highest semantic version within 1.21.x

In this example, we’re using a semantic versioning policy to select the highest patch version within the 1.21.x range. This is a common and recommended approach for production environments to ensure stability while still getting updates. If you wanted the absolute latest tag, you could use alphabetical: {} or numerical: {} with a different pattern. For more complex scenarios, refer to the Flux Image Automation documentation.

git add .
git commit -m "Add ImagePolicy for nginx"
git push origin main

Verify

Check if the ImagePolicy has been created and has resolved a latest image tag.

flux get imagepolicies -n flux-system

Expected output (resolved image will vary):

NAME          IMAGE   LAST UPDATE             RESOLVED IMAGE   READY   MESSAGE
nginx-policy  nginx   2023-10-27T10:05:00Z    nginx:1.21.6     True    successfully resolved image 'nginx:1.21.6'

Step 5: Configure ImageUpdateAutomation

The ImageUpdateAutomation custom resource is the heart of this process. It orchestrates the entire automation flow: it watches ImagePolicy resources, detects when a new image is resolved, finds the corresponding marker in your Git repository, updates the manifest, commits the change, and pushes it back to Git. This is where the magic happens, effectively closing the GitOps loop.

Add this to your ./clusters/my-cluster/image-automation.yaml file:

# ./clusters/my-cluster/image-automation.yaml (append to existing file)
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcd@example.com
        name: fluxcd
      messageTemplate: '{{ .Updated.String }}' # Commit message template
  interval: 1m0s # How often to check for updates and commit
  sourceRef:
    kind: GitRepository
    name: flux-system
  update:
    # Path to the directory containing the manifests that Flux should update
    path: ./apps/nginx
    strategy: Set

Key parameters:

• git.checkout.ref.branch: The branch Flux should update.
• git.commit.messageTemplate: Defines the commit message when Flux pushes changes. {{ .Updated.String }} is a common choice, showing the updated image.
• interval: How often the automation runs.
• sourceRef: Points to the GitRepository containing your manifests.
• update.path: Crucially, this is the path within your Git repository where Flux will look for files to update. This should correspond to the path where your deployment.yaml (with the # {"$imagepolicy": "flux-system:nginx-policy"} marker) resides.

git add .
git commit -m "Add ImageUpdateAutomation"
git push origin main

Verify

Check if the ImageUpdateAutomation resource is ready.

flux get imageupdateautomations -n flux-system

Expected output:

NAME          LAST RUN                LAST COMMIT   LAST PUSH               READY   MESSAGE
flux-system   2023-10-27T10:10:00Z    main/a1b2c3d4   2023-10-27T10:10:00Z    True    Automation run succeeded

Step 6: Trigger an Image Update

Now, let’s simulate a new image being published. We’ll push a new tag for our NGINX image to Docker Hub. Since we’re using the public NGINX image, we can’t push to it directly. Instead, we’ll manually change the ImageRepository to point to a different, higher version of NGINX, or you can use your own image where you control the tags.

For demonstration, let’s pretend a new nginx:1.21.7 exists. We will temporarily modify our ImagePolicy to only filter for 1.21.7 to force Flux to pick it up, then revert it. In a real scenario, you’d just push a new image tag to your private registry.

Edit ./clusters/my-cluster/image-automation.yaml to temporarily change the filter pattern in nginx-policy:

# ./clusters/my-cluster/image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: nginx-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: nginx
  filterTags:
    # Temporarily change pattern to force a specific tag
    pattern: '^1\.21\.7$' 
  policy:
    semver:
      range: '1.21.x'

git add .
git commit -m "Force NGINX 1.21.7 for demo"
git push origin main

Alternatively, if you have your own image, you would simply build and push a new tag:

docker build -t myregistry.com/myorg/myapp:1.0.1 .
docker push myregistry.com/myorg/myapp:1.0.1

Verify

Watch your Git repository for a new commit from Flux. Then, check your deployment.

# Check Git history (e.g., on GitHub website or with git log)
# You should see a commit like: "nginx:1.21.6@sha256:..." or "nginx:1.21.7@sha256:..."

# Check the deployment image
kubectl get deployment nginx-app -n default -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected output (example, will vary based on resolved image):

nginx:1.21.7

You can also check the Flux logs to see the automation in action:

kubectl logs -f -n flux-system deploy/image-automation-controller

You should see log entries indicating that the controller is detecting changes, updating manifests, and committing them to Git.

Once you’ve verified the update, remember to revert the filterTags in your ImagePolicy back to the original pattern if you want to resume monitoring the broader 1.21.x series, then commit and push again.

Production Considerations

• Image Pull Secrets: For private registries, ensure Flux has access to the necessary ImagePullSecrets. These secrets should be referenced by the workload (Deployment, StatefulSet) and potentially by the Flux ImageRepository if the image reflector needs to authenticate to scan tags.
• Tagging Strategy: Carefully consider your image tagging strategy. Semantic versioning (e.g., 1.2.3) is highly recommended. Avoid using mutable tags like latest in production, as they can lead to unpredictable deployments. Flux’s ImagePolicy works best with well-defined, immutable tags.
• Granular Automation: You can have multiple ImageRepository and ImagePolicy resources for different images or different update strategies (e.g., critical security patches vs. minor feature updates).
• Commit Message Templates: Use descriptive commit message templates in ImageUpdateAutomation to easily track automated updates in your Git history. Including the old and new image tags is very helpful.
• Rate Limiting: Be mindful of Docker Hub’s rate limits if you’re pulling public images frequently. For production, consider using a caching proxy or mirroring images to a private registry.
• Monitoring and Alerting: Set up monitoring for your Flux controllers, especially the Image Reflector and Image Automation controllers. Alerts for failed scans or automation runs are crucial. You can leverage tools like Prometheus and Grafana for this. For advanced observability, consider using eBPF Observability with Hubble to monitor network interactions related to image pulls.
• Rollback Strategy: While Flux automates updates, ensure you have a clear rollback strategy. If an automated image update introduces a bug, you’ll need a way to quickly revert to a previous working version by reverting the commit in Git. Flux will then synchronize back to the reverted state.
• Integration with CI/CD: Image automation complements your CI/CD pipeline. Your CI pipeline should build and push new images with unique tags. Flux then takes over for deployment. Consider integrating Sigstore and Kyverno to ensure only signed and trusted images are deployed.
• Resource Limits: Ensure Flux controllers have appropriate CPU and memory limits/requests in the flux-system namespace to prevent resource exhaustion.

Troubleshooting

1. Issue: Flux controllers are not running or are in a CrashLoopBackOff state.

   Solution: Check the logs of the problematic controller pod in the flux-system namespace. Common issues include incorrect Git repository URL, invalid SSH keys, or network connectivity problems. Ensure your Git PAT or SSH key is correctly configured and has the necessary permissions.

   kubectl logs -f -n flux-system deploy/source-controller
   kubectl describe pod -n flux-system <pod-name>
   

2. Issue: ImageRepository status shows False or “failed scan”.

   Solution: This usually means Flux cannot access the container registry or cannot find the specified image. Verify the image field in your ImageRepository is correct. If it’s a private registry, ensure Flux has the correct ImagePullSecret configured and mounted, and that the secret is in the same namespace as the ImageRepository (flux-system by default).

   flux get imagerepositories -n flux-system -o yaml
   kubectl logs -f -n flux-system deploy/image-reflector-controller
   

3. Issue: ImagePolicy is not resolving a new image tag.

   Solution: Check the filterTags and policy sections of your ImagePolicy. The pattern might not be matching the new image tags, or the policy might not be selecting the desired tag. Test your regex pattern with the actual image tags available in your registry. Ensure the ImageRepositoryRef points to the correct ImageRepository.

   flux get imagepolicies -n flux-system -o yaml
   

4. Issue: Deployment image is not updating in the cluster, but Flux shows a successful automation run.

   Solution: This indicates that Flux successfully updated the manifest in Git, but the Kustomization or HelmRelease responsible for applying that manifest to the cluster is not picking up the change. Verify the interval of your Kustomization and ensure its sourceRef is correctly pointing to the Git repository. Also, check the path in the Kustomization to ensure it covers the updated manifest.

   flux get kustomizations -n flux-system
   flux reconcile kustomization <kustomization-name> -n flux-system --with-source
   

5. Issue: Flux is making commits to Git, but the image tag is not changing in the YAML.

   Solution: The most common reason is a missing or incorrect image automation marker. Ensure your deployment’s image field has the exact comment: # {"$imagepolicy": "flux-system:<your-image-policy-name>"}. Also, double-check that the ImageUpdateAutomation‘s update.path correctly points to the directory containing the manifest to be updated.

   # Check your deployment.yaml file content in Git
   # Check the ImageUpdateAutomation resource
   flux get imageupdateautomations -n flux-system -o yaml
   

6. Issue: Flux is updating the image in Git, but it’s not the latest version or the desired version.

   Solution: Review your ImagePolicy carefully. The filterTags pattern or the policy (e.g., semver.range) might be too restrictive or selecting an unexpected tag. Ensure your image tags follow a consistent format that the policy can correctly interpret. You might need to adjust the regex or semantic version range.

   flux get imagepolicies -n flux-system -o yaml
   

FAQ Section

1. What is the difference between Flux CD and Argo CD?

   Both Flux CD and Argo CD are popular GitOps tools for Kubernetes. While they share the core principle of GitOps (desired state in Git, actual state in cluster), they have different architectures and feature sets. Flux CD is built on a set of independent controllers, while Argo CD is a single application. Flux tends to be more declarative and Git-centric, often preferring Kustomize, whereas Argo CD has a richer UI and supports various templating tools out-of-the-box. For more insights into CI/CD and deployment strategies, consider how tools like Kubernetes Gateway API can enhance traffic management for these deployments.

2. Can Flux CD automate updates for Helm charts?

   Yes, Flux CD can automate updates for Helm charts. For Helm releases, instead of annotating a Deployment, you would update the ImagePolicy reference directly within your HelmRelease resource. Flux’s Helm controller will then render the chart with the new image tag. You can find more details in the Flux CD Helm documentation.

3. How does Flux handle rollbacks if a new image causes issues?

   Flux CD itself doesn’t automatically roll back deployments based on application health. Its primary function is to synchronize the cluster state with Git. If a new image causes an issue, the standard GitOps practice is to revert the commit in your Git repository that introduced the problematic image tag. Flux will then detect this reversion and apply the previous, working state to your cluster. This maintains Git as the single source of truth.

4. Is it safe to use latest tag with Flux Image Automation?

   While technically possible to configure ImagePolicy to select the latest tag, it is generally discouraged, especially in production. The latest tag is mutable and can change without warning, making deployments unpredictable and rollbacks difficult. It’s best practice to use immutable, versioned tags (e.g., 1.0.0, v2.1-rc1) and leverage Flux’s semantic versioning or alphabetical policies to manage updates safely. For advanced scaling and cost optimization with such deployments, consider exploring tools like Karpenter for Kubernetes cost optimization.

5. How can I use Flux Image Automation with private container registries?

   To use private registries, you need to provide Flux with credentials. This is typically done by creating a Kubernetes Secret of type kubernetes.io/dockerconfigjson containing your registry credentials. This secret should be in the flux-system namespace. You then reference this secret in your ImageRepository resource using the secretRef field. For example:

   apiVersion: image.toolkit.fluxcd.io/v1beta1
   kind: ImageRepository
   metadata:
     name: my-private-app
     namespace: flux-system
   spec:
     image: myregistry.com/myorg/myapp
     interval: 1m0s
     secretRef:
       name: my-registry-secret
   

Cleanup Commands

To remove Flux CD and all its components from your cluster, as well as the resources created during this tutorial, follow these steps:

# Remove Flux CD from the cluster
flux uninstall --namespace=flux-system --keep-namespace=false

# Remove the manifests from your Git repository (optional, but good practice)
# Navigate to your Git repository root
# git rm -rf clusters/my-cluster apps/nginx
# git commit -m "Remove Flux CD and demo app"
# git push origin main

# Delete the Kubernetes namespace for the application if not default
# kubectl delete namespace default # BE CAREFUL: This deletes everything in default namespace

Confirm Flux components are gone:

kubectl get ns flux-system
# Should show "NotFound"

Next Steps / Further Reading

• Explore more advanced Flux CD features, such as multi-tenancy, secrets management with SOPS, and notifications. The Flux CD documentation on repository structure is an excellent resource.
• Deep dive into Kubernetes concepts to better understand the underlying platform.
• Learn about GitOps principles and how they can transform your development and operations workflows.
• Investigate how Flux CD integrates with other CNCF projects like Prometheus for monitoring or OPA/Kyverno for policy enforcement. For instance, combining image automation with policies to ensure compliance is a powerful pattern, as seen in Securing Container Supply Chains with Sigstore and Kyverno.
• Consider using a service mesh like Istio Ambient Mesh to manage traffic, security, and observability for your automatically updated applications.

Conclusion

Flux CD’s image automation capabilities are a game-changer for continuous delivery in Kubernetes. By leveraging Git as the single source of truth and automating the process of detecting new images, updating manifests, and applying changes, Flux significantly reduces operational overhead and speeds up the delivery of new features and critical updates. This guide has walked you through the complete setup, from bootstrapping Flux to triggering your first automated image update. Embrace this powerful GitOps pattern to build a more efficient, reliable, and scalable cloud-native platform.