Automate Image Promotion with ArgoCD Updater

Introduction

In the fast-paced world of cloud-native development, continuous delivery is paramount. However, manually updating image tags in Kubernetes manifests, especially across multiple environments like development, staging, and production, can be a tedious, error-prone, and time-consuming process. This friction often leads to slower deployment cycles and increased operational overhead. What if you could automate the promotion of new container images across your environments, directly from your GitOps repository?

Enter ArgoCD Image Updater. As an extension to the popular GitOps continuous delivery tool ArgoCD, Image Updater allows you to automatically detect new image versions in your container registries and update your Kubernetes manifests in Git, triggering a seamless ArgoCD synchronization. This powerful combination transforms your image promotion strategy from a manual chore into an automated, GitOps-driven workflow, ensuring that your applications are always running the latest, approved images without human intervention.

# Install ArgoCD Image Updater
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

# Create a sample application in ArgoCD
argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default

# Annotate an ArgoCD Application for Image Updater
argocd app set guestbook \
  -p image.tag=latest \
  --annotation argocd-image-updater.argoproj.io/image-list="guestbook=ghcr.io/argoproj/argocd-example-apps/guestbook" \
  --annotation argocd-image-updater.argoproj.io/guestbook.update-strategy="latest" \
  --annotation argocd-image-updater.argoproj.io/guestbook.allow-tags="^v\\d+\\.\\d+\\.\\d+$" \
  --annotation argocd-image-updater.argoproj.io/write-back-target="argocd"

# Trigger an immediate update check
kubectl -n argocd patch argocd.argoproj.io/argocd --type=json -p='[{"op": "replace", "path": "/spec/image", "value": "quay.io/argoproj/argocd:v2.9.3"}]'
# (This is an example for ArgoCD itself, for an app, you'd push a new image tag to the registry)

Prerequisites

Before diving into the installation and configuration of ArgoCD Image Updater, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (version 1.18+ recommended).
• kubectl: The Kubernetes command-line tool configured to connect to your cluster.
• ArgoCD Installation: A functional ArgoCD instance deployed in your cluster. This guide assumes ArgoCD is installed in the argocd namespace.
• Git Repository: A Git repository hosting your Kubernetes manifests that ArgoCD is configured to deploy. You’ll need push access to this repository for Image Updater to write back changes.
• Container Registry: Access to a container registry (e.g., Docker Hub, Google Container Registry, GitHub Container Registry, Quay.io) where your application images are stored.
• Basic GitOps Knowledge: Familiarity with GitOps principles and how ArgoCD operates.
• Helm (Optional): If you plan to install ArgoCD or Image Updater via Helm, ensure you have it installed.

Step-by-Step Guide: Automated Image Promotion with ArgoCD Image Updater

Step 1: Install ArgoCD Image Updater

The first step is to deploy the ArgoCD Image Updater component into your Kubernetes cluster. It typically runs alongside your ArgoCD instance, watching for new image tags and updating your Git repository.

Image Updater runs as a separate deployment that periodically scans your configured ArgoCD applications. When it detects a new image tag in a specified container registry that matches your criteria, it updates the corresponding image tag in your Git repository. This change in Git then triggers ArgoCD to synchronize and deploy the new image.

We’ll install it using the provided manifest, which creates the necessary Deployment, ServiceAccount, Role, RoleBinding, and ConfigMap within the argocd namespace.

# Ensure you are in the correct namespace context, or specify -n argocd
kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

Verify: Check if the ArgoCD Image Updater pod is running and healthy.

kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-image-updater

Expected Output:

NAME                                  READY   STATUS    RESTARTS   AGE
argocd-image-updater-XXXXX-YYYYY   1/1     Running   0          XmYs

Step 2: Configure ArgoCD Image Updater with Registry Credentials (if needed)

If your container images are hosted in private registries, ArgoCD Image Updater needs credentials to pull image metadata (tags). These credentials are stored as Kubernetes Secrets and referenced in the Image Updater’s configuration.

For this example, we’ll assume a public registry or that your cluster’s node IAM roles (for AWS ECR, GCP GCR, Azure ACR) are configured correctly. If you need private registry access, you’d create a secret. The secret type should be kubernetes.io/dockerconfigjson or Opaque with .dockerconfigjson key.

Below is an example for Docker Hub. Replace <your-docker-username> and <your-docker-password> with your actual credentials. For other registries, consult their documentation for the correct docker login format.

# Create a docker config.json file
mkdir -p ~/.docker
echo "{ \"auths\": { \"docker.io\": { \"auth\": \"$(echo -n "<your-docker-username>:<your-docker-password>" | base64 -w0)\" } } }" > ~/.docker/config.json

# Create the secret in the argocd namespace
kubectl create secret generic argocd-image-updater-secret \
    --from-file=.dockerconfigjson=~/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson \
    -n argocd

# Clean up local file
rm ~/.docker/config.json

Verify: Check that the secret has been created.

kubectl get secret argocd-image-updater-secret -n argocd -o yaml

Expected Output (snippet):

apiVersion: v1
data:
  .dockerconfigjson: eyJhdXR... # (base64 encoded string)
kind: Secret
metadata:
  name: argocd-image-updater-secret
  namespace: argocd
type: kubernetes.io/dockerconfigjson

Next, you would typically configure Image Updater to use this secret. This is done by editing the argocd-image-updater-config ConfigMap in the argocd namespace.

# Patch the argocd-image-updater-config ConfigMap to reference the secret
# This adds a new entry under 'registries.conf'
kubectl patch configmap argocd-image-updater-config -n argocd --type merge -p '{"data":{"registries.conf":"registries:\n  - name: Docker Hub\n    prefix: docker.io\n    api_url: https://index.docker.io/v1/\n    credentials: <secret:argocd-image-updater-secret/>\n    default: true"}}'

Verify: Confirm the ConfigMap has been updated.

kubectl get configmap argocd-image-updater-config -n argocd -o yaml

Expected Output (snippet):

apiVersion: v1
data:
  registries.conf: |
    registries:
      - name: Docker Hub
        prefix: docker.io
        api_url: https://index.docker.io/v1/
        credentials: 
        default: true
kind: ConfigMap
metadata:
  name: argocd-image-updater-config
  namespace: argocd

Step 3: Prepare an ArgoCD Application for Image Updater

ArgoCD Image Updater works by adding specific annotations to your ArgoCD Application resources. These annotations tell Image Updater which images to track, what update strategy to use, and where to write back the changes.

For this step, we’ll use a simple guestbook application from the ArgoCD example apps repository. First, ensure you have an ArgoCD Application defined. If not, create one. If you’re using a private Git repository, ensure ArgoCD has access credentials configured.

We’ll create a basic ArgoCD application pointing to a simple guestbook example.

# Ensure ArgoCD CLI is installed and configured
# If not, follow: https://argoproj.github.io/cd/cli_installation/

# Login to ArgoCD (replace with your ArgoCD server address)
argocd login <YOUR_ARGOCD_SERVER_IP_OR_HOSTNAME> --username admin --password <YOUR_ARGOCD_PASSWORD> --insecure --grpc-web

# Create a sample ArgoCD application
argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --revision HEAD \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default \
  --sync-policy automated \
  --auto-prune --self-heal

# Wait for the app to sync
argocd app wait guestbook --health --sync

Verify: Check the application status in ArgoCD.

argocd app get guestbook

Expected Output (snippet):

Name:               guestbook
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://<YOUR_ARGOCD_SERVER>/applications/guestbook
Repo:               https://github.com/argoproj/argocd-example-apps.git
Path:               guestbook
Head:               HEAD
Sync Policy:        Automated
Sync Status:        Synced
Health Status:      Healthy

GROUP        KIND        NAMESPACE  NAME          STATUS  HEALTH   
apps         Deployment  default    guestbook-ui  Synced  Healthy  
core         Service     default    guestbook-ui  Synced  Healthy  

Step 4: Annotate the ArgoCD Application

Now, we’ll add the necessary annotations to the guestbook ArgoCD Application resource. These annotations instruct ArgoCD Image Updater on how to track and update images.

• argocd-image-updater.argoproj.io/image-list: Specifies the images to track. Format: <image-name>=<registry/repository>.
• argocd-image-updater.argoproj.io/<image-name>.update-strategy: Defines how to select the new tag (e.g., latest, semver, digest).
• argocd-image-updater.argoproj.io/<image-name>.allow-tags: A regular expression to filter allowed tags. This is crucial for controlling promotions (e.g., only promoting specific release versions).
• argocd-image-updater.argoproj.io/write-back-target: Specifies where Image Updater should write back the updated image tag. Common options are argocd (updates the Application resource) or git (pushes to the Git repository). For true GitOps, git is preferred.
• argocd-image-updater.argoproj.io/git-branch: The target branch in your Git repository for the write-back operation.

We will configure the Image Updater to track the guestbook image and update it to the latest semantic version tag found, writing back to the Git repository on the main branch. Make sure your Git repository has a main branch, or adjust accordingly.

Important: For write-back-target: git, Image Updater needs Git credentials. This is typically configured via the argocd-image-updater-ssh-secret or argocd-image-updater-git-secret secrets, which hold SSH keys or HTTPS tokens. For simplicity in this example, we’ll use write-back-target: argocd, which updates the Application resource itself. For production, git is highly recommended, requiring proper Git credentials for Image Updater.

Let’s update our ArgoCD Application to track the guestbook image.

# Annotate the ArgoCD Application resource
# Replace 'ghcr.io/argoproj/argocd-example-apps/guestbook' if your image is hosted elsewhere
# Note: The example app image uses tags like 'v1.0.0', 'v1.0.1', 'v1.0.2', etc.
kubectl annotate applications.argoproj.io guestbook -n argocd \
  argocd-image-updater.argoproj.io/image-list="guestbook=ghcr.io/argoproj/argocd-example-apps/guestbook" \
  argocd-image-updater.argoproj.io/guestbook.update-strategy="semver" \
  argocd-image-updater.argoproj.io/guestbook.allow-tags="^v\\d+\\.\\d+\\.\\d+$" \
  argocd-image-updater.argoproj.io/write-back-target="argocd" \
  --overwrite

Verify: Check the annotations on the ArgoCD Application.

kubectl get application guestbook -n argocd -o yaml

Expected Output (snippet):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/guestbook.allow-tags: ^v\d+\.\d+\.\d+$
    argocd-image-updater.argoproj.io/guestbook.update-strategy: semver
    argocd-image-updater.argoproj.io/image-list: guestbook=ghcr.io/argoproj/argocd-example-apps/guestbook
    argocd-image-updater.argoproj.io/write-back-target: argocd
  name: guestbook
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
  # ... other fields

Step 5: Trigger an Update and Observe

Now that everything is configured, ArgoCD Image Updater will periodically check for new image tags. To observe an update, you would typically push a new image with a higher semantic version tag to your configured container registry. Since we’re using a public example, we can simulate this by manually setting an older tag and then letting the updater find a newer one.

First, let’s manually set the image tag in the ArgoCD application to an older version (e.g., v1.0.0) so the updater has something to “update” from. This is done by modifying the spec.source.helm.parameters or spec.source.kustomize.images depending on your manifest structure. For the simple guestbook, it’s typically a parameter.

Let’s check the current image tag used by the guestbook application. This might require inspecting the deployed Deployment or the source manifests directly.

The guestbook example uses a Helm chart structure, where the image tag is defined in values.yaml. When ArgoCD deploys it, it uses the default image ghcr.io/argoproj/argocd-example-apps/guestbook:v1. To make Image Updater work, we need to explicitly tell ArgoCD which parameter to update.

We need to add an annotation to specify the path to the image tag within the application’s manifest. For Helm charts, this is usually image.tag parameter.

# Add annotation to specify the image path within the Helm chart
kubectl annotate applications.argoproj.io guestbook -n argocd \
  argocd-image-updater.argoproj.io/guestbook.helm.image-name="guestbook.image.repository" \
  argocd-image-updater.argoproj.io/guestbook.helm.image-tag="guestbook.image.tag" \
  --overwrite

# Now, manually set an older version to simulate a starting point
# This will update the ArgoCD Application's parameters.
argocd app set guestbook -p guestbook.image.tag=v1.0.0 --revision HEAD

Verify: Check the application parameters and the annotations.

kubectl get application guestbook -n argocd -o yaml | grep -A 5 "guestbook.image"
kubectl get application guestbook -n argocd -o yaml | grep "helm.image"

Expected Output (snippet):

# from first command
    helm:
      parameters:
      - name: guestbook.image.repository
        value: ghcr.io/argoproj/argocd-example-apps/guestbook
      - name: guestbook.image.tag
        value: v1.0.0 # This should be v1.0.0 now
    path: guestbook
# from second command
    argocd-image-updater.argoproj.io/guestbook.helm.image-name: guestbook.image.repository
    argocd-image-updater.argoproj.io/guestbook.helm.image-tag: guestbook.image.tag

Now, let Image Updater do its job. It runs on a schedule (default 3 minutes). You can check its logs.

# Get the image updater pod name
UPDATER_POD=$(kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-image-updater -o jsonpath='{.items[0].metadata.name}')

# Stream logs
kubectl logs -f $UPDATER_POD -n argocd

Look for messages indicating that it’s checking the guestbook application and potentially performing an update. You should see something like:

...
time="2023-10-27T10:00:00Z" level=info msg="Starting image update check for application guestbook" application=guestbook
time="2023-10-27T10:00:00Z" level=info msg="Found 1 images to check for application guestbook" application=guestbook
time="2023-10-27T10:00:01Z" level=info msg="Checking image 'ghcr.io/argoproj/argocd-example-apps/guestbook' with current tag 'v1.0.0'" application=guestbook image_name=guestbook
time="2023-10-27T10:00:02Z" level=info msg="Found new image 'ghcr.io/argoproj/argocd-example-apps/guestbook:v1.0.2' for image 'guestbook'" application=guestbook image_name=guestbook new_tag=v1.0.2 old_tag=v1.0.0
time="2023-10-27T10:00:02Z" level=info msg="Successfully updated image 'guestbook' to 'v1.0.2' for application 'guestbook'" application=guestbook image_name=guestbook new_tag=v1.0.2 old_tag=v1.0.0
time="2023-10-27T10:00:02Z" level=info msg="Application guestbook update successful" application=guestbook
...

After a successful update, ArgoCD will detect the change in its Application resource and re-sync, deploying the new image.

Verify: Check the ArgoCD Application status and the deployed image.

argocd app get guestbook

Expected Output (snippet):

...
Parameters:
  Name:                  guestbook.image.repository
  Value:                 ghcr.io/argoproj/argocd-example-apps/guestbook
  Name:                  guestbook.image.tag
  Value:                 v1.0.2   # <-- This should now be the latest version found by Image Updater
...

You can also inspect the deployed Kubernetes Deployment:

kubectl get deployment guestbook-ui -n default -o yaml | grep "image:"

Expected Output:

      image: ghcr.io/argoproj/argocd-example-apps/guestbook:v1.0.2 # <-- This should be the updated image

Congratulations! You’ve successfully configured ArgoCD Image Updater to automatically promote a new image version. This automated process significantly streamlines your continuous delivery pipeline, reducing manual effort and potential errors. For more advanced networking configurations in Kubernetes, consider reviewing our guide on Kubernetes Gateway API vs Ingress.

Production Considerations

Implementing ArgoCD Image Updater in a production environment requires careful planning and robust configuration to ensure stability, security, and traceability.

1. Git Write-Back Strategy: While write-back-target: argocd is easy for demos, in production, always prefer write-back-target: git. This ensures that your Git repository remains the single source of truth, and every image promotion is a commit in Git, providing a full audit trail. For this, Image Updater needs Git credentials (SSH key or HTTPS token) configured as a Kubernetes Secret.
2. Git Branching Strategy: Define a clear branching strategy for image promotions. For example, Image Updater could push to a dev branch, and a separate CI/CD pipeline or manual review process would promote changes from dev to staging, and then to main/production. Use argocd-image-updater.argoproj.io/git-branch to specify the target branch.
3. Tag Filtering (allow-tags): Always use robust regular expressions with allow-tags to prevent accidental promotion of unwanted tags (e.g., experimental builds, old versions). For example, ^v\d+\.\d+\.\d+$ for semantic versioned tags or ^prod-\d{8}-\d{6}$ for date-stamped production builds.
4. Registry Credentials Security: Store registry credentials (.dockerconfigjson) in Kubernetes Secrets with appropriate RBAC restrictions. Ensure the Image Updater ServiceAccount has read-only access to these secrets. For cloud-native registries (ECR, GCR, ACR), leverage IAM roles for service accounts (IRSA for EKS, Workload Identity for GKE, Workload Identity for AKS) to avoid long-lived credentials.
5. Rate Limiting and Throttling: Be mindful of rate limits imposed by your container registry. If you have many applications and frequently push new images, Image Updater might hit these limits. Configure the argocd-image-updater-config ConfigMap with appropriate registries.<name>.ratelimit settings if necessary.
6. Observability and Alerting: Integrate Image Updater logs into your centralized logging solution (e.g., Loki, Fluentd). Set up alerts for failed updates, authentication issues, or unexpected image tag promotions. Tools like eBPF Observability with Hubble can provide deeper network insights if issues arise.
7. Rollback Strategy: While Image Updater automates promotion, always have a clear rollback strategy. If a new image causes issues, you should be able to quickly revert the Git commit made by Image Updater or manually pin to a previous working tag.
8. Testing Environments: Always promote images through a series of testing environments (dev -> staging -> production). Image Updater can be configured to target different branches or ArgoCD applications for each environment.
9. Resource Management: Monitor the resource consumption of the argocd-image-updater pod. Adjust CPU and memory limits/requests as needed based on the number of applications and images it tracks.
10. Security Scanning: Ensure all images pushed to your registry are scanned for vulnerabilities before Image Updater picks them up. Integrating Sigstore and Kyverno can provide strong supply chain security by enforcing image signing and verification.

Troubleshooting

Here are common issues encountered with ArgoCD Image Updater and their solutions:

1. Image Updater pod is CrashLoopBackOff or Pending.

   Issue: The pod fails to start or keeps restarting.

   Solution: Check the pod logs for specific error messages.

   kubectl logs -f -n argocd <argocd-image-updater-pod-name>
   kubectl describe pod -n argocd <argocd-image-updater-pod-name>
   

   Common causes include misconfigured ConfigMap (e.g., invalid YAML syntax in registries.conf) or missing ServiceAccount permissions. Ensure the ServiceAccount argocd-image-updater has the necessary RBAC permissions to read ArgoCD Application resources and secrets.

2. Image Updater doesn’t find new image tags.

   Issue: New images are pushed to the registry, but the ArgoCD application is not updated.

   Solution:

   1. Check Updater Logs: Look for messages like “Starting image update check…” and “Checking image…” to confirm it’s processing your application.

      kubectl logs -n argocd <argocd-image-updater-pod-name> | grep "application=<your-app-name>"

   2. Verify Annotations: Double-check that annotations on the ArgoCD Application are correct, especially image-list, update-strategy, and allow-tags. A typo can prevent detection.
   3. Registry Credentials: Ensure Image Updater has correct credentials for the registry. Test manual docker login with the same credentials.
   4. Tag Filtering: The allow-tags regex might be too restrictive. Test your regex with a regex tester against your actual tags.
   5. Image ID vs. Tag: Ensure your application’s manifest actually references an image tag (e.g., nginx:1.21.0) and not just an image ID or a fixed base image that doesn’t change (e.g., nginx@sha256:...). Image Updater tracks tags.
3. “Failed to push changes to Git” or “Failed to update application resource”.

   Issue: Image Updater finds a new tag but cannot write back the changes.

   Solution:

   1. Git Write-Back: If write-back-target: git, ensure the Git secret (argocd-image-updater-git-secret or argocd-image-updater-ssh-secret) is correctly configured and the provided credentials have push access to the target repository and branch. Check Image Updater logs for Git-specific errors.
   2. ArgoCD Write-Back: If write-back-target: argocd, ensure the Image Updater ServiceAccount has RBAC permissions to patch ArgoCD Application resources in the argocd namespace.
   3. Branch Protection: If using write-back-target: git, check if the target Git branch has protection rules that prevent direct pushes. Image Updater needs to bypass these or push to a temporary branch that triggers a pull request.
4. ArgoCD doesn’t sync after Image Updater changes.

   Issue: Image Updater successfully updates the Git repository or ArgoCD Application, but ArgoCD itself doesn’t deploy the new image.

   Solution:

   1. ArgoCD Sync Policy: Ensure the ArgoCD Application has an appropriate sync policy (e.g., Automated with Auto-Sync or Self-Heal enabled). If it’s manual, you’ll need to trigger the sync manually.
   2. Refresh Interval: ArgoCD has a refresh interval for checking Git changes. Ensure it’s not set to an excessively long period.
   3. Parameter Override: If you’re using Helm or Kustomize, ensure the image tag is being correctly overridden by the parameter Image Updater is changing. Sometimes, a hardcoded image in a sub-chart or base manifest can prevent the override from taking effect.
5. Image Updater is slowing down or consuming too many resources.

   Issue: The argocd-image-updater pod shows high CPU/memory usage or takes a long time to complete checks.

   Solution:

   1. Increase Update Interval: Adjust the interval in the argocd-image-updater-config ConfigMap to reduce the frequency of checks, especially for applications that don’t update frequently.
   2. Optimize Registry Access: If using private registries, ensure network latency is low. For public registries