Kubernetes Backups with Velero: Recover Fast

Introduction

In the dynamic world of Kubernetes, applications are often distributed, stateful, and mission-critical. While Kubernetes offers incredible resilience and self-healing capabilities for individual components, it doesn’t inherently protect your entire application’s data or its configuration from catastrophic failures like accidental deletions, cluster-wide outages, or even ransomware attacks. Imagine losing all your persistent data, critical configurations, and application deployments due to an unfortunate incident. The thought alone is enough to send shivers down any SRE’s spine.

This is where Velero steps in. Velero is an open-source tool designed to safely back up and restore your Kubernetes cluster resources and persistent volumes. It provides a robust solution for disaster recovery, cluster migration, and even replicating development environments. By leveraging Velero, you can ensure that your applications, their data, and their configurations are protected, allowing for quick recovery and minimal downtime in the face of unexpected events. This guide will walk you through setting up and using Velero for comprehensive backup and disaster recovery in your Kubernetes environment.

Prerequisites

Before diving into Velero, ensure you have the following:

• Kubernetes Cluster: A running Kubernetes cluster (v1.16 or higher). You can use Minikube, Kind, or a cloud-managed cluster like EKS, GKE, or AKS.
• kubectl: The Kubernetes command-line tool, configured to connect to your cluster. Refer to the official kubectl installation guide.
• Velero CLI: Download and install the Velero client CLI from the Velero GitHub releases page.
• Object Storage: Access to an S3-compatible object storage bucket (e.g., AWS S3, Google Cloud Storage, Azure Blob Storage, MinIO). You’ll need credentials with read/write access to this bucket.
• Cloud Provider Credentials: Depending on your cloud provider, you’ll need appropriate IAM roles or service accounts configured for Velero to interact with object storage and potentially snapshot persistent volumes.
• Helm (Optional but Recommended): For easier Velero installation and management. Install Helm by following the official Helm documentation.

Step-by-Step Guide

Step 1: Install Velero CLI

The Velero CLI is essential for interacting with Velero backups and restores. It allows you to create backups, monitor their status, and initiate restores from your local machine.

First, download the latest Velero release for your operating system. Replace v1.12.0 with the latest stable release.

# Download the Velero CLI
VELERO_VERSION="v1.12.0" # Check for the latest stable release
OS_TYPE="linux" # or "darwin" for macOS, "windows" for Windows

wget https://github.com/vmware-tanzu/velero/releases/download/${VELERO_VERSION}/velero-${VELERO_VERSION}-${OS_TYPE}-amd64.tar.gz

# Extract the archive
tar -xvzf velero-${VELERO_VERSION}-${OS_TYPE}-amd64.tar.gz

# Move the velero binary to your PATH
sudo mv velero-${VELERO_VERSION}-${OS_TYPE}-amd64/velero /usr/local/bin/

# Verify the installation
velero version --client-only

Verify:

You should see the client version information. The server version will be unknown until Velero is installed in the cluster.

Client:
        Version: v1.12.0
        Git Commit: 97892a0951a134a4925829621379e1e8386c9f2b

Step 2: Prepare Object Storage Credentials

Velero needs access to an S3-compatible object storage bucket to store backup archives. This involves creating a bucket and generating credentials (e.g., IAM user with programmatic access in AWS) that have read and write permissions to that bucket. It’s crucial to follow the principle of least privilege when setting up these credentials.

For AWS, you would create an IAM user with a policy like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-velero-bucket/*",
                "arn:aws:s3:::your-velero-bucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        }
    ]
}

Save your AWS access key ID and secret access key in a file named credentials-velero:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

For other cloud providers, the process is similar. For Google Cloud Storage, you’d use a service account key file. For Azure Blob Storage, you’d use a storage account and access key.

Step 3: Install Velero in Your Kubernetes Cluster

Now, install the Velero server component into your Kubernetes cluster. We’ll use the Velero CLI for this, which simplifies the process by generating the necessary Kubernetes resources.

Specify your cloud provider, the bucket name, and the path to your credentials file. For AWS, the provider is aws. For GCP, it’s gcp. For Azure, it’s azure.

# For AWS
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.9.0 \
    --bucket your-velero-bucket-name \
    --secret-file ./credentials-velero \
    --use-node-agent \
    --snapshot-location-config region=us-east-1 \
    --backup-location-config region=us-east-1

# For GCP (assuming you have a service account key file named 'gcp-credentials.json')
# velero install \
#     --provider gcp \
#     --plugins velero/velero-plugin-for-gcp:v1.9.0 \
#     --bucket your-velero-bucket-name \
#     --secret-file ./gcp-credentials.json \
#     --use-node-agent \
#     --snapshot-location-config project=your-gcp-project-id \
#     --backup-location-config project=your-gcp-project-id

# For Azure (assuming you have your storage account name and resource group)
# export AZURE_SUBSCRIPTION_ID=""
# export AZURE_RESOURCE_GROUP=""
# export AZURE_STORAGE_ACCOUNT_ACCESS_KEY=""
# export AZURE_STORAGE_ACCOUNT_NAME=""
# velero install \
#     --provider azure \
#     --plugins velero/velero-plugin-for-azure:v1.9.0 \
#     --bucket your-velero-bucket-name \
#     --secret-file ./credentials-velero \
#     --use-node-agent \
#     --snapshot-location-config apiTimeout=5m,resourceGroup=${AZURE_RESOURCE_GROUP} \
#     --backup-location-config resourceGroup=${AZURE_RESOURCE_GROUP},storageAccount=${AZURE_STORAGE_ACCOUNT_NAME}

Verify:

Check if the Velero pod is running in the velero namespace.

kubectl get pods -n velero

# Expected Output:
# NAME                      READY   STATUS    RESTARTS   AGE
# velero-xxxxxxxxx-yyyyy    1/1     Running   0          2m
# velero-node-agent-zzzzz   1/1     Running   0          2m

Also, verify that Velero is reporting the correct server version:

velero version

# Expected Output:
# Client:
#         Version: v1.12.0
#         Git Commit: 97892a0951a134a4925829621379e1e8386c9f2b
# Server:
#         Version: v1.12.0
#         Git Commit: 97892a0951a134a4925829621379e1e8386c9f2b

Step 4: Deploy a Sample Application

To demonstrate Velero’s capabilities, let’s deploy a simple stateful application. We’ll use a basic Nginx deployment with a PersistentVolumeClaim (PVC) to simulate real-world data. This will allow us to back up both the application’s configuration and its data.

Create a namespace for our sample application:

kubectl create namespace sample-app

Now, deploy the Nginx application with a PVC:

# sample-app.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
  namespace: sample-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-persistent-storage
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-persistent-storage
        persistentVolumeClaim:
          claimName: nginx-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: sample-app
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Apply the YAML:

kubectl apply -f sample-app.yaml

Write some data to the Nginx persistent volume to simulate user data:

POD_NAME=$(kubectl get pods -n sample-app -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n sample-app -- sh -c 'echo "Hello from Nginx, this is important data!" > /usr/share/nginx/html/index.html'

Verify:

Check if the application components are running and the data is present.

kubectl get all -n sample-app

# Expected Output (similar to):
# NAME                                 READY   STATUS    RESTARTS   AGE
# pod/nginx-deployment-xxxxxxxxx-yyyyy   1/1     Running   0          2m

# NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
# service/nginx-service   ClusterIP   10.100.100.100           80/TCP         2m

# NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
# deployment.apps/nginx-deployment   1/1     1            1           2m

# NAME                                       DESIRED   CURRENT   READY   AGE
# replicaset.apps/nginx-deployment-xxxxxxxxx   1         1         1       2m

# Verify data
kubectl exec -it $POD_NAME -n sample-app -- cat /usr/share/nginx/html/index.html

# Expected Output:
# Hello from Nginx, this is important data!

Step 5: Create a Velero Backup

With our sample application running, we can now create our first Velero backup. This backup will include all Kubernetes resources within the sample-app namespace, as well as a snapshot of its persistent volume.

Create a backup named nginx-app-backup, specifically including the sample-app namespace.

velero backup create nginx-app-backup --include-namespaces sample-app --wait

The --wait flag will keep the command running until the backup operation is complete, showing its progress.

Verify:

Check the status of your backup and list all available backups.

velero backup get

# Expected Output (similar to):
# NAME               STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
# nginx-app-backup   Completed   0        0          2023-10-26 10:30:00 +0000 UTC   29d       default            

velero backup describe nginx-app-backup

# Expected Output (detailed information including included resources, PV snapshots, etc.):
# Name:         nginx-app-backup
# Namespace:    velero
# Labels:       velero.io/backup-name=nginx-app-backup,velero.io/created-by=velero
# Annotations:  velero.io/source-cluster-k8s-major-version=1,velero.io/source-cluster-k8s-minor-version=27
#               velero.io/source-cluster-k8s-gitversion=v1.27.4
# Phase:        Completed

# Namespaces:
#   Included:  sample-app
#   Excluded:  

# Resources:
#   Included:        *
#   Excluded:        
#   Cluster-scoped:  auto

# Label selector:  

# Storage Location:  default
# Snapshot Location: default

# Velero-Native Snapshot PVs:
#   pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (cloud-provider-specific-snapshot-id)

# TTL:  720h0m0s
# Hooks:  
# ...

You can also check your object storage bucket to see the backup tarball and metadata files.

Step 6: Simulate Disaster: Delete the Namespace

To test our disaster recovery capabilities, let’s simulate a disaster by completely deleting the sample-app namespace and all its contents.

kubectl delete namespace sample-app --wait=false # --wait=false to not wait for deletion completion

Verify:

Confirm that the namespace and its resources are gone.

kubectl get all -n sample-app

# Expected Output:
# Error from server (NotFound): namespaces "sample-app" not found

Step 7: Restore from Backup

Now for the moment of truth: restoring our application from the Velero backup. This will recreate the namespace, deploy the application, and restore the persistent volume with its data.

velero restore create --from-backup nginx-app-backup --wait

Verify:

Check the status of the restore operation and then verify the application and its data.

velero restore get

# Expected Output (similar to):
# NAME                         BACKUP             STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
# nginx-app-backup-xxxxxxxxx   nginx-app-backup   Completed   0          0        2023-10-26 10:45:00 +0000 UTC   

velero restore describe nginx-app-backup-xxxxxxxxx

# Expected Output (detailed information including restored resources, PVs, etc.):
# Name:         nginx-app-backup-xxxxxxxxx
# Namespace:    velero
# Labels:       velero.io/backup-name=nginx-app-backup,velero.io/restore-name=nginx-app-backup-xxxxxxxxx
# Annotations:  
# Phase:        Completed
# ...

# Verify application is back
kubectl get all -n sample-app

# Expected Output (similar to what you saw in Step 4):
# NAME                                 READY   STATUS    RESTARTS   AGE
# pod/nginx-deployment-xxxxxxxxx-yyyyy   1/1     Running   0          2m

# NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
# service/nginx-service   ClusterIP   10.100.100.100           80/TCP         2m

# NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
# deployment.apps/nginx-deployment   1/1     1            1           2m

# Verify data is restored
POD_NAME=$(kubectl get pods -n sample-app -l app=nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n sample-app -- cat /usr/share/nginx/html/index.html

# Expected Output:
# Hello from Nginx, this is important data!

Congratulations! You’ve successfully performed a backup and restore operation using Velero.

Step 8: Schedule Automatic Backups

Manual backups are good for one-off tasks, but for true disaster recovery, you need automated, scheduled backups. Velero allows you to define schedules for regular backups.

Create a daily backup schedule for the sample-app namespace at 1:00 AM UTC.

velero schedule create daily-sample-app-backup \
    --schedule "0 1 * * *" \
    --include-namespaces sample-app \
    --ttl 720h0m0s # Keep backups for 30 days

The --ttl (Time To Live) flag specifies how long the backup should be retained. After this period, Velero will automatically delete the backup from object storage.

Verify:

Check the status of your scheduled backup.

velero schedule get

# Expected Output (similar to):
# NAME                        STATUS    CREATED                         SCHEDULE    BACKUP TTL   LAST BACKUP   SELECTOR
# daily-sample-app-backup   Enabled   2023-10-26 11:00:00 +0000 UTC   0 1 * * *   720h0m0s            

# You can also check the Kubernetes CronJob created by Velero:
kubectl get cronjob -n velero

# Expected Output (similar to):
# NAME                      SCHEDULE    SUSPEND   FORBIDDEN   LAST SCHEDULE   AGE
# daily-sample-app-backup   0 1 * * *   False     False                 1m

Velero will automatically create backups according to this schedule. You can see the created backups with velero backup get once they start running.

Production Considerations

Implementing Velero in a production environment requires careful planning beyond basic installation:

• Security Best Practices:
  • Least Privilege: Ensure Velero’s IAM role/service account has only the necessary permissions for object storage and volume snapshots.
  • Secrets Management: Store Velero credentials securely using Kubernetes Secrets and consider tools like external-secrets or Vault.
  • Network Policies: Restrict network access to the Velero pod using Kubernetes Network Policies to prevent unauthorized access to backup data.
• Backup Strategy:
  • Granularity: Decide whether to back up entire clusters, specific namespaces, or individual resources. Velero supports all these options.
  • Retention Policies: Define appropriate TTLs for your backups based on compliance requirements and recovery point objectives (RPO).
  • Backup Frequency: Schedule backups to meet your RPO. For critical data, consider more frequent backups.
  • Pre/Post Hooks: Use Velero’s backup hooks to quiesce databases or flush caches before a snapshot, ensuring data consistency.
• Storage Location Redundancy:
  • Store backup data in a different region or even a different cloud provider than your primary cluster for true disaster recovery.
  • Utilize object storage versioning to protect against accidental backup deletions.
• Monitoring and Alerting:
  • Monitor Velero backup job status, errors, and warnings. Integrate with your existing monitoring stack (Prometheus, Grafana).
  • Set up alerts for failed backups or restores.
  • Consider using eBPF Observability with Hubble for deeper insights into network interactions during backup/restore.
• Testing Disaster Recovery:
  • Regularly perform full disaster recovery drills to ensure your backups are valid and your restore process works as expected. Don’t wait for a real disaster to find out your backups are corrupted or your process is flawed.
  • Test restores to a separate, isolated cluster.
• Resource Management:
  • Velero itself consumes resources. Ensure its pods have adequate CPU and memory requests/limits.
  • For large clusters with many PVCs, consider the impact of snapshot operations on your storage backend.
• Volume Snapshot Class: Ensure you have a default VolumeSnapshotClass configured for your cluster if you rely on CSI snapshots.

Troubleshooting

Here are some common issues you might encounter with Velero and their solutions:

1. Velero pod not running or in CrashLoopBackOff:

   Issue: The Velero server pod or node agent pod isn’t starting correctly.

   Solution:

   1. Check pod logs for errors:

   
   kubectl logs -f -n velero deploy/velero
   kubectl logs -f -n velero ds/velero-node-agent
               

   2. Common causes include incorrect cloud provider credentials, insufficient IAM permissions, or misconfigured backup/snapshot locations. Verify your credentials-velero file and your cloud provider’s IAM roles.

   3. Ensure your BackupStorageLocation and VolumeSnapshotLocation resources are correctly configured:

   
   kubectl get backupsstoragelocation default -n velero -o yaml
   kubectl get volumesnapshotlocation default -n velero -o yaml
               

2. Backup fails with “Failed” status:

   Issue: A Velero backup operation completes with a Failed status.

   Solution:

   1. Get more details about the failure:

   
   velero backup describe <backup-name>
   velero backup logs <backup-name>
               

   2. Look for specific error messages. Common issues include:

   • Permissions errors: Velero’s IAM role lacks permissions to list resources, create snapshots, or write to the S3 bucket.
   • Resource not found: If using --include-resources or --include-namespaces, ensure the specified resources/namespaces exist.
   • Volume snapshot issues: The cloud provider might be having issues, or Velero lacks permissions to create snapshots (e.g., EC2 permissions for AWS).
   • Network connectivity: Velero pod cannot reach the object storage endpoint. Check network policies if you’re using something like Cilium WireGuard Encryption.
3. Restore fails or resources are not created:

   Issue: A restore operation fails, or some resources are missing after a restore.

   Solution:

   1. Describe and log the restore operation:

   
   velero restore describe <restore-name>
   velero restore logs <restore-name>
               

   2. Check for “warnings” or “errors” sections in the describe output. Common causes:

   • Resource conflicts: If you’re restoring into a cluster where some resources (e.g., namespaces, CRDs) already exist, Velero might skip them. Use --existing-resource-policy update or ensure a clean target cluster.
   • PersistentVolumeClaim (PVC) issues: The storage class might not exist in the target cluster, or there aren’t enough resources to provision the PV. Ensure the storage class name matches or use restore hooks to modify PVCs.
   • Admission controllers: Mutating or validating admission webhooks might be preventing resource creation. Temporarily disable problematic webhooks or configure them to ignore the Velero namespace during restore.
4. Scheduled backups are not running:

   Issue: Your Velero schedules are created, but no backups are being generated.

   Solution:

   1. Check the Velero schedule status:

   
   velero schedule get
               

   2. Inspect the underlying Kubernetes CronJob:

   
   kubectl get cronjob -n velero <schedule-name> -o yaml
   kubectl describe cronjob -n velero <schedule-name>
               

   3. Look for errors in the CronJob’s events. Ensure the Velero server pod is healthy and running. Timezone issues or incorrect cron syntax can also prevent schedules from triggering.

5. Backups are too large or taking too long:

   Issue: Velero backups are consuming excessive storage or taking an unreasonable amount of time to complete.

   Solution:

   1. Filter resources: Use --include-namespaces, --exclude-namespaces, --include-resources, --exclude-resources, or --selector to back up only what’s necessary. For example, you might exclude monitoring namespaces or temporary resources.

   2. Exclude PVs: For ephemeral data, you might exclude certain PVCs from snapshotting using --exclude-pvs. For specific data, consider application-level backups for databases rather than just volume snapshots.

   3. Check network bandwidth: Ensure the Velero pod has sufficient network bandwidth to upload data to object storage. For large clusters, network bottlenecks can be a significant factor. Consider your cloud provider’s network performance limits.

FAQ Section

1. What exactly does Velero back up?

   Velero backs up two main types of data:

   • Kubernetes Cluster Resources: This includes deployments, services, ConfigMaps, Secrets, PersistentVolumeClaims (PVCs), namespaces, custom resource definitions (CRDs), and more. It essentially captures the state of your cluster’s API objects.
   • Persistent Volumes (PVs): For stateful applications, Velero can optionally take snapshots of the underlying persistent volumes. It integrates with cloud provider snapshot APIs (e.g., EBS snapshots for AWS, disk snapshots for GCP/Azure) via plugins.
2. Can Velero back up specific namespaces or exclude certain resources?

   Yes, Velero offers fine-grained control over what gets backed up. You can use flags like --include-namespaces, --exclude-namespaces, --include-resources, --exclude-resources, and --selector (to filter by labels) when creating a backup. This is crucial for managing backup size and scope.

3. What’s the difference between a full cluster backup and a namespace backup?

   A full cluster backup (velero backup create my-full-backup without --include-namespaces) backs up almost all cluster-scoped and namespaced resources. A namespace backup (velero backup create my-ns-backup --include-namespaces my-app-ns) only backs up resources within the specified namespaces, including associated cluster-scoped resources like PersistentVolumes if they are claimed by a PVC in that namespace.

4. Does Velero support cross-cluster or cross-cloud migration?

   Yes, Velero is an excellent tool for cluster migration. You can back up a cluster (or specific namespaces) in one environment (e.g., a development cluster or an on-prem cluster) and restore it to another (e.g., a production cloud cluster) as long as the target cluster has Velero installed and can access the same object storage bucket. You might need to adjust StorageClasses or other cloud-specific configurations during migration.

5. How does Velero handle application consistency during backup?

   For applications that require strong consistency (like databases), simply snapshotting the volume might not be enough if the application is actively writing data. Velero supports

   Was this article helpful?

   Yes No