Kubernetes Backup & DR with Velero

**Verify:**
Ensure your `credentials-velero` file is created and contains the correct AWS access key ID and secret access key. *Remember to replace `YOUR_AWS_ACCOUNT_ID` and `my-velero-backup-bucket-12345` with your actual values.*

cat ./credentials-velero

Expected output:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Step 3: Install Velero in Your Kubernetes Cluster using Helm

Installing Velero into your cluster is typically done via Helm, which simplifies the deployment and configuration. You'll need to specify your cloud provider, the S3 bucket, region, and provide the credentials. Velero will create a `velero` namespace and deploy its server components there.

First, add the Velero Helm repository.

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update

Now, install Velero. Make sure to replace `my-velero-backup-bucket-12345`, `us-east-1`, and the content of `credentials.secretContents.cloud` with your actual values and the base64-encoded content of your `credentials-velero` file.

# Base64 encode your credentials file
export VELERO_CREDENTIALS_B64=$(base64 -w 0 ./credentials-velero)

# Install Velero using Helm
helm install velero vmware-tanzu/velero \
    --namespace velero \
    --create-namespace \
    --set "configuration.provider=aws" \
    --set "configuration.backupStorageLocation.bucket=my-velero-backup-bucket-12345" \
    --set "configuration.backupStorageLocation.region=us-east-1" \
    --set "configuration.volumeSnapshotLocation.config.region=us-east-1" \
    --set "credentials.secretContents.cloud=${VELERO_CREDENTIALS_B64}" \
    --set "serviceAccount.server.annotations.iam.amazonaws.com/role=arn:aws:iam::YOUR_AWS_ACCOUNT_ID:role/VeleroIAMRole" # Optional: If using IAM Roles for Service Accounts (IRSA)
    --set "backups.enableCSI=true" # Enable CSI snapshot support if your cluster uses CSI drivers

**Note on IRSA:** If your Kubernetes cluster is running on AWS EKS and you've configured IAM Roles for Service Accounts (IRSA), you can remove the `credentials.secretContents.cloud` line and instead provide an IAM role ARN via `serviceAccount.server.annotations.iam.amazonaws.com/role`. This is a more secure way to manage credentials.

**Verify:**
Check if the Velero Pods are running in the `velero` namespace.

kubectl get pods -n velero

Expected output:

NAME                      READY   STATUS    RESTARTS   AGE
velero-7dd4c54784-abcde   1/1     Running   0          2m
velero-restic-xyzabc      1/1     Running   0          2m # Only if you enable restic for filesystem backups

Step 4: Create a Sample Application for Backup

To demonstrate Velero's capabilities, let's deploy a simple Nginx application with a PersistentVolumeClaim (PVC) in a new namespace. This will simulate a typical stateful application.

# Save as nginx-app.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: my-app-namespace
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
  namespace: my-app-namespace
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: my-app-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-storage
          mountPath: /usr/share/nginx/html
      volumes:
      - name: nginx-storage
        persistentVolumeClaim:
          claimName: nginx-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: my-app-namespace
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Deploy the application:

kubectl apply -f nginx-app.yaml

**Verify:**
Check if the application components are running in the `my-app-namespace`.

kubectl get all -n my-app-namespace

Expected output:

NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-78f5f68fd8-abcde   1/1     Running   0          2m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/nginx-service      ClusterIP   10.96.123.45     <none>        80/TCP    2m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   1/1     1            1           2m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-78f5f68fd8   1         1         1       2m

NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/nginx-pvc            Bound    pvc-12345678-abcd-efgh-ijkl-1234567890ab   1Gi        RWO            gp2            2m

Step 5: Perform a Velero Backup

Now that Velero is installed and our sample application is running, let's create our first backup. You can back up entire clusters, specific namespaces, or even select resources using labels.

To back up our `my-app-namespace` and its associated PersistentVolumes, use the `velero backup create` command.

velero backup create nginx-app-backup --include-namespaces my-app-namespace --wait

The `--wait` flag will make the CLI wait until the backup operation completes, showing you the status.

**Verify:**
Check the status of your backup using `velero backup get`.

velero backup get

Expected output:

NAME               STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
nginx-app-backup   Completed   0        0          2023-10-27 10:30:00 +0000 UTC   29d       default            <none>

You can also get more detailed information about the backup:

velero backup describe nginx-app-backup

This will show you all the resources included in the backup, the associated volume snapshots, and any events or errors.

Step 6: Simulate a Disaster (Delete the Namespace)

To test our recovery capabilities, let's simulate a disaster by deleting the `my-app-namespace`. This will remove all the deployments, services, and PVCs associated with our Nginx application.

kubectl delete namespace my-app-namespace

**Verify:**
Confirm that the namespace and its resources are gone.

kubectl get all -n my-app-namespace

Expected output:

Error from server (NotFound): namespaces "my-app-namespace" not found

Step 7: Restore from the Velero Backup

With the disaster simulated, it's time to bring our application back using the backup we created. Velero makes restoration straightforward.

To restore the entire `nginx-app-backup`:

velero restore create --from-backup nginx-app-backup --wait

The `--wait` flag will again show you the progress.

**Verify:**
Check the status of the restore operation and then verify if your application resources are back.

velero restore get

Expected output:

NAME                       BACKUP             STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
nginx-app-backup-20231027   nginx-app-backup   Completed   0          0        2023-10-27 10:45:00 +0000 UTC   <none>

Now, check the application resources:

kubectl get all -n my-app-namespace

You should see your Nginx deployment, service, and PVC recreated and running, just as they were before the deletion.

NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-78f5f68fd8-abcde   1/1     Running   0          1m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/nginx-service      ClusterIP   10.96.123.45     <none>        80/TCP    1m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   1/1     1            1           1m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-78f5f68fd8   1         1         1       1m

NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/nginx-pvc            Bound    pvc-12345678-abcd-efgh-ijkl-1234567890ab   1Gi        RWO            gp2            1m

Congratulations! You have successfully set up Velero, backed up a Kubernetes application, simulated a disaster, and restored it.

Production Considerations

Deploying Velero in a production environment requires careful planning beyond a basic setup.

* **Backup Strategy:**
* **Frequency:** Define RPO (Recovery Point Objective) and schedule backups accordingly (e.g., hourly, daily). Velero supports scheduled backups.
* **Retention:** Set appropriate retention policies for your backups to manage storage costs and compliance.
* **Scope:** Decide what to back up (entire cluster, specific namespaces, or resources). Exclude ephemeral data.
* **Velero Schedules:** Use `velero schedule create` for automated backups.

velero schedule create daily-full-backup --schedule="0 1 * * *" --include-cluster-resources --ttl 720h0m0s

* **Storage Location:**
* **Redundancy:** Use highly available and durable object storage (e.g., S3 with cross-region replication).
* **Security:** Implement strong encryption (at rest and in transit) for your backup data. Ensure your cloud provider credentials have the principle of least privilege.
* **Network Policies:** If you have strict network segmentation, ensure that Velero's pods can reach the object storage endpoint. For guidance on securing your cluster's network, refer to our Kubernetes Network Policies: Complete Security Hardening Guide.
* **Volume Snapshots vs. Restic:**
* **Volume Snapshots:** Ideal for cloud-managed PersistentVolumes (EBS, GPD, Azure Disk) as they are efficient and application-agnostic. They require a CSI driver for your storage class.
* **Restic:** For volumes that don't support snapshots (e.g., hostPath, NFS), or when you need granular file-level backups. Restic performs backups by backing up the contents of Pod volumes directly to object storage. It adds overhead but provides universal volume backup capability. Enable it during Velero installation or later via `velero install --use-restic`.
* **Monitoring and Alerting:**
* Monitor Velero backup and restore job statuses. Integrate with your existing monitoring solutions (Prometheus, Grafana).
* Set up alerts for failed backups or restores. Velero exposes Prometheus metrics.
* For advanced observability, especially for network traffic related to backups, tools leveraging eBPF Observability with Hubble can provide deep insights.
* **Testing Disaster Recovery:**
* Regularly test your backup and restore procedures on a separate, non-production cluster. This validates your strategy and ensures your RTO (Recovery Time Objective) is met.
* Document your disaster recovery plan thoroughly.
* **Security:**
* Rotate Velero's cloud credentials regularly.
* Consider using IAM Roles for Service Accounts (IRSA) on AWS EKS, Workload Identity on GKE, or Managed Identities on Azure AKS for credential-less access to object storage.
* Integrate Velero with your cluster's security policies, potentially using Sigstore and Kyverno for policy enforcement on deployed images.
* **Cluster Migration:** Velero is an excellent tool for migrating applications between clusters, even across different cloud providers, by restoring backups to a new cluster.
* **Pre/Post-Backup Hooks:** For complex applications, use Velero's backup hooks to quiesce applications before a backup (e.g., flush databases) and unquiesce them afterward.

Troubleshooting

Here are common issues you might encounter with Velero and how to resolve them.

1. **Velero Pod Not Running:**
* **Issue:** The `velero` pod in the `velero` namespace is in `Pending`, `CrashLoopBackOff`, or `Error` state.
* **Solution:**
1. Check pod logs: `kubectl logs -n velero deploy/velero`
2. Check pod events: `kubectl describe pod -n velero `
3. Common causes: Incorrect cloud provider credentials, insufficient IAM/Service Account permissions, misconfigured S3 bucket name/region, network issues preventing access to object storage.
4. Verify your `BackupStorageLocation` and `VolumeSnapshotLocation` custom resources: `kubectl get bsl -n velero -o yaml` and `kubectl get vsl -n velero -o yaml`.

2. **Backup Fails with "InvalidProvider" or "NoCredentialFound":**
* **Issue:** Velero cannot access your object storage or volume snapshot APIs.
* **Solution:**
1. Double-check the `credentials.secretContents.cloud` value during Helm installation (ensure it's base64 encoded correctly and contains valid keys).
2. Verify the IAM policy attached to your Velero service account or user has all the necessary permissions for S3/GCS/Azure Blob and EC2/GCE/Azure Disk APIs.
3. If using IRSA/Workload Identity, ensure the role ARN is correct and the trust policy allows the Velero service account.
4. Ensure your `configuration.backupStorageLocation.bucket` and `region` settings are correct.

3. **Volume Snapshot Fails:**
* **Issue:** Backups complete, but volume snapshots show errors or are skipped.
* **Solution:**
1. Ensure your Kubernetes cluster has a CSI driver installed and configured for your storage class, and that it supports volume snapshots.
2. Verify that `backups.enableCSI=true` was set during Velero installation.
3. Check Velero pod logs for CSI-related errors.
4. Ensure the Velero IAM role has permissions for `ec2:CreateSnapshot`, `ec2:DeleteSnapshot`, `ec2:DescribeSnapshots`, etc.
5. Some storage classes might not support snapshots, requiring Restic for volume backups.

4. **Restic Backups Not Working:**
* **Issue:** Restic backups fail or are not performed.
* **Solution:**
1. Ensure Restic was enabled during Velero installation (`--set useRestic=true` or `--set backups.enableRestic=true` for newer Helm charts).
2. Check if the `velero-restic-daemonset` is running on all nodes: `kubectl get pods -n velero -l app.kubernetes.io/component=restic`.
3. Verify the `hostPath` volume mounts for Restic in the DaemonSet are correctly configured for your node OS.
4. Check the logs of the Restic pods for errors.
5. Ensure that the `PersistentVolumeClaim` you are trying to back up with Restic is correctly annotated: `velero.io/backup-volumes: `.

5. **Restore Fails or Incomplete:**
* **Issue:** After a restore, some resources are missing or the application doesn't start correctly.
* **Solution:**
1. Check the restore logs: `velero restore logs `.
2. Describe the restore for detailed status and warnings: `velero restore describe `.
3. Look for errors related to resource conflicts (e.g., if you restored to a cluster that already had some resources with the same names).
4. Check if `restore.spec.preserveNodePorts` or `restore.spec.restorePVs` were set as needed.
5. Ensure the target cluster has the necessary StorageClasses or other dependencies (e.g., custom resource definitions) that were present in the source cluster.

6. **Scheduled Backups Not Running:**
* **Issue:** Your `velero schedule` shows as created, but no backups are being generated.
* **Solution:**
1. Check the `velero` pod logs for any errors related to schedules.
2. Verify the cron schedule syntax is correct.
3. Ensure the `velero` deployment is healthy and has sufficient resources.
4. Check the status of the schedule: `velero schedule get`. If it shows `LastBackup` as ``, there might be an issue.

7. **"Backup not found" Error:**
* **Issue:** You try to restore, but Velero says the backup doesn't exist.
* **Solution:**
1. Verify the backup name is correct: `velero backup get`.
2. Ensure your `BackupStorageLocation` is correctly configured and pointing to the S3 bucket where the backup files are stored.
3. If you migrated Velero or the S3 bucket, you might need to re-sync: `velero backup sync`.

FAQ Section

1. **What is the difference between Velero and a cloud provider's snapshot feature?**
Cloud provider snapshots (e.g., AWS EBS snapshots) only capture the state of PersistentVolumes. Velero, on the other hand, backs up *both* your Kubernetes cluster resources (Deployments, Services, ConfigMaps, etc.) and PersistentVolumes (using cloud provider snapshots or Restic). This means Velero provides a complete application-consistent backup, whereas raw volume snapshots alone don't know anything about the Kubernetes resources they belong to.

2. **Can Velero back up cluster-scoped resources?**
Yes, Velero can back up cluster-scoped resources like `CustomResourceDefinitions`, `ClusterRoles`, `StorageClasses`, and `Namespaces`. By default, it backs up namespaces and their contents. To include other cluster-scoped resources, you can use the `--include-cluster-resources` flag with `velero backup create`.

3. **How does Velero handle applications with multiple PersistentVolumes?**
Velero automatically identifies all PersistentVolumeClaims (PVCs) associated with the resources being backed up. If volume snapshots are enabled and supported by the underlying storage, Velero will trigger snapshots for all relevant PVs. If Restic is used, it will back up the data within specified volumes.

4. **Is Velero suitable for migrating applications between different Kubernetes clusters or cloud providers?**
Absolutely! Velero is a popular choice for cluster migration. You can back up an application from one cluster (e.g., an on-premise cluster) and restore it to another (e.g., an EKS cluster on AWS). Velero handles the recreation of Kubernetes resources and data, even translating cloud-specific volume types if necessary (e.g., restoring an EBS snapshot to a GCE Persistent Disk might require manual intervention or specific plugin support). For more complex migrations or advanced networking patterns, you might also consider solutions like the Kubernetes Gateway API.

5. **What's the best way to secure Velero's backups?**
Security is paramount.
* **Access Control:** Use the principle of least privilege for the IAM user/role Velero uses to access object storage and volume APIs.
* **Encryption:** Configure your object storage bucket to use server-side encryption (SSE-S3, SSE-KMS). Velero also supports client-side encryption with Velero's encryption features.
* **Network Security:** Restrict network access to your object storage bucket. If you're using a private endpoint or VPC endpoint for S3, ensure Velero's pods can reach it. For advanced networking within Kubernetes, consider solutions like Cilium WireGuard Encryption to secure pod-to-pod traffic.
* **Audit:** Regularly audit access logs for your object storage bucket.

Cleanup Commands

To remove the resources created during this tutorial:

1. **Delete the sample application namespace:**

kubectl delete namespace my-app-namespace

2. **Delete Velero backups:**

velero backup delete nginx-app-backup --confirm

This will also delete the corresponding objects in your S3 bucket.