Introduction

In the ephemeral world of containers, where pods can be spun up and torn down in a blink, managing persistent data poses a unique challenge. By default, data written inside a container is lost when the container restarts or is deleted. This behavior is perfectly acceptable for stateless applications, but what about databases, message queues, or stateful microservices that require their data to outlive the containers that produce it?

Enter Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) in Kubernetes. These powerful constructs provide an abstraction layer for storage, decoupling the lifecycle of storage from the lifecycle of pods. However, the true magic behind this abstraction, especially in modern Kubernetes clusters, lies with Container Storage Interface (CSI) Drivers. CSI is a standardized API that allows Kubernetes to interface with various storage systems, from cloud-specific block storage to network file systems and even custom on-premises solutions, in a generic way. This guide will demystify PVs, PVCs, and how CSI drivers enable flexible and robust persistent storage for your Kubernetes workloads.

# Check CSI drivers
kubectl get csidrivers

# Get StorageClasses
kubectl get sc

# Create a StorageClass (example for AWS EBS)
kubectl apply -f storageclass.yaml

# Create a PVC
kubectl apply -f pvc.yaml

# Create a Pod using the PVC
kubectl apply -f pod-with-pvc.yaml

# Clean up
kubectl delete -f pod-with-pvc.yaml -f pvc.yaml -f storageclass.yaml

Prerequisites

To follow this guide, you’ll need:

• A running Kubernetes cluster (e.g., Minikube, Kind, or a cloud-managed cluster like GKE, EKS, AKS).
• kubectl installed and configured to interact with your cluster. You can find installation instructions on the official Kubernetes documentation.
• Basic understanding of Kubernetes concepts like Pods, Deployments, and YAML manifests.
• For cloud-specific examples, appropriate IAM permissions to provision storage resources (e.g., EBS volumes on AWS, Persistent Disks on GCP).

Step-by-Step Guide: Provisioning Persistent Storage with CSI

Step 1: Understand the CSI Driver Landscape

Before you even think about creating PVs or PVCs, it’s crucial to understand that your Kubernetes cluster needs a way to talk to your underlying storage infrastructure. This is where CSI drivers come in. CSI is an industry standard for exposing arbitrary block and file storage systems to containerized workloads on orchestrators like Kubernetes. Cloud providers like AWS, GCP, and Azure provide their own CSI drivers for their native storage services (e.g., AWS EBS CSI driver, GCE Persistent Disk CSI driver). There are also CSI drivers for network file systems like NFS, CephFS, and many others. Most managed Kubernetes services will have a default CSI driver pre-installed and configured.

You can check which CSI drivers are installed and recognized by your cluster. The output will show you the registered CSI drivers, which are essentially Kubernetes API objects representing the driver’s capabilities and identity.

kubectl get csidrivers

Verify: You should see a list of CSI drivers. For example, on a GKE cluster, you might see:

NAME                   ATTACHREQUIRED   PODINFOONMOUNT   STORAGECAPACITY   TOKENREQUESTS   REQUIRESREPUBLISH   MODES        AGE
pd.csi.storage.gke.io  true             true             false             <unset>         false               Persistent   2d

If you’re on a self-managed cluster or using a specific storage solution, you might need to install the relevant CSI driver. For instance, if you’re running on bare metal and want to use Longhorn, you’d install its CSI driver. For more on advanced networking and storage, consider exploring topics like Cilium WireGuard Encryption which can secure data in transit between your storage and workloads.

Step 2: Define a StorageClass

A StorageClass is a crucial Kubernetes resource that abstracts away the details of the underlying storage provider. It defines a “class” of storage, specifying parameters like the provisioning mechanism (which CSI driver to use), reclaim policy, and volume binding mode. When you request storage via a PVC, you can specify a StorageClass, and Kubernetes will use it to dynamically provision a PV for you. This dynamic provisioning is one of the most powerful features of CSI drivers.

Let’s define a StorageClass. For demonstration purposes, we’ll use an example for AWS EBS. If you are on GCP, you would use pd.csi.storage.gke.io and appropriate parameters. If you are on Minikube, you might use standard, which often uses a hostPath or an in-tree driver.

Here’s an example storageclass.yaml for AWS EBS gp2 volumes:

# storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-csi
provisioner: ebs.csi.aws.com # This tells Kubernetes to use the AWS EBS CSI driver
parameters:
  type: gp2 # General purpose SSD (gp2)
  fsType: ext4 # Filesystem type
reclaimPolicy: Delete # When the PVC is deleted, the PV and underlying volume are deleted
volumeBindingMode: Immediate # Provision volume immediately upon PVC creation
allowVolumeExpansion: true # Allow users to expand the volume capacity

Apply this StorageClass to your cluster:

kubectl apply -f storageclass.yaml

Verify: Check if the StorageClass was created successfully:

kubectl get sc gp2-csi

Expected output:

NAME      PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
gp2-csi   ebs.csi.aws.com   Delete          Immediate           true                   5s

Step 3: Create a PersistentVolumeClaim (PVC)

A PersistentVolumeClaim (PVC) is a request for storage by a user. It specifies the desired size, access modes (e.g., ReadWriteOnce, ReadOnlyMany), and optionally a StorageClass. Kubernetes then finds a suitable PV or dynamically provisions one using the specified StorageClass and binds it to the PVC. This binding makes the storage available for pods.

Let’s create a PVC named my-pvc requesting 5Gi of storage from our gp2-csi StorageClass. This pvc.yaml will be used by our application.

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce # This volume can be mounted as read-write by a single node
  storageClassName: gp2-csi # Reference our StorageClass
  resources:
    requests:
      storage: 5Gi # Request 5 Gigabytes of storage

Apply the PVC:

kubectl apply -f pvc.yaml

Verify: Check the status of your PVC. It should eventually transition to Bound, meaning a PV has been provisioned and bound to it.

kubectl get pvc my-pvc

Expected output (after a short delay for provisioning):

NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-pvc   Bound    pvc-1a2b3c4d-e5f6-7g8h-9i0j-1k2l3m4n5o6p   5Gi        RWO            gp2-csi        15s

You can also inspect the associated PersistentVolume:

kubectl get pv

This will show the PV dynamically created by the CSI driver.

Step 4: Mount the PVC in a Pod or Deployment

Now that you have a PVC, you can consume it in your pods or deployments. You simply reference the PVC by its name in the volumes section of your pod spec, and then mount that volume into your container’s filesystem at a specified mountPath.

Here’s a simple Nginx deployment that uses our my-pvc to store its web content:

# pod-with-pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-data-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-data
  template:
    metadata:
      labels:
        app: nginx-data
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: html-storage # Name of the volume mount
          mountPath: /usr/share/nginx/html # Path inside the container where the volume will be mounted
      volumes:
      - name: html-storage # Name of the volume, matching the volumeMounts name
        persistentVolumeClaim:
          claimName: my-pvc # Reference our PVC by name

Apply the deployment:

kubectl apply -f pod-with-pvc.yaml

Verify:
1. Check the deployment and pod status:

kubectl get deploy nginx-data-deployment
kubectl get pod -l app=nginx-data

2. Once the pod is running, you can exec into it and verify the mount point. Let’s create a simple index.html file to test persistence.

POD_NAME=$(kubectl get pod -l app=nginx-data -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $POD_NAME -- /bin/bash -c "echo '
Hello from Persistent Storage!
' > /usr/share/nginx/html/index.html"
kubectl exec -it $POD_NAME -- ls /usr/share/nginx/html

Expected output from ls:

index.html

3. To further confirm persistence, delete the pod and let the deployment recreate it. The index.html file should still be there:

kubectl delete pod $POD_NAME # This will trigger a recreation by the deployment
sleep 10 # Give it time to restart
NEW_POD_NAME=$(kubectl get pod -l app=nginx-data -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it $NEW_POD_NAME -- cat /usr/share/nginx/html/index.html

Expected output:

Hello from Persistent Storage!

This confirms that your data is indeed persistent and survives pod restarts, thanks to the PVC and underlying CSI-provisioned volume.

Production Considerations

Deploying persistent storage in production requires careful planning beyond just getting a volume to mount. Here are critical factors:

1. StorageClass Parameters:
   • Performance Tiers: Most CSI drivers allow specifying performance characteristics (e.g., IOPS, throughput). Don’t just use the default. For critical databases, you’ll need higher-tier storage.
   • Reclaim Policy: Delete is default for many, but for sensitive data, Retain might be preferred to prevent accidental data loss, requiring manual cleanup.
   • Volume Binding Mode: Immediate provisions volumes as soon as the PVC is created. WaitForFirstConsumer delays provisioning until a pod actually uses the PVC, which can be better for topology-aware scheduling.
2. Backup and Restore: Persistent volumes are a single point of failure without a robust backup strategy. Implement regular backups of your volumes. Tools like Velero can help with Kubernetes-native backup and restore.
3. Snapshots: CSI drivers support volume snapshots. Leverage this for quick recovery points, especially before major application upgrades.
4. Volume Expansion: Ensure your StorageClass has allowVolumeExpansion: true if you anticipate needing to increase storage capacity without downtime.
5. Access Modes:
   • ReadWriteOnce (RWO): Most common, a single node can mount read-write.
   • ReadOnlyMany (ROM): Multiple nodes can mount read-only.
   • ReadWriteMany (RWX): Multiple nodes can mount read-write. This typically requires a network file system (like NFS, CephFS, or cloud file storage like AWS EFS, GCP Filestore) and is not supported by block storage.

   Choose wisely based on your application’s concurrency needs.

6. Topology Awareness: For cloud providers, ensure your pods are scheduled in the same availability zone as your block storage for optimal performance and to avoid cross-zone data transfer costs. CSI drivers and WaitForFirstConsumer can help with this.
7. Monitoring: Monitor volume utilization, IOPS, and latency. Integrate with your existing observability stack. For more on advanced observability, check out our guide on eBPF Observability with Hubble.
8. Security: Encrypt your volumes at rest. Most cloud CSI drivers support this. Also, ensure appropriate network policies are in place to restrict access to applications using the storage. Our Network Policies Security Guide offers in-depth advice.
9. Cost Optimization: Choose the right storage class and size your volumes appropriately. Over-provisioning storage can lead to unnecessary costs. Consider tools like Karpenter for Kubernetes Cost Optimization, which can help manage node resources efficiently, indirectly impacting storage access costs.

Troubleshooting

1. PVC stuck in Pending state:

   Problem: Your PersistentVolumeClaim stays in the Pending state and doesn’t bind to a PersistentVolume.

   Solution: This usually means Kubernetes couldn’t provision a PV for your PVC.

   1. Check Events: The most common reason is an issue with the StorageClass or the underlying CSI driver.

      kubectl describe pvc my-pvc
      

      Look for events at the bottom. It might indicate that the provisioner failed or no suitable PV was found.

   2. Verify StorageClass: Ensure the StorageClass exists and its provisioner field is correct and corresponds to an installed CSI driver.

      kubectl get sc
      kubectl describe sc <your-storage-class-name>
      

   3. Check CSI Driver Pods: Ensure the CSI driver’s controller and node agent pods are running correctly in the kube-system namespace (or wherever they are installed).

      kubectl get pods -n kube-system -l app=csi-<driver-name>
      

      Check their logs for errors.

   4. Permissions: For cloud providers, ensure the Kubernetes worker nodes (or the service account associated with the CSI driver) have the necessary IAM permissions to create storage resources.
2. Pod fails to start with “Volume not found” or “Failed to attach volume”:

   Problem: Your pod is in Pending or CrashLoopBackOff, and logs/events show issues attaching the volume.

   Solution:

   1. Check PVC Status: Ensure the PVC is Bound. If it’s Pending, refer to the previous troubleshooting step.

      kubectl get pvc my-pvc
      

   2. Pod Events: Describe the pod to see specific errors related to volume attachment.

      kubectl describe pod <pod-name>
      

      Look for messages like “Multi-Attach error” if you’re trying to use a RWO volume with multiple pods, or issues with the CSI driver’s node plugin.

   3. Node Affinity/Topology: If using block storage (RWO) with topology constraints (e.g., AWS EBS in a specific AZ), ensure your pod is scheduled to a node in the same AZ as the volume. WaitForFirstConsumer volume binding mode helps with this.
   4. CSI Node Plugin: The CSI driver usually has a node-specific component. Ensure it’s running and healthy on the node where the pod is trying to mount the volume.
3. Volume data is lost or not persistent:

   Problem: Data written to the mounted volume disappears after a pod restart or deletion.

   Solution:

   1. Verify Mount Path: Double-check that your application is actually writing data to the correct mountPath specified in the volumeMounts.
   2. Reclaim Policy: If your StorageClass or PV has a reclaimPolicy: Delete, the underlying storage will be deleted when the PVC is deleted. Ensure you understand this behavior. For sensitive data, Retain is safer, but requires manual PV cleanup.
   3. Ephemeral Volumes: Ensure you are using a PersistentVolumeClaim and not an ephemeral volume type like emptyDir, which is temporary.
   4. Application Logic: Sometimes the application itself might be misconfigured and not writing to the mounted path or deleting its own data.
4. Volume expansion fails:

   Problem: You tried to expand a PVC, but it’s stuck or failed.

   Solution:

   1. StorageClass allowVolumeExpansion: Ensure your StorageClass has allowVolumeExpansion: true.

      kubectl get sc <your-storage-class-name> -o yaml | grep allowVolumeExpansion
      

   2. CSI Driver Support: Not all CSI drivers support online volume expansion. Check the documentation for your specific CSI driver.
   3. Filesystem Expansion: Even if the underlying block device expands, the filesystem on it might need to be expanded. Some CSI drivers handle this automatically, others might require a pod restart or manual action within the container.
   4. Pod Restart: For many block storage types, the pod using the volume needs to be restarted for the expanded capacity to be recognized by the OS inside the container.
5. Performance issues with persistent volumes:

   Problem: Applications using persistent volumes are experiencing slow I/O.

   Solution:

   1. StorageClass Parameters: Review your StorageClass parameters. Are you using the correct performance tier (e.g., gp3 instead of gp2 for AWS EBS, or a higher IOPS tier)?

      kubectl describe sc <your-storage-class-name>
      

   2. Application I/O Patterns: Understand if your application is doing small random reads/writes or large sequential operations. Optimize the storage type accordingly.
   3. Node-Volume Proximity: Ensure the node where your pod is running is in the same availability zone/region as the provisioned volume to minimize latency.
   4. Network Saturation: Check network performance on your nodes. If you’re using network-attached storage, network bottlenecks can impact storage performance.
   5. Monitoring: Use cloud provider metrics (e.g., CloudWatch for AWS EBS, Stackdriver for GCP PD) and Kubernetes metrics to monitor volume IOPS, throughput, and latency.

FAQ Section

1. What is the difference between a PersistentVolume (PV) and a PersistentVolumeClaim (PVC)?

   A PersistentVolume (PV) is an actual piece of storage in the cluster, provisioned by an administrator or dynamically by a CSI driver. It’s a cluster-wide resource. A PersistentVolumeClaim (PVC) is a request for storage by a user or application. It specifies requirements like size and access mode. Kubernetes then finds a suitable PV or provisions a new one to fulfill the PVC.

2. Why do I need CSI drivers? Can’t I just use in-tree volume plugins?

   CSI drivers are the modern, recommended way to manage storage in Kubernetes. In-tree volume plugins (e.g., awsElasticBlockStore, gcePersistentDisk) are deprecated and being removed from Kubernetes. CSI offers several advantages:

   • Out-of-tree development: CSI drivers can be developed and updated independently of the Kubernetes release cycle.
   • Extensibility: Anyone can write a CSI driver for any storage system without modifying the core Kubernetes code.
   • Standardization: It provides a common interface for all storage vendors.

   For more on modern Kubernetes networking, see our guide on Kubernetes Gateway API vs Ingress.

3. Can I share a PersistentVolume between multiple pods?

   Yes, but it depends on the accessModes of the PV/PVC and the capabilities of the underlying storage.

   • ReadWriteOnce (RWO): Can only be mounted by a single node as read-write. Most block storage (e.g., EBS, GCP PD) supports this.
   • ReadOnlyMany (ROM): Can be mounted by many nodes as read-only.
   • ReadWriteMany (RWX): Can be mounted by many nodes as read-write. This typically requires network file storage like NFS, CephFS, or cloud file storage (e.g., AWS EFS, Azure Files).

   Ensure your chosen storage system and CSI driver support the desired access mode.

4. What happens to my data if I delete a PVC?

   It depends on the reclaimPolicy of the associated PersistentVolume:

   • Delete (default for dynamic provisioning): The underlying storage volume (e.g., EBS volume) is deleted along with the PV. This is common for temporary or easily reproducible data.
   • Retain: The underlying storage volume and the PV are kept intact. The PV enters a Released state, and the data is preserved. This is useful for sensitive data where you want to manually manage deletion or recovery.

   Always verify the reclaimPolicy before deleting PVCs with critical data.

5. How do I backup and restore data from a PersistentVolume?

   Kubernetes itself doesn’t provide a built-in backup solution for PVs. You typically need external tools or methods:

   • Volume Snapshots: Many CSI drivers support creating volume snapshots, which are point-in-time copies of your data. These can be used for quick recovery.
   • Cloud Provider Backups: Leverage the backup features of your cloud provider’s storage service (e.g., AWS EBS snapshots, GCP Persistent Disk snapshots).
   • Third-party Tools: Tools like Velero (GitHub) are popular for backing up and restoring Kubernetes resources, including persistent volumes, across clusters or to object storage.
   • Application-level Backups: For databases, often the best approach is to use the database’s native backup mechanisms (e.g., logical backups, streaming replication).

   For securing your supply chain and cluster, consider solutions like Sigstore and Kyverno.

Cleanup Commands

To clean up the resources created in this guide, execute the following commands in reverse order of creation:

# Delete the deployment
kubectl delete -f pod-with-pvc.yaml

# Delete the PersistentVolumeClaim
kubectl delete -f pvc.yaml

# Delete the StorageClass
kubectl delete -f storageclass.yaml

# Optional: If you created any manual PVs (not in this guide), delete them
# kubectl delete pv <pv-name>

Verify: Ensure all resources are gone:

kubectl get deploy,pvc,pv,sc

You should see that nginx-data-deployment, my-pvc, and gp2-csi are deleted. The dynamically created PV for my-pvc should also be gone due to the reclaimPolicy: Delete.

Next Steps / Further Reading

• Explore different StorageClass parameters and their impact on your storage.
• Learn about Volume Snapshots for backup and recovery.
• Investigate Volume Expansion to grow your persistent volumes.
• Deep dive into specific CSI drivers for your chosen infrastructure (e.g., GCP PD CSI Driver, AWS EBS CSI Driver).
• For stateful applications, consider using StatefulSets, which are designed to manage stateful applications and automatically provision storage for each replica using volumeClaimTemplates.
• If you’re dealing with demanding machine learning workloads, understanding storage for large datasets is crucial. You might find our guide on Running LLMs on Kubernetes: GPU Scheduling Best Practices relevant, as it often involves complex storage requirements.
• For advanced service mesh deployments, explore how storage integrates with solutions like Istio Ambient Mesh, especially for data plane components.

Conclusion

Persistent storage is a cornerstone for running stateful applications reliably on Kubernetes. By leveraging Persistent Volumes, Persistent Volume Claims, and the flexible Container Storage Interface (CSI) drivers, you can seamlessly integrate your Kubernetes cluster with a vast array of storage solutions, from cloud-native block storage to network file systems. This abstraction not only simplifies storage management but also enables dynamic provisioning, elasticity, and enhanced data durability.

Understanding these concepts is vital for any production-grade Kubernetes deployment. As you continue your Kubernetes journey, remember that while containers are ephemeral, your data doesn’t have to be. With CSI, Kubernetes empowers you to build robust, scalable, and data-persistent applications, unlocking the full potential of your containerized infrastructure.