How to reclaim an existing PV in state "Released" with AKS

Introduction

In this blog post, I will demonstrate how a PersistentVolume in state Released, can be reclaimed on AKS. Consider the following scenario...

Scenario

A seemingly dangling PersistentVolumeClaim gets deleted accidentally.

Maybe because the Used By: <none> information revealed by kubectl describe pvc gave the impression it's not in use anymore or you overlooked the PVC is still in a bound state.

The referenced StorageClass has the ReclaimPolicy set to Retain, and therefore the PersistentVolume, including the Managed Disk, still exists.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-csi-retain
parameters:
  cachingmode: ReadOnly
  skuName: StandardSSD_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Now, the PersistentVolume is in state Released and has claimRef to a non-existing PVC. You'd like to reclaim and attach the storage to a new Pod.

> kubectl get pv 

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM           STORAGECLASS         REASON   AGE
pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            Retain           Released   app/pvc-disk1   managed-csi-retain            146m

Step by Step

High-level steps

First, we will delete the PV and create a new, static one pointing to the existing managed disk. Then, we will create a new PVC that a Pod can reference again.

Deleting the PV

kubectl delete pv pvc-36ac28d3-e1af-4009-9b74-968b974f601c

Create a new PV

Create a PV definition, and replace the attributes below to match your case.

  • metadata.name
  • spec.capacity.storage
  • spec.storageClassName
  • spec.csi.volumeAttributes (resource id of your managed disk)
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: disk.csi.azure.com
  name: pvc-36ac28d3-e1af-4009-9b74-968b974f601c
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: managed-csi-retain
  csi:
    driver: disk.csi.azure.com
    readOnly: false
    volumeHandle: /subscriptions/<subscription-id>/resourceGroups/<aks-infrastructure-resource-group>/providers/Microsoft.Compute/disks/pvc-36ac28d3-e1af-4009-9b74-968b974f601c
    volumeAttributes:
      fsType: ext4
pvc-36ac28d3-e1af-4009-9b74-968b974f601c.yaml

Create the object and make sure everything goes smoothly. Please note, that the status should now be Avaiable.

> kubectl apply -f pvc-36ac28d3-e1af-4009-9b74-968b974f601c.yaml 
persistentvolume/pvc-36ac28d3-e1af-4009-9b74-968b974f601c created

> kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS         REASON   AGE
pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            Retain           Available           managed-csi-retain            4s

Add a new PVC

Next, create a new PVC definition pointing to the PV. Make sure to use your desired namespace. Then, apply it with kubectl apply -f pvc-disk1.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-disk1
  namespace: <your-namespace>
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  volumeName: pvc-36ac28d3-e1af-4009-9b74-968b974f601c
  storageClassName: managed-csi-retain
pvc-disk1.yaml

You should now have a PV and PVC in state Bound.

> kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS         REASON   AGE
pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            Retain           Bound    <your-namespace>/pvc-disk1   managed-csi-retain            15m

> kubectl get pvc 
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
pvc-disk1   Bound    pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            managed-csi-retain   2m

Consume the PVC

You can now create a Pod referencing and consuming the PVC.

apiVersion: v1
kind: Pod 
metadata:
  name: demo
  namespace: app
spec:
  containers:
  - name: demo
    image: nginx:latest
    volumeMounts:
    - name: datadir
      mountPath: "/mnt/azure"
  volumes:
  - name: datadir
    persistentVolumeClaim:
      claimName: pvc-disk1
pod.yaml

Conclusion

This article discussed how a persistent volume in state Released could be reclaimed and reattached to a Pod. This was possible by creating a new, static PV pointing to the existing Azure-managed disk.

I hope you found this article helpful! Thanks for reading! 😎

Further reading

Dynamic Volume Provisioning
Dynamic volume provisioning allows storage volumes to be created on-demand. Without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, and then create PersistentVolume objects to represent them in Kubernetes. The…
Create a persistent volume with Azure Disks in Azure Kubernetes Service (AKS) - Azure Kubernetes Service
Learn how to create a static or dynamic persistent volume with Azure Disks for use with multiple concurrent pods in Azure Kubernetes Service (AKS)