How to reclaim an existing PV in state "Released" with AKS

Introduction

In this blog post, I will demonstrate how a PersistentVolume in state Released, can be reclaimed on AKS. Consider the following scenario...

Scenario

A seemingly dangling PersistentVolumeClaim gets deleted accidentally.

Maybe because the Used By: <none> information revealed by kubectl describe pvc gave the impression it's not in use anymore or you overlooked the PVC is still in a bound state.

The referenced StorageClass has the ReclaimPolicy set to Retain, and therefore the PersistentVolume, including the Managed Disk, still exists.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-csi-retain
parameters:
  cachingmode: ReadOnly
  skuName: StandardSSD_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Now, the PersistentVolume is in state Released and has claimRef to a non-existing PVC. You'd like to reclaim and attach the storage to a new Pod.

> kubectl get pv 

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM           STORAGECLASS         REASON   AGE
pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            Retain           Released   app/pvc-disk1   managed-csi-retain            146m

Step by Step

High-level steps

First, we will delete the PV and create a new, static one pointing to the existing managed disk. Then, we will create a new PVC that a Pod can reference again.

Deleting the PV

kubectl delete pv pvc-36ac28d3-e1af-4009-9b74-968b974f601c

Create a new PV

Create a PV definition, and replace the attributes below to match your case.

metadata.name
spec.capacity.storage
spec.storageClassName
spec.csi.volumeAttributes (resource id of your managed disk)

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: disk.csi.azure.com
  name: pvc-36ac28d3-e1af-4009-9b74-968b974f601c
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: managed-csi-retain
  csi:
    driver: disk.csi.azure.com
    readOnly: false
    volumeHandle: /subscriptions/<subscription-id>/resourceGroups/<aks-infrastructure-resource-group>/providers/Microsoft.Compute/disks/pvc-36ac28d3-e1af-4009-9b74-968b974f601c
    volumeAttributes:
      fsType: ext4

pvc-36ac28d3-e1af-4009-9b74-968b974f601c.yaml

Create the object and make sure everything goes smoothly. Please note, that the status should now be Avaiable.

> kubectl apply -f pvc-36ac28d3-e1af-4009-9b74-968b974f601c.yaml 
persistentvolume/pvc-36ac28d3-e1af-4009-9b74-968b974f601c created

> kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS         REASON   AGE
pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            Retain           Available           managed-csi-retain            4s

Add a new PVC

Next, create a new PVC definition pointing to the PV. Make sure to use your desired namespace. Then, apply it with kubectl apply -f pvc-disk1.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-disk1
  namespace: <your-namespace>
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  volumeName: pvc-36ac28d3-e1af-4009-9b74-968b974f601c
  storageClassName: managed-csi-retain

pvc-disk1.yaml

You should now have a PV and PVC in state Bound.

> kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS         REASON   AGE
pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            Retain           Bound    <your-namespace>/pvc-disk1   managed-csi-retain            15m

> kubectl get pvc 
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
pvc-disk1   Bound    pvc-36ac28d3-e1af-4009-9b74-968b974f601c   5Gi        RWO            managed-csi-retain   2m

Consume the PVC

You can now create a Pod referencing and consuming the PVC.

apiVersion: v1
kind: Pod 
metadata:
  name: demo
  namespace: app
spec:
  containers:
  - name: demo
    image: nginx:latest
    volumeMounts:
    - name: datadir
      mountPath: "/mnt/azure"
  volumes:
  - name: datadir
    persistentVolumeClaim:
      claimName: pvc-disk1

pod.yaml

Conclusion

This article discussed how a persistent volume in state Released could be reclaimed and reattached to a Pod. This was possible by creating a new, static PV pointing to the existing Azure-managed disk.

I hope you found this article helpful! Thanks for reading! 😎