I was recently cleaning up some orphaned vSphere Container Volumes and while I was able to compare the Persistent Volumes (PV) IDs against the ones currently in use by a couple of Kubernetes clusters, I wanted to be 100% sure before removing them, especially since I planned bulk delete using PowerCLI’s Remove-CnsVolume cmdlet.
You can view all provisioned vSphere Container Volumes in the vSphere UI by navigating to vSphere Cluster->Monitor->Cloud Native Storage->Container Volumes. In addition to the volume details, you also get access to all of the associated metadata and labels, very similar to what you would see using kubectl, which is quite nice!
While reviewing the PV IDs in the UI, I noticed an attribute named Kubernetes Cluster, which appeared to identify the Kubernetes (K8s) cluster that owned the volume. How do I find this Kubernetes Cluster ID?
After a bit of trial and error, I discovered that the Kubernetes Cluster ID shown in the vSphere UI corresponds to the Cluster ID maintained by the vSphere CSI driver, making it easy to map a vSphere Container Volume back to its originating Kubernetes cluster.
Using PowerCLI, we can first fetch all unique k8s Cluster ID from available PVs by running the following:
$pv = Get-CnsVolume $pv.ExtensionData.Metadata.ContainerCluster.ClusterID | Select-Object -Unique
Once I have the list of k8s Cluster ID, I can then login to k8s cluster in question and compare the ID, which you can run the following command:
kubectl -n kube-system get cm vsphere-csi-cluster-id -o yaml
In my case, I used this to validate both my VCF Management Services (VCFMS) and VCF Automation (VCFA) deployments, ensuring that none of the orphaned PVs I was about to remove were associated with either environment, since I had a recent re-deployment.
Once you have identified the k8s Cluster ID, you can use the following PowerCLI snippet to remove stale/orphaned PV volumes:
$clusterIdPvToRemove = "dc88861b-393b-4b0e-8b2a-db5416b3f02d"
$pv | where {$_.ExtensionData.Metadata.ContainerCluster.ClusterID -eq $clusterIdPvToRemove} | Remove-CnsVolume -Confirm:$false
Thanks for the comment!