Yesterday, I began upgrading my VMware Cloud Foundation (VCF) environment from version 9.0.1 to the latest 9.0.2 release. After successfully upgrading VCF Operations Fleet Manager, I then proceeded to upgrade VCF Operations, which includes running a pre-validation to ensure everything was okay before we can proceed.
As you can see from the screenshot above, I ran into the following error: "No space left on device" which was pretty vague, but I figured it had to do with my VCF Operations instance, since I was attempting to upgrade it ...
After SSH'ing into my VCF Operations instance, I did not see anything alarming when running the diskfree (df -h) command, there was plenty of space!
This was puzzling. I decided to log into VCF Operations Fleet Manager, since it was performing the validation in the hopes I might find a clue or at least understand what it is checking. You can view the logs under /var/log/vmware/vrlcm.log and after a quick search from the bottom, I saw where the UI error might be referencing/
The only issue is that the partition that is being check is NOT of VCF Operations but VCF Operations Fleet Manager itself!
I was able to verify the same output by running diskfree command on VCF Operations Fleet Manager and was able to correlate the information.
Now that we know where and what the issue was, the resolution was to simply hot extend VCF Operations Fleet Manager. Login to VCF Operations UI and navigate Fleet Management->Lifecycle->VCF Management->Settings and click on Extend Storage to increase the volume, which I added an additional 100GB.
Once the storage expansion had completed, I was able to re-run the validation, which was successful and I was then able to successfully upgrade VCF Operations to 9.0.2!
Although the Extend storage is happening online, depending on storage type and the amount of additional GB, it would be wise schedule a maintenance for this operation because it can be very long before it finish.
In my lab with NFS storage Extend of addition 900GB, took 9 hours to complete.
To notes for people working through this. 1) you can't have a snapshot of your FM when doing the expansion. Give me some slack, I took a snap before updating and forgot to remove it. 2) you have to have a valid certificate structure for this workflow to work.
vCenter> FM VM > expand Hard disk 2. SSH to FM, /var/lib/vlcm-common/autogrow.sh.