Removing NSX CPU/Memory reservations when deploying a VMware Cloud Foundation (VCF) Management or Workload Domain

Now that you can deploy a VMware Cloud Foundation (VCF) Management Domain onto a single ESXi host for homelab purposes, another challenge that can arise is simply having enough resources to deploy either a Management or Workload Domain.

While you do need to meet some minimum bar in terms of the amount of CPU and memory resources required to run the various components within a VCF Management or Workload Domain, I found that NSX resource requirements does add another challenge. The NSX Unified Appliance comes configured with both CPU and memory reservations, this means unless you can reserve both the required CPU and Memory for NSX, you can not power on NSX and the deployment will continue to retry until it eventually gives up. 

Again, for a production environment, this is not a problem but for homelab or testing purposes, this can further restrict users from exploring the VCF solution. 

I figured why not just reduce or even remove the CPU and memory reservations from the NSX OVA within the VMware Cloud Builder which is responsible for the initial VCF bringup? Well, I quickly found out why as the ISO which contains all the OVA is mounted as read-only 🙂

Luckily, all hope is not lost and this is where I learned to leverage a neat Linux kernel capability called OverlayFS which can help us! With a bit of trial/error, I was able to remove the CPU and memory reservations from the NSX appliance that is used during the deployment of a VCF Management Domain. In addition, I also ran into the same challenge when deploying a VCF Workload Domain and luckily, that was a much easier solution to figure out.

Disclaimer: This is not supported by VMware, use at your own risk. As of writing this blog post, this trick is functional on latest VCF 4.5 release.

Cloud Builder

Step 1 - Before starting a VCF Management Domain deployment, SSH to the Cloud Builder VM and unmount the default /mnt/iso path by running the following command:

umount /mnt/iso

Step 2 - Load the overlay kernel module by running the following command:

modprobe overlay

Note: This setting does not persist across a reboot

Step 3 - Create the following directories which will be used for the overlay as well as mounting the VCF ISO image to an alternative path:

mkdir -p /overlay/{upper,work}
mkdir -p /root/oldiso

Step 4 - Mount VCF ISO to our alternative directory that we had created in the previous step:

mount -o loop /opt/vmware/vcf/iso/sddc-foundation-bundle.iso /root/oldiso

Step 5 - Mount our overlay which we can make edits under /overlay/work/work by running the following command:

mount -t overlay -o lowerdir=/root/oldiso,upperdir=/overlay/upper,workdir=/overlay/work overlay /mnt/iso

Step 6 - Before we can change NSX OVA, we need to first convert it to an OVF by running the following commands:

NSX_FILENAME="nsx-unified-appliance-3.2.1.2.0.20541216"
ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification /root/oldiso/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ova /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ovf

Step 7 - Next, we remove the manifest file since we are going to make changes to the NSX OVF and also remove the resource reservations from the OVF file by running the following commands:

rm /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.mf
sed -i '/ <rasd:Reservation>.*/d' /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ovf

Step 8 - Now we just convert the NSX OVF back to an OVA by running the following command:

ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ovf /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ova

Step 9 - Clean up the unnecessary files by deleting the previous VMDK and OVF file:

rm /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/*.vmdk
rm /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/*.ovf

Step 10 - Lastly, update both the owner and permissions so that Cloud Builder can access our newly created file:

chown nobody:nogroup -R /mnt/iso
chmod -R 755 /mnt/iso

At this point, you are now ready to start your VCF deployment and NSX will now be deployed without CPU and memory reservations!

SDDC Manager

Step 1 - Before starting a VCF Workload Domain deployment, SSH to the SDDC Manager VM using the vcf user account and then switch to root by running the following command and providing the credentials to the vcf user:

su -

Step 2 - Change into the NSX OVA directory and set the following environment variable based on the filename of the NSX OVA:

cd /nfs/vmware/vcf/nfs-mount/base-install-images/nsxt_ova
NSX_FILENAME="nsx-unified-appliance-3.2.1.2.0.20541216"

Step 3 - Convert the OVA to an OVF by running the following command:

ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification ${NSX_FILENAME}.ova ${NSX_FILENAME}.ovf

Step 4 - Next, we remove the manifest file and OVA since we are going to make changes to the NSX OVF and also remove the resource reservations from the OVF file by running the following commands:

rm ${NSX_FILENAME}.mf
rm ${NSX_FILENAME}.ova
sed -i '/ <rasd:Reservation>.*/d' ${NSX_FILENAME}.ovf

Step 5 - Now we just convert the NSX OVF back to an OVA by running the following command:

ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification ${NSX_FILENAME}.ovf ${NSX_FILENAME}.ova

Step 6 - Lastly, update both the owner and permissions so that Cloud Builder can access our newly created file and clean up the the files that we no longer need:

chown vcf:vcf ${NSX_FILENAME}.ova
chmod 755 ${NSX_FILENAME}.ova
rm *.ovf
rm *.vmdk

At this point, you are now ready to start your VCF Workload Domain deployment and NSX will now be deployed without CPU and memory reservations!