WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple
You are here: Home / Automation / Removing NSX CPU/Memory reservations when deploying a VMware Cloud Foundation (VCF) Management or Workload Domain

Removing NSX CPU/Memory reservations when deploying a VMware Cloud Foundation (VCF) Management or Workload Domain

02.22.2023 by William Lam // 3 Comments

Now that you can deploy a VMware Cloud Foundation (VCF) Management Domain onto a single ESXi host for homelab purposes, another challenge that can arise is simply having enough resources to deploy either a Management or Workload Domain.

While you do need to meet some minimum bar in terms of the amount of CPU and memory resources required to run the various components within a VCF Management or Workload Domain, I found that NSX resource requirements does add another challenge. The NSX Unified Appliance comes configured with both CPU and memory reservations, this means unless you can reserve both the required CPU and Memory for NSX, you can not power on NSX and the deployment will continue to retry until it eventually gives up. 

Again, for a production environment, this is not a problem but for homelab or testing purposes, this can further restrict users from exploring the VCF solution. 

I figured why not just reduce or even remove the CPU and memory reservations from the NSX OVA within the VMware Cloud Builder which is responsible for the initial VCF bringup? Well, I quickly found out why as the ISO which contains all the OVA is mounted as read-only 🙂


Luckily, all hope is not lost and this is where I learned to leverage a neat Linux kernel capability called OverlayFS which can help us! With a bit of trial/error, I was able to remove the CPU and memory reservations from the NSX appliance that is used during the deployment of a VCF Management Domain. In addition, I also ran into the same challenge when deploying a VCF Workload Domain and luckily, that was a much easier solution to figure out.

Disclaimer: This is not supported by VMware, use at your own risk. As of writing this blog post, this trick is functional on latest VCF 4.5 release.

Cloud Builder

Step 1 - Before starting a VCF Management Domain deployment, SSH to the Cloud Builder VM and unmount the default /mnt/iso path by running the following command:

umount /mnt/iso

Step 2 - Create the following directories which will be used for the overlay as well as mounting the VCF ISO image to an alternative path:

mkdir -p /overlay/{upper,work}
mkdir -p /root/oldiso

Step 3 - Mount VCF ISO to our alternative directory that we had created in the previous step:

mount -o loop /opt/vmware/vcf/iso/sddc-foundation-bundle.iso /root/oldiso

Step 4 - Before we can change NSX OVA, we need to first convert it to an OVF by running the following commands:

NSX_FILENAME="nsx-unified-appliance-3.2.1.2.0.20541216"
ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification /root/oldiso/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ova /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ovf

Step 5 - Next, we remove the manifest file since we are going to make changes to the NSX OVF and also remove the resource reservations from the OVF file by running the following commands:

rm /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.mf
sed -i '/ <rasd:Reservation>.*/d' /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ovf

Step 6 - Now we just convert the NSX OVF back to an OVA by running the following command:

ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ovf /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ova

Step 7 - Clean up the unnecessary files by deleting the previous VMDK and OVF file:

rm /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/*.vmdk
rm /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/*.ovf

Step 8 - Update both the owner and permissions so that Cloud Builder can access our newly created file:

chown nobody:nogroup -R /mnt/iso
chmod -R 755 /mnt/iso

Step 9 - Load the overlay kernel module by running the following command:

modprobe overlay

Note: This setting does not persist across a reboot

Step 10 - Finally, mount our overlay which we can make edits under /overlay/work/work by running the following command:

mount -t overlay -o lowerdir=/root/oldiso,upperdir=/overlay/upper,workdir=/overlay/work overlay /mnt/iso

At this point, you are now ready to start your VCF deployment and NSX will now be deployed without CPU and memory reservations!

SDDC Manager

Step 1 - Before starting a VCF Workload Domain deployment, SSH to the SDDC Manager VM using the vcf user account and then switch to root by running the following command and providing the credentials to the vcf user:

su -

Step 2 - Change into the NSX OVA directory and set the following environment variable based on the filename of the NSX OVA:

cd /nfs/vmware/vcf/nfs-mount/base-install-images/nsxt_ova
NSX_FILENAME="nsx-unified-appliance-3.2.1.2.0.20541216"

Step 3 - Convert the OVA to an OVF by running the following command:

ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification ${NSX_FILENAME}.ova ${NSX_FILENAME}.ovf

Step 4 - Next, we remove the manifest file and OVA since we are going to make changes to the NSX OVF and also remove the resource reservations from the OVF file by running the following commands:

rm ${NSX_FILENAME}.mf
rm ${NSX_FILENAME}.ova
sed -i '/ <rasd:Reservation>.*/d' ${NSX_FILENAME}.ovf

Step 5 - Now we just convert the NSX OVF back to an OVA by running the following command:

ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification ${NSX_FILENAME}.ovf ${NSX_FILENAME}.ova

Step 6 - Lastly, update both the owner and permissions so that Cloud Builder can access our newly created file and clean up the the files that we no longer need:

chown vcf:vcf ${NSX_FILENAME}.ova
chmod 755 ${NSX_FILENAME}.ova
rm *.ovf
rm *.vmdk

At this point, you are now ready to start your VCF Workload Domain deployment and NSX will now be deployed without CPU and memory reservations!

More from my site

  • ESXi on GMKtec NucBox K11
  • Quick Tip - VMware Cloud Foundation (VCF) Bringup fails without persistent ESX-OSData
  • Enhancements to VMware Cloud Foundation (VCF) & vSphere Automated Lab Deployment Scripts
  • vSAN ESA hardware mock VIB for physical ESXi deployment for VMware Cloud Foundation (VCF)
  • Quick Tip - Easily host VMware Cloud Foundation (VCF) Offline Depot using Python SimpleHTTPServer with Authentication

Categories // Automation, Home Lab, NSX, VMware Cloud Foundation Tags // VMware Cloud Foundation

Comments

  1. *protectedAbbed Sedkaoui says

    03/19/2023 at 10:31 am

    Thank you for the tricks, i was wishing for that to exist and you made it happen!
    Just tried and there is a folder created by ovftool so we have to change Step 7,8,9 like that.

    Step 7
    rm /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}/${NSX_FILENAME}.mf
    sed -i '/ .*/d' /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}/${NSX_FILENAME}.ovf

    Step 8
    ovftool --acceptAllEulas --allowExtraConfig --allowAllExtraConfig --disableVerification /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}/${NSX_FILENAME}.ovf /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}.ova

    step 9
    rm -rf /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/${NSX_FILENAME}/

    Cheers
    Abbed

    Reply
    • *protectedAbbed Sedkaoui says

      03/21/2023 at 11:03 am

      Finally got it working, in addition to previous folder change,
      by updating ownership and permission for the work folder as well.
      in Step 10
      chown nobody:nogroup -R /overlay/work/work
      chown nobody:nogroup -R /mnt/iso
      chmod -R 755 /overlay/work/work
      chmod -R 755 /mnt/iso

      And the last Step that did the trick for me was to copy work to upper.

      root@vcf-m01-cb01 [ ~ ]# ls -l /mnt/iso/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/
      total 10248252
      -rwxr-xr-x 1 nobody nogroup 10494218240 Sep 28 17:46 nsx-unified-appliance-3.2.1.2.0.20541216.ova

      cp /overlay/work/work/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/nsx-unified-appliance-3.2.1.2.0.20541216.ova /overlay/upper/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/nsx-unified-appliance-3.2.1.2.0.20541216.ova

      Then upper naturally showed up on merged mnt folder.

      root@vcf-m01-cb01 [ ~ ]# ls -l /mnt/iso/sddc-foundation-bundle-4.5.0.0-20612863/nsxt_ova/
      total 10248252
      -rwxr-xr-x 1 nobody nogroup 10494204928 Mar 21 16:11 nsx-unified-appliance-3.2.1.2.0.20541216.ova

      Reply
  2. *protectedSho says

    09/22/2024 at 6:25 am

    Do you know how to use NSX Manager with 1-node in VI Domain?

    Reply

Thanks for the comment!Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...