Several users in the community have reported running into a memory error issue when attempting to upgrade their standalone ESXi hosts using this ESXCLI trick, which has been possible since ESXi 5.1 back in 2012.
[MemoryError]
Please refer to the log file for more details.
Initially, I was not able to reproduce in my lab environment at work, which was running ESXi 8.0 Update 2, but I still got an error message but it was different:
Got no data from process.
Command "LANG=en_US.UTF-8 /usr/lib/vmware/esxcli-software sources.profile.list -d "https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml" "
еxited with error code: 1
After checking my homelab environment which was also running ESXi 8.0 Update 2, I did see the same MemoryError message as reported by the community when using ESXCLI to list the available ESXi Image Profiles from VMware's online patch repository.
I was about to report this ESXCLI issue when I saw that an internal PR had already been filed and Engineering is currently investigating the problem, which I have also asked for a KB to be published for awareness and resolution tracking purposes.
From the preliminary assessment, the issue is due to the fixed amount of memory (300MB) that is assigned to the ESXCLI process when it runs and because the VMware's online patch repository has gotten bigger with each release, additional memory is required to parse the current repository, which exceeds the fixed limit. You can see this limi by running: grep 'mem=' /usr/lib/vmware/esxcli-software
The fixed memory limit for ESXCLI was only introduced in ESXi 8.x and is not observed in ESXi 7.x, which would also explain why I was not seeing this problem when using the latest ESXi 7.0 Update 3 host.
The current guidance for this problem is to either update/upgrade using vCenter Server OR if you have a standalone ESXi host, point ESXCLI to a local ESXi offline bundle ZIP file rather than using the online VMware patch repository which would avoid parsing the repo's metadata. The other nice benefit of using the local patch method is that you only have to download the update once rather than pulling it down multiple times if you have more than a single ESXi host to update.
Alternatively, if this is just your homelab and you only have a single ESXi host like myself, there is a workaround that I was able to figure out that would allow you to still use online VMware patch repository but this is is definitely not officially supported.
Login to the ESXi Shell via SSH and run the following command, which will update the configured ESXCLI memory limit from 300MB to 500MB, which should be sufficient to now use ESXCLI to update/upgrade your ESXi host:
esxcli system settings advanced set -o /VisorFS/VisorFSPristineTardisk -i 0 cp /usr/lib/vmware/esxcli-software /usr/lib/vmware/esxcli-software.bak sed -i 's/mem=300/mem=500/g' /usr/lib/vmware/esxcli-software.bak mv /usr/lib/vmware/esxcli-software.bak /usr/lib/vmware/esxcli-software -f esxcli system settings advanced set -o /VisorFS/VisorFSPristineTardisk -i 1
Brilliant, thanks William. While I did not run into that issue it should help everyone else that did.
WORKED GREAT!!! I have a HOME lab with one ESX node and THANK YOU!!
NICE!!! Thank you!
Much appreciated.
Success at long last 🙂
Hmm what license are u using in your homelab? 😉
Dude, he works for VMware, so I guess he is using whatever licensing he wants to generate for himself. I know Broadcom are tight, but I doubt they made him sign up to a VMUG Advantage sub!
Works fine, thanks man!
thanks, works fine 🙂
Thanks, worked perfectly!
Thanks 🙂
Hi William, very useful article, as usual...
Working on a DELL PE R440 with a Dell-customised ESXi 8.0 U2b, trying to update the Dell Addon component to A05, I had to increase that limit to 700 MB...
Thanks!
Hi,
What are the steps to increate mem allocation form vcenter?
Thanks,
Save my day. Thank you very mutch
GREAT!!! Thx!!!!
What does line 1 and 4 do?
Thank you! Worked for me too.
Much appreciated! I have saved this into an executable script file because it seems that the value gets reset with every reboot of the ESXi host. Many thanks!
Thank you very much for this great article. I ran into this error and didn't know why. You helped me to solve the issue.
Thanks this have saved me lots of work.
Thanks William got exacly this same problem with Memory error when i try update my esx 8.0 to Build 24280767
William, you are the boss. Guys like you save us tons of time and effort. Great respect. Jay
Apperciate the comment Jay, just helping our users, especially as I'm still an active user myself on a regular basis 🙂
Doesn't modifying the backup for the esxcli-software and then overwriting the original defeat the purpose of making a backup?
i just made the backup and modified the original, if something broke i could restore the original from the backup.
Thank you! I have successfully updated to ESXi 8.0 Update 3d.
Thanks a lot. Now Update 3d is installed
Is the Broadcom depot disliking access by proxy intermediated ESXCLI access?
Doing
esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --proxy=http://example.com:80
works but
esxcli software sources profile list -d https://hostupdate.broadcom.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --proxy=http://example.com:80
throws a 403 error, but running a browser through the same depot URL and proxy comes back fine with the correct xml file and no cert errors. Running the ESXCLI command proxyless also works fine too, but usually this host has no direct internet access so I can't depend on that.
afaik, the endpoint is hostupdate.vmware.com NOT hostupdate.broadcom.com