Like most folks, I was excited about the release of vSphere 7.0 Update 2 and I was ready to upgrade my personal homelab, which was running on vSphere 7.0 Update 1c. However, after starting my VCSA upgrade in the VAMI UI, it quickly failed with the following error message: Exception occurred in install precheck phase
Joy … I just attempted to upgrade my VCSA (7.0u1c) in my personal homelab to #vSphere70Update2 and ran into “Exception occurred in install precheck phase” … pic.twitter.com/4mkvxHxdRl
— William Lam (@lamw) March 9, 2021
Given the release had just GA'ed less than an hour ago and everyone was probably hammering the site, I figured I would wait and then try again.
Issue
I normally use the "Stage and Install" option and after accepting the EULA and providing the required credentials, the upgrade began.
Almost immediately, it failed with the same error and the only option was to click on the Resume button which I did.
The upgrade seemed to have resumed and pre-check results passed as you can see from the screenshot below. However, after hitting continue, I would be taken right back to the pre-check error as noted earlier. I repeated this a few times and realize it was stuck in this loop.
I even attempted to reboot the VCSA, maybe my environment had some issue but the results were the same. There were also a couple of suggestions on removing the following state file /etc/applmgmt/appliance/software_update_state.conf on the VCSA filesystem which apparently was a solution that had worked. However, in this case it did not do anything and I still ran into the issue.
At this point, I was not interested poking around or randomly removing files as I heard few other folks were running into this problem. I generated a support bundle and file an internal bug to have Engineering take a look.
Disclaimer: If you are currently facing this problem in a Production environment, please stop and file a VMware Support Request so that this can be officially tracked and you can have GSS reference PR 2732490. The workaround below has only been confirmed by the community and should only be attempted on non-production or homelabs environment where you are comfortable with the workaround.
Workaround
Several folks have shared with me a workaround that they have verified which requires the upgrading the VCSA using the VCSA Shell. I have personally not applied this workaround as I wanted to leave my environment in its existing state, in case VMware Engineering need to troubleshoot live. For those interested, below are the instructions to apply the workaround.
Step 1 - Login to the VAMI UI and under Updates, only select the "Stage" option to download the 7.0 Update 2 updates.
Step 2 - SSH to the VCSA and remove /etc/applmgmt/appliance/software_update_state.conf file
Step 3 - Run the following command to install the staged 7.0 Update 2 software:
software-packages install --url --acceptEulas
Once this operation completes, you should now have a successfully upgraded VCSA to 7.0 Update 2. I will update this blog post as I learn more from Engineering and if this workaround is official, I will ask the team to publish a KB and also mention this in the 7.0 Update 2 release notes.
I ran into this today. The fix for me was staging and then upgrading in two steps, rather than one. The install then proceeded until 80% when it threw an error that I was able to continue past. No issues with my second vCenter.
Two vCenters have been upgraded today. It should be easier than this. Don't they test?!
William - It would be helpful to know what version your VCSA was at prior to the upgrade attempt to 7.0.2? I recently ran into this same issue when upgrading VCSA 7.0.1a to 7.0.1d. I'll refer readers to your Disclaimer above - STOP and call GSS if you're experiencing this on a production system! In my case, GSS had me follow the procedure in https://kb.vmware.com/s/article/67179 to remove the software_update_state.conf file, then mount the full patch -fp.iso to my VCSA vm appliance, then staging the packages and upgrading in two separate steps via CLI as described here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vcenter.upgrade.doc/GUID-5FCA78EC-8637-43A4-8B28-24624E4D5EBA.html . One thing not mentioned is that you will get prompted to enter SSO credentials during the upgrade process, but the CLI upgrade went smoothly. GSS indicated to me that this was caused a bug in the source VCSA bits of my previous 7.0.1a install, and that this bug was resolved in the newer versions - so future upgrades from VAMI GUI should work as normal. I have NOT tried this 7.0.2 upgrade yet in that environment.
Thanks for sharing Patrick. I actually did mention my version, it’s 7.0u1c 🙂
My apologies - my eyes skimmed right over it. Reading Is Fundamental! Now I need to test from 7.0.1d and see if I run into the same issue 😉
Fixed for me as well, thanks!
I've tried the workaround to update from 7.0 Update 1d. The update ends with the following error: Mismatch:
summary: PostgreSQL failed to run upgrade command. So even it's my lab, I decide to wait for an update of vCSA 7.0 Update2. 🙂
Hit this exact issue twice on the last two times I patched the VCSA in our lab.
The same workaround (patch ISO) worked each time, and appears to be working this time.
I wanted to update my lab yesterday to 7.0 U2. from 7.0 U1a
First I tried via VAMI and got the same error like above.
Removing the file etc/applmgmt/appliance/software_update_state.conf did not resolve the issue.
But also via CLI I could not update, if I use the --ulr switch I got:
"Answers for following questions have to be provided to install phase:
Question:
ID: vmdir.password
Text: Single Sign-On administrator password
Description: For the first instance of the identity domain, this is the password given to the Administrator account. Otherwise, this is the password of the Administrator account of the replication partner.
Allowed values:
Default value:
"
I do not get a prompt to enter something.
(Command> software-packages stage --url https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.1.00100.latest/ --acceptEulas
)
Then I downloaded the 7.0 U2 iso and mounted it to VCSA and run the update via CLI with the --iso switch, I got:
"ISO unmounted successfully
CD drives do not have valid patch iso."
It sounds like you may have downloaded the install .iso and not the Full Patch iso? Try the VMware-vCenter-Server-Appliance-7.0.2.00000-17694817-patch-FP.iso available on the VMware patch site: https://my.vmware.com/group/vmware/patch
@ Patrick Long: thx, with the patch iso the upadte to 7.0 U2 worked.
There has been a running bug in 7.0 since GA just in a slightly different form. If you install any updates using the CLI then VAMI will no longer install updates at all failing with errors or not showing any updates available.
I had cases open and was told this would be fixed in U2 sounds like fixes for it may have broken something else.
I was able to complete the update today. Here is a short document, I created for future reference.
http://www.infotechram.com/index.php/2021/03/16/how-to-update-vcenter-manually-using-cli/
Ram
Thank you William, the workaround worked as a charm. In my case I stumbled on some other issues like:
"Error in method invocation {'id': 'com.vmware.appliance.update.checksum_verification_failed', 'default_message': 'Checksum verification failed', 'args': []}"; so changing the URL at update Section overcomes this one: https://10.15.9.4:5480/#/ui/update/availableUpdates
https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.1.00100.latest/
Afterwards, once staged and prechecked/validated the Install process starts to never end. After an snapshot rollback the only way to resolve it so far is to follow on the posted solution at the article.
Got the exact same issue in a prod vcenter; going to raise a ticket right now, will reference your GSS ticket. thanks for that!
Get the same issue but still no luck with your instruction, always failed in the "copying software packages" stage when it comes to 66/67 package out of 150.
Had an odd experience with this issue just now. I got the error as everyone else here. Then started to google the issue, and of course stumbed upon this post.
Then after a while, without doing anything,i noticed i got tossed out into the login screen of the VCSA, and when trying to log in again, it failed.
I then tried opening vcenter normally, and got a message about upstream issue.
This reminded me a bit about what happens while an update is going on, so i let it rest for a while, and god different errors when trying to enter vcenter.
Left it for about an hour, and when i tried again, it was updated! So even though i got the error, and didn't try any of the workarounds, it ended up updating to Update 2 after all!
I tried a couple times using the Management site Update (including separately pre-checking, staging, installing... precheck always passed, staging always succeeded but install always hung up and I had to recover by deleting the update state conf file). Now I'm using the cli and software-packages command, it had a couple separate steps but I did get it to install successfully that way.
I also had this issue. I followed the workaround in this KB and it worked for me:
https://kb.vmware.com/s/article/83145?lang=en_US
Had the same issue, but the latest patch 7.0.2.00100 fixed it.
YO, Niels. This KB fixed it for me as well.
https://kb.vmware.com/s/article/83145?lang=en_US
Many thanks.
I've hit this issue 5-6 times already, had to reverse the order of the steps in the workaround.
Delete that file first
then pre-check (this was a required step for me)
then stage
then install
the above worked okay for me
I used command line and iso
Update went fine and after reboot
vmdir.password is not set ; aborting installation
SAD SAD
any idea for this one ?
same there !! the vcenter is so buggy ! any solution yet ?
Had the same issues on two separate vcenters. I had to restart the VAMI because it was stuck in a failure loop, and changing the address to get out of the loop worked but the staging option would fail.
After restarting the VAMI the staging option worked on both servers, and then was able to procceed with the workaround.
Thank You!
Hi,
I have the same issue during upgrade of my test vCenter to 7 Update 3.
My vCenter version is 7.0.2 Build: 18455184 and was testing the upgrade to 7.0.3
Will try the KB : https://kb.vmware.com/s/article/83145?lang=en_US
Tell you if it's work for me.
I had a similar problem when upgrading my test installation from 7.0.3.00000 (7.0U3) to 7.0.3.00100 (7.0U3a).
I tried all the root shell tricks mentioned above regarding software_update_state.conf, result was always: "Stage path file doesn't exist".
This command helped:
software-packages unstage
After that, this worked:
software-packages stage --iso --acceptEula
and thus this as well:
software-packages install --staged
Changing the Update Mirror on the VCSA (settings sections) to:
https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/7.0.3.00100.latest/
worked for me. The upgrade went through and the mirror can be set to "standard" afterwards.
Seems like this is an on going issue that vmware has not fixed yet.
I faced the issue in my upgrade from 7.0 U1 to U2 and today again from 7.0U2b to 7.0U2d
the fix is as below.
At around 75 % of the install ( as shown in the GUI), keep checking the permission of the below file.
It is supposed to look like this:
root@HRC-1C-VC01 [ /usr/lib ]# ls -al /usr/lib/libsas*
-rwxr-xr-x 1 root root 112240 May 12 2020 /usr/lib/libsasl2.so.3.0.0
At some point (75 % of the installer) , the permission will change as below:
root@HRC-1C-VC01 [ /usr/lib ]# ls -al /usr/lib/libsas*
-rw-r----- 1 root root 112240 May 12 2020 /usr/lib/libsasl2.so.3.0.0
Immediately run the below command to correct the permission:
chmod 755 /usr/lib/libsasl2.so.3.0.0
Let the installer run.
This worked for me on 2 different versions.
the issues is still there .. holy !
HI,
I use this VMware KB and it fix the issue for me
https://kb.vmware.com/s/article/83145
I tried to run the command "software-packages install --url --acceptEulas" but I received the following error:
-bash: software-packages: command not found
You need to run this in appliancesh, since it shows bash, you need to run that command first then run software-package
Thanks William!