I got a question from my buddy Paudie O'Riordan this morning where he was noticing a strange issue while trying to upgrade his ESXi hosts from 6.0 to 6.5 (all on the VMware HCL). Like many of our customers, he runs ESXi on USB device and when he attempted to upgrade using ESXi Scripted Install (Go Automation!), he was surprised to find that his USB device was not being detected.
Interestingly, I had literally just finished answering a similar question on our internal Socialcast forum and I had wondered if Paudie was also seeing the same problem. The issue looks to be related to the new USB Native Driver (vmkusb) that was introduced in ESXi 6.5 where is it is unable to claim the specific USB device.
Although you can disable the USB Native Driver and fall back to the legacy driver as mentioned in this VMware KB 2147650, but because this is happening during the installation/upgrade process, it can get a bit tricky.
Luckily, we have a workaround. We just need to disable the Native Drivers for the initial installation which will allow us to perform the upgrade and then we can disable the vmkusb driver and re-enable the rest of the Native Drivers post-install. The last step is needed especially if you rely on some of the other Native Drivers, which was the case for Paudie as his vSAN setup was using them. Below are the instructions on the workaround.
Step 1 - Append preferVmklinux=TRUE to the ESXi's boot.cfg file whether that is on a USB device which has your ESXi installer or on your PXE Server for ESXi Scripted Installations.
Here is an example of what boot.cfg should look like:
kernelopt=runweasel preferVmklinux=TRUE
Step 2 - Boot up ESXi and you should now be able to see the USB device and continue with your upgrade.
Step 3 - Once ESXi has been successfully upgraded, go ahead and run the following commands to re-enable the Native Drivers and disable the vmkusb driver (reboot required for changes to go into effect):
esxcli system settings kernel set -s preferVmklinux -v FALSE
esxcli system module set --enabled false -m vmkusb
reboot
Note: If you are doing an ESXi Scripted Installation, you can actually append just the first two commands (replace esxcli with localcli as hostd is not running) as part of the Kickstart %post section. This would then allow you to automate the entire upgrade using this fix without having to perform any manual steps or additional reboots.
Any guides on how to update 6.0 to 6.5 ?
Thanks for a great resource! - Ed
William - that same issue "bricked" a couple of hosts in my lab! I used VUM to update ESXi to 6.5 (Dell Customized) on a cluster of R710s. Because the ESXi install was on USB, the update was staged, remediation initiated, but upon boot-up the install could not be found causing the nodes to reboot over and over infinitely because an upgrade/update flag must have been invoked but the USB driver within the staged update could not be located. Nothing I could do at this point without deep understanding of the boot args so I had to manually install 6.5 to spinning disk which was the only storage detected at the time of boot. Obviously this may not be an option in production. I hope VMware can incorporate your fix in the future because I am a bit scared to discover USB incompatibility in production!
Thanks, William!
I have server ESXi6 with USB flash boot. I did upgrade through VUM to 6.5. On reboot had error: Device naa.2** can't be read.
On start clean install local USB flash drive not found.
I did run server, on ESXi boot menu did press "Shift+O", add "preferVmklinux=TRUE" to the end of string. After this upgrade was finished. Now a host normal boot from 4GB USB flash drive.
Hi William,
Do you know if there is a kernel option similar to this to disable the vmw_ahci module as well on boot?
Loosing the boot option from USB keys and slow ache performances are my main two problems with ESXi 6.5 - everything else is great.
No, but you can disable it as part of Kickstart installation if you want it Automated OR just disable it post-install
Hi William,
Thanks for your workaround. However after using it I noted a strange behavior when I'm trying to set scratch location.
Despite having properly set ScratchConfig.ConfiguredScratchLocation and rebooting my servers the ScratchConfig.CurrentScratchLocation is always set to /scratch and doesn't change.
Do you know what may cause this behavior?
Thanks,
Lukasz
i bought a hp dl 380 g7 server and installed esxi 6.5 to internal usb. but after restarting it never recognized there is internal usb and trying to find a boot disk. is there any setup in hp dl 380 g7 in bios or something else. installation was successful without any issues.
Hi William,
I am trying scripted upgrade from esxi 6.0 to esxi 6.5 on a remote disk (LUN);
Seeing the similar issue "upgrade --firstdisk specified, but no suitable disk was found". How ever If I do "install" esxi 6.5 instead of "upgrade" it works fine.
I tried with "kernelopt=runweasel prefervmklinux=TRUE formatwithmbr ks=http://10.235.0.12/ks_mirror/ESX65/ks.cfg"
Below is the ks.cfg
[[email protected] tftpboot]# cat /var/www/html/ks_mirror/ESX65/ks.cfg
........
#install --firstdisk --overwritevmfs
upgrade --firstdisk
network --bootproto=dhcp
%firstboot --interpreter=busybox
Awesome. Saved us!
Hi, you trick worked for me too. I booted by 6.0 host via iLO with the 6.5 installer mounted as a virtual CDROM, booted it, did the "shift+O" thing and it detected the USB stick running 6.0 and was able to upgrade.