WilliamLam.com

Re: Host is in a VSAN enabled cluster but does not have VSAN service enabled

by // Leave a Comment

I recently noticed a couple of people hitting a warning message when configuring VSAN and specifically when running VSAN in a Nested ESXi environment (which is not officially supported by VMware). The warning message is displayed on the summary page of the ESXi host which states the following:

Host is in a VSAN enabled cluster but does not have VSAN service enabled

vsan-minimal-memory
The reason you are seeing this issue is related to the amount of memory you have allocated for your Nested ESXi VM. During the VSAN Beta, the minimal amount of memory was 4GB but it looks like that has changed to 5GB with the GA of vSphere 5.5 Update 1 last week. I know we could have done better job with the error message and communicate the actual underlying issue (will ensure we have an FR filed for this).

However, the fix is quite simple, just shutdown your Nested ESXi VM and then change it to 5GB and this message will go away. It is also worth noting that as you increase the number of disks and disk groups in your ESXi hosts, there will be an increase in memory. I would highly recommend you take a look at the official VSAN Design & Sizing guide to properly size out your real VSAN environments.

For basic functional testing and education of VSAN (not including running additional VMs), running a Nested ESXi VM with 5GB will be sufficient. You can also take a look at the my VSAN Nested ESXi OVF template which can just download and install ESXi 5.5 Update 1 without any issues.

How to properly clone a Nested ESXi VM?

by // 53 Comments

I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.

UPDATE (07/01/21) - As of ESXi 7.0 Update 2, cloning an ESXi boot volume (Nested or Physical) is no longer safe and can lead to data corruption. Please refer to the following two VMware KB articles for more information on this topic https://kb.vmware.com/kb/84280 and https://kb.vmware.com/kb/84349 

First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.

Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.

  • The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
  • The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.

First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:

sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf

To ensure the file is persisted, run /sbin/auto-backup.sh

Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.

If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:

esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]

w00t! VMware Tools for Nested ESXi!

by // 42 Comments

I have been working with Nested ESXi since it original inception and this technology has greatly benefited me and the entire VMware community, especially when it comes to learning about VMware software and being able to easily prototype something before installing it on actual hardware. However, one thing that I felt that has been missing for awhile now is the ability to run an instance of VMware Tools within a Nested ESXi VM. I have personally been asking for this feature for a couple of years and I know many in the VMware community have expressed interests as well.

I am super excited to announce that VMware has just released a new Fling that provides you with a VIB that you can install VMware Tools inside a Nested ESXi host. I originally showed a demo of this at VMworld Barcelona in my vBrownBag Tech Talk and as I mentioned we would be releasing this as a VMware Fling very soon. So here it is!

UPDATE (08/20/15) - An updated version of VMware Tools for Nested ESXi was just published, make sure to download latest version and you can find more details here.

Requirements:

  • Nested ESXi running 5.0, 5.1 or 5.5 

Installation:

To install the VIB, you simply just need to download it and upload the VIB it to your Nested ESXi datastore and then run the following commands:

esxcli system maintenanceMode set -e true
esxcli software vib install -v /vmfs/volumes/[VMFS-VOLUME-NAME]/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f
esxcli system shutdown reboot -r "Installed VMware Tools"

You can also install the VIB directly from VMware.com if you have direct or proxy internet connectivity from your ESXi host by running the following commands:

esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi_tools_for_guests/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f

Once the VIB has been successfully installed, you will need to reboot the host for the changes to take effect. To verify, you can now login to either your vSphere Web/C# Client and you should now see the status for VMware Tools for your Nested ESXi host showing green and the IP Address of the Nested ESXi host should be displayed.

So why would you want to do this? Well, there’s a couple of reasons actually. The first one is pretty basic, which is when I need to reboot or shutdown a Nested ESXi VM, instead of having to jump into the VM console or SSH into ESXi host, I could just right click in the vSphere Web/C# Client and just say shutdown or reboot. I also tend to do all sorts of craziness in my lab (I’m sure this is an understatement for folks that know me) and may often break networking connectivity to my Nested ESXi VM. In vSphere 5.0, we introduced the Guest Operations API (formally known as VIX API) which is now part of the vSphere API. This API is actually quite handy as it allows you to perform guest operations within the VM without needing network connectivity as it relies on the fact that VMware Tools is running (pretty cool stuff!).

Here is a screenshot demonstrating the executing of vmkfstools through the Guest Operations API to one of my Nested ESXi VM:

A couple of things to note:

  • If you install VMware Tools on Nested ESXi VM, you will NOT be able to just right click in the UI and say install/upgrade
  • If you wish to integrate this into you ESXi image, you can take a look at a community tool  called ESXi-Customizer created by Andreas Peetz which I have used in the past and works great. Image Builder does not support raw VIBs, only zip files which may need to contain additional metadata information. If you want to create an offline bundle instead to then use Image Builder to create your custom ISO, Andreas has a new tool you can take a look at here.

Finally, if you have any feedback (likes/dis-likes), thanks, comments please head over to the VMware's Fling page for VMware Tools for Nested ESXi and leave a comment. I am sure the Jim Mattson the engineer who built this Fling would greatly appreciate any feedback you may have.