Search Results for: kickstart

Does VSAN work with Free ESXi?

07.22.2014 by William Lam // 8 Comments

I recently had to re-provision one of my VSAN lab environments using my recently shared ESXi 5.5 VSAN Kickstart. I usually specify a license key within the Kickstart so I do not have to license the ESXi host later. This actually got me wondering on whether VSAN would in fact work with Free ESXi aka vSphere Hypevisor? Being a curious person, I of course had to test this in the lab 🙂

Needless to say, if you want to properly evaluate or use VSAN in production, you should go through the supported method of using vCenter Server as it provides a simple and intuitive management interface for VSAN. More importantly, having the ability to create individual VM Storage Policies that can be applied on a per VMDK basis based on SLA's for your given application or Virtual Machine.

Disclaimer: This is not officially supported by VMware and running ESXi without a VSAN license is against VMware's EULA.

Since we do not have a vCenter Server, we will need to be able to fully configure VSAN without it. Luckily, we know of a way to "bootstraping" VSAN onto an ESXi host without vCenter Server and I will be leveraging that blog post to test this scenario with Free ESXi.

Prerequisite:

3 ESXi 5.5 hosts already installed and licensed with vSphere Hypervisor (Free ESXi) License
SSH Enabled

Step 1 - SSH to the first ESXi host and run the following ESXCLI command to create a VSAN Cluster:

esxcli vsan cluster join -u $(python -c 'import uuid; print str(uuid.uuid4());')

Step 2 - Run the following ESXCLI command to make a note of the VSAN Cluster UUID (highlighted in green in the screenshot above) which will be needed later:

esxcli vsan cluster get

Step 3 - Enable VSAN Traffic for VMkernel interface you plan on using for VSAN traffic by running the following ESXCLI command:

esxcli vsan network ipv4 add -i vmk0

Step 4 - Run the following command to view a list of disks that are eligible for use with VSAN. You will need a minimum of 1xSSD and 1xMD

vdq -q

Step 5 - Using the information from vdq, we will now create our VSAN Disk Group which will contain the SSD/MD's to be used for VSAN. Use the following ESXCLI command and substituting in the SSD/MD Names (please refer to the screenshot above for an example):

esxcli vsan storage add -s [SSD] -d [MD]

Step 6 - To ensure you have properly configured a VSAN Disk Group, you can run the following ESXCLI command to confirm:

esxcli vsan storage list

At this point, we now have a single ESXi host configured with VSAN Datastore, we can also confirm this by running the following ESXCLI command:

esxcli storage filesystem list

Step 7 - Repeat Steps 3-6 on the remainder two ESXi hosts

Step 8 - Finally, we now need to join the remainder ESXi hosts to the VSAN Cluster. We will need the VSAN Cluster UUID that we recorded earlier and specify that in the following ESXCLI command on each of the remainder ESXi hosts:

esxcli vsan cluster join -u [VSAN-CLUSTER-UUID]

If we now login to all of our ESXi hosts using the vSphere C# Client, we will see a common VSAN Datastore that is shared among the three ESXi hosts. To prove that that VSAN is in fact working, we can create a Virtual Machine and ensure we can power it on as seen in the screenshot below. By default, VSAN has a "Default" policy which defines FTT (Number of host failures to tolerate) set to 1 and assuming you have at least 3 ESXi hosts, all Virtual Machines will be protected by default.

Even though you can run VSAN using Free ESXi and leveraging the default VM Storage Policy that is built into VSAN for protecting Virtual Machines, you are only exercising a tiny portion of the potential that VSAN can bring when consuming it through vCenter Server. As mentioned earlier, you will not have the ability to create specific VM Storage Policies and assign them based on the specific SLAs and be able to easily monitor their compliance and remediation. The management of VSAN Cluster for adding additional capacity or serviceability is also quite limited without vCenter Server, though it can be definitely be done it is much easier with just a couple of clicks in the vSphere Web Client or a simple API call.

How to properly clone a Nested ESXi VM?

12.06.2013 by William Lam // 54 Comments

I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.

UPDATE (07/01/21) - As of ESXi 7.0 Update 2, cloning an ESXi boot volume (Nested or Physical) is no longer safe and can lead to data corruption. Please refer to the following two VMware KB articles for more information on this topic https://kb.vmware.com/kb/84280 and https://kb.vmware.com/kb/84349

First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.

Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.

The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.

First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:

sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf

To ensure the file is persisted, run /sbin/auto-backup.sh

Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.

If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:

esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]

How to automate vFRC configurations using the command-line in ESXi

11.20.2013 by William Lam // 1 Comment

While working on my vSphere Flash Read Cache (vFRC) articles last week, I wanted to be able to quickly build out my vSphere environment so that vFRC was fully configured as part of my ESXi installation using a Kickstart script. This would allow me to simply add my ESXi hosts into vCenter Server and not have to go through the vSphere Web Client for each host configuring vFRC. Now of course the vSphere Web Client is not the only option to configure vFRC, you can also use the vSphere APIs by creating your own script or even using the new vFRC PowerCLI cmdlets as an alternative.

However, I was interested in creating a very simple script that I could easily integrate with my kickstart deployment as that is what I am using for automated provisioning of my Nested ESXi hosts. With a bit of research and some trial/error, I have come up with a process that can be fully automated from the command-line of ESXi. In my environment I have a Nested ESXi host that contains three SSD's (4GB each) which will be used to construct my Virtual Flash Resource.

Note: Jump to the very bottom for a completely automated script to configure vFRC for your ESXi host.

Step 1 -You will want to list out the available SSD devices on your ESXi host, you can do so by using the following ESXCLI command:

esxcli storage vflash device list

You will need to make a note of the device ID's as they will be required in the sub-sequent steps.

Step 2 - Next we will need to partition our devices before we can create VFFS (Virtual Flash File System) and we will need to calculate the end sector if we wish to consume the entire device. To do so, we will need to use the partedUtil command and specify the "getptbl" option to identify some information.

partedUtil getptbl /vmfs/devices/disks/naa.6000c2932c4ed8a540b6e9f0be9e1009

You will need to make a note of the first three numbers which represents number of cylinders, number of heads and number of sectors per track. To calculate the end sectors, the equation will be the following: (Number of Cylinders x Number of Heads x Number of Sectors Per Track) - 1

In our example we have (522*255*63)-1 which gives us 8385929

To create the partition, we will again use the partedUtil and specify "setptbl" option by running the following command (ensure to replace your end sector value):

partedUtil setptbl /vmfs/devices/disks/naa.6000c2932c4ed8a540b6e9f0be9e1009 "gpt" "1 2048 8385929 AA31E02A400F11DB9590000C2911D1B8 0"

For more details on using the partedUtil command, please refer here and here.

Since my other two devices are exactly the same size, I can just re-use the command and replace the device path. Ensure all devices that you wish to use in your Virtual Flash Resource is partition before moving onto the next step.

Step 3 - We will now create our VFFS volume which only needs to be created on one of the devices. In this example, I have chosen to use the first SSD device as shown in "esxcli storage vflash device list". To create the VFFS volume we will use the vmkfstools tool just like we would if we were creating a VMF volume but instead use the "vmfsl" type.

Run the following command to create your VFFS volume, you will need to append :1 to the end of the SSD device to specify the partition you created earlier as well as a display name of the volume which I chose vffs-$(hostname -s) which will use the short hostname of the ESXi host

vmkfstools -C vmfsl /vmfs/devices/disks/naa.6000c2932c4ed8a540b6e9f0be9e1009:1 -S vffs-$(hostname -s)

Step 4 - Once you have your VFFS volume created, you can extend it with additional SSD devices by using vmkfstools and specifying the -Z option. The syntax for the command is the SSD device partition you wish to add followed by the source SSD device containing the VFFS volume.

Here is an example of the command:

vmkfstools -Z /vmfs/devices/disks/naa.6000c29498be5c56231d631d9c6cbee8:1 /vmfs/devices/disks/naa.6000c2932c4ed8a540b6e9f0be9e1009:1

You will be prompted on whether you want to extend and to confirm enter value of 0.

You will need to do this for all SSD devices you partition earlier to be part of the same VFFS volume.

Step 5 - To confirm that everything was configured correctly, we will use vmkfstools to query our VFFS volume by running the following command and specifying the path to our VFFS volume:

vmkfstools -Ph /vmfs/volumes/vffs-vesxi55-10

From the output we should see the filesystem for the volume is of type VFFS and we should also see the three SSD devices that is backing this VFFS volume as shown in screenshot above.

Step 6 - Finally to make this new VFFS volume visible to the ESXi host, we will need to refresh the ESXi storage system and we can do so by running the following vim-cmd:

vim-cmd hostsvc/storage/refresh

At this point, we now have a fully configured VFFS volume. If you jump right into the vSphere Web Client expecting to see your new Virtual Flash Resource on your newly configured ESXi host, you might be in for a surprise! You will actually NOT see the VFFS volume that we just configured which stumped me initially.

It turns out simply creating a VFFS volume does not automatically equate to configuring a Virtual Flash Resource. You still need to configure the ESXi host to add the Virtual Flash Resource based on your VFFS volume and in my opinion that seems to be quite odd and counter-intuitive. Today there is no CLI command to add the Virtual Flash Resource, you would need to use either the vSphere Web Client or use the vFRC vSphere API. If you login to the vSphere Web Client and configure a Virtual Flash Resource, you will see the VFFS volume that we have created and you just need to select it and it will automatically add it.

This is not very ideal if you want to completely automate vFRC configurations and I decided to leverage my knowledge of the vFRC vSphere APIs and create a very simple python script that would call into the ESXi host's MOB and issue the HostConfigureVFlashResource() method. This was sort of a quick/dirty way to call the vSphere API and add in the Virtual Flash Resource.

Disclaimer: These scripts are provided as examples, please test these scripts in your development/test environment before running them in production.

To make this really useful I have created two scripts that can be embedded into either a kickstart script or executed manually. The script will automatically perform the above operations above as well as configure the Virtual Flash Resource without any user input/intervention.

The main script is called configurevFRC.sh which is a shell script that performs the majority of the work and it then it calls the python script which is called addVirtualFlashResource.py (ensure you change the password variable in the script) for adding the Virtual Flash Resource. You need to download both scripts and run them on the ESXi Shell.

Here is the contents of configurevFRC.sh (you can download both scripts using the links above):
Here is a sample execution of configurevFRC.sh script:

In the future I hope we can completely automate vFRC configurations from the command-line as we can using the vSphere Web Client or vSphere APIs. For now, this solution will help get you around the limitations we have in the command-line utilities.

HostConfigureVFlashResource