WilliamLam.com

How to Format and Create VMFS5 Volume using the CLI in ESXi 5

by // 39 Comments

VMware always recommends formatting and creating a new VMFS volume using the vSphere Client as it automatically aligns your VMFS volume. However, if you do not have access to the vSphere Client or you wanted to format additional VMFS volumes via a kickstart, you can do so using the CLI and the partedUtil under /sbin.

~ # /sbin/partedUtil
Not enough arguments
Usage:
Get Partitions : get
Set Partitions : set ["partNum startSector endSector type attr"]*
Delete Partition : delete Resize Partition : resize
Get Partitions : getptbl
Set Partitions : setptbl

With ESXi 5, an MBR (Master Boot Record) partition table is no longer used and has been replaced with a GPT (GUID Partition Table) partition table. There is also only one block size of 1MB versus the 2,4 and 8 that was available in ESX(i) 4.x

We can view the partitions of a device by using the "getptbl" option and ensure we don't have an existing VMFS volume:

~ # /sbin/partedUtil "getptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0"
gpt
652 255 63 10485760

Next we will need to create a partition by using the "setptbl" option:

/sbin/partedUtil "setptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0" "gpt" "1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0"

The "setptbl" accepts 3 arguments:

  • diskName
  • label
  • partitionNumber startSector endSector type/GUID attribute

The diskName in this example is the full path to the device which is /vmfs/devices/disks/mpx.vmhba1:C0:T2:L0

The label will be gpt

The last argument is actually a string comprised of 5 individual parameters:

  • partitionNumber - Pretty straight forward
  • startSector - This will always be 2048 for 1MB alignment for VMFS5
  • endSector - This will need to be calculated based on size of your device
  • type/GUID - This is the GUID key for a particular partition type, for VMFS it will always be AA31E02A400F11DB9590000C2911D1B8

To view all GUID types, you can use the "showGuids" option:

~ # /sbin/partedUtil showGuids
Partition Type       GUID
vmfs                 AA31E02A400F11DB9590000C2911D1B8
vmkDiagnostic        9D27538040AD11DBBF97000C2911D1B8
VMware Reserved      9198EFFC31C011DB8F78000C2911D1B8
Basic Data           EBD0A0A2B9E5443387C068B6B72699C7
Linux Swap           0657FD6DA4AB43C484E50933C84B4F4F
Linux Lvm            E6D6D379F50744C2A23C238F2A3DF928
Linux Raid           A19D880F05FC4D3BA006743F0F84911E
Efi System           C12A7328F81F11D2BA4B00A0C93EC93B
Microsoft Reserved   E3C9E3160B5C4DB8817DF92DF00215AE
Unused Entry         00000000000000000000000000000000

Once you have the 3 arguments specified, we can now create the partition:

~ # /sbin/partedUtil "setptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0" "gpt" "1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0"
gpt
0 0 0 0
1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0

UPDATE (01/15) - Here is a quick shell snippet that you can use to automatically calculate End Sector as well as creating the VMFS5 volume:

partedUtil mklabel ${DEVICE} msdos
END_SECTOR=$(eval expr $(partedUtil getptbl ${DEVICE} | tail -1 | awk '{print $1 " \\* " $2 " \\* " $3}') - 1)
/sbin/partedUtil "setptbl" "${DEVICE}" "gpt" "1 2048 ${END_SECTOR} AA31E02A400F11DB9590000C2911D1B8 0"
/sbin/vmkfstools -C vmfs5 -b 1m -S $(hostname -s)-local-datastore ${DEVICE}:1

Note: You can also use the above to create a VMFS-based datastore running on a USB device, however that is not officially supported by VMware and performance with USB-based device will vary depending on the hardware and the speed of the USB connection. 

We can verify by running the "getptbl" option on the device that we formatted:

~ # /sbin/partedUtil "getptbl" "/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0"
gpt
652 255 63 10485760
1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Finally we will now create the VMFS volume using our favorite vmkfstools, the syntax is the same as previous release of ESX(i):

~ # /sbin/vmkfstools -C vmfs5 -b 1m -S himalaya-SSD-storage-3 /vmfs/devices/disks/mpx.vmhba1:C0:T2:L0:1
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs5 file system on "mpx.vmhba1:C0:T2:L0:1" with blockSize 1048576 and volume label "himalaya-SSD-storage-3".
Successfully created new volume: 4dfdb7b0-8c0dcdb5-e574-0050568f0111

Now you can refresh the vSphere Client or run vim-cmd hostsvc/datastore/refresh to view the new datastore that was created.

How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5

by // 66 Comments

With the release of vSphere 5, one of the most sought out feature from VMware is the ability to run nested 64bit and Hyper-V guest virtual machines in a virtual ESXi instance. Previous to this, only 32bit virtual machines were supported as the VT-x/AMD-V Hardware Virtualization CPU instructions could not be virtualized and presented to the virtual ESX(i) guest. This feature is quite useful for home and lab setups in testing new features or studying for VMware certifications and running multiple vESX(i) instances.

You will still be required to have a 64bit capable system and CPU and you will need to be running ESXi 5.0, this will not work for ESX(i) 4.x or older.

The above diagram depicts the various levels of inception where pESXi is your physical ESXi 5.0 hosts. We then create a vESXi 5.0 host which will contain the necessary Hardware Virtualization CPU instructions to support a 64bit nested guest OS which I've created as another ESXi host called vvESXi.
Note: You will not be able to run a 4th level nested 64bit VM (I have tried by further passing the HV instructions in the nested guest) and it will just boot up and spin your CPUs for hours.
This feature by default is disabled in ESXi 5.0, to enable this virtualized HV (Hardware Virtualization) you will need to add the following string vhv.allow = "TRUE"  to /etc/vmware/config of your Physical ESXi 5.0 host
Once the configuration change has been made, the feature goes into effect right away. A reboot of the system is not necessary. To verify, you should now be able to power on a 64bit guest OS and see that the HV instructions bits are being passed into the guestOS which will then allow you to run a nested 64bit guestOS. You can also verify by looking in the vmware.log file of the virtual machine and grep for the string "monitorControl.vhv" and if you see the following message, then Virtualized HV is not enabled.
In the past to run a virtual ESX(i) instance, a few advanced .vmx configuration entries were needed as documented here. With ESXi 5.0, if you are using virtual hardware version 8, then you do not need to make any additional changes. If you are using hardware version 4 or 7, then you will need to add a few changes to the VM's configuration file.
Creating vESXi 5.0 Instance using Hardware Version 8:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client
2. Once the VM has been created, edit the settings of the VM and change over to the "Options" and now have the ability to select a new guestOS type: VMware ESXi 5.x or VMware ESXi 4.x under the "Other" section.

Note: I'm not sure why these two additional guestOS type is not available from the default creation menu, but are available after the initial VM shell is created.

3. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance as denoted from the picture below

Creating vESXi 5.0 Instance using Hardware Version 4/7:
1. To create a virtual ESXi 5.0 instance, start off by just creating a standard RHEL5/6 64bit VM using the vSphere Client

2. Now you will need to add the following advanced .vmx parameter:  monitor.virtual_exec = "hardware" which can be done through the vSphere Client and/or editing the .vmx parameter manually.

3. Next you will need to add some cpuid bits, depending if you are running an Intel or AMD CPU, the respective entries are required:

Intel Hosts:

cpuid.1.ecx = "----:----:----:----:----:----:--h-:----"

AMD Hosts:

cpuid.80000001.ecx.amd = "----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd = "hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"

4. You are now ready to install ESXi 5.0 in this new vESXi host and then you can create and power on nested 64bit guestOS within that vESXi instance

By using a VM that is hardware version 8, you can easily automate the creation of vESXi 5.0 instance by changing the guestOS string in the .vmx parameter to "vmkernel" and the above configurations other than "vhv" string needed for either an Intel or AMD system are automatically configured.

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.  

Creating a vHyper-V  Instance on physical ESXi 5.0:
1. To create a virtual Hyper-V instance, start off by creating a Windows 2008 Server R2 64bit VM using the vSphere Client

2. If you are using Hardware Version 7, you will need to follow the instructions in "Creating vESXi 5.0 Instance using Hardware Version 4/7" to add the additional parameters to the VM. If you are using Hardware Version 8, you just need to change the guestOS type to VMware ESXi 5.0

3. You need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE

4. You are now ready to install Hyper-V in a virtual machine and you can also spin up nested 64bit guestOSes in this virtual Hyper-V instance.

As you can see, now you can even run Hyper-Crap, err I mean Hyper-V as a virtualized guest under ESXi 5.0. I did not get a chance to try out Xen, but I'm sure with the ability to virtualize the Hardware Virtualization instructions, you should be able to run other types of hypervisors for testing purposes.

This is a really awesome feature but note that this is not officially supported by VMware, use at your own risk.

How to Disable a vmnic in ESX(i)

by // 8 Comments

There was another interesting thread today on the VMTN community forums about disabling an unused vmnic from an ESXi host due to false alarms being generated from HP SIM. Due to the specific hardware version, disabling the vmnic through the BIOS was not an option and there were no alternatives from the discussion.

I decided to dig a bit to see what I could find and stumbled upon a neat little utility called vmkchdev (VMkernel Change Device?) and from what I can tell provides a method passing a particular device to be controlled by either the VMkernel or as a passthrough device to a virtual machine (think VMDirect Path).

Disclaimer: Please note, this is using an undocumented utility. You should test this out in a development/lab environment before using and you may want to also contact VMware support to get their blessings***

~ # vmkchdev
Usage:
-s (scan device)
-v (give device to vmkernel)
-p (give device to passthru/VM)
-l (list device state)
-L (list device state with details)
[0x][seg:[bus[:slot[.func]]]]

What I found while testing the utility is by passing it over as a passthrough device, the vmnic is actually unrepresented to the VMkernel and does not show up under network adapters or even the unused/unlinked adapter list in the vSwitch configurations. It seems that it is just masks the device away from the VMkernel as you can still see the active configuration in esx.conf and you can see the device listed using vmkchdev.

Here is an example of an unused physical nic vmnic1 that we would like to disable and unpresent to an ESXi host.

Here is the output from esxcfg-nics:

First we need to identify the vmnic's PCI slot, we do by running the "-l" or list operation and searching for the particular vmnice device.

~ # vmkchdev -l | grep vmnic1
000:002:01.0 8086:100f 15ad:0750 vmkernel vmnic1

Next we will pass the device from the VMkernel to passthrough/VM using the "-p" flag and specifying the PCI slot, which in this case it is 000:002:01.0. We will also need to refresh the network section so the changes are reflected in the vSphere Client by using vim-cmd.

~ # vmkchdev -p 000:002:01.0
~ # vim-cmd hostsvc/net/refresh

If we list our vmnic again using vmkchdev, you will notice the device is now owned by passthru versus the VMkernel.

~ # vmkchdev -l | grep vmnic1
000:002:01.0 8086:100f 15ad:0750 passthru vmnic1

Now if we check the output of esxcfg-nics and the vSphere Client, you will notice that vmnic1 is no where to be found

If you would like to enable or re-present the disabled vmnic, you just need to pass the device back over to VMkernel by using the "-v" flag.

~ # vmkchdev -v 000:002:01.0
~ # vim-cmd hostsvc/net/refresh

Your vmnic should now re-appear on all your screens and any existing NIC teams that may have exists is automatically restored. This trick actually works on both a used and unused vmnic

If you are trying to do this on ESX, vmkchdev actually has an additional option called "console" for the Service Console.

[root@esx41-1 ~]# vmkchdev
Usage:
-s (scan device)
-c (give device to console)
-v (give device to vmkernel)
-p (give device to passthru/VM)
-l (list device state)
-L (list device state with details)
[0x][seg:[bus[:slot[.func]]]]

I found that you need to pass the vmnic from VMkernel to Console, passing it to passthru/VM will not work and an error is thrown if you do. Again, you can easily re-enable by passing it back to the VMkernel

vmkchdev -c 000:002:01.0

If you would like to automatically persist this change across reboots, specifically for ESXi as changes are not saved. You will need to add the following lines to /etc/rc.local which will execute the disabling of the vmnic's after bootup.

/sbin/vmkchdev -p 000:002:01.0
vim-cmd hostsvc/net/refresh

You will also need to run /sbin/auto-backup.sh to ensure the changes to /etc/rc.local are saved and reloaded upon the next reboot. For ESX, you can place it in /etc/rc.local without having to do anything extra as the changes persists across reboots for classic ESX

Now you can play hide and seek with your vmnic's without resorting to a system reboot or touching the BIOS. Though ideally, if you do have unused devices, you should definitely disable them in the BIOS if you have the option. On a side note, this might be a fun trick to play on one of your co-workers by hiding all vmnics 😉