ESXCLI – Page 4

Home Labs made easier with VSAN 6.0 + USB Disks

03.04.2015 by William Lam // 23 Comments

VSAN 6.0 includes a large number of new enhancements and capabilities that I am sure many of you are excited to try out in your lab. One of the challenges with running VSAN in a home lab environment (non-Nested ESXi) is trying to find a platform that is both functional and cost effective. Some of the most popular platforms that I have seen customers use for running VSAN in their home labs are the Intel NUC and the Apple Mac Mini. Putting aside the memory constraints in these platforms, the number of internal disk slots for a disk drive is usually limited to two. This would give you just enough to meet the minimal requirement for VSAN by having at least a single SSD and MD.

If you wanted to scale up and add additional drives for either capacity purposes or testing out a new configurations, you are pretty much out of luck, right? Well, not necessary. During the development of VSAN 6.0, I came across a cool little nugget from one of the VSAN Engineers where USB-based disks could be claimed by VSAN which could be quite helpful for testing in a lab environment, especially using the hardware platforms that I mentioned earlier.

For a VSAN home lab, using cheap consumer USB-based disks which you can purchase several TB's for less than a hundred dollars or so and along with USB 3.0 connectivity is a pretty cost effective way to enhance hardware platforms like the Apple Mac Mini and Intel NUCs.

Disclaimer: This is not officially supported by VMware and should not be used in Production or evaluation of VSAN, especially when it comes to performance or expected behavior as this is now how the product works. Please use supported hardware found on the VMware VSAN HCL for official testing or evaluations.

Below are the instructions on how to enable USB-based disks to be claimable by VSAN.

Step 1 - Disable the USB Arbitrator service so that USB devices can been seen by the ESXi host by running the following two commands in the ESXi Shell:

/etc/init.d/usbarbitrator stop
chkconfig usbarbitrator off

Step 2 - Enable the following ESXi Advanced Setting (/VSAN/AllowUsbDisks) to allow USB disks to be claimed by VSAN by running the following command in the ESXi Shell:

esxcli system settings advanced set -o /VSAN/AllowUsbDisks -i 1

Step 3 - Connect your USB-based disks to your ESXi host (this can actually be done prior) and you can verify that they are seen by running the following command in the ESXi Shell:

vdq -q

Step 4 - If you are bootstrapping vCenter Server onto the VSAN Datastore, then you can create a VSAN Cluster by running "esxcli vsan cluster new" and then contribute the storage by adding the SSD device and the respective USB-based disks using the information from the previous step in the ESXi Shell:

esxcli vsan storage add -s t10.ATA_____Corsair_Force_GT________________________12136500000013420576 -d mpx.vmhba32:C0:T0:L0 -d mpx.vmhba33:C0:T0:L0 -d mpx.vmhba34:C0:T0:L0 -d mpx.vmhba40:C0:T0:L0

If we take a look a the VSAN configurations in the vSphere Web Client, we can see that we now have 4 USB-based disks contributing storage to the VSAN Disk Group. In this particular configuration, I was using my Mac Mini which has 4 x USB 3.0 devices that are connected and providing the "MD" disks and one of the internal drives that has an SSD. Ideally, you would probably want to boot ESXi from a USB device and then claim one of the internal drives along with 3 other USB devices for the most optimal configuration.

As a bonus, there is one other nugget that I discovered while testing out the USB-based disks for VSAN 6.0 which is another hidden option to support iSCSI based disks with VSAN. You will need to enable the option called /VSAN/AllowISCSIDisks using the same method as enabling USB-based disk option. This is not something I have personally tested, so YMMV but I suspect it will allow VSAN to claim an iSCSI device that has been connected to an ESXi host and allow it to contribute to a VSAN Disk Group as another way of providing additional capacity to VSAN with platforms that have restricted number of disk slots. Remember, neither of these solutions should be used beyond home labs and they are not officially supported by VMware, so do not bother trying to do anything fancy or running performance tests, you are just going to let your self down and not see the full potential of VSAN 🙂

Want to issue a VAAI UNMAP operation using the vSphere Web Client?

09.18.2014 by William Lam // 3 Comments

Recently, I have seen several requests from both customers and partners wanting to be able to run the VAAI UNMAP operation from within the vSphere Web Client. For those of you not familiar with the VAAI UNMAP operation, I recommend you check out this blog post by my colleague Cormac Hogan. Today, the only way to issue the UNMAP operation is by using ESXCLI either remotely or in the ESXi Shell. There is currently not a vSphere API for this operation and therefore it would be difficult to build a native vSphere Web Client Plugin to provide this functionality.

Having said that, one way to provide this capability is through the use of a vCenter Orchestrator (vCO) Workflow which can remotely execute an ESXCLI command whether that is going through ESXCLI using a Linux jump box or through PowerCLI using a Windows jump box. Starting with vSphere 5.5, you can now extend the vSphere Web Client and attach a vCO Workflow to a vSphere Object and be able to execute the workflow right from the vSphere Web Client. This is great if you are already using vCO, but for those that are not, it can be somewhat complex to setup along with a steep learning curve depending on your experience.

Today, there was an exciting announcement from my Automation buddy, Alan Renouf for a new VMware Fling called PowerActions for the vSphere Web Client. This new Fling allows you to easily extend the vSphere Web Client in the following ways:

Access a PowerCLI console directly in the vSphere Web Client
Ability to run a context aware PowerCLI script directly from the vSphere Web Client

The prerequisite for setting up PowerActions is no different than vCO calling a PowerCLI script, you just need a Windows "jump-box" that has PowerCLI installed along with PowerActions. The added benefit, is that you do need to setup another piece of infrastructure like vCO if you are not already using it. This made setting up PowerActions extremely easy to setup and even I was able get it up and running in under 5minutes (minus a quick RTFM moment :)).

Given the number of inquiries regarding VAAI UNMAP operation via the vSphere Web Client, I thought that would be a great use case for my first PowerActions script! Below are the instructions on creating the VAAI UNMAP script for PowerActions:

Step 1 - Click on the PowerCLI Scripts option on the left hand side of the Object Navigator and then click on the "New Script" Icon. Select Datastore as the context aware object for the script.

Step 2 - Provide a name and description for the script. Also make sure to select "Action".

Step 3 - Copy and paste the following script from https://github.com/lamw/vghetto-scripts/blob/master/powershell/unmap-poweraction.ps1 inside the script window and then save the script. What the script does is takes the Datastore object and retrieves a list of ESXi hosts that has access to the Datastore and then randomly selects one of the host. This is required because ESXCLI operations on a per host level and we use that information to pass into Get-EsxCli cmdlet to issue the VAAI UNMAP operation.

Step 4 - To test the script, you just need to right click on a VMFS Datastore and click on PowerCLI->Execute a Script

Note: Please be aware of the impact when running a UNMAP operation, you may want to run this on a non-production datastore for testing purposes or during off hours when your workload may not be as busy.

Step 5 - Select the VAAI UNMAP script you just created and once selected and you will be prompted to specify the number of VMFS blocks to unmap per iteration which is exactly the same input when manually ESXCLI.

At this point, if everything was successful the VAAI UNMAP operation should begin and you can tail /var/log/hostd.log to see the UNAMP operation. Once completed, you should see the prompt return true.

As you can see, it was extremely easy to create my own PowerAction script that expose new functionality and making it available within the vSphere Web Client. I think this is going to be a pretty popular Fling and remember if this is something you would like to see officially in the product, be sure to leave a comment on the PowerAction for vSphere Web Client Fling page, product managers are listening! The only feedback I have is that I would love to see this get extended beyond just PowerCLI and into a generic script extension framework, just imagine the possibilities!

Does VSAN work with Free ESXi?

07.22.2014 by William Lam // 8 Comments

I recently had to re-provision one of my VSAN lab environments using my recently shared ESXi 5.5 VSAN Kickstart. I usually specify a license key within the Kickstart so I do not have to license the ESXi host later. This actually got me wondering on whether VSAN would in fact work with Free ESXi aka vSphere Hypevisor? Being a curious person, I of course had to test this in the lab 🙂

Needless to say, if you want to properly evaluate or use VSAN in production, you should go through the supported method of using vCenter Server as it provides a simple and intuitive management interface for VSAN. More importantly, having the ability to create individual VM Storage Policies that can be applied on a per VMDK basis based on SLA's for your given application or Virtual Machine.

Disclaimer: This is not officially supported by VMware and running ESXi without a VSAN license is against VMware's EULA.

Since we do not have a vCenter Server, we will need to be able to fully configure VSAN without it. Luckily, we know of a way to "bootstraping" VSAN onto an ESXi host without vCenter Server and I will be leveraging that blog post to test this scenario with Free ESXi.

Prerequisite:

3 ESXi 5.5 hosts already installed and licensed with vSphere Hypervisor (Free ESXi) License
SSH Enabled

Step 1 - SSH to the first ESXi host and run the following ESXCLI command to create a VSAN Cluster:

esxcli vsan cluster join -u $(python -c 'import uuid; print str(uuid.uuid4());')

Step 2 - Run the following ESXCLI command to make a note of the VSAN Cluster UUID (highlighted in green in the screenshot above) which will be needed later:

esxcli vsan cluster get

Step 3 - Enable VSAN Traffic for VMkernel interface you plan on using for VSAN traffic by running the following ESXCLI command:

esxcli vsan network ipv4 add -i vmk0

Step 4 - Run the following command to view a list of disks that are eligible for use with VSAN. You will need a minimum of 1xSSD and 1xMD

vdq -q

Step 5 - Using the information from vdq, we will now create our VSAN Disk Group which will contain the SSD/MD's to be used for VSAN. Use the following ESXCLI command and substituting in the SSD/MD Names (please refer to the screenshot above for an example):

esxcli vsan storage add -s [SSD] -d [MD]

Step 6 - To ensure you have properly configured a VSAN Disk Group, you can run the following ESXCLI command to confirm:

esxcli vsan storage list

At this point, we now have a single ESXi host configured with VSAN Datastore, we can also confirm this by running the following ESXCLI command:

esxcli storage filesystem list

Step 7 - Repeat Steps 3-6 on the remainder two ESXi hosts

Step 8 - Finally, we now need to join the remainder ESXi hosts to the VSAN Cluster. We will need the VSAN Cluster UUID that we recorded earlier and specify that in the following ESXCLI command on each of the remainder ESXi hosts:

esxcli vsan cluster join -u [VSAN-CLUSTER-UUID]

If we now login to all of our ESXi hosts using the vSphere C# Client, we will see a common VSAN Datastore that is shared among the three ESXi hosts. To prove that that VSAN is in fact working, we can create a Virtual Machine and ensure we can power it on as seen in the screenshot below. By default, VSAN has a "Default" policy which defines FTT (Number of host failures to tolerate) set to 1 and assuming you have at least 3 ESXi hosts, all Virtual Machines will be protected by default.

Even though you can run VSAN using Free ESXi and leveraging the default VM Storage Policy that is built into VSAN for protecting Virtual Machines, you are only exercising a tiny portion of the potential that VSAN can bring when consuming it through vCenter Server. As mentioned earlier, you will not have the ability to create specific VM Storage Policies and assign them based on the specific SLAs and be able to easily monitor their compliance and remediation. The management of VSAN Cluster for adding additional capacity or serviceability is also quite limited without vCenter Server, though it can be definitely be done it is much easier with just a couple of clicks in the vSphere Web Client or a simple API call.