Search Results for: NUC

Home Labs made easier with VSAN 6.0 + USB Disks

03.04.2015 by William Lam // 23 Comments

VSAN 6.0 includes a large number of new enhancements and capabilities that I am sure many of you are excited to try out in your lab. One of the challenges with running VSAN in a home lab environment (non-Nested ESXi) is trying to find a platform that is both functional and cost effective. Some of the most popular platforms that I have seen customers use for running VSAN in their home labs are the Intel NUC and the Apple Mac Mini. Putting aside the memory constraints in these platforms, the number of internal disk slots for a disk drive is usually limited to two. This would give you just enough to meet the minimal requirement for VSAN by having at least a single SSD and MD.

If you wanted to scale up and add additional drives for either capacity purposes or testing out a new configurations, you are pretty much out of luck, right? Well, not necessary. During the development of VSAN 6.0, I came across a cool little nugget from one of the VSAN Engineers where USB-based disks could be claimed by VSAN which could be quite helpful for testing in a lab environment, especially using the hardware platforms that I mentioned earlier.

For a VSAN home lab, using cheap consumer USB-based disks which you can purchase several TB's for less than a hundred dollars or so and along with USB 3.0 connectivity is a pretty cost effective way to enhance hardware platforms like the Apple Mac Mini and Intel NUCs.

Disclaimer: This is not officially supported by VMware and should not be used in Production or evaluation of VSAN, especially when it comes to performance or expected behavior as this is now how the product works. Please use supported hardware found on the VMware VSAN HCL for official testing or evaluations.

Below are the instructions on how to enable USB-based disks to be claimable by VSAN.

Step 1 - Disable the USB Arbitrator service so that USB devices can been seen by the ESXi host by running the following two commands in the ESXi Shell:

/etc/init.d/usbarbitrator stop
chkconfig usbarbitrator off

Step 2 - Enable the following ESXi Advanced Setting (/VSAN/AllowUsbDisks) to allow USB disks to be claimed by VSAN by running the following command in the ESXi Shell:

esxcli system settings advanced set -o /VSAN/AllowUsbDisks -i 1

Step 3 - Connect your USB-based disks to your ESXi host (this can actually be done prior) and you can verify that they are seen by running the following command in the ESXi Shell:

vdq -q

Step 4 - If you are bootstrapping vCenter Server onto the VSAN Datastore, then you can create a VSAN Cluster by running "esxcli vsan cluster new" and then contribute the storage by adding the SSD device and the respective USB-based disks using the information from the previous step in the ESXi Shell:

esxcli vsan storage add -s t10.ATA_____Corsair_Force_GT________________________12136500000013420576 -d mpx.vmhba32:C0:T0:L0 -d mpx.vmhba33:C0:T0:L0 -d mpx.vmhba34:C0:T0:L0 -d mpx.vmhba40:C0:T0:L0

If we take a look a the VSAN configurations in the vSphere Web Client, we can see that we now have 4 USB-based disks contributing storage to the VSAN Disk Group. In this particular configuration, I was using my Mac Mini which has 4 x USB 3.0 devices that are connected and providing the "MD" disks and one of the internal drives that has an SSD. Ideally, you would probably want to boot ESXi from a USB device and then claim one of the internal drives along with 3 other USB devices for the most optimal configuration.

As a bonus, there is one other nugget that I discovered while testing out the USB-based disks for VSAN 6.0 which is another hidden option to support iSCSI based disks with VSAN. You will need to enable the option called /VSAN/AllowISCSIDisks using the same method as enabling USB-based disk option. This is not something I have personally tested, so YMMV but I suspect it will allow VSAN to claim an iSCSI device that has been connected to an ESXi host and allow it to contribute to a VSAN Disk Group as another way of providing additional capacity to VSAN with platforms that have restricted number of disk slots. Remember, neither of these solutions should be used beyond home labs and they are not officially supported by VMware, so do not bother trying to do anything fancy or running performance tests, you are just going to let your self down and not see the full potential of VSAN 🙂

VMware officially releases vibddi for vSphere 4.1

09.18.2011 by William Lam // Leave a Comment

There were several product releases last week that got a lot of buzz on the inter-tube:

However, VMware actually released an additional product last week which snuck under the radar, vibddi.

I actually wrote about this unsupported and undocumented utility last year: How to inject custom drivers into an ESXi 4.1 image using vibddi? vibddi (pronounced vib d-d-i) stands for VIB (vSphere Installation Bundle) Disk Dump Image and it is a utility to help users easily customize ESXi images with custom drivers. This utility first appeared in the vSphere Auto Deploy appliance and it looks like VMware has finally released it as an official tool to support vSphere 4.1 image customization. You also may have heard about the new Image Builder tool with the release of vSphere 5, the origins of that utility actually came from vibddi.

If you are still using vSphere 4.1 and need to inject or modify drivers, I would highly recommend you take a look at the tool as it is extremely simple to use. For more details, please check out the new VMware KB article 2003316 documenting the details of the utility or my blog post. If you are using vSphere 5, you will need to use Image Builder as vSphere 4.1 is not supported and vice-a-versa with ESXi 5 with vibddi.

Note: There are some changes in the latest vibddi utility compared to the one found in the vSphere Auto Deploy such as injecting custom kickstart configuration file or license file. If you rely on these features, you may want to use the older version or manually update these after the system build.

How to compile Busybox for ESXi ... kind of Part 2

02.15.2011 by William Lam // Leave a Comment

Continuing from part1 of How to compile Busybox for ESXi ... kind of Part 1 we found some challenges in compiling VMware's Busybox version found in ESXi OSS Source Code. In this article, we will go through process of compiling the latest version of Busybox which is currently 1.18.3 to run on ESXi.

Again, before getting started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!

The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.

Step 1

Download busybox-1.18.3.tar.bz2 from the Busybox's website and SCP it to your build system.

Step 2

You will also need to install the following packages, you can do so using yum if you are using CentOS or RHEL and have a proper repository configured. You can use the following:

yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64

Step 3

Extract the contents of busybox-1.18.3.tar.bz2 using the following command:

tar -xvjpf busybox-1.18.3.tar.bz2

Step 4

You will now change into the busybox-1.18.3 directory and from here you have a few ways of building Busybox. To get a list of build options, use the following command:
make help

You can customize the build of Busybox by enabling and disable specific applets to be compiled. If you want to build Busybox with no applets (not really useful), you can run the following command:

make allnoconfig
make

Once the compilation is complete, you will now have a Busybox binary in which you can run but it does nothing useful:

You can also perform the exact opposite by enabling all options which can be called using the "allnoconfig" or "defconfig", you can run the following command:

make defconfig
make

Once the compilation is complete, you now have Busybox binary which has all the available applets by default:

Enabling all applets, the Busybox binary still comes out to be less than 900k. If you wanted to create a custom Busybox with specific applets, you can manually edit the .config file which is where the features are either enabled or disabled. This can get very tedious, luckily there is a make option which allows for an interactive menu on selecting the applets you would like to include in the build.

Step 5

To enable the interactive menu, you will run the following command:

make menuconfig

From here, you will be able to configure the Busybox settings and the various applet types by their functions such as networking or mail utilities. One feature that is actually disabled in the VMware's version of Busybox is the support for large files, if you tried to tar up a file that was larger than 2GB and then extract the file, you will notice you get an error regarding the file size. The reason is VMware decided to not enable this feature in their Busybox build.

Step 6

Once you are done selecting or de-selecting all your Busybox settings and applets, you will need to save your configuration which is stored in the .config.

Step 7

Once you have exited from the interactive Busybox menu, you are now ready to build your custom version of Busybox, you will run the following command:

make

Step 8

If everything went well, you now should have a busybox binary in your current working directory. Here is an example of my custom Busybox image:

If you would like to customize the version information found when just running busybox command without any arguments, you can edit the Makefile and append your custom text to EXTRAVERSION variable.

Now you just need to re-run the "make" command and your new busybox binary will include the additional version information.

Now before you jump and start creating your own Busybox binary, I must throw in another caveat that re-iterates the title of this post. Just because you have Busybox built with all these applets, it does not mean it will run 100% on ESXi, the reason being is there are specific dependencies that the non-VMware Busybox applets may rely on which are just not available with ESXi. This is one of the reasons why VMware did not just enable all applets from Busybox to begin with, your mileage will vary depending on the particular command you have enabled.

Here is an example of my custom Busybox applet "crontab" using the "-l" flag which lists any active cron entries for the particular user:

It runs just fine on ESXi as you can see, but let's try using the "-e" flag which if you are familiar with using crontab you will know that is the option to edit your cron entries:

Ooops, what's this? It looks like this particular operation of crontab relies on vfork which is not implemented. I have seen other similar errors with other applets relying on particular dependencies in the environment that just does not exist with ESXi. I have not tested every single applet and some may work 100% while others may be partially functional, you will need to test and verify for yourself.

If you feel adventurous, go ahead and download Busybox and start playing with it. I would highly recommend that you test this in your vSphere development environment before trying this on a production host. I would also recommend that you do NOT replace the default Busybox binary found on ESX(i) as you can run into some serious issues. Since the Busybox binary is self-contained, you can store it in /tmp or somewhere more permanent such as a local or remote VMFS datastore and renaming the file will also remove any confusion.

If you find other utilities that you feel that should be included in the default Busybox applet and do not want to resort to something like Poor man's traceroute for ESXi, be sure to submit your feedback to either your VMware rep or file a feature request.

If you are interesting to learning more about Busybox, check out their FAQ page and this page for more details.

Again, another warning:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!