vSphere – Page 16

Community stories of VMware & Apple OS X in Production: Part 9

10.09.2014 by William Lam // Leave a Comment

Company: Connell Insurance
Software: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Bryan, thanks for your time this afternoon. I know you have been pretty busy these last couple of weeks and I am glad we got some time with you to chat about your environment. Before we get started, can you please introduce yourself and what you are currently responsible for?

[Bryan] - Hi William. Thanks for the interest! My name is Bryan Linton. I'm currently the IT Director for Connell Insurance, an independent Insurance Agency in Southwest Missouri. We’ve been around for a long time but have been growing more rapidly in recent years. We currently have around 40 employees.

We have most of our systems in a secure off-site datacenter, but I still need some support systems onsite, and that's where your project bringing together inexpensive Mac Minis with ESXi caught my attention.

[William] - Can you tell us a bit more about your Mac Mini infrastructure? What is the hardware and software configuration and the type of workload you are currently running on them?

[Bryan] – First of all, I didn't buy the "server" versions of the Mac Mini - I just ordered the standard Mac Minis with stock RAM and storage. My only exception to going stock was on the CPU. For that, I went all-out and ordered the i7 quad-core processors since I knew I’d be using them as servers. But I bought a large SSD, a data doubler kit that allowed me to mount the second hdd, and 16GB of RAM, and installed all those myself, since it was cheaper than spec’ing it that way from Apple.

I also bought a 16GB low-profile USB flash drive to use as my install point and boot device. It's working fine booting and running ESXi 5.5. I’ve always used ESXi embedded on my servers going all the way back to version 3.5, so I was already comfortable and familiar with booting and running ESXi from a flash drive. So to summarize the hardware, that's a quad-core i7, 16GB of RAM, a 1TB spinning HDD and a 500GB SSD as datastores, booting ESXi 5.5 from a tiny flash drive that barely protrudes from its USB port.

On the software side, I downloaded your turnkey ISO for ESXi 5.5 for the Mac Mini. Your advice to enable support for the Thunderbolt Ethernet adapter was easy to follow, and that gave me two 1GigE NICs for whatever I need. The installation was about as simple as you can get.

My workload currently is very light. We just opened a branch office, so that’s where I’m using the Mac Mini. I installed a DC for the site, and have set up just one other support machine so far. It runs the management software for our video surveillance, alarm system remote administration, and the controller software for my WiFi access points. I’ll probably add (or move) other support roles to this machine before long. It's not working that hard and to be honest, is probably faster than the 6-year-old Xeon-based server currently filling the support-system role inside our main office. The two offices are very well-connected, so I can move around support systems almost without concern for which site uses them most. So I do foresee loading up the Mac Mini with more work.

Here is a picture of Bryan's setup:

[William] - What was the deciding factor on choosing a Mac Mini versus Mac Pro or any other platform? Given the Mac Mini is consumer grade hardware and is currently not a supported platform, were there any architecture decisions you had to make on the infrastructure or application side to accommodate for this fact?

[Bryan] – The Mac Pro is fantastic hardware, so if you need heavy-duty power behind your VMs it's certainly worth looking at. But if you really beef up a Mac Pro you're back in the cost realm of what server hardware typically costs. I honestly didn’t look at the compatibility or support of the Mac Pro with ESXi because we didn't need that kind of power for our new branch office.

As for other platforms considered, we’re largely a Dell shop. We use their small-form-factor desktops for our user workstations. I considered using one of our retired workstations as a “server”, but was afraid it would be too slow, even with an SSD, and it didn't have any more RAM capacity than the Mac Mini. Plus it's obviously bigger and less power-efficient, and it’s probably less hardware-compatible with ESXi. That’s why we ultimately chose the Mac Mini.

The main limitation of the consumer-grade hardware in the Mac Mini, for us, was RAM. The Xeon processors in my "proper" servers running in our CoLo aren't ever taxed nearly as much as the other compute resources, so to me the i7 quad-core CPU seemed more than adequate. Having the RAM maxed out at only 16GB, though, made me put extra thought into how I can make the best use of ESXi’s transparent page sharing between VMs. We’re mostly a Windows shop, so it made sense to me to standardize on a single Windows Server version for a given Mac Mini, and try to keep all those VMs at the same patch level. That way, the odds are better that many of the core OS pages in RAM will be identical, meaning the use of TPS will increase, and more RAM will be available to run applications (or even more VMs). We chose Server 2012 Standard as the go-to Windows server OS, and I look forward to loading up the Mac Mini to see just how far ESXi’s advanced memory management techniques will let me push it.

The other constraint was network connectivity. For example, I have just 2 NICs, and I'm currently in the process of working out a backup strategy. I think I'm going to use a backup appliance running on the Mac Mini with a relatively inexpensive NAS as the backup destination, and if needed I can dedicate a physical NIC to that, but with only two NICS, I have to think a lot about network traffic management. I have a couple of other ideas that involve using a thunderbolt dock and/or USB 3.0 NICs to increase the number of USB ports and NICs available to me. The big question there is driver support in ESXi, but I haven’t yet researched or tested those ideas at all.

[William] - You mentioned the current workload is pretty light and you plan to deploy additional Window Servers. What type of workloads will these Virtual Machines be running? Do you see a need to scale up your Mac Mini infrastructure from what you have today?

[Bryan] - I'm not yet ready to put critical production data on them. Not until I have a better feel for what backup looks like, and until I really push the limits of the hardware resources. But support systems like I mentioned above - Wi-Fi controller, NVR for surveillance (with the data stored off-box), things like AV management, Spiceworks, syslog servers, additional DCs to provide a global catalog server, a DFS namespace server, and maybe a replica file server synced with DFS replication are all good possible candidates for running on a Mac Mini.

We have an email archiving system that has to run somewhere. Its data is stored on a remote share and DB, but the processing can happen anywhere, and Mac Mini will handle it fine. That lets me keep resources free for the mission-critical apps that run on my "real" servers that DO contain our critical production data. If I can improve user experience by offloading non-critical support systems to the Mac Mini, there's less resource contention on my vCenter-managed hosts, and user responsiveness should benefit from that.

I also have software firewalls that I use to create a "double-router, Dual-NAT" environment so I can run or build machines in a test, isolated environment, with internet connectivity, with the IP addresses they use (or will use) in production without conflict. They run in perfect isolation. The Mac Mini can host those software firewalls along with any machines that are either being built, or perhaps being restored from backup for exploratory purposes, or even for testing of upgrades or new software in a mirror environment before it goes into production.

As for scaling up, so far I haven't considered adding a Mac Mini ESXi host to my vCenter environment. But I might consider that if two things happened:

Apple starts supporting more RAM in the Mac Mini.
VMware decides to support the Mac Mini hardware officially.

Actually, as I get more time and experience using this setup, number 2 may diminish in importance. Time will tell.

[William] - Your last reply was quite interesting. You mentioned you have not considered adding your Mac Mini ESXi hosts to vCenter Server? I’m curious to hear why the support of Mac Mini would dictate the ease of centralize management? Is it from a support standpoint that you did not want to do this or additional licenses?

[Bryan] - My Dell ESXi servers are managed by vCenter, but I have three of them, which is my license limit currently. If I had a free slot for the Mac Mini I'd certainly use it. But I'd have a hard time justifying the purchase of *additional* licenses for vCenter to bring machines under management that aren't even officially supported. But yeah - if I had the licenses I'd have no hesitation in managing them via vCenter. I get around it currently thru the use of shared data stores.

[William] - Bryan, it was really great chatting with you and thanks again for sharing your experiences on how you have leveraged VMware and Apple Mac Mini’s in your production environment. Here is the last question that I have asked all my past interviewees, do you have any final advice or words of wisdom for someone looking to embark on a similar journey? Any particular resources you would recommend for someone to start with?

[Bryan] - Well, virtuallyGhetto is THE place to go for those resources. Without your pages I would not have made the leap. My advice would be: Don't expect it to do what a $5,000 investment will do. If you plan to run mission-critical apps or host production data, KNOW what your backup and recovery process looks like and TEST it.

If you understand you're striking out somewhat on your own, and you don't mind being a pioneer for the fun of it, do your due diligence, and if it seems like a fit, enjoy it! It's honestly fun to show people my rack. "That's our server. It's running ESXi 5.5." "...Really?!?"

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

VMware has the best platform to run latest Windows 10 Desktop, Server & Hyper-V Tech Preview!

10.08.2014 by William Lam // 6 Comments

I am constantly amazed at the number of guest operating systems that is supported on VMware products like VMware vSphere our Enterprise Hypervisor, vCloud Air our public cloud offering which runs on vSphere and our desktop products such as VMware Fusion and Workstation. If we just look at vSphere alone, it currently "lists" 101 supported guest operating systems! (full list below) However, this is actually a tiny subset of what is actually supported on vSphere as new guest OSes are constantly being added to the support matrix. This also does not include any pre-released operating systems like the recent Apple OS X Yosemite (10.10) Tech Preview. Heck, you can even run Windows 3.11 if you really want to as shown by my fellow VMware colleague Chris Colotti.

To get the complete list of currently supported operating systems for vSphere or any other VMware products, you will want to check the VMware HCL for Guest Operating Systems. Running a filter on latest ESXi 5.5 Update 2 release for all Guest OSes, we can see that the total number of supported Guest OSes is astounding 231! I know this number is even greater as we probably can not capture every single x86 Guest OS that exists out there today which can run on VMware.

Getting back to the topic of this post, I know Microsoft has recently released a new Tech Preview of their upcoming Windows platform dubbed Windows 10 (not a typo, they decided to skip Windows 9) and I know some of you may be interested in trying out their latest release. What better way than to run it on VMware? I know there was a blog or two about running Windows 10 on vSphere, however there was some incorrect information about not being able to install VMware Tools or getting the optimized VMXNET3 driver working. I decided to run all three flavors (Windows 10 Desktop, Server and Hyper-V) on the latest vSphere 5.5 release (should work on previous releases of 5.5) and will share the Virtual Machine configuration.

Note: You can also run Windows 10 Tech Preview on both VMware Fusion and Workstation, take a look at this article for more details. These are great options in addition to vSphere and vCloud Air.

Windows 10 Desktop:

GuestOS: Windows 8 64-bit
Virtual HW: vHW10
Network Driver: VMXNET3
Storage Controller: LSI Logic SAS

Windows 10 Server:

GuestOS: Windows 2012 64-bit
Virtual HW: vHW10
Network Driver: VMXNET3
Storage Controller: LSI Logic SAS

Windows 10 Hyper-v:

GuestOS: Windows 2012 64-bit
Virtual HW: vHW10
Network Driver: VMXNET3
Storage Controller: LSI Logic SAS
CPU Advanced Setting: Enable VHV
VM Advanced Setting: hypervisor.cpuid.v0

For more details about running Hyper-V and the last two advanced settings, please take a look at this article on running other Hypervisors.

If you look closely at this last screenshot, you will see that I am not only running Windows 10 Hyper-V within a VM on ESXi, but I am also running a Nested Windows 10 VM within this Hyper-V VM! How cool is that!? Not sure there are good use cases for this, but if you wanted to, you could! In my opinion (although I may be bias because I work for VMware, but results speak for itself), VMware truly provides the best platform to the widest variety of x86 guest operating systems that exists.

Here are the guest operating systems that are currently "listed" in vSphere today that can be selected:

Apple Mac OS X 10.5 (32-bit)
Apple Mac OS X 10.5 (64-bit)
Apple Mac OS X 10.6 (32-bit)
Apple Mac OS X 10.6 (64-bit)
Apple Mac OS X 10.7 (32-bit)
Apple Mac OS X 10.7 (64-bit)
Apple Mac OS X 10.8 (64-bit)
Apple Mac OS X 10.9 (64-bit)
Asianux 3 (32-bit)
Asianux 3 (64-bit)
Asianux 4 (32-bit)
Asianux 4 (64-bit)
CentOS 4/5/6 (32-bit)
CentOS 4/5/6/7 (64-bit)
Debian GNU/Linux 4 (32-bit)
Debian GNU/Linux 4 (64-bit)
Debian GNU/Linux 5 (32-bit)
Debian GNU/Linux 5 (64-bit)
Debian GNU/Linux 6 (32-bit)
Debian GNU/Linux 6 (64-bit)
Debian GNU/Linux 7 (32-bit)
Debian GNU/Linux 7 (64-bit)
FreeBSD (32-bit)
FreeBSD (64-bit)
IBM OS/2
Microsoft MS-DOS
Microsoft Small Business Server 2003
Microsoft Windows 2000
Microsoft Windows 2000 Professional
Microsoft Windows 2000 Server
Microsoft Windows 3.1
Microsoft Windows 7 (32-bit)
Microsoft Windows 7 (64-bit)
Microsoft Windows 8 (32-bit)
Microsoft Windows 8 (64-bit)
Microsoft Windows 95
Microsoft Windows 98
Microsoft Windows NT
Microsoft Windows Server 2003 (32-bit)
Microsoft Windows Server 2003 (64-bit)
Microsoft Windows Server 2003 Datacenter (32-bit)
Microsoft Windows Server 2003 Datacenter (64-bit)
Microsoft Windows Server 2003 Standard (32-bit)
Microsoft Windows Server 2003 Standard (64-bit)
Microsoft Windows Server 2003 Web Edition (32-bit)
Microsoft Windows Server 2008 (32-bit)
Microsoft Windows Server 2008 (64-bit)
Microsoft Windows Server 2008 R2 (64-bit)
Microsoft Windows Server 2012 (64-bit)
Microsoft Windows Vista (32-bit)
Microsoft Windows Vista (64-bit)
Microsoft Windows XP Professional (32-bit)
Microsoft Windows XP Professional (64-bit)
Novell NetWare 5.1
Novell NetWare 6.x
Novell Open Enterprise Server
Oracle Linux 4/5/6 (32-bit)
Oracle Linux 4/5/6/7 (64-bit)
Oracle Solaris 10 (32-bit)
Oracle Solaris 10 (64-bit)
Oracle Solaris 11 (64-bit)
Other (32-bit)
Other (64-bit)
Other 2.4.x Linux (32-bit)
Other 2.4.x Linux (64-bit)
Other 2.6.x Linux (32-bit)
Other 2.6.x Linux (64-bit)
Other 3.x Linux (32-bit)
Other 3.x Linux (64-bit)
Other Linux (32-bit)
Other Linux (64-bit)
Red Hat Enterprise Linux 2.1
Red Hat Enterprise Linux 3 (32-bit)
Other (32-bit)
Red Hat Enterprise Linux 3 (64-bit)
Red Hat Enterprise Linux 4 (32-bit)
Red Hat Enterprise Linux 4 (64-bit)
Red Hat Enterprise Linux 5 (32-bit)
Red Hat Enterprise Linux 5 (64-bit)
Red Hat Enterprise Linux 6 (32-bit)
Red Hat Enterprise Linux 6 (64-bit)
Red Hat Enterprise Linux 7 (32-bit)
Red Hat Enterprise Linux 7 (64-bit)
SCO OpenServer 5
SCO OpenServer 6
SCO UnixWare 7
SUSE Linux Enterprise 10 (32-bit)
SUSE Linux Enterprise 10 (64-bit)
SUSE Linux Enterprise 11 (32-bit)
SUSE Linux Enterprise 11 (64-bit)
SUSE Linux Enterprise 12 (32-bit)
SUSE Linux Enterprise 12 (64-bit)
SUSE Linux Enterprise 8/9 (32-bit)
SUSE Linux Enterprise 8/9 (64-bit)
Serenity Systems eComStation 1
Serenity Systems eComStation 2
Sun Microsystems Solaris 8
Sun Microsystems Solaris 9
Ubuntu Linux (32-bit)
Ubuntu Linux (64-bit)
VMware ESX 4.x
VMware ESXi 5.x

How to automate VM deployment from large USB keys using ESXi Kickstart?

10.08.2014 by William Lam // 8 Comments

During VMworld US, I had the opportunity to speak with several customers to learn about their VMware environment and some of the challenges they were facing. In some scenarios, I was able to offer a solution or a different way of solving the problem. For others, it was primarily feedback on how we can better improve some of our capabilities/features or specific feature requests they would like to see get added.

One interesting challenge that arose from a class of customers who manages hundreds of remote sites is the ability fully automate the provisioning of an ESXi host as well as set of Virtual Machines as part of the initial deployment. The provisioning is all done through Kickstart (unattended installation of ESXi) and usually from a USB device but it could also be from a custom ISO. One ask that kept coming up was the support for larger USB key support within ESXi so that it could be used to include additional payload.

As some of you may or may not know, ESXi can only access USB devices within the ESXi Shell formatted using the FAT16 filesystem which allows for a maximum file size of 2GB for each partition. However, this limitation is only for the ESXi Shell itself and for the size of the ESXi installation media, this is more than sufficient. If you wish to leverage larger USB keys which has increased significantly in recent years from 32GB, 64GB and even 128GB, you can directly pass that into any guest OS through the USB Arbitrator Service (enabled by default) and there you will be able to consume the entire capacity of the USB device. The challenge is how do you go about bootstrapping ESXi as well as the initial set of Virtual Machines with these limitations and completely automated using an ESXi Kickstart?

Over the years I have seen some really creative solutions to solving this problem and funny enough, right before VMworld I had several folks reach out asking similar questions. I decided to take a look and also build upon some earlier work done by a fellow VMware SE (Tim S) to come up with a completely automated solution that would scale to any size USB device and hopefully make it easy to extend if needed.

For this project, I used a 64GB USB key which I received from the folks over at Micron who I visited in the Solution Exchange during VMworld US (these guys are doing some really awesome stuff with VSAN and an All-Flash array, be sure to check them out).

Here is a diagram of the partition structure for the 64GB USB key which I will explain further:

The first partition is 2GB using a FAT16 filesystem and this is used to store the actual ESXi media along with an embedded ESXi Kickstart configuration file. You can easily reference a remote Kickstart if you wish, but for simplicity purposes and to support some of the requested use cases from customers, I have embedded it.

The second partition is also 2GB using a FAT16 filesystem and this is used to store a tiny VM which I am calling a "Service VM". This VM needs to be small enough to fit the partition and will be used to read the remainder capacity of the USB device which will be using a more capable filesystem type. I have decided to store a pre-configure vMA appliance which is tarred up to reduce the disk footprint.

The third and final partition will consume the remainder capacity of the USB device, in this case it would be 60GB and using a FAT32 partition which can support up to 2TB for a single volume. This is where additional Virtual Machines would be stored and accessed by the "Service VM".

As you can probably guess, the idea is to install ESXi as you normally would to a local disk or directly onto the USB device in which case an additional partition would be required. As part of the installation, the "Service VM" would be boot strapped as it would be visible within the ESXi Shell and registered and powered on during first bootup. A first boot script could then be included in the guestOS which can receive some details about the ESXi deployment which could be hard coded (not recommended) or dynamically discovered as I have implemented it. The USB device would then be passed directly to this "Service VM" to mount and then it would be able to deploy the remainder Virtual Machines which would be stored in this larger partition.

Here is the complete ESXi Kickstart which implements what has been discussed so far and I have also included a break down of the kickstart below:

vmaccepteula
install --firstdisk --overwritevmfs
rootpw vmware123
reboot

network --bootproto=static --ip=192.168.1.200 --netmask=255.255.255.0 --gateway=192.168.1.1 --hostname=mini.primp-industries.com --nameserver=192.168.1.1 --addvmportgroup=1

%post --interpreter=busybox

# stop USB Arbitrator service to access USB device in ESXi Shell
/etc/init.d/usbarbitrator stop

# copy service VM to local VMFS datastore
cp /vmfs/volumes/SERVICEVM/vMA.tar.gz /vmfs/volumes/datastore1
tar -zvxC /vmfs/volumes/datastore1 -f /vmfs/volumes/datastore1/vMA.tar.gz
rm -f /vmfs/volumes/datastore1/vMA.tar.gz

# add guestinfo property for ESXi IP Address for adv. VM deployment
ESXI_IP=$(localcli network ip interface ipv4 get | grep vmk0 | awk '{print $2}')
echo "guestinfo.esxi_ip = ${ESXI_IP}" >> /vmfs/volumes/datastore1/vMA/vMA.vmx

%firstboot --interpreter=busybox

# Ensure hostd is ready
while ! vim-cmd hostsvc/runtimeinfo; do
sleep 10
done

# enable & start SSH
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

# enable & start ESXi Shell
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

# Suppress ESXi Shell warning
esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

# rename datastore1
vim-cmd hostsvc/datastore/rename datastore1 mini-local-datastore-1

# Register VM
vim-cmd solo/registervm /vmfs/volumes/mini-local-datastore-1/vMA/vMA.vmx

# connect USB device via passthrough
USB_DEV_NAME="Alcor Micro Corp"
USB_DEV_BUSID=$(lsusb | grep "${USB_DEV_NAME}" | awk '{print $2}' | cut -c 2)
vim-cmd vmsvc/device.connusbdev 1 "path:${USB_DEV_BUSID}/0/1 version:2"

# power on VM
vim-cmd vmsvc/power.on 1

Line12 - Need to disable the USB Arbitrator Service so the USB device can be seen by ESXi since it is by default made ready to be exposed to a VM. The service will be automatically re-enabled after the installation of ESXi which will allow for the VM to connect to the USB device.

Line15-17 - Copy the "Service VM" from USB device to local VMFS datastore1. In the example, I have pre-configured the vMA appliance tarred up the VMX and its respective VMDK.

Line20-21 - Extract the ESXi IP Address and sets a custom guestInfo property so the "Service VM" knows where to deploy the additional VMs to

Line26-29 - This checks to ensure hostd is up and running before continuing on

Line45 - Register the "Service VM" within ESXi

Line49-50 - Identify the USB device ID which will be required to mount to the "Service VM". You will need to update USB_DEV_NAME based on the USB device you are using

Line51 - Connect the USB Device to "Service VM"

Line54 - Power on the "Service VM"

At this point, you should be able to access the USB device from within the "Service VM". We can easily verify this by running the following command:

sudo fdisk -l

As seen in the screenshot above, we can see our three partitions and third is the one with our FAT32 partition which contains a couple of Virtual Machines that I want to deploy. Of course, this partition can contain anything you wish to store, so the sky is the limit!

To mount the USB device and the specific partition, we will create a temporarily directory and issue the mount command by running these two commands:

sudo mkdir -p /mnt/USB;sudo mount /dev/sdb3 /mnt/USB

For my USB key, I have stored both the VCSA and NSX Manager OVA which can then be deployed using ovftool. The last part to be able to make this as seamless and automated as possible is to be able to identify the ESXi host information. If you recall earlier, we had set a custom guestInfo property within our "Service VM". This custom property can then be read by the guestOS leveraging VMware Tools and provides the IP Address to the guest. You can easily set other metadata information but to be able to deploy these additional OVA's, we would need to know the IP Address of the ESXi host and this makes it so you do not need to hard code anything (perhaps ESXi host credentials).

To retrieve this custom property, you will need to run the following command:

vmtoolsd --cmd "info-get guestinfo.esxi_ip"

With these last few guestOS commands, you will be able to create a firstboot script which will automatically mount the appropriate USB partition and deploy these additional Virtual Machines. This is just one of the many possibilities on how you can deploy additional VMs as part of your ESXi Kickstart deployment. Hopefully this solution provides a base in which you can easily customize based on your own requirements.