WilliamLam.com

Why is my VSAN Component maximum showing less than 3000?

by // Leave a Comment

This is a question that I have seen come up on several occasions in both the VMTN Community forums as well as in our internal Socialcast group. I have not seen anyone blog about this topic yet and figure I would share the answer since this was a question I had asked myself when I had initially setup VSAN. If you are not familiar with VSAN Components, I highly recommend you check out Cormac Hogan's blog article VSAN Part 4: Understanding Objects and Components.

In vSphere 5.5 Update 1, the maximum number of supported components for VSAN is 3000 which is a per ESXi host maximum. What some folks are noticing when they run the RVC vsan.check_limits command on their VSAN Cluster, they are finding out that the maximum is coming up much lower as seen in the example below.

/localhost/VSAN-Datacenter/computers> vsan.check_limits VSAN-Cluster/
2015-01-28 15:34:25 +0000: Gathering stats from all hosts ...
2015-01-28 15:34:27 +0000: Gathering disks info ...
+--------------------------------+-------------------+-------------------------------------------+
| Host                           | RDT               | Disks                                     |
+--------------------------------+-------------------+-------------------------------------------+
| vesxi55-3.primp-industries.com | Assocs: 30/20000  | Components: 8/750                         |
|                                | Sockets: 17/10000 | naa.6000c2932c3f51f04e4cd395f4a11752: 8%  |
|                                | Clients: 3        | naa.6000c294f6496a99ad756857b9b06f01: 0%  |
|                                | Owners: 5         |                                           |
| vesxi55-2.primp-industries.com | Assocs: 10/20000  | Components: 8/750                         |
|                                | Sockets: 13/10000 | naa.6000c294bde5987d60398e0305978b00: 9%  |
|                                | Clients: 0        | naa.6000c292a964255b82410099360a9b27: 0%  |
|                                | Owners: 0         |                                           |
| vesxi55-1.primp-industries.com | Assocs: 24/20000  | Components: 8/750                         |
|                                | Sockets: 15/10000 | naa.6000c298b69006b820e367b5fde97cbf: 11% |
|                                | Clients: 3        | naa.6000c29db3f272cfb7fb4d08bffad3ab: 0%  |
|                                | Owners: 3         |                                           |
+--------------------------------+-------------------+-------------------------------------------+

The reason for this is actually due to the amount of physical memory available to each ESXi host. If you are running VSAN in a Nested ESXi environment like I am in the example above, I only have 8GB of memory configured for each ESXi host. The number of supported VSAN Components will definitely differ from an actual physical host with more memory and the nice thing about vsan.check_limits command is that it is dynamic in nature based on the actual available resources. Funny enough, the majority of the questions actually came from folks who ran VSAN in a Nested Environment, so this would explain why this question keeps popping up.

If I run the same RVC command on an environment where VSAN was running on real hardware with a decent amount of memory which most modern systems these days have, then I can see the VSAN Component maximum is properly displaying the 3000 limit as expected in the example below.

/localhost/datacenter01/computers> vsan.check_limits vsan-cluster01/
2015-01-28 15:28:47 +0000: Querying limit stats from all hosts ...
2015-01-28 15:28:49 +0000: Fetching VSAN disk info from esx021.vmwcs.com (may take a moment) ...
2015-01-28 15:28:49 +0000: Fetching VSAN disk info from esx022.vmwcs.com (may take a moment) ...
2015-01-28 15:28:49 +0000: Fetching VSAN disk info from esx024.vmwcs.com (may take a moment) ...
2015-01-28 15:28:51 +0000: Done fetching VSAN disk infos
+---------------------------+--------------------+---------------------------------------------------------------------------------+
| Host                      | RDT                | Disks                                                                           |
+---------------------------+--------------------+---------------------------------------------------------------------------------+
| esx021.vmwcs.com          | Assocs: 223/45000  | Components: 97/3000                                                             |
|                           | Sockets: 132/10000 | t10.ATA_____WDC_WD1002FAEX2D00Z3A0________________________WD2DWCATRC061926: 18% |
|                           | Clients: 14        | t10.ATA_____KINGSTON_SH103S3480G__________________00_50026B7226017C69____: 0%   |
|                           | Owners: 29         |                                                                                 |
| esx022.vmwcs.com          | Assocs: 252/45000  | Components: 96/3000                                                             |
|                           | Sockets: 143/10000 | t10.ATA_____KINGSTON_SH103S3480G__________________00_50026B7226017CA2____: 0%   |
|                           | Clients: 14        | t10.ATA_____WDC_WD1002FAEX2D00Z3A0________________________WD2DWCATRC050466: 19% |
|                           | Owners: 38         |                                                                                 |
| esx024.vmwcs.com          | Assocs: 197/45000  | Components: 96/3000                                                             |
|                           | Sockets: 122/10000 | t10.ATA_____ST2000DL0032D9VT166__________________________________5YD73PRP: 8%   |
|                           | Clients: 17        | t10.ATA_____KINGSTON_SH103S3480G__________________00_50026B7226017C5B____: 0%   |
|                           | Owners: 22         |                                                                                 |
+---------------------------+--------------------+---------------------------------------------------------------------------------+

The lesson here is that even though I am a huge supporter of using Nested ESXi to learn about new products, features and how they work from a functional perspective, there is no amount of Nested ESXi testing that can ever replace actual testing of real hardware.

Thunderbolt Storage for ESXi

by // 47 Comments

Screen Shot 2015-01-20 at 9.11.51 PMA question that I frequently receive is whether ESXi supports Thunderbolt-based storage devices. This is especially interesting for folks running ESXi on an Apple Mac Mini due to the limited number of IO connections the Mac Minis' have. If you look on VMware's HCL, you will not find any supported Thunderbolt Storage devices nor are there any that are being actively tested with ESXi, at least as far as I know.

Having said that, generally speaking from an ESXi host point of view, the Thunderbolt interface is just seen as an extended PCIe bus. This means that whatever storage device is connected on the other end can work with ESXi as long as there is a driver in ESXi that can communicate with that devices. This is analogous to having a RAID card and having the proper device driver on ESXi to see its storage.

Even though VMware is not actively testing Thunderbolt-based storage devices, there are a few folks out in the community who have and have been successful. I wanted to share these stories with the community for those that might be interested in this topic and hopefully others who have had similar success can also share more details about their setup.

UPDATE (09/12/16) - ESXi Thunderbolt Driver to Fibre Channel Storage from ATTO

Disclaimer: All solutions listed below are from the community and decisions to purchase based on these solutions will be at your own risk. I hold no responsibility if the listed solutions do not work for whatever reason.

Solution #1 - Pegaus R6 Thunderbolt Storage Enclosure

This was the first Thunderbolt storage device that I had ever seen confirmed publicly to work with ESXi after installing a STEX driver VIB. You can find more details here.

Solution #2 - Sonnet Echo Express III-R Rackmount Thunderbolt 2 Expansion Chassis & RacMac Mini Enclosure

This next solution was recently shared with me from Marc Huppert who has recently expanded his home lab. Marc combined a Thunderbolt expansion chassis with a Mac Mini chassis to exposed Fibre Channel storage to his Mac Minis. You can find more details here.

Solution #3 - xMac Mini Server Enclosure

I came across this solution while searching online which also uses another Mac Mini Thunderbolt expansion chassis connected to Fibre Channel based storage. You can find more details here.

Solution #4 - Sonnet xMac Pro Server Enclosure

Thanks to Joshua for sharing his solution. You can find more details in the comments here.

Solution #5 - LaCie Rugged Thunderbolt drives

Thanks to Philip for sharing his solution. You can find more details in the comments here.

Solution #6 - ARC-8050T2 Thunderbolt 2 RAID

Thanks to Jason for sharing his solution. You can find more details in the comments here.

Solution #7 - Another Sonnet xMac Pro Server Enclosure + EMC VNX

Thanks to Johann for sharing his solution. You can find more details here.

Solution #8 - LaCie Little Big Disk Thunderbolt 2 with 2013 Mac Pro w/ESXi 6.0

Thanks to Thomas for sharing his solution. You can find more details here.

Solution #9 - Sonnet Echo Express III with Mac Pro 6,1 and ATTO ExpressSAS H680 w/ESXi 6.0

Thanks to Grasshopper for sharing details here and here.

Solution #10 - OWC ThunderBay 4 RAID 4-Bay External Drive w/Dual Thunderbolt 2

Thanks to Gregg Green for sharing his use of the Thunderbay with Mac Mini 2012

If there are other Thunderbolt-based storage devices that you or others have had success with ESXi, feel free to leave a comment with details and I will add it to the post. If there are any Thunderbolt storage device vendors that would like to send me a demo unit, I would be more than happy to give the system a test to see if it works with ESXi 🙂

Completely automating vCenter Server Appliance (VCSA) 5.5 Configurations

by // 8 Comments

As promised, here is a new script called configureVCSA55.sh that I have put together after learning about a couple new VCSA automation tips here and here. This script will fully automate the configuration of a vCenter Server Appliance (VCSA) 5.5 and once the script has completed, you will have a fully functional vCenter Server Appliance. There are several variables at the top of the script that you will want to edit prior to running the script.

Here is a summary of the high level operations the script is performing and not all operations will be performed, it will depend on the variables that you have configured.

  • Accept EULA
  • vSphere Inventory Size Configuration
  • Active Directory Configuration (optional)
  • DNS Search Domain Configuration
  • NTP Configuration
  • vCenter Server Database Configuration
  • vSphere SSO Configuration
  • vSphere SSO Identity Source Configuration for Active Directory (optional)
  • Active Directory default Identity Source Configuration (optional)
  • VMware Telemtry Configuration (optional)

To run the script, you can either SCP the script to a newly deployed VCSA and run it locally in the shell or remotely via SSH using the following command:

ssh root@[VC-IP] < configureVCSA55.sh

completely-automate-configuration-vcsa55.0
I almost never go through a manual configuration of the VCSA anymore (since 5.0) as it just takes way too long! Hopefully you will find this script handy when needing to quickly test something or automating the deployment of a few dozen VCSA which I know of a few customers that are doing on a regular basis 🙂