The other day I needed to increase the capacity on one of my Nested VSAN Datastores as one of our users required a larger VSAN datastore than it was initially configured for. I was expecting to be able to just increase the size of the underlying VMDKs like I would for a traditional Nested ESXi environment and rescan in ESXi to pick up the new capacity without any downtime. It turns out, this is was not exactly the case for a Nested VSAN environment.

Disclaimer: Nested Virtualization is not officially supported by VMware

When you first setup VSAN, regardless of how the disks were claimed, VSAN will consume the entire device (SSD or MD). The capacity that VSAN initially detects will then be used to create the necessary partition as part of the VSAN Disk Group creation. VSAN assumes that the capacity for the underlying devices would never change as in the "real" world, disks do not auto-magically get larger 🙂 and this is a valid assumption. In a Nested ESXi environment however, it can auto-magically get larger but VSAN was not built for this use case. What ends up happening is that the underlying devices can be "hot-extended" but the existing VSAN Disk Group can not detect this new capacity.

Having said that, there are two ways you can increase your VSAN datastore:

Option 1 - If you wish to preserve your VSAN Datastore, you can hot-add additional VMDK(s) to your existing VSAN Disk Group or if it is full, you can create a new disk group and add additional VMDK(s). This will modify your setup slightly if you wanted a particular set of disk groups but will allow you to preserve your data.

Option 2 - The latter option requires the deletion and re-creation of the VSAN Datastore which is not ideal if you already have data on it. You will need to increase the capacity of the underlying VMDKs and then re-create your VSAN Datastore, but this way you can keep the existing number of disks and disk groups you initially created your Nested ESXi environment with.

In my scenario, I could not destroy the VSAN Datastore as I had someone using it and so I opted for option #1. Here is what my configuration looked like before which was a single VSAN Disk Group with 1xSSD and 1xMD:

I then added an additional 10GB VMDK to each of my Nested ESXi hosts and issue a rescan so the ESXi host would pickup the new device:

In just a few seconds, I can see my new storage device. I can now head over to the VSAN management page which is located at the vSphere Cluster and once I refresh, I can see that VSAN has automatically added the new "MD" into the existing disk group and my storage has automatically expanded!

I recently noticed a couple of people hitting a warning message when configuring VSAN and specifically when running VSAN in a Nested ESXi environment (which is not officially supported by VMware). The warning message is displayed on the summary page of the ESXi host which states the following:

Host is in a VSAN enabled cluster but does not have VSAN service enabled

The reason you are seeing this issue is related to the amount of memory you have allocated for your Nested ESXi VM. During the VSAN Beta, the minimal amount of memory was 4GB but it looks like that has changed to 5GB with the GA of vSphere 5.5 Update 1 last week. I know we could have done better job with the error message and communicate the actual underlying issue (will ensure we have an FR filed for this).

However, the fix is quite simple, just shutdown your Nested ESXi VM and then change it to 5GB and this message will go away. It is also worth noting that as you increase the number of disks and disk groups in your ESXi hosts, there will be an increase in memory. I would highly recommend you take a look at the official VSAN Design & Sizing guide to properly size out your real VSAN environments.

For basic functional testing and education of VSAN (not including running additional VMs), running a Nested ESXi VM with 5GB will be sufficient. You can also take a look at the my VSAN Nested ESXi OVF template which can just download and install ESXi 5.5 Update 1 without any issues.