In this article, I will provide a step by step walk through on how to setup and configure single VSAN node that will allow you to deploy a vCenter Server onto a VSAN datastore. This initial "bootstrapping" can help when initially building out your VSAN cluster and can come in handy for greenfield deployments and potentially for brownfield deployments as well. Before getting started, make sure you have taken a look at How to bootstrap vCenter Server onto a single VSAN node Part 1.
Environment:
- 3 physical host
- Each host as a small iSCSI boot LUN for ESXi installation (this can be another local disk or USB/SD card)
- Each host has single SSD and SATA disk (minimum)
Step 1 - Install ESXi 5.5 onto your physical hosts, we technically only need one host to begin the process but you will probably want to have two additional hosts ready unless you do not care about your vCenter Server being able to recover if there is any hardware issues.
Step 2 - You will need to modify the default VSAN storage policy on the ESXi host in which you plan to provision your vCenter Server. It looks like this behavior changed during the VSAN beta and when VSAN was GA'ed yesterday with vSphere 5.5 Update 1. You will need to run the following two ESXCLI commands to enable "force provisioning":
esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"
esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"
You can confirm you have the correct VSAN default policy by running the following ESXCLI command:
~ # esxcli vsan policy getdefault
Policy Class Policy Value
------------ --------------------------------------------------------
cluster (("hostFailuresToTolerate" i1))
vdisk (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmnamespace (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmswap (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
We start off with our first ESXi host and as you can see from the screenshot below, we do not have additional datastores that we can use to provision our vCenter Server.
Step 3 - You will need to identify the disks that you will be using on the first ESXi host to contribute to the VSAN datastore. You can do so by running the following ESXCLI command:
esxcli storage core device list
To get specific details on a particular device such as identifying whether it is an SSD or regular HDD, you can specify the -d option and the device name.
Once you have identified the disks you will be using, make a note of the the disks names as they will be needed in the upcoming steps. As mentioned in my environment, I only have a single SSD and single HDD and their respective device names are naa.50026b72270126ff & naa.5000c500331bca77
Step 4 - Before we can create our VSAN datastore, we need to first create a VSAN cluster. One of the parameters that is needed when going through this "bootstrapping" method without a vCenter Server is a unique UUID to identify the VSAN cluster. The UUID is in the format of "nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn" where n is a hexidecimal value. You can easily generate one within the ESXi Shell by leveraging the following Python snippet
python -c 'import uuid; print(str(uuid.uuid4()));'
Step 5 - To create a VSAN cluster, we will use the following ESXCLI command and specify the UUID from the previous step for the -u option:
esxcli vsan cluster join -u UUID
UPDATE (02/11/15) - In vSphere 6, you no longer have to perform step 4 to generate a UUID. There is now a new ESXCLI command which will automatically create a VSAN Cluster and generate a UUID automatically by running the following command:
esxcli vsan cluster new
Once the VSAN cluster has been created, you can retrieve information about the VSAN cluster by running the following ESXCLI command:
esxcli vsan cluster get
Step 6 - Next we need to add the disks from our ESXi host to create our single node VSAN datsatore. To do so, we will need the disk device names from our earlier step for both SSD and HDDs and run the following ESXCLI command:
esxcli vsan storage add -s SSD-DISK-ID -d HDD-DISK-ID
The -d option specifies regular HDD disks and the -s option specifies an SSD disk. If you have more than one HDD disk, you will need to specify multiple -d entries. You can also take a look at the disks being contributed to the VSAN datatore by running the following ESXCLI command:
esxcli vsan storage list
Step 7 - To save us one additional step, you can also enable the VSAN traffic type on the first ESXi host using ESXCLI and you can also do this for the other two hosts in advance. This step does not necessary have to be done now as it can be done later when the vCenter Server is available and using the vSphere Web Client. You will need to either create or select an existing VMkernel interface to enable the VSAN traffic type and you can do so by running the following ESXCLI command:
esxcli vsan network ipv4 add -i VMK-INT
At this point, you now have a valid VSAN datastore for your single ESXi host! You can verify this by logging into the vSphere C# Client and you should see the VSAN datastore mounted to your ESXi host.
At this point, you are now ready to deploy your vCenter Server 5.5 onto the VSAN datastore. The next series of steps outline the deployment of the VCSA for completeness of the article.
Step 8 - Deploy the VCSA 5.5 OVA/OVF onto the VSAN datastore and power on the VM.
UPDATE: You skip Steps 9-11 by leveraging ovftool 4.0 to inject the required OVF properties when deploying the VCSA, take a look at this article for more details.
Step 9 - Since you can not configure the OVF properties for the VCSA, you will notice that networking is not configured (unless you happen to have DHCP on the network). If you are like most Enterprise customers, you will not have DHCP running in your environment and you will need to configure a static IP.
Step 10 - Login to the VCSA console and we will use the following VAMI CLI /opt/vmware/share/vami/vami_set_network to configure the IP Address for the VCSA. Here is an example of what that command would look like:
/opt/vmware/share/vami/vami_set_network eth0 STATICV4 172.24.68.14 255.255.255.0 172.24.68.1
For more details on the syntax, you can refer to this blog article here. At this point, you should be able to ping your VCSA and verify connectivity.
Step 11 (Optional) - In addition to IP connectivity, you may also want to configure your DNS Server and DNS search domain before configure the VCSA application. You can also do this by using the following VAMI CLI /opt/vmware/share/vami/vami_set_dns and for search domain, you would need to add the entry to /etc/resolve.conf
Step 12 - You now are ready to configure the VCSA. Open a browser and connect to https://[VCSA-IP]:5480 and proceed through the VCSA setup wizard.
Step 13 - Once the VCSA has been configured, you can now login to the vSphere Web Client and create a Datacenter object and then a vSphere Cluster and enable VSAN. Make sure you also enter your VSAN beta license key under the "Manage" section of the vSphere Cluster before you can use VSAN.
Step 14 - Add all three of your ESXi hosts to the vSphere Cluster. If you recall earlier we had enabled the VSAN traffic type on our first ESXi host and if you did not run the command on the remainder ESXi hosts, you will need to do so using the vSphere Web Client under the "Networking" section of each ESXi host
Step 15 - Once all three ESXi hosts have been added to the vSphere Cluster, we should now see their local storage contributed to the VSAN datastore under the "General" tab
Step 16 (Optional) - If for whatever reason the disks do not get automatically claimed, you can click on "Disk Management" and manually claim them. If you selected "Automatic" mode when enabling VSAN, the disks on each ESXi host should automatically be handled by VSAN. However, they may not be claimed if the disks are being seen as "remote" versus "local" devices.
Step 17 - The final thing I would recommend is to configure the VCSA to automatically startup and shutdown when the ESXi host reboots. To do so, login to the ESXi host using the vSphere C# Client and click on "Virtual Machine Startup/Shutdown" under the Configuration tab.
So there you have it! You are now running the vCenter Server on top of the VSAN datastore without having to initially setup a local VMFS or rely on an external NFS volume to deploy your vCenter Server and build up to the full VSAN cluster. By leveraging this bootstrap method, you can easily standup a fully self contained storage and compute cluster which is ideal for an SMB or ROBO environment. The best part of about this setup is that the VCSA will use the default VSAN storage policy which is to tolerate at least one failure and as you add your 2nd and 3rd ESXi host, you will automatically have resiliency for the VCSA.
Thank you! That is great. I looked at this because I was having issues with doing a vMotion of the vCenter Server to the vSAN. I am doing a lab via VMware Workstation. I feel as if my performance is the reason for why vMotion is not working properly. I had issues with where I began the vMotion and it tells me it failed, but noticed that my vSAN capacity was almost full and it seemed to have somehow copied a chunk of files and left them there. Not sure how that happened, so I blew away the environment and started from scratch again. I will give this a try and see how it goes! Thanks again!
This worked great in combination with your article on how to simulate an SSD drive with a HDD, allowing me to stand up a 3 node vCenter Cluster on old test hardware in my lab. Now, the only warning I have in the entire vCenter Cluster with HA and vSAN is the one where my logs are not on persistent storage, since I booted off of 4GB USB keys to setup the Hosts (since I only had two SAS drives per Host, one I made a pseudo-SSD and combined with the remaining one HDD to make a vSAN datastore). I'm wondering if there is a way to move my logs onto the vsanDatastore that gets created, but I've not found a way so far (mkdir want make a directory on a vSAN datastore, so the normal method of moving logs to a datastore doesn't work).
Another awesome VSAN article once again. Just have a quick question. You state in step 16 to use Virtual Machine startup rules..etc, but this is a host only setting, so are you saying to use Affinity Must rule to keep the VCSA to a specific host? If that's the case then we don't have any HA protection for vCenter.
Wouldn't we want to use a Cluster level VM start priority instead?
No, this is only a temporary configuration in case the host need to be rebooted. Once you have vCenter Server up and running, then you should use the cluster rules to ensure the VCSA will automatically start back up. We would not want to tie it to a particular node 🙂
Ahhh..ok! Loving the VSAN templates and the quick and easy way to classify SSD!
Not sure why but I can only add 2 of the 3 hosts. They are all identical, when I go into claim disks or a disk group is just says completed but only 2 of 3 are eligible after the task completes.
SIGH.......Licensing
Now, that's a quite useful post the time I needed it! I have just started a 3-node lab with each node having one SSD and 3 1TB SATA disks. I have followed the steps above (even had to delete the partitions of the HDDs) but then when I run the esxcli command to add the 3 HDDs and the 1 SSD to the vsanDatastore, it was stuck there for ages (ok, I am not that patient..)
Any thoughts on where to look?
Take a look at /var/log/vmkernel.log to see if there's any issues when the VSAN partitions are being created. You may also want to check your physical drives and make sure there's no issues
hm.. it shows a great deal of kernel timeouts for the drives and the output is *exactly* what this guy saw: http://serverfault.com/questions/585944/upgrade-reinstall-vmware-vsan-from-beta
Now, These drives were initially setup as a Raid group which I changed in the BIOS and set them as AHCI mode. I removed the (1) partition they all had. ESXi sees them all. I haven't tried the solution of this guy above though. The second node has the same behavior. So, these two ESXi nodes are stuck on "usbarbitrator start" while booting today! The 3rd node is OK (never tried to create a VSAN there).
I will give a try with the remediation that this guy did but it its strange if that worked for him. The 3 nodes have ESXi 5.5u1 build 1623387. Never played with the beta VSAN.
No luck. I have tried a few combinations with the HDDs but it's the same painful result. I have removed all partitions from the disks, I've put them behind RAID0 (but it was a Volume).. So, I'm pretty tired after a week of trying. I may end up taking out the disks and building a tower box as NFS. What a pitty... what a waste of HDD/SSD resources (from my side that is..)
But the posting here is really nice 🙂 Thanks William!
BTW, the esxcli commands in Step2 worked for me without the "\" and with single quotes in the outside:
esxcli vsan policy setdefault -c vdisk -p '(("hostFailuresToTolerate" i1) ("forceProvisioning" i1))'
esxcli vsan policy setdefault -c vmnamespace -p '(("hostFailuresToTolerate" i1) ("forceProvisioning" i1))'
regards,
Thanos
I guess this might be an obvious question, but have you confirmed that all components (storage controllers, MD, SSD) are on the VSAN HCL? If not, then it would not surprise me that you're seeing issues.
If all the hardware is on the VSAN HCL, another thing you may want to double check if HCL provides is if you're running the latest/recommended firmware or drivers. If you're not using the correct or unsupported hardware, then you will most likely run into trouble
I'm almost sure that the components are not on the HCL 🙂 I mean the server is not on the ESXi5.x HCL but it runs (I got the server after someone else posted that 5.5 does boot on it 🙂 VSAN is another story. Yes, the onboard SATA controller is recognized from 5.5 without any tricks or hacks or extra VIBs. The SSD and the disks are 'normal' SATA variants. Nothing strange there but maybe the HDD firmware or probably the controller's firmware would need some upgrade (I doubt there's one available. This model is EOL.. Anyways, I will try to find some firmware if possible as the last resort. Then, it's NFS (sigh)
..Unless the VSAN team (a couple of buildings away) helps me hack it somehow to work... there are a few guys that want to try VSAN in these old puppies. It would make people happy 🙂 Well, anyways..
Hey William,
This was a fantastic article. I actually found it after I had already created the local vmfs, installed VC, created switching, and created the disk groups on two of the ESXi servers. However I was able to run those commands and enable force provisioning and then storage vmotion Virtual Center to the VSAN datastore, reverse the commands, delete the local vmfs and add the remaining disks to a disk group and to VSAN. Thank you so much, that would have been a tricky spot to get out of!
Hi William,
step 6 doesn't work for me. When trying to add the SSD and non-SSD disks to the cluster i get the following error:
Disk: naa.6b......is an SSD and can not be used as a VSAN HDD
My SSD is a RAID0 SSD which I had to claim to be an SSD first. Everything else I did like you suggested in this post.
Did you saw this before?
Make sure there are no partitions on it and it's also local device. Recommend you run "vdq -q" command to see if that's the case 🙂
Thanks for replying so fast. I made a unnecessary mistake by using the wrong identifier.
Oh and thumps up for this article, your whole blog and not to forget your Healthcheck script. Useful as hell!
Hi William,
I've got a question:
Imagine that I've followed your procedure to bootstrap vCenter on the VSAN Datastore on a single host and I haven't changed the default SP from "Force Provisioning" to something else after the deployement. So not having done this, this means that the VM components will not be replicated as they should be by default with a RAID1 configuration for example.
Now I need to reset the system configuration of the host, using the dcui, for a particular reason.
How can I get back the vCenter VM that was created on the VSAN Datastore using the bootstrap method ?
A reset system configuration wouldn't change anything on the SSD and MD layouts so, I'm wondering if I will get the VM back after adding back the SSD and MD to the VSAN Datastore.
Thanks
Interesting question. It's not something I've tried before, but given reset doesn't affect the VSAN Data, your VM will be intact. However, it may not power on due to not meeting the FTT policy. Generally, if the FTT policy fails, the provision is halted, but in this case you've already provisioned the bits so there might be a chance it'll complain or not allow you to power on, so you just need to change the policy. This is my guess, something you would want to try out
Hi William,
Got the confirmation that the VM data is back again and accessible once you add the host back to the VSAN cluster, using the command "esxcli vsan cluster join"
You don't even need to re-add manually the SSD's and MD's to the VSAN Datastore.
It works pretty well and the VM starts without any issue.
This is a very good point for VSAN.
Anyway I would recommend your procedure JUST for Customers who only have 1 SSD and 1 MD per hosts and no external Storage.
For Customers having more than 1 SSD and 1 MD, I believe it should be easier just to deploy vCenter on one Datastore formed by one of the MD and then after migrate the vCenter to the VSAN DS and reuse the MD for VSAN, once free.
Thanks William !
Hi William
I have an issue when adding the disks to the single node VSAN datastore.
i ran:
esxcli storage nmp satp rule add --satp VMW_SATP_LOCAL --device mpx.vmhba1:C0:T0:L0 --option "enable_local enable_ssd"
esxcli storage nmp satp rule add --satp VMW_SATP_LOCAL --device mpx.vmhba1:C0:T1:L0 --option "enable_local"
esxcli storage core claiming reclaim -d mpx.vmhba1:C0:T0:L0
esxcli storage core claiming reclaim -d mpx.vmhba1:C0:T1:L0
It sees the SSD as true.
I get the ID's returne by the core device list command, but, when i run the:
esxcli vsan storage add -s vml.0000000000766d686261313a303a30 -d vml.0000000000766d686261313a313a30
it returns that vml.0000000000766d686261313a313a30 Disk not found.
what's the catch?
thanks
Hi William,
Thanks for this article, I've followed it and have completed building my 4 host VSAN cluster successfully. I have one question though. On step 2 the default VSAN storage policy on the ESXi host where VCSA is bootstrapped is modified. Does that need to be reversed once the cluster is up and running? If so, how is that accomplished?
To echo everyone else, great article. In the beginning you enable forceProvisioning Should. that be disabled when you're all done? If so, how do you do that? Thank you, Zach
Thank you for sharing, I’m trying to setup a single node VSAN cluster for my home lab. Should I go with 1x240GB or 2x240GB or 1x480GB disk for the SSD layer?. I will also have 3x1TB HDD.
On 6.7 U1 I get a syntax error when generating the UUID. The following command works okay with an extra set of brackets around the string:
python -c 'import uuid; print (str(uuid.uuid4()))'
LOL then I read the note about this not being required in 6.x
i have tried your class. and success. i met some problem ,but i fixed. problem such as one host vmk cannot enable vsan service ,such as one host vmk cannot enable vsan service. finally, i create a new cluster and move nodes to new cluster.