WilliamLam.com

How to recover VCSA 5.5 from an expired administrator account?

by // 9 Comments

Last week I wrote about a new security feature in the new VCSA 5.5 where the administrator account (root) password will now expire automatically after 90 days of powering on the VCSA if the password is not changed before then. This new enhancement is to ensures that administrative passwords are rotated routinely for good security practices. However, in the event that you forget to change the password before the expiration, you can still recover the VCSA and this article will walk you through that process.

As a lab exercise, I have configured my root password to expire in one day and purposely let it expire. If you try to login to the VAMI UI, you will get an "Unable to authenticate user" error and you will see something similar if you login to the SSH console. Ideally, this message should be a bit more descriptive to say something like the password has expired (which I have filed an internal bug for).

Requirements:

  • You will need console access to your VCSA
  • You will also need a Linux LiveCD, I personally like using KNOPPIX

Step 1 - Mount the Linux LiveCD to your VCSA and boot into the image. You will need to bring up a terminal shell. The version I am using has a menu and I just select the "shell" option.

Step 2 - Once you are in the terminal, you will need to switch to the root user by running the following command:

su -

Step 3 - Next, we need to mount the VCSA root partition which will be /dev/sda3 to /mnt directory by running the following command:

mount /dev/sda3 /mnt

Step 4 - We now need to edit /etc/shadow file on our VCSA which is located in /mnt/etc/shadow to disable the account lock. You will need to use an editor such as vi to open up the file.

You need to delete "x" in the 2nd field and the numeric value on the 5th field (if it exists, this should be the number of days for expiration, default is 90) for the root user account. The screenshot above shows what values needs to be deleted. Once you have made the changes, go ahead and save the file.

Step 5 - Reboot the VCSA and now you can login to both the VAMI UI interface as well as the SSH console.

Note: If you had the password expiration feature enabled, it has now been disabled for you to login. If you wish to re-enable it, you will need to configure it in the VAMI UI or through the CLI. Please refer to this article here for more details.

How to bootstrap vCenter Server onto a single VSAN node Part 2?

by // 47 Comments

In this article, I will provide a step by step walk through on how to setup and configure single VSAN node that will allow you to deploy a vCenter Server onto a VSAN datastore. This initial "bootstrapping" can help when initially building out your VSAN cluster and can come in handy for greenfield deployments and potentially for brownfield deployments as well. Before getting started, make sure you have taken a look at How to bootstrap vCenter Server onto a single VSAN node Part 1.

Environment:

  • 3 physical host
  • Each host as a small iSCSI boot LUN for ESXi installation (this can be another local disk or USB/SD card)
  • Each host has single SSD and SATA disk (minimum)

Step 1 -  Install ESXi 5.5 onto your physical hosts, we technically only need one host to begin the process but you will probably want to have two additional hosts ready unless you do not care about your vCenter Server being able to recover if there is any hardware issues.

Step 2 - You will need to modify the default VSAN storage policy on the ESXi host in which you plan to provision your vCenter Server. It looks like this behavior changed during the VSAN beta and when VSAN was GA'ed yesterday with vSphere 5.5 Update 1. You will need to run the following two ESXCLI commands to enable "force provisioning":

esxcli vsan policy setdefault -c vdisk -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"
esxcli vsan policy setdefault -c vmnamespace -p "((\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i1))"

You can confirm you have the correct VSAN default policy by running the following ESXCLI command:

~ # esxcli vsan policy getdefault
Policy Class  Policy Value
------------  --------------------------------------------------------
cluster       (("hostFailuresToTolerate" i1))
vdisk         (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmnamespace   (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
vmswap        (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))

We start off with our first ESXi host and as you can see from the screenshot below, we do not have additional datastores that we can use to provision our vCenter Server.

Step 3 - You will need to identify the disks that you will be using on the first ESXi host to contribute to the VSAN datastore. You can do so by running the following ESXCLI command:

esxcli storage core device list

To get specific details on a particular device such as identifying whether it is an SSD or regular HDD, you can specify the -d option and the device name.

Once you have identified the disks you will be using, make a note of the the disks names as they will be needed in the upcoming steps. As mentioned in my environment, I only have a single SSD and single HDD and their respective device names are naa.50026b72270126ff & naa.5000c500331bca77

Step 4 - Before we can create our VSAN datastore, we need to first create a VSAN cluster. One of the parameters that is needed when going through this "bootstrapping" method without a vCenter Server is a unique UUID to identify the VSAN cluster. The UUID is in the format of "nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn" where n is a hexidecimal value. You can easily generate one within the ESXi Shell by leveraging the following Python snippet

python -c 'import uuid; print(str(uuid.uuid4()));'

Step 5 - To create a VSAN cluster, we will use the following ESXCLI command and specify the UUID from the previous step for the -u option:

esxcli vsan cluster join -u UUID

UPDATE (02/11/15) - In vSphere 6, you no longer have to perform step 4 to generate a UUID. There is now a new ESXCLI command which will automatically create a VSAN Cluster and generate a UUID automatically by running the following command:

esxcli vsan cluster new

Once the VSAN cluster has been created, you can retrieve information about the VSAN cluster by running the following ESXCLI command:

esxcli vsan cluster get

Step 6 - Next we need to add the disks from our ESXi host to create our single node VSAN datsatore. To do so, we will need the disk device names from our earlier step for both SSD and HDDs and run the following ESXCLI command:

esxcli vsan storage add -s SSD-DISK-ID -d HDD-DISK-ID

The -d option specifies regular HDD disks and the -s option specifies an SSD disk. If you have more than one HDD disk, you will need to specify multiple -d entries. You can also take a look at the disks being contributed to the VSAN datatore by running the following ESXCLI command:

esxcli vsan storage list

Step 7 - To save us one additional step, you can also enable the VSAN traffic type on the first ESXi host using ESXCLI and you can also do this for the other two hosts in advance. This step does not necessary have to be done now as it can be done later when the vCenter Server is available and using the vSphere Web Client. You will need to either create or select an existing VMkernel interface to enable the VSAN traffic type and you can do so by running the following ESXCLI command:

esxcli vsan network ipv4 add -i VMK-INT

At this point, you now have a valid VSAN datastore for your single ESXi host! You can verify this by logging into the vSphere C# Client and you should see the VSAN datastore mounted to your ESXi host.

At this point, you are now ready to deploy your vCenter Server 5.5 onto the VSAN datastore. The next series of steps outline the deployment of the VCSA for completeness of the article.

Step 8 - Deploy the VCSA 5.5 OVA/OVF onto the VSAN datastore and power on the VM.

UPDATE: You skip Steps 9-11 by leveraging ovftool 4.0 to inject the required OVF properties when deploying the VCSA, take a look at this article for more details.

Step 9 - Since you can not configure the OVF properties for the VCSA, you will notice that networking is not configured (unless you happen to have DHCP on the network). If you are like most Enterprise customers, you will not have DHCP running in your environment and you will need to configure a static IP.

Step 10 - Login to the VCSA console and we will use the following VAMI CLI /opt/vmware/share/vami/vami_set_network to configure the IP Address for the VCSA. Here is an example of what that command would look like:

/opt/vmware/share/vami/vami_set_network eth0 STATICV4 172.24.68.14 255.255.255.0 172.24.68.1

For more details on the syntax, you can refer to this blog article here. At this point, you should be able to ping your VCSA and verify connectivity.

Step 11 (Optional) - In addition to IP connectivity, you may also want to configure your DNS Server and DNS search domain before configure the VCSA application. You can also do this by using the following VAMI CLI /opt/vmware/share/vami/vami_set_dns and for search domain, you would need to add the entry to /etc/resolve.conf

Step 12 - You now are ready to configure the VCSA. Open a browser and connect to https://[VCSA-IP]:5480 and proceed through the VCSA setup wizard.

Step 13 - Once the VCSA has been configured, you can now login to the vSphere Web Client and create a Datacenter object and then a vSphere Cluster and enable VSAN. Make sure you also enter your VSAN beta license key under the "Manage" section of the vSphere Cluster before you can use VSAN.

Step 14 - Add all three of your ESXi hosts to the vSphere Cluster. If you recall earlier we had enabled the VSAN traffic type on our first ESXi host and if you did not run the command on the remainder ESXi hosts, you will need to do so using the vSphere Web Client under the "Networking" section of each ESXi host

Step 15 - Once all three ESXi hosts have been added to the vSphere Cluster, we should now see their local storage contributed to the VSAN datastore under the "General" tab

Step 16 (Optional) - If for whatever reason the disks do not get automatically claimed, you can click on "Disk Management" and manually claim them. If you selected "Automatic" mode when enabling VSAN, the disks on each ESXi host should automatically be handled by VSAN. However, they may not be claimed if the disks are being seen as "remote" versus "local" devices.

Step 17 - The final thing I would recommend is to configure the VCSA to automatically startup and shutdown when the ESXi host reboots. To do so, login to the ESXi host using the vSphere C# Client and click on "Virtual Machine Startup/Shutdown" under the Configuration tab.

So there you have it! You are now running the vCenter Server on top of the VSAN datastore without having to initially setup a local VMFS or rely on an external NFS volume to deploy your vCenter Server and build up to the full VSAN cluster. By leveraging this bootstrap method, you can easily standup a fully self contained storage and compute cluster which is ideal for an SMB or ROBO environment. The best part of about this setup is that the VCSA will use the default VSAN storage policy which is to tolerate at least one failure and as you add your 2nd and 3rd ESXi host, you will automatically have resiliency for the VCSA.

How to bootstrap vCenter Server onto a single VSAN node Part 1?

by // 18 Comments

By now, I am sure you have heard about VMware Virtual SAN (VSAN) and you are probably anxious to give it a spin once the beta becomes publicly available in the very near future. I have been doing some testing in my lab with VSAN, not Nested VSAN, but on actual physical hardware. While getting started, I hit an interesting challenge given my physical hardware configuration and also this being a greenfield deployment.

Let me explain by what I mean by this. In my lab, I have three physical hosts and each contains a single SSD and single SATA drive. Each host has been provisioned with a small 5GB iSCSI boot LUN that is used to install ESXi (this could have also been another local disk or even USB/SD card). Though VSAN itself is built into the VMkernel, the management of the VSAN cluster, configurations and policies are all performed through vCenter Server. So for a greenfield deployment, you would need to first deploy a vCenter Server which would then require you to consume at least one of the local disks. This is the good ol chicken and egg problem!

In my environment, this was a problem because I only have a single SSD and SATA disk and I would not be able to setup a VSAN datastore for all three hosts at once. This meant I had to do the following steps:

  1. Create a local VMFS volume on the first ESXi host
  2. Deploy vCenter Server and then create a VSAN Cluster
  3. Add the two other ESXi host to the VSAN Cluster
  4. Storage vMotion the vCenter Server to the VSAN Datastore
  5. Destroy the local VMFS datastore on first ESXi host (existing VMFS partitions will not work with VSAN) & delete partitions
  6. Add the first ESXi host to VSAN Cluster

As you can see this can get a bit complicated and potentially error prone when needing to destroy VMFS volumes ...

I figured there had to be a better way and I was probably not going to be the only one hitting this scenario for a greenfield and even potentially for a brownfield deployments. In talking to Christian Dickmann, a Tech Lead for the VSAN project, I learned about a really cool feature of VSAN in which you can actually bootstrap vCenter Server onto a single VSAN node! This was possible due to the tight integration of VSAN within the VMkenel and best part about this solution is that it is fully SUPPORTED by VMware. From an operational perspective, this deployment workflow is much easier and intuitive than the process listed above. This also allows you to maximize the use of your hardware investment by running both your core infrastructure VMs as well as your regular workloads all on the VSAN datastore which is great for small or ROBO offices.

In my environment, I start out with a single ESXi 5.5 host which contains a single SSD and SATA disk and I create single VSAN node from that ESXi host and contribute its storage to the VSAN datastore. I then deploy a vCenter Server for which I am using the VCSA (vCenter Server Appliance) for quick and easy deployment. The default policy for VSAN is to automatically ensure there is at least one additional replica of the VM as new ESXi compute nodes join the VSAN cluster.

Once the vCenter Server is online, I can then create a vSphere Cluster and enable it with VSAN and add all three ESXi 5.5 hosts to the vSphere Cluster. This will then contribute all their storage to the VSAN datastore all while the vCenter Server is happily running. Once the other ESXi hosts join the VSAN cluster, we will automatically get replication between the other nodes to ensure our vCenter Server is replicated and of course you can change this policy.

As you can see this is much simpler setup than having to start out with an existing VMFS or even NFS datastore to initially store the vCenter Server and then create the VSAN datstore and migrate the vCenter Server. I also like how I can start deploying my infrastructure with a single ESXi host and then slowly bring in additional ESXi hosts (just make sure you do it in timely fashion as you have a SPOF until then). In part two of this article, I will go into more details on how to configure the single VSAN node and bootstrap vCenter Server. In the meantime, if you have not checked out these awesome articles by some of my VMware colleagues, I would highly recommend you give them a read, especially Cormac's awesome VSAN series!

Here is How to bootstrap vCenter Server onto a single VSAN node Part 2?

If you are interested in testing out VSAN, be sure to sign up for the beta here!

Cormac Hogan

Duncan Epping

Dave Hill