WilliamLam.com

Using Google Authenticator To Provide Two-Factor Authentication For ESXi

by // 12 Comments

Last year, I came across an interesting open source project called Google Authenticator, which provides two-factor authentication by using both a PAM (Pluggable Authentication Module) module and a mobile application for generating one-time passcodes. This project sparked my curiosity and I wanted to see if I could get Google Authenticator to run on ESXi to provide additional security by requiring two-factor authentication when attempting to login to the ESXi Shell. After several attempts, I failed to get Google Authenticator working on ESXi and eventually gave up. I reported the behavior I saw in my environment in the Issues section of the project which did not yield any responses and I thought that was the end of that experiment.

Almost eight months later, I received a surprising email from two VMware engineers who were also interested in the Google Authenticator project and were able to figure out how to get Google Authenticator to work on ESXi. In sharing their findings, it turns out that the solution was actually quite simple and it just required commenting out three lines of C Macros in the Google Authenticator source code (tweak is documented in this blog post here). I was able to confirm the engineers findings in my home lab and was also able to build a custom Google Authenticator VIB for ESXi to help with the setup.

Disclaimer: This is not officially supported by VMware, use at your own risk

Installing Google Authenticator Custom VIB / Offline Bundle

Prerequisite:

  • Ensure that your ESXi host's clock is in sync with a proper time source (skew should be < 4minutes)
  • Keep a separate SSH connection open to your ESXi host, in case something goes wrong you can easily revert the changes else you can potentially lock yourself out

Step 1 - Download either the Google Authenticator VIB vghetto-google-auth.vib or offline bundle vghetto-google-auth.zip and upload it to the datastore of your ESXi host

Step 2 - You will need to change the acceptance level of your ESXi host to Community Supported as this is a requirement for any custom VIBs created. Run the following ESXCLI command:

esxcli software acceptance set --level CommunitySupported

Step 3 - To install Google Authenticator VIB, you will need to run the following ESXCLI command and specify the full datastore path of the VIB:

esxcli software vib install -v /vmfs/volumes/mini-local-datastore-1/vghetto-google-auth.vib -f

To install the Google Authenticator offline bundle, you will run the same command but instead of using the -v argument, you will specify the -d

Step 4 - You can verify the Google Authenticator was installed successfully by running the following ESXCLI command:

esxcli software vib get -n vGhetto-goog-auth

 

Configuring Google Authenticator & ESXi Configurations

Step 1 - Download the Google Authenticator app for your mobile phone. In this example, I am using the iPhone's Google Authenticator mobile app.

Step 2 - Next you will need to configure Google Authenticator for the ESXi host, run the google-authenticator command in the ESXi Shell which will start the setup.

You should see a URL as well as the secret key which you will need to enter into your Google Authenticator mobile app. You can either manually add your ESXi host into the mobile app by entering the secret key OR copy and paste the URL into a web browser which provides a QRC code that the mobile app can just read.

For all the prompted questions, you can use yes for the defaults.

Step 3 - You will need to add the following configuration to your SSHD configuration under /etc/ssh/sshd_config

ChallengeResponseAuthentication yes

Step 4 - You will also need to add the following entry to the following PAM configuration files /etc/pam.d/login and /etc/pam.d/sshd

auth   required   pam_google_authenticator.so

To add the entry into both files on the ESXi Shell, run the following two commands:

sed -i -e '3iauth       required     pam_google_authenticator.so\' /etc/pam.d/login
sed -i -e '3iauth       required     pam_google_authenticator.so\' /etc/pam.d/sshd

Note:  To ensure the above configuration persists after a reboot, you will need to add the two sed commands to /etc/rc.local.d/local.sh for ESXi 5.1 or /etc/rc.local for ESXi 5.0 hosts which will automatically add the entries upon bootup. 

Finally, you will need to restart the SSH daemon for the changes to go into effect by running the following command:

/etc/init.d/SSH restart

Step 5 - To validate that everything was configured correctly, open a new SSH session to your ESXi host. Instead of seeing the usual password prompt, you should now see a verification code prompt. Open up your Google Authenticator mobile app and enter the code that is displayed for your ESXi host and then enter the root password.

If everything was correct, you should now be authorized and logged into your ESXi host

Though the configuration could be a bit more automated including adding each ESXi host to your mobile application, this is a pretty nifty and free solution to provide two-factor authentication for your ESXi hosts. I am curious to see if others are interested in such functionality within ESXi or if this would be useful? Feel free to leave a comment.

Big thanks to VMware engineers who helped me get this solution to work!

How To "Pause" (Not Suspend) A Virtual Machine In ESXi?

by // 14 Comments

Last week I received a very interesting question from a fellow blogger asking whether it was possible to "pause" (not suspend) a virtual machine running on ESXi. Today ESXi only supports the suspend operation which saves the current memory state of a virtual machine to disk. With a "pause" operation, the memory state of the virtual machine is not saved to disk, it is still preserved in the physical memory of the ESXi host. The main difference with a "pause" operation is the allocated memory is not released and this allows you to quickly resume a virtual machine almost instantly at the cost of holding onto physical memory.

The use case for this particular request was also quite interesting. The user had an NFS server that housed about 200 virtual machines that needed to be restarted and the goal was to minimize the impact to his virtual machines as much as possible. He opted out from suspending the virtual machines as it would have taken too long and decided on a more creative solution. He filled up the remainder capacity on the datastore which in effect caused all virtual machines to halt their I/O operations. Though not an ideal solution IMHO, this allowed him to restart the NFS server and then run a script for the virtual machines to retry their I/O operation once the NFS server was available again.

Based on the above scenario, he asked if it was possible to "pause" the virtual machines similar to a capability Hyper-V provides today which would have provided him a quicker way to resume the virtual machines. Thinking about the question for a bit, a virtual machine is just a VMX process running in ESXi and I wondered if this process could be paused like a UNIX/Linux process using the "kill" command. Well, it turns out, it can be!

Disclaimer: This is not officially supported by VMware, use at your own risk.

Using the kill command, you can pause the VMX process by sending the STOP signal and to resume the VMX process, you can send the CONT signal. Before getting started, you will need to identify the PID (Process ID) for the virtual machine's VMX process.

There are two methods of identifying the parent VMX PID, the easiest is using the following ESXCLI command:

esxcli vm process list


The PID for the virtual machine will be listed under the "VMX Cartel ID" and in this example I have a virtual machine called vcenter51-1 and on the right I am pinging the system to verify it is up and running. An alternative way of identifying the PID is to use "ps" by running the following command:

ps -c | grep -v grep | grep [vmname]

Note: Make sure you identify the parent PID of the virtual machine if you are using the above command as you will see multiple entries for the different VMX sub-processes.

To pause the VMX process, run the following command (substitute your PID):

kill -STOP [pid]

To resume VMX process, run the following command:

kill -CONT [pid]

Here is a screenshot of pausing and then resuming the virtual machine. You can also see where the pings stop as the virtual machine is paused and then resumed. Once the virtual machine was resumed, it operated exactly where it left off with no issues as far as I can tell.
 
Note: I have found that if you have VM monitoring enabled, there maybe issues resuming the virtual machine. This should only be done if you have VM monitoring disabled as it may not be properly aware that the VMX process being paused on purpose.

Though it is possible to pause a virtual machine, I am not sure I see too many valid use cases for this feature? Are there are use cases where this feature would actually be beneficial, feel free to leave a comment if you believe there are. For now, this is just another neat "notsupported" trick 😉

vCloud Director Simulator

by // 8 Comments

I have received several questions from folks asking if there are other VMware solutions that could work with the vCenter Server Simulator (vcsim) which I recently wrote an article about back in 2012. Well, the answer is I do not know, as other solutions have not really been tested out with vcsim. The goal of vcsim was primarily focused at the vSphere layer and I suspect that other VMware solutions can probably leverage what vcsim has done but probably will not work (today) as some solutions require a fair amount of communication between itself and the vCenter Server. Now having said all that, you might be wondering about the title of this article?

The one solution that I am aware of today that works with vcsim is vCloud Director and this is merely a proof of concept than anything else, so I would like to set the proper (low) expectations. The available functionality from vCloud Director is quite limited when using it with vcsim and I have also ran into several issues which I will go into more detail later. For vCloud Director to be able to consume the simulated inventory from vcsim, a few tweaks are required in both vcsim and the vCloud Director database.

Disclaimer: This is not officially supported by VMware, use at your own risk.

Requirements:

  • VCSA 5.1 (vCenter Server Appliance)
  • VCNS 5.1 (vCloud Networking and Security)
  • VCD 5.1 Appliance
  • configureVCSimulator.sh script

I highly recommend you take a look at my vCenter Server Simulator article before getting started as it gives you some background that is required later on but you will not be required to configure the VCSA in advanced as there are some additional configurations that are needed. I will also be using the vCloud Director appliance which I recommend for quick ease of deployment and setup.

Configurations: 

Step 1 - Deploy VCSA and configure it as you would normally (do not touch the vcsim configurations, as that is handled in the next step using a script that is provided)

Step 2 - Download the configureVCSimulator.sh script and upload it to the VCSA. By default, vcsim's XML configuration files is set to ESXi 4.0 and we will need to adjust the version numbers to reflect either 5.0 or 5.1 as well as modify the hypervisor type from embeddedEsx to esx which is required by the simulator code. Go ahead and run the script on the VCSA which will setup the vcsim as well as modify the necessary files. Once the script has completed, if you wish to change the default inventory, go ahead and modify /etc/vmware-vpx/vcsim/model/initInventory.cfg and then restart the vCenter Server service by running service vmware-vpxd restart

Here is a screenshot of the script running:

Note: If you are interested in the files that are being modified, you can take a look at the shell script for more details. 

Step 3 - Deploy and configure VCNS appliance as you would normally. Make sure you also register the vCenter Server in the VCNS UI that you just deployed as this is needed before moving forward to the next step.

Step 4 - Deploy the VCD appliance using the embedded database and power on the system. Once the system is ready, go ahead and SSH to the host, we will need to execute a few SQL queries which will allow vCloud Director to support simulated hosts as well as the particular ESXi versions. First we need to switch over to the Oracle user, run the following command:

su - oracle

Next we will login to the VCD database using sqlplus and the default credentials which should be vcloud/VCloud, run the following command:

sqlplus vcloud/VCloud

The first SQL statement will allow VCD to support simulated ESXi hosts, type in the following SQL statement:

update config set value='1' where name='vzSim50Supported';

The next two SQL statements will define the version of ESXi that VCD will support, type in the following SQL statements that are applicable to you:

insert into os values (seq_config.NextVal, 'vmnix-x86', 'VMware ESX', '5.1.0', 0, 1);
insert into os values (seq_config.NextVal, 'vmnix-x86', 'VMware ESX', '5.0.0', 0, 1);

Finally, we just need to type "quit" to exit from sqplus and then type "exit" to logoff as the oracle user and this will bring us back to root context. The last thing we need to do is restart the vCloud Director service for the changes to go into effect by running the following command

service vmware-vcd restart

Here is a screenshot of running through the VCD database changes:

Step 5 - Once the vCloud Director service has been successfully restarted, we are now ready to setup our vCloud Director Simulator environment. Start off by adding both your VCSA and VCNS to the vCloud Director under Resources tab as you would normally.

Next, click on Provider VDCs to create your Pvdc and when you arrive at the ESXi host preparation step, you can type in anything for the username/password (since these are simulated ESXi host).

After you click next or finish, you then click into the Hosts tab to watch the ESXi host preparation. This is where you will notice the first issue, the preparation should finish almost immediately as thgey are simulated hosts, so no VCD agent is installed or services needing to be enabled, but from what I have seen this process can take up to 30minutes if not longer. You will need to be patient when you get to this point and do not try to cancel the process and just let it sit and finish. To ensure your environment is setup correctly, make sure the following three columns show green check marks: Enable, Ready and Available during the preparation.

This is probably a good time to get some coffee or tea and check out other cool articles on virtuallyghetto.com 😉

Once the ESXi hosts have completed preparation, you should see a green check box under the Status column which means you are now ready to create an Organization which is a pre-requisite before creating an Organization VDC. I will assume you know how to proceed from here and depending if you setup an ESXi 5.0 or ESXi 5.1 environment in vcsim, you will need to choose the appropriate virtual machine compatibility (virtual hardware) to support in your OrgVDC.

Note: If you setup an ESXi 5.0 environment, you may see an error with too many connected datastores, to resolve this, you need to either decrease the number of hosts in your simulated environment or use ESXi 5.1

You can probably guess the last step is to start provisioning VMs/vApps in our vCloud Director instance, but sadly this is where you will face another issue. What I have found is that when you deploy a new vApp, an exception is thrown regarding a NULL pointer and VCD will fail on the deployment.

However, if we take a look at our vcsim environment, you will see that the VM actually does get created but this still poses a problem for us in VCD, as the task is still not considered successful.

I have not been able to figure out why and I know from running through several tests, there was a case where I was able to successfully provision a VM in the VCD simulator. This might be related to the version of virtual hardware selected, I had more luck using either 7 or 8 but not so much with 9. Unfortunately, without provisioning capabilities, this may not be as useful as one would like but I encourage you to still give this a try if you are interested.

The vCloud API can also be access in the simulated environment, but remember that the operations will be limited to what has been shown and will probably work for most of the inventory based API calls, but again, YMMV.

Though vCloud Director Simulator is somewhat limited today and with a few known issues, it is a proof of concept that other VMware solutions can leverage what vcsim has provided as a base foundation. If you think this is something that is useful to have or have other use cases for, please leave a comment. You never know, this could be a VMware Fling one day if there is enough interest from the community.