WilliamLam.com

SSH Keys & Lockdown Mode Caveat in ESXi 5

by // 6 Comments

Hopefully everyone is familiar with ESXi's Lockdown Mode and what it means from a security standpoint. Here is a table of the behavior between normal and Lockdown Mode:

In ESXi 5, the use of SSH keys is officially supported without having to manually create any hacks to preserve .ssh directory as you did with prior releases of ESXi. If you use Lockdown Mode, there is an additional caveat to be aware of in which the use of SSH keys is able to by-pass the Lockdown Mode configuration for an ESXi 5 host.

Here is a quick example demonstrating the process from VMware's VCVA (vCenter Virtual Appliance):

Step 1 - Create SSH keys

vcenter50-1:~ # ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
a1:8c:24:60:0e:fe:9a:cf:4a:35:17:d6:89:ba:08:9d root@vcenter50-1
The key's randomart image is:
+--[ DSA 1024]----+
|o. |
|=. o . |
| o. .+ o. |
| ..+oo.. . |
|. E=..o S |
|. = + |
| = . |
|. o |
| ..o |
+-----------------+

Step 2 - Copy SSH public keys over to destination ESXi 5 host into the authorized file under /etc/ssh/keys-root/authorized_keys

vcenter50-1:~ # scp .ssh/id_dsa.pub root@vesxi50-4:/etc/ssh/keys-root/authorized_keys
Password:
id_dsa.pub 100% 606 0.6KB/s 00:00

Step 3 - Enable Lockdown Mode via vCenter

Step 4 - SSH into locked down ESXi 5 host utilizing SSH keys

By default the support of SSH is enabled, you will need to manually disable it to ensure that you are fully lockdown when you choose to enable Lockdown Mode. To disable SSH key support, you just need to comment the following line in /etc/ssh/sshd_config

# AuthorizedKeysFile /etc/ssh/keys-%u/authorized_keys

The change takes effect right away and you do not need to restart the SSH daemon.

If you are interested in automatically disabling the use of SSH keys or you would like to copy an existing SSH key into your ESXi 5 host via kickstart, take a look at Automating ESXi 5.x Kickstart Tips & Tricks for more details.

ESXi Lockdown mode does not play nice with vMA

by // 1 Comment

Today on the VMTN community forums, a user identified an interesting side effect when using vMA's vilogger and enabling lockdown mode on an ESXi host. What the user found was the vilogger daemon stopped collecting logs when lockdown mode was enabled for an ESXi host. At first, I thought lockdown mode should have no affect on vilogger, as it only disables the "root" account from accessing ESXi host other than from the DCUI (Directo Console User Interface).

I replicated the setup in my lab using an ESXi host that was being managed by vMA via vi-fastpass (fpauth) and enabled vilogger for this host. I verified log collection was functional before enabling lockdown mode on the ESXi host, right away vilogger stopped collecting logs when lockdown mode was enabled. When using the "vilogger list" command, the status of the ESXi host goes from "collecting" to "No Permission". I found this to be quite odd and verified what the user was describing in his environment.

Next was to take a look at the vilogger logs which is stored in /var/log/vmware/vma/vilogd.log and I found the same "No Permission" error.

I decided to login to the ESXi host and tailed the hostd logs to see what was going on when lockdown mode was being enabled. What I found was pretty surprising to me, there was a task that removed permissions from the vi-adminXX user account, I was pretty sure at this point, the culprit was related to lockdown mode.

I decided to take a look at VMware's documentation to see what the behavior of Lockdown Mode was and the following snippet taken from vSphere's online documentation explains it all:

The text highlighted in red is the key to the issue the user is facing and specifically the very last section where it states:

you cannot run vCLI commands from an administration server, from a script, or from vMA against the host

This meant that not only the root account was locked out, but all other accounts found on the ESXi host whether they are custom from your environment or from auxiliary systems such as VMware vMA, would be completely disabled. What is even more interesting, even read-only accounts would no longer function, they too had to go to vCenter to be re-proxied to specific ESXi host.

This has a few implications for users considering Lockdown Mode:

  1. All scripts including resxtop and user authentication must go through vCenter. If vCenter went down, you have no remote way to access your ESXi host. This also meant that you could not remotely start up vCenter if it was hosted in a virtual machine but rather from DCUI after enabling Local Tech Support Mode
  2. The use of vMA's vilogger is completely useless when Lockdown Mode is enabled for ESXi host. Users may want to consider setting up a traditional syslog server and have the logs forwarded from the ESXi host

IMHO, I don't think Lockdown Mode should crippled the vilogger functionality, the logging is a "Read" operation and I think re-configuring it to "read-only" role should have suffice. I also think that VMware could have done a better job working with the vMA engineers to support this functionality and have some documentation regarding this issue. For now, if you rely on any type of automation that goes directly to an ESXi host and you are thinking about Lockdown mode, you may want to think twice.