William Lam

How to automatically add ESX(i) host to vCenter in Kickstart

03.29.2011 by William Lam // 65 Comments

While recently updating my Automating Active Directory Domain Join in ESX(i) Kickstart article, it reminded me an old blog post by Justin Guidroz who initially identified a way to add an ESXi host to vCenter using python and the vSphere MOB. The approach was very neat but was not 100% automated as it required some user interaction with the vSphere MOB to identify certain API properties before one could potentially script it within a kickstart installation.

I decided to revisit this problem as it was something I had investigated awhile back. There are numerous ways on getting something like this to work in your environment, but it all boils down to your constraints, naming convention and provisioning process. If you have a well defined environment and utilizing a good naming structure and can easily identify which vCenter a given ESX(i) host should be managed from, then this can easily be integrated into your existing kickstart with minor tweaks. This script was tested on vCenter 4.1 Update 1 and ESXi 4.1 and 4.1 Update1.

UPDATE (03/29/2011): Updated the IP Address extraction to use gethostbyname and added proper logout logic after joining vCenter.

UPDATE (02/01/2013): I have provided a download link to the joinvCenter.py script below as there have been some funky formatting issues when displaying the script. For ESXi 5.x hosts, you will need to ensure httpClient is enabled (disabled by default) on the ESXi firewall else it will not be able to connect to your vCenter Server. Please refer below for the instructions.

There are a few steps that are necessary before we get started and a recommended one for those that have security concerns around this solution.

Step 1 - You will need to extract some information from the vCenter server in which you would like your ESX(i) hosts to join. You will need to generate an inventory path to the vCenter cluster which will take the form of: [datacenter-name]/host/[cluster-name], this will automatically locate the managed object ID of your vCenter cluster which is required as part of the host add process. This was a manual process in Justin's original solution.

In this example, I have a datacenter called "Primp-Skunkworks" and a cluster under that datacenter called "Primp-Skunkworks-Cluster", the inventory path will look like the following:

"Primp-Skunkworks/host/Primp-Skunkworks-Cluster"

You will need this value to populate a variable in the script which will be described a little bit later

Step 2 - As you may have guessed, to add an ESX(i) host to vCenter, you will need to connect to vCenter server and use an account that has the permission to add a host. It is recommended that you do not use or expose any administrative accounts for this as the credentials are stored within the script unencrypted. A work around is to create a service account whether that is a local account or an Active Directory account with only the permission to add an ESX(i) host to a vCenter cluster. You will create a new role, in this example I call it "JoinvCenter" and you just need to provide the Host->Inventory->Add host to cluster privilege.

Once you have created the role, you will need to assign this role to the service account user either globally in vCenter if you want to add to multiple cluster or a given datacenter/cluster.

Now that we have the pre-requisites satisfied, we will need to populate a few variables within the script which will be used in your %post section of ESX(i) kickstart configuration file.

This variable defines the name of your vCenter server, please provide the FQDN:

This variable define the vCenter cluster path which was generated earlier:

These variables define the server account credentials used to add an ESX(i) host to vCenter. You will need to run the following command to encode the selected password. You will need access to a system with python interpreter to run the following command:

python -c "import base64;
print base64.b64encode('MySuperDuperSecretPasswordYo')"

Note: This does not encrypt your password but obfuscate it slightly so that you are not storing the password in plain text. If a user has access to the encoded hash, it is trivial to decode it.

These variables define the ESX(i) root credentials which is required as part of the vCenter add process. If you do not want to store these in plain text, you will also need to encode them using the command in previous section:

We are now all done and ready to move forward with the actual script which will be included in your kickstart configuration. As a sanity check, you can run this script manually on an existing ESX(i) host to ensure that the process works before testing in kickstart. For ESXi 5.x hosts, ensure httpClient firewall ruleset is enabled by following ESXCLI command:

esxcli network firewall ruleset set -e true -r httpClient

You should also ensure this is the very last script to execute as I ran into a race condition while the root password was being updated automatically from the default 999.* scripts. To ensure this is the very last script, set the --level to something like 9999 in your %firstboot stanza

Download: joinvCenter.py

To aide in troubleshooting, the script also outputs the details to syslog and on ESX(i), it will be stored in /var/log/messages and you can just search for the string "GHETTO-JOIN-VC". If everything is successful, after %firstboot section has completed, you should be able to see an ESX(i) host join vCenter and the following in the logs.

Tips: You should only see "Success" messages, if you see any "Failed" messages, something went wrong. If you are still running into issues, make sure your ESX(i) host has it's hostname configured with FQDN and you should see an error on your vCenter server if it fails to whether it's due to hostname and/or credentials. You can also redirect the output of the script to local VMFS volume for post-troubleshooting.

Depending on your provision process and how you determine which ESX(i) host should join which vCenter/cluster, you can easily add logic in the main kickstart configuration file to automatically determine or extract from a configuration file and dynamically update joinvCenter.py script prior to execution.

I would like to thank Justin Guidroz and VMTN user klich for their contributions on the python snippets that were used in the script.

FYI - I am sure the python code could be cleaner but I will leave that as an exercise for those more adept to python. My python-fu is not very strong 😉

How to enable SSH security banner on ESXi

03.19.2011 by William Lam // 3 Comments

There was a pretty interesting question on the VMTN forums this week about adding a security banner to ESXi prior to user authentication via SSH. With classic ESX, this can easily be enabled by updating the "Banner" entry in /etc/sshd/sshd_config and specifying the path to the file containing your banner. With ESXi, OpenSSH is not being used for the SSH daemon/client, but rather a server/client with a much smaller footprint called Dropbear.

Dropbear not only provides a server/client but it also includes key generator/converter and secure copy (scp) all bundled into a single binary under /sbin/dropbearmulti. To access the different functions, you would just need to create a symlink to (dropbear,dbclient,dropbearkey,dropbearconvert and scp) which is already enabled by default from VMware with the exception of dropbear which is for the server. You can see the dropbear process spawn when you are logged into the Busybox Console (Tech Support Mode) using "ps" command.

Here is what that looks like:

What you may not notice is that there are some additional arguments being passed to the dropbear utility. Dropbear is launched using inetd and it's configurations are defined in the /etc/inetd.conf file.

You may wonder if there are some additional options that dropbear supports? The answer is yes and we can check for these options by manually creating a symlink from /bin/dropbear to /sbin/dropbearmulti and running dropbear -h option.

Note: The symlink creation of "dropbear" is not needed to enable security banner

As you can see there are a ton of additional options including support for a banner file using the -b option.

To enable this, we will first create a banner and for this example, I will be storing it under /etc/banner

Next, we will need to update the dropbear arguments to include the banner file, you will need to update /etc/inetd.conf and specify the path to your banner file.

Next you will need to restart inetd process and use kill -HUP command.

Now if you try to login to your ESXi host via SSH, you should now see the new banner be presented prior to authenticating to the host.

Now before you jump off and start thinking about a creative banner, you should note that changes within the ESXi Busybox Console are not always preserved and persisted across reboots. This is not a well known fact and the reason for this is ESXi is loaded into memory after it boots up.

There are certain configuration files (e.g. /etc/inetd.conf) that are automatically backed up through a cronjob which looks for particular files under /etc that have been marked with the stickybit. A user can not manually mark a file with the stickybit and have it automatically backed up, it requires one additional file which is implemented by the VisorFS. ESXi creates a copy of these stickybit files and renames the original as .#filename. The backup process will then look for any .#* files and back those up. Due to this special permission mechanism, you can not manually create/touch files with this format as explained by a VMware employee on this VMTN thread.

An alternative to this, which is one that I have used in the past is to update the /etc/rc.local file which is automatically backed up. Entries in this file will be executed after the host has booted up and it is the perfect place to re-create our banner file as it will not be persisted across reboots. You will create a simple here document in the script which contains the contents of your banner file, in this example, I am storing it in /etc/banner

Once you have updated and saved /etc/rc.local file, you will need to manually run a backup to ensure we have a good backup in case the host reboots. You can do so by running /sbin/auto-backup.sh which will create the latest backup and store it under /bootbank. You now have successfully enabled a security banner on ESXi and it persist through reboots.

Note: You could also have stored your banner under local VMFS datstore and/or other datastore, but it is probably best that you keep it under normal filesystem paths.