Automated VMware Cloud Foundation (VCF) host commission using ESXi Kickstart

ESXi Scripted Installation (Kickstart) has been my go-to method for achieving zero-touch provisioning of ESXi hosts at scale, which I had started using back in the ESX 2.5 days when I was a customer! Having worked at some very larger enterprises, I got the opportunity to experience and manage a variety of environments for automated ESXi provisioning.

For more than a decade, I have written hundreds of articles about ESXi kickstart and how it can help solve a variety of use cases stemming from my own background to some of the unique requirements that have come up from some of our largest VMware customers. To date, some of my favorite ESXi kickstart solutions includes my 2014 blog post in automating VM deployments using a USB device which became the basis for my USB to SDDC project in 2017.

While playing with the latest VMware Cloud Foundation (VCF) 5.1 Holodeck release (currently in Beta), I was thinking about the current VCF host commissioning workflow, which is a multi-step process after an ESXi host has been provisioned where you need to manually (or using automation) to add the hosts to SDDC Manager before they can be consumed for either expanding and/or deploying a new workload domain.

I thought, why could we not just skip this step all together and that was when I had the idea of just incorporating the VCF host commissioning workflow automatically as part of an ESXi Kickstart installation! 😀

There are a number of ways to achieve to the desired outcome that I have described above and for my solution, I wanted it to be dynamic in terms of which SDDC Manager an ESXi host would automatically commission itself to. The way I accomplished was to host a simple host mapping text file (sddcm-mapping.txt) on a web server that would contain the ESXi FQDN to the SDDC Manager (including service account token and network pool ID) details which would then be used by the specific ESXi host to remotely invoke the VCF Commission REST API.

Here is an example of the sddcm-mapping.txt file:

esxi-9.vcf.sddc.lab,sddc-manager.vcf.sddc.lab,EGoXCs16YY8pzN4cWEJfakmzN4wMfNog,662944fd-b7cc-4f2b-9b1c-484f986692ca
esxi-10.vcf.sddc.lab,sddc-manager.vcf.sddc.lab,EGoXCs16YY8zwN4cWEJfakmzN4wMfNog,662944fd-b7cc-4f2b-9b1c-484f986692ca
esxi-11.vcf.sddc.lab,sddc-manager.vcf.sddc.lab,EGoXCs16YY8zwN4cWEJfakmzN4wMfNog,662944fd-b7cc-4f2b-9b1c-484f986692ca
esxi-12.vcf.sddc.lab,sddc-manager.vcf.sddc.lab,EGoXCs16YY8zwN4cWEJfakmzN4wMfNog,662944fd-b7cc-4f2b-9b1c-484f986692ca

To generate an SDDC Manager service account which just requires the Operator role, please see this blog post for more details. For simplicity purposes, I used the built API Explorer within SDDC Manager to identify the Operator role ID and creation of the service account as shown in the screenshot below.

The rest of the automation magic is contained within %firstboot section of your ESXi Kickstart as shown in the reference implementation below:

vmaccepteula
rootpw VMware123!
install --firstdisk --overwritevmfs
network --bootproto=dhcp --device=vmnic0
reboot

%firstboot --interpreter=busybox

# Ensure TLS certificate matches ESXi FQDN
/sbin/generate-certificates

# Ensure hostd is ready
while ! vim-cmd hostsvc/runtimeinfo; do
sleep 10
done

# Allow outbound 443 to SDDC Manager
esxcli network firewall ruleset set -e true -r httpClient

# Ensure NTP is running and properly configured
esxcli system ntp set -e true -s 10.0.0.221

# Retrieve configuration file to determine which SDDC Manager to commission ESXi host
wget http://10.0.0.250/sddcm-mapping.txt -O /tmp/sddcm-mapping.txt

SDDCMADDRESS=$(grep $(hostname) /tmp/sddcm-mapping.txt | awk -F ',' '{print $2}')
SDDCMTOKEN=$(grep $(hostname) /tmp/sddcm-mapping.txt | awk -F ',' '{print $3}')
SDDCMNETORKPOOLKD=$(grep $(hostname) /tmp/sddcm-mapping.txt | awk -F ',' '{print $4}')

cat > /tmp/sddcm-host-commission.py <<EOF
import socket
import sys
# Add the required path
sys.path.append("/usr/lib/vmware/vsan/perfsvc/")
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning

# Disable SSL warnings
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

sddcManagerAddress = "${SDDCMADDRESS}"
sddcManagerToken = "${SDDCMTOKEN}"
sddcManagerNetworkPoolId = "${SDDCMNETORKPOOLKD}"
esxiPassword = "VMware123!"
esxiFQDN = socket.getfqdn()

def main():
    # Create a session for making API requests
    session = requests.Session()
    session.verify = False  # Disabling SSL verification

    # Retrieve SDDC Manager API access token
    payload = {"apiKey": sddcManagerToken}
    headers = {"Content-Type": "application/json"}

    try:
        response = session.post("https://" + sddcManagerAddress + "/v1/tokens", json=payload, headers=headers)
        response.raise_for_status()  # Raise an exception for bad response codes
        access_token = response.json().get('accessToken')

        if access_token:
            # Prepare payload for commissioning ESXi host
            host_payload = [{"fqdn": esxiFQDN, "username": "root", "password": esxiPassword ,
                             "storageType": "VSAN", "networkPoolId": sddcManagerNetworkPoolId}]
            headers['Authorization'] = "Bearer " + access_token

            # Comission ESXi host
            response = session.post("https://" + sddcManagerAddress + "/v1/hosts", json=host_payload, headers=headers)
            response.raise_for_status()

            print("ESXi Host Commission successful")
        else:
            print("Failed to retrieve access token.")
    except requests.RequestException as e:
        print("Error occurred during ESXi host comission request:", e)

if __name__ == "__main__":
    main()
EOF

python sddcm-host-commission.py

Once the ESXi host has rebooted from the initial installation, it will then attempt to run the firstboot script which will download the sddcm-mapping.txt file and it will check whether there is a configuration entry that directs it to commission itself to a specific SDDC Manager. If an entry is found, it will then use the credentials and construct the required payload and invoke the VCF host commission REST API and if everything was setup correctly, you should now see a task within your SDDC Manager which has been initiated by the ESXi host after it has finished provisioning, completely automated and zero touch user interaction. 😎

As you can imagine, this could be very useful for those provisioning a large number of ESXi hosts for expanding and/or creating new Management and Workload Domains, especially in a distributed manner across multiple VCF Instances and this can remove one additional manual step in the overall provisioning process. I certainly would love to see this as a native feature of ESXi Kickstart and supported by the VCF platform in the future 🙂