Using the new vSphere Guest OS Customization with cloud-init in vSphere 7.0 Update 3

In vSphere 7.0 Update 3, the vSphere Guest Operating System (OS) Customization Engine has added native support for using cloud-init, which is an industry standard for customizing Linux-based operating systems. This additional GuestOS customization option is currently only available when using the vSphere API, either vSphere SOAP API or vSphere REST API and is not available for consumption when using the vSphere UI.

As part of a recent project, I was exploring some of the customization options and since I had not played with this specific API before, I figured this would be a good exercise. I quickly found that it was not very user intuitive in getting started, especially with the lack of end-to-end examples since this can only be consumed using the vSphere API. I also came across a number of different VMware KBs (here, here and here) that outlined various requirements and constraints when using cloud-init which also added to the confusion.

The high level requirements for using the new vSphere Guest OS Customization with cloud-init is the following:

• vSphere 7.0. Update 3 or later (vCenter and ESXi)
• VMware Tools running 11.3 or later
• cloud-init running 21.1 or later

Note: Although VMware PhotonOS does support cloud-init natively, it is not a supported operating system when using the new vSphere GuestOS Customization with cloud-init due to how cloud-init has been integrated. For customers that require customization via cloud-init with PhotonOS, should continue using either the seed ISO option or the GuestInfo OVF option.

In this blog post, I will explore the complete end-to-end workflow from preparing a GuestOS for customization to applying the actual vSphere customization spec using the new cloud-init option. In addition, I have also created a simple PowerShell script which demonstrates the use of the vSphere REST API on constructing the required specification for using the new cloud-init option and this should hopefully help folks understand how the underlying API works with a working example.

Prepare GuestOS for Customization

Step 1 - Install the desired GuestOS that has support for cloud-init using OS vendor ISO. In the example below, I have been successful in using both Ubuntu 21.10 and Ubuntu 22.04 with all default options selected.

Step 2 - Add the following to entry to the bottom of /etc/cloud/cloud.cfg configuration file:

disable_vmware_customization: false

Step 3 - Delete the following files that is located under /etc/cloud/cloud.cfg.d/

rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg
rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg

Step 4 - Run the cloud-init clean operation:

/usr/bin/cloud-init clean --logs

Step 5 - Shutdown the Guest OS

To help simplify the GuestOS preparation steps, I have created the a quick shell script that you can use that will automatically apply all the required changes listed above and then shutdown the GuestOS.

#!/bin/bash

grep "disable_vmware_customization: false" /etc/cloud/cloud.cfg > /dev/null 2>&1
if [ $? -eq 1 ]; then
    echo "disable_vmware_customization: false" >> /etc/cloud/cloud.cfg
fi

rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg
rm -rf /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg

/usr/bin/cloud-init clean --logs
shutdown -h now

At this point, you can either apply the vSphere GuestOS Customization directly onto this VM or simply turn this VM into a vSphere Template or Content Library Image to become the base image for deploying new VMs that can then be customized using cloud-init.

Apply GuestOS Customization

Step 1 - Create your desired cloud-init metadata file (must be JSON format) that contains the desired network configuration for your VM. Below is an example configuration which I have named metadata-ubuntu.json and simply configures a static IPv4 address for the network adapter labeled ens160.

{
    "instance_id": "vsphere-gosc-cloud-init-configured",
    "local_hostname": "vsphere-gosc-cloud-init-configured",
    "network": {
        "version": 2,
        "ethernets": {
            "ens160": {
                "gateway4": "192.168.30.1",
                "addresses": ["192.168.30.231/24"],
                "dhcp4": "false",
                "nameservers": {
                    "search": ["primp-industries.local"],
                    "addresses": ["192.168.30.2"]
                }
            }
        }
    }
}

Step 2 - Download the vsphere_guestos_customization_using_cloud_init.ps1 PowerShell example and edit the variables to match your environment. You will also need to obtain the MoRef ID for the VM you wish to apply the vSphere GuestOS Customization. You can do this by using the GET VM API or simply looking at the URL when selecting the desired VM using the vSphere UI, which will be in the format of vm-X, where X is some numeric value (e.g. vm-36040).

Step 3 - Run the PowerShell script which will create the vSphere GuestOS Customization spec from the supplied cloud-init metadata file and apply that to the desired VM. If the operation was successful, you should see a message like the following:

The script also has a debug option to display full REST API payload for those interested in using this API with another API client such as REST, python, go, etc.

Step 3 - Finally, the last step is to power on the VM and you should also see the start and completed guest OS customization tasks within the vSphere UI as shown in the screenshot below.

Once the customization has completed, you can verify that the correct networking has been applied to the desired VM.

Troubleshooting

There are other useful logs located in /var/run/cloud-init and /var/lib/cloud/instance that can also be helpful but during my exploration of the new vSphere GuestOS Customization with cloud-init, I found the following logs to be useful in aiding with the debugging.

• /var/log/vmware-imc/toolsDeployPkg.log - Used to confirm that VMware Tools has detected the GuestOS customization CAB file exists in the VM home directory and has been copied into GuestOS
• /var/log/cloud-init-output.log - Used to confirm the actual processing of cloud-init and customization
• /var/run/cloud-init/instance-data.json - Used to confirm metadata payload that cloud-init has processed
• /var/lib/cloud/instance/user-data.txt - Used to confirm the userdata and ensuring its in the right format