In vSphere 7.0 Update 3, the vSphere Guest Operating System (OS) Customization Engine has added native support for using cloud-init, which is an industry standard for customizing Linux-based operating systems. This additional GuestOS customization option is currently only available when using the vSphere API, either vSphere SOAP API or vSphere REST API and is not available for consumption when using the vSphere UI.
As part of a recent project, I was exploring some of the customization options and since I had not played with this specific API before, I figured this would be a good exercise. I quickly found that it was not very user intuitive in getting started, especially with the lack of end-to-end examples since this can only be consumed using the vSphere API. I also came across a number of different VMware KBs (here, here and here) that outlined various requirements and constraints when using cloud-init which also added to the confusion.
The high level requirements for using the new vSphere Guest OS Customization with cloud-init is the following:
- vSphere 7.0. Update 3 or later (vCenter and ESXi)
- VMware Tools running 11.3 or later
- cloud-init running 21.1 or later
Note: Although VMware PhotonOS does support cloud-init natively, it is not a supported operating system when using the new vSphere GuestOS Customization with cloud-init due to how cloud-init has been integrated. For customers that require customization via cloud-init with PhotonOS, should continue using either the seed ISO option or the GuestInfo OVF option.
In this blog post, I will explore the complete end-to-end workflow from preparing a GuestOS for customization to applying the actual vSphere customization spec using the new cloud-init option. In addition, I have also created a simple PowerShell script which demonstrates the use of the vSphere REST API on constructing the required specification for using the new cloud-init option and this should hopefully help folks understand how the underlying API works with a working example.
Prepare GuestOS for Customization
Step 1 - Install the desired GuestOS that has support for cloud-init using OS vendor ISO. In the example below, I have been successful in using both Ubuntu 21.10 and Ubuntu 22.04 with all default options selected.
Step 2 - Add the following to entry to the bottom of /etc/cloud/cloud.cfg configuration file:
disable_vmware_customization: false
Step 3 - Delete the following files that is located under /etc/cloud/cloud.cfg.d/
rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg
rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
Step 4 - Run the cloud-init clean operation:
/usr/bin/cloud-init clean --logs
Step 5 - Shutdown the Guest OS
To help simplify the GuestOS preparation steps, I have created the a quick shell script that you can use that will automatically apply all the required changes listed above and then shutdown the GuestOS.
#!/bin/bash grep "disable_vmware_customization: false" /etc/cloud/cloud.cfg > /dev/null 2>&1 if [ $? -eq 1 ]; then echo "disable_vmware_customization: false" >> /etc/cloud/cloud.cfg fi rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg rm -rf /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg /usr/bin/cloud-init clean --logs shutdown -h now
At this point, you can either apply the vSphere GuestOS Customization directly onto this VM or simply turn this VM into a vSphere Template or Content Library Image to become the base image for deploying new VMs that can then be customized using cloud-init.
Apply GuestOS Customization
Step 1 - Create your desired cloud-init metadata file (must be JSON format) that contains the desired network configuration for your VM. Below is an example configuration which I have named metadata-ubuntu.json and simply configures a static IPv4 address for the network adapter labeled ens160.
{ "instance_id": "vsphere-gosc-cloud-init-configured", "local_hostname": "vsphere-gosc-cloud-init-configured", "network": { "version": 2, "ethernets": { "ens160": { "gateway4": "192.168.30.1", "addresses": ["192.168.30.231/24"], "dhcp4": "false", "nameservers": { "search": ["primp-industries.local"], "addresses": ["192.168.30.2"] } } } } }
Step 2 - Download the vsphere_guestos_customization_using_cloud_init.ps1 PowerShell example and edit the variables to match your environment. You will also need to obtain the MoRef ID for the VM you wish to apply the vSphere GuestOS Customization. You can do this by using the GET VM API or simply looking at the URL when selecting the desired VM using the vSphere UI, which will be in the format of vm-X, where X is some numeric value (e.g. vm-36040).
Step 3 - Run the PowerShell script which will create the vSphere GuestOS Customization spec from the supplied cloud-init metadata file and apply that to the desired VM. If the operation was successful, you should see a message like the following:
The script also has a debug option to display full REST API payload for those interested in using this API with another API client such as REST, python, go, etc.
Step 3 - Finally, the last step is to power on the VM and you should also see the start and completed guest OS customization tasks within the vSphere UI as shown in the screenshot below.
Once the customization has completed, you can verify that the correct networking has been applied to the desired VM.
Troubleshooting
There are other useful logs located in /var/run/cloud-init and /var/lib/cloud/instance that can also be helpful but during my exploration of the new vSphere GuestOS Customization with cloud-init, I found the following logs to be useful in aiding with the debugging.
- /var/log/vmware-imc/toolsDeployPkg.log - Used to confirm that VMware Tools has detected the GuestOS customization CAB file exists in the VM home directory and has been copied into GuestOS
- /var/log/cloud-init-output.log - Used to confirm the actual processing of cloud-init and customization
- /var/run/cloud-init/instance-data.json - Used to confirm metadata payload that cloud-init has processed
- /var/lib/cloud/instance/user-data.txt - Used to confirm the userdata and ensuring its in the right format
Thanks for the detailed post! I followed your instructions and was successfully able to customize an Ubuntu 22 machine using the Cloud-Init GOSC API as per your example.
However, I encountered this issue after rebooting my VM: https://kb.vmware.com/s/article/71264.
Can you confirm whether your also encountered this issue in your environment?
I've reviewed the source code for the cloud-init OVF datasource (https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceOVF.py) and from what I understand it is expected behaviour that due to there being no customization data on subsequent boots, cloud-init falls back to DatasourceNone, which results in DHCP networking.
Your userdata could have script content to disable cloud-init after customisation, I guess?
Thanks for the detailed post! I followed your instructions and was successfully able to customize an Ubuntu 20 and ubuntu 21 machine using the Cloud-Init GOSC API as per your example.
However, I try with Centos 7 - but have error: 400 bad request. ( I see Centos7 cannot upgrade to open-vm-tools version 11.3 and cloud-init 21.1)
I hope u can help me!
your indications differ from original docs (https://cloudinit.readthedocs.io/en/latest/topics/datasources/vmware.html);
the steps 2,3,4 in particular. which is the right one?
One things the VMware documentation doesn't make at all clear is how you put multi-line text (be that cloud-init data or the Linux custom script) into a JSON string value. Reading your script is the metadata "string" value not really a string, but in fact nested JSON?
Oh I just read your screenshot showing the contents of the API request, and apparently the "metadata" field is escaped JSON, and the "userdata" field is YML in a string, with \n for LF?
was helpful, thanks
Hello,
When I send a PUT request to the "/api/vcenter/vm/vm-1000/guest/customization" address, I receive the following error:
{
"error_type": "SERVICE_UNAVAILABLE",
"messages": []
}
I cannot make this work. Is there any chance this is no longer working correctly in 7.0.3 01800?
Turned out that if you try this with the OS version set to Debian 11, it wont work Setting it to Other Linux (64-Bit) fixes the issue and the cloud-init doesn't have an issue. Seems to work fine with Ubuntu 64-Bit as well.
Thanks William, that is all good, however, how can a customer with a few templates in v7 U2 who has just updated to v7 U3 overcome the error message below, which has rendered all their templates unusuable? Is that fixed in V8?
the selected customization specification contains cloud-init data. cloud-init specifications are not supported in vSphere client currently. You should use the API to apply it to the guest OS, or select another specification from the list below.