Infinite possibilities with new VM Service CloudInit transport for vSphere with Tanzu in vSphere 8

When the VM Service capability (part of Sphere with Tanzu) was first introduced back in vSphere 7.0 Update 2a, I was really excited for the possibilities this feature could unlock for both DevOps personas but also for our VI Admins. Currently, the VM Service can only deploy two specific OVF images (CentOS and Ubuntu) that are pre-built by VMware and distributed from the VMware marketplace.

While the potential for the VM Service is definitely there, our customers and even our partners need the ability to create their own custom images and using approved operating systems that they have built and harden based on the needs of their organizations. Even though I was able to get the VM Service to deploy a non-default image like a Nested ESXi VM using a couple of tricks, there needs to be a much easier and supported way to create and deploy non-default VMware OS images and this is where vSphere 8 can now help 😀

In addition to the existing vmMetadata transport options (ExtraConfig and OvfEnv) for customizing a VM using the VM Service, vSphere with Tanzu using vSphere 8 now supports an additional transport type called CloudInit. This new vmMetadata transport is what allows users to deploy and customize their own OVF images for any Linux operating system that has support for Cloud-Init 21.3 or later. The new CloudInit transport implements the Cloud-init Datasource for VMware GuestInfo feature, which I highly recommend folks give a read as it provides additional background.

OK, enough talking about what the feature is, let's now see VM Service and the new CloudInit transport in action! Imagine you needed to deploy a Rocky Linux 9 VM using the VM Service because that is the new Linux standard for your organization. I chose Rocky Linux 9 as that is a fairly recent Linux OS and is not part of the default OS type that the VM Service supports today, however you can use any Linux distribution that meets the Cloud-init requirements. So, how do you go about accomplishing this?

Step 0 - vSphere with Tanzu must already be enabled and configured on a vSphere 8 Cluster (vCenter Server and ESXi).

Step 1 - Install Rocky Linux 9 OS into a standard VM using any method you are comfortable with (manual or automated).

Step 2 - Prepare the OS by following these steps outlined in the "Prepare GuestOS for Customization" section and once the VM is shutdown, you are now ready to start using it with the VM Service.

Step 3 - Convert the Rocky Linux 9 VM into a vSphere Content Library OVF Template. If you do not have a vSphere Content Library, go ahead and create a local library and then right click on the VM and select the Clone VM as a Template Library and select OVF as the type and the desired content library to transfer the VM image to.

Step 4 - Associate the content library to the VM Service by navigating to Workload Management and then select the vSphere Namespace that you have already created. See the official VMware documentation if you need more guidance.

Once the content library has been sync'ed by the VM Operator, which powers the VM Service, you should be able to see your custom Rocky Linux 9 VM when running the following kubectl command:

kubectl get vmimage

Note: You will see that these custom OVF images will show image not supported and that is expected as these are not official images produced by VMware.

At this point we are now ready to deploy our custom Rocky Linux 9 image using the VM Service!

Step 5 - Next, we create our VM Service deployment YAML for our Rocky Linux 9 image that we wish to deploy. Below is an example of what the file should look like and you will need to replace the environment specific variables with the name within your own setup. The networking is already handled as part of the VM Service, so we only need to focus our attention on the OS or application specific configuration, which can either be a ConfigMap or Secret that contains the Cloud-init user-data. For our customization example, I will simply create the following file /tmp/vm-service-message.txt in the guest OS that contains a very special message.

Note: Please refer to the official Cloud-init documentation for more details and what is supported in the user-data field.

Here is an example deployment YAML that uses a ConfigMap to store the user-data:

apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
  name: rocky9-01
  labels:
    app: rocky9-01
  annotations:
    vmoperator.vmware.com/image-supported-check: disable
spec:
  imageName: rocky-linux-9
  className: best-effort-medium
  powerState: poweredOn
  storageClass: tanzu-storage-policy
  networkInterfaces:
  - networkType: vsphere-distributed
    networkName: workload-1
  vmMetadata:
    configMapName: rocky9-01
    transport: CloudInit
---
apiVersion: v1
kind: ConfigMap
metadata:
    name: rocky9-01
data:
  user-data: |
    #cloud-config
    write_files:
    - encoding: gz+b64
      content: H4sIAA6K1GIAA/NIzcnJV0grys9VCM/MyclMzPVJzNVLzs/lAgD/QoszGgAAAA==
      path: /tmp/vm-service-message.txt

Here is an example deployment YAML that uses a Secret to store the user-data:

apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
  name: rocky9-01
  labels:
    app: rocky9-01
  annotations:
    vmoperator.vmware.com/image-supported-check: disable
spec:
  imageName: rocky-linux-9
  className: best-effort-medium
  powerState: poweredOn
  storageClass: tanzu-storage-policy
  networkInterfaces:
  - networkType: vsphere-distributed
    networkName: workload-1
  vmMetadata:
    secretName: rocky9-01
    transport: CloudInit
---
apiVersion: v1
kind: Secret
metadata:
    name: rocky9-01
stringData:
  user-data: |
    #cloud-config
    write_files:
    - encoding: gz+b64
      content: H4sIAA6K1GIAA/NIzcnJV0grys9VCM/MyclMzPVJzNVLzs/lAgD/QoszGgAAAA==
      path: /tmp/vm-service-message.txt

Note: The use of the vmoperator.vmware.com/image-supported-check: disable annotation is required and allows the VM Service to deploy our custom image versus one that has been built by VMware. In the future, this annotation may not be needed.

Once you have created the deployment YAML file, you can begin the deployment by running the following command passing in the name of the VM Service deployment YAML using the following:

kubectl apply -f rocky-linux-9-vm.yaml

If everything was setup correctly, you should see the VM Service deploy a new VM based on the selected VM template in your content library. Once the networking is up, you can verify that customization has been applied by SSH'ing to the VM's IP Address and locating the file that should be stored in /tmp/vm-service-message.txt

For those of you that are familiar with the VM Service, you will find the new Cloudinit transport to be very straight forward and that is by design. The only requirement is a Linux operating system that can run Cloud-init 21.3 or later and you can deploy those VMs using the VM Service.

Some of you may have also noticed in the various screenshots above, that I have deployed both Linux (Rocky Linux 9 & Ubuntu 22.04) and Windows (10 & 2019) images. While you CAN use the VM Service in vSphere 8 to deploy Microsoft Windows images, static network configuration for Windows is currently not supported with Cloud-Init, which would prevent the VM Service from configuring the networking on the Windows VM with the allocated workload address for proper connectivity. This is why you see a 192.168.x.x address rather than a 10.20.x, which is the actual vSphere with Tanzu workload address range. I know this is an area the VM Service Engineering team is looking into and hopefully in the near future, we will also be able to deploy a fully customized Windows VM as well!

The VM Service is now ready for prime time and I can not wait to hear how our customers will leverage this exciting new feature, whether that is for Developers or DevOps persona to VI Admins who are interested in a much simpler and modern way to deploy VM workloads using a Kubernetes-based interface.