An interesting vSphere Lifecycle Manager (vLCM) question that has already come up a few times in the past two weeks is whether vLCM can remediate (patch/upgrade) a specific group of ESXi host(s)?
Today, there are two ways in which vLCM can remediate ESXi hosts within a vSphere Cluster. The first is to simply remediate all ESXi hosts within a vSphere Cluster using the Remediate All option.
If a user wishes to exclude specific ESXi host(s) from being remediated, then the only other option is to manually remediate each individual host. To do so, select an ESXi host from the list and then use the Remediate option in the Actions menu as shown in the screenshot below.
For a small environment, manual remediation might be okay for those willing to click through the vSphere UI but it is certainly less than ideal and you are also not taking advantage of the automation and orchestration that vLCM provides in the first place.
Luckily, there is a third option which is to use the vLCM REST API to specify the desired list of ESXi host(s) that want to patch/upgrade and best of all, this is fully automated!
The vLCM Apply Clusters Software Task API is what the vLCM UI calls when remediating all or a specific ESXi host within a vSphere Cluster. The API payload accepts a list of ESXi host Managed Object Reference (MoRef) IDs, so you can see how this API can be used to selectively choose which set of ESXi hosts to remediate.
Note: While you can specify a list of ESXi hosts to patch/update, in vSphere 7.x, the remediation is still performed sequentially with one host at a time. In vSphere 8.x, parallel remediation will be supported and can be configured from 1 to 96. From my limited testing, it also seems the order of ESXi MoRef ID is also the order in which vLCM starts the remediation.
To demonstrate how this vLCM API works, below are examples using the vCenter REST API via cURL and PowerCLI.
cURL
# Get VMware API Session ID
curl -X POST -H "Authorization: Basic ${BASIC_AUTH_TOKEN}" https://vcsa.primp-industries.local/api/session
# Patch specific ESXi hosts
curl -X POST -H "vmware-api-session-id: ${VMWARE_SESSION_ID}" -H "Content-Type: application/json" \
-d '{"accept_eula":true,"hosts":["host-84028"]}'\
"https://vcsa.primp-industries.local/api/esx/settings/clusters/${CLUSTER_ID}/software?action=apply&vmw-task=true"
PowerCLI
Step 1 - Connect using both the Connect-VIServer and Connect-CISServer cmdlet. The next section will store the desired name of the vSphere Cluster you wish to perform the vLCM operation on and retrieve the vSphere Cluster Managed Object Reference (MoRef) ID which will be needed in Step 3.
$CLUSTER_NAME="vLCM-Cluster"
$CLUSTER_ID=(Get-Cluster -Name $CLUSTER_NAME).Id.replace("ClusterComputeResource-","")
Step 2 - Retrieve the the desired ESXi Managed Object Reference (MoRef) IDs that you wish to patch using vLCM.
(Get-Cluster -Name $CLUSTER_NAME | Get-VMHost) | select Name,Id
Step 3 - Create a $hostIds variable which is an array containing the desired ESXi MoRef IDs from the previous step. After that, the next section creates the vLCM apply spec and then performs the apply/patch operation.
# List of ESXi host (MoRef ID) to patch from Step 2
$hostIds = @("host-84025")
# Create vLCM Apply Spec
$esxClusterSoftwareService = Get-CisService -Name com.vmware.esx.settings.clusters.software
$spec = $esxClusterSoftwareService.Help.'apply$task'.spec.Create()
$spec.accept_eula = $true
$spec.hosts = $hostIds
# Run vLCM Apply/Patch operation
$esxClusterSoftwareService.'apply$task'($CLUSTER_ID, $spec)
I think a nice enhancement for the vLCM UI in the future is to offer a menu that automatically selects all ESXi hosts to remediate, but allowing a user to override and uncheck or select specific hosts to remediate. For now, the vLCM API is the best option over manually remediating each host which can be extremely time consuming with management of maintenance mode enter/exit, reboot, etc.
Hi William
Any idea what (service?) I can restart on the VC, if remediation on a specific node is hanging ? There are prod workloads on the esxi, so I'd prefer not restart services there.
You can restart vmware-vpxd which is main VC service. Warning, any task that may be in flight will automatically get cancelled. You may also want to consider file SR and have support take a look if you have SnS
Hey William,
We are developing a vRO workflow to patch individual ESXi hosts utilizing the vLCM API - this is exactly what I needed. Thanks!
I do have a question about the vmware-api-session-id key. Fully matured we will have multiple workflows running in parallel for different clusters, each with the need to generate and use a key but no way to pass a previously generated key to a new instance of the workflow.
When a new key is generated using the same VC credentials as a previous key, is the old key overwritten? Or, in other words, will the old key stop working once a new one is generated?
Looks like the answer is here:
https://blogs.vmware.com/performance/2022/07/endpoint-limits-for-concurrent-rest-requests-from-vcenter-apis.html
How is the $BASIC_AUTH_TOKEN obtained? I know it's base64 encoded and think it should be uid:pwd, but it's getting vague when realms come into play. I'm assuming it's the same method of authentication as for recent versions of vCenter 7.0.
See https://developer.vmware.com/apis/vsphere-automation/latest/cis/api/session/post/ and yes, Basic Auth is industry standard b64 encoding of user:pass 🙂
Hello,
While Remediation the vib install is failing due to invalid or corrupted signature error.
befor we use to provide --nosigcheck while installing the vib, but now how we can do this please provide your view?
Hi William,
I have a use case where I need to configure an image and remediate hosts that are standalone - not part of a cluster. I have the following code so far:
$datacenterName = "(insert dc name here)"
$esxiImageName = "8.0 U1c - 22088125"
$esxiComponentName = "VMware Tools Async Release"
$esxiComponentVersion = "12.3.5"
$vendorAddOnName = "HPE-Custom-AddOn"
$vendorAddOnVer = "801.0.0.11.4.0-6"
$esxiBaseImage = Get-LcmImage -Type BaseImage -Version $esxiImageName
$esxiComponent = Get-LcmImage -Type Component | where {$_.Name -eq $esxiComponentName -and $_.Version -eq $esxiComponentVersion}
$esxiVendorAddOn = Get-LcmImage -Type VendorAddOn -Name $vendorAddOnName -Version $vendorAddOnVer
Set-VMHost -VMHost esx01.lab.local -BaseImage $esxiBaseImage -Component $esxiComponent -VendorAddOn $esxiVendorAddOn
This all runs great however I can't find a way to remediate the image because in step 3 of this article a Cluster is needed.
If you look at Step 3, you'll see the API endpoint we're calling from vLCM is "clusters.software" which is the vLCM API to apply at Cluster level (to be expected).
For standalone ESXi hosts, you need to refer to vLCM API docs and specifically there's "hosts.software" https://developer.vmware.com/apis/vsphere-automation/latest/esx/api/esx/settings/hosts/host/softwareactionapplyvmw-tasktrue/post/ which takes the same payload but is for standalone ESXi host, so you just need to use the correct API for the operation you're interested in.
Give that a shot
I believe I have what I need:
$vmhost = Get-Vmhost esx01.lab.local
$spec=Initialize-SettingsHostsEnablementSoftwareEnableSpec -SkipSoftwareCheck $false
Invoke-SetHostEnablementSoftwareAsync -Host $vmhost.ExtensionData.MoRef.Value -SettingsHostsEnablementSoftwareEnableSpec $spec
Then to remediate:
$spec = Initialize-SettingsHostsSoftwareApplySpec -AcceptEula $true
$vmhost = Get-Host -Name $hostName
Invoke-ApplyHostSoftwareAsync -Host $vmhost.ExtensionData.MoRef.Value -SettingsHostsSoftwareApplySpec $spec
Hi WIlliam,
Going back to running this against a host in a cluster, when I run $esxClusterSoftwareService.'apply$task'($CLUSTER_ID, $spec) I get the following:
A server error occurred: 'com.vmware.vapi.std.errors.invalid_argument': . Check $Error[0].Exception.ServerError for more details.
At line:1 char:1
+ $esxClusterSoftwareService.'apply$task'($CLUSTER_ID, $spec)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], CisServerException
+ FullyQualifiedErrorId : VMware.VimAutomation.Cis.Core.Types.V1.CisServerException
If I check the contents of the variable $spec:
commit hosts accept_eula
------ ----- -----------
host-1937934 True
Hi William,
Found a solution in case it helps someone else:
$Spec = Initialize-SettingsClustersSoftwareApplySpec -Hosts $vmhostmof -AcceptEula:$true
Invoke-ApplyClusterSoftwareAsync -Cluster $CLUSTER_ID -SettingsClustersSoftwareApplySpec $Spec
$CLUSTER_ID=(Get-Cluster -Name $CLUSTER_NAME).Id.replace("ClusterComputeResource-","")
isnt its a bit ugly?
$CLUSTER_ID=((Get-Cluster -Name $CLUSTER_NAME).ExtensionData.MoRef).value