Clearing resource locks with VCF 9.0 Installer

While deploying a new VMware Cloud Foundation (VCF) 9.0 Fleet, I had observed the VCF Installer task for checking the deployment status for VCF Automation (VCFA) had taken longer than expected and it eventually failed to complete.

While the deployment of VCFA will take some time, especially if you are using consumer grade hardware like I was, I was actually monitoring the granular progress within VCF Operations (VCFO) under Fleet Management->Lifecycle->VCF Management->Tasks and you should see an active task labeled VCF Automation which is representing the deployment of VCFA.

Perhaps it was my impatience ... but I had noticed the VCFA task had already completed but the VCF Installer still did not progress further after waiting maybe 10-15 minutes.

I had the (not so) brilliant idea of attempting to force a re-check of the task by simply rebooting the VCF Installer ...

Once the VCF Installer had started back up, I figured I could just retry the deployment and it should pickup where it had left off and see that VCFA deployment has complete and then complete the VCF installation!

Nope ... I got myself into a new problem because retry would always fail with the following error message: Failed to acquire resource locks

You may have noticed the VCF Installer will acquire various resource lock for the different operations and I guess as part of the forced reboot, the lock was still held and was not properly released 🤦

Fortuatnely, I was able to locate some documentation regarding resource lock management and there was a way to clear the resource lock using the VCF Installer API.

Disclaimer: If this is for a Production environment, please open an SR with Broadcom Support to get further guidance.

Step 1 - We need to SSH to the VCF Installer since that will be the quickest way to invoke the VCF Installer Resource Locks API

Step 2 - We need to authenticate the VCF Installer endpoint and retrieve an access token which you can run the following cURL snippet (replace the password your VCF Installer password):

TOKEN=$(curl -s -H 'Content-Type:application/json' https://localhost/v1/tokens -d '{"username" : "admin@local","password":"VMware1!VMware1!"}' -k | jq -r '.accessToken')

Step 3 - We now need to retrieve the resource that is currently locked, which we can do so by running the following cURL snippet:

curl -k -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" -X GET https://localhost/v1/resource-locks

Here is a screenshot of what to expect from running the above command:

You should ideally only get back one resource lock and in my example, it contained the following:

{
    "elements": [
        {
            "id": "037f4c56-bce5-461e-8836-0e8708cf6bb8",
            "resourceType": "system",
            "resourceId": "SYSTEM",
            "resourceName": "SYSTEM",
            "operationId": "252db974-ee15-4a77-bf55-25a9f14d5282",
            "serviceId": "elm-drift-detection"
        }
    ]
}

Step 4 - From the above output, we need to construct a new payload to remove the resource lock which must be in the following format:

{
    "serviceId": "elm-drift-detection",
    "resources": [
        {
            "resourceType": "system",
            "resourceName": "SYSTEM"
        }
    ],
    "operationId": "252db974-ee15-4a77-bf55-25a9f14d5282"
}

You create your payload in a file and then reference that in the cURL command or you can do this with one-liner cURL like the following:

curl -k -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" -X DELETE https://localhost/v1/resource-locks -d '{"serviceId": "elm-drift-detection","resources": [{"resourceType": "system","resourceName": "SYSTEM"}],"operationId": "252db974-ee15-4a77-bf55-25a9f14d5282"}'

Here is a screenshot if the command complete successfully:

With the resource lock now removed, I was able to restart the deployment task using the VCF Installer and immediately it was able to validate the VCFA deployment had completed and took me the end of the VCF Installer workflow! 😮‍💨

Perhaps I should be a bit more patient next time ... but at least I learned something new! 😅

Thanks for the comment!Cancel reply