I was helping a fellow colleague yesterday with an OVA question and I came to learn about an upcoming feature in the popular OVFTool utility that would allow for a new method of uploading a remote OVF/OVA to either a vCenter and/or ESXi endpoint.
Historically, when you upload an OVF/OVA whether that is stored locally or remotely from a URL, the data path will actually transfer through the system running the OVFTool between the source and destination, which is ultimately the ESXi host which performs the actual download. Although the OVF/OVA data is not actually stored on your local system, the traffic is proxied through your system and can add an unnecessary hop if the remote OVF/OVA URL can directly be accessed by ESXi host.
A new --pullUploadMode flag has been introduced in the latest OVFTool 4.4.1 release, which will allow ESXi host to directly download (pull) from the remote OVF/OVA URL, assuming it has connectivity. In addition to version of OVFTool, you will also need to have either ESXi 6.7 or 7.0 environment for this new feature to work.
Disclaimer: Although this feature is available in latest OVFTool release, it is still under development and should be considered a Beta feature in case folks are interested in trying it out.
Since the ESXi host is directly downloading from the remote source, there are two additional security verification that has already been implemented. The first is an additional vSphere Privilege called "Pull from URL" which is under the vApp section. Without this, you will get a permission denied error.
Secondly, in addition to specifying the new CLI option, you will also need to provide another flag called --sourceSSLThumbprint which should include the SHA1 hash of endpoint hosting the OVF/OVA. This is an additional verification to ensure the validity of the server hosting the OVF/OVA.
Here is an example of deploying my latest ESXi 7.0 Update 1 Virtual Appliance OVA which is remotely hosted. The quickest way to obtain the SHA1 thumbprint is simply opening browser to based URL which is https://download3.vmware.com/
You will need to replace the space with ":" (colon), so the final string should look like BA:C6:4E:D9:AD:D4:53:B5:86:5A:5D:70:36:CF:89:93:D1:6C:F9:63
Note: The SHA1 thumbprint example shown above is only valid as of Oct 2020, as TLS certificates are replaced periodically, the SHA1 hash will change.
Here is an example OVFTool command to deploy from the remote URL
ovftool \
--X:logFile="ovftool.log" \
--acceptAllEulas \
--allowAllExtraConfig \
--allowExtraConfig \
--noSSLVerify \
--sourceSSLThumbprint="B2:52:9E:4D:57:9F:EA:53:4D:A0:0B:7F:D4:7E:55:91:56:C0:64:BB" \
--name="Nested-ESXi-7.0-Update-1-Appliance" \
--datastore=sm-vsanDatastore \
--net:"VM Network"="VM Network" \
--pullUploadMode \
https://download3.vmware.com/software/vmw-tools/nested-esxi/Nested_ESXi7.0u1_Appliance_Template_v1.ova \
'vi://*protected email*:[email protected]/Primp-Datacenter/host/Supermicro-Cluster'
If we switch over to the vSphere UI, we should see a new task called "Download remote files" which indicates the new pull method is being leveraged. One thing to note is that because ESXi is now performing the download directly, progress may not be known by the OVFTool client, since it is not longer the source for the data transfer. Another thing to be aware of is that OVFTool itself has built-in retry logic in case there is a slight interruption during the data transfer with the current mechanisms. In the "pull" scenario, there is no retry by ESXi and so depending on connectivity, its possible deployments can fail and complete re-transfer would be required.
This is a great improvement! Thanks for pointing it out! Also, William, congrats on your 9 year anniversary as a VMware employee!
Very interesting.
But from your description I can imagine a number of possible improvements, eg
Use a directory service to list and select available machines
A label alias that maps to the ivf id instead of the number string,
Progress bar or similar displaying progress of remote download
Thanks for writing this post William.
Does this mean that I could get vSphere to pull directly from a jFrog Artifactory to get an OVA to deploy in vCenter?
If so, how exactly?
If jFrog provides an accessible URL, then yes 🙂
I'd recommend you give it a try and experiment a bit yourself
Hello, William!
Ovftool can pullUploadMode. I really want to teach this mode to the community.vmware.vmware_deploy_ovf ansible module. Do you know if there is public documentation for this API method?
Yes, see https://vdc-download.vmware.com/vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/vim.HttpNfcLease.html#pullFromUrls
Hey William.
I went through the documentation and rewrote community.vmware.vmware_deploy_ovf to use this method. However, during testing, I ran into a problem when deploying from an HTTPS source.
Loading vCenter fails with "HTTP communication could not be completed with status 404" error.
At first I thought that I was passing sslThumbprint in the wrong format to the HttpNfcLeaseSourceFile dictionary and tried different options (including the one in your example SHA-1 "XX:XX:..:XX"). However, I was surprised to find that ovftool behaves in a similar way, returning the same error:
Error: Message is: HTTP communication could not be completed with status 404,
Fault cause: vim.fault.HttpFault
Then I thought that maybe the problem is in my HTTPS server (I use regular IIS based on Windows Server 2022 in my lab) and tried to reproduce your own example (changing sslThumbprint to the current one):
ovftool --
X:logFile="ovftool.log" --acceptAllEulas --allowAllExtraConfig --allowExtraConfig --noSSLVerify --sourceSSLThumbprint="B2:52:9E:4D:57:9F:EA:53:4D:A0:0B:7F: D4:7E:55:91:56:C0:64:BB" --name="Nested-ESXi-7.0-Update-1-Appliance" --datastore=vsanDatastore --net:"VM Network"="MNG" --pullUploadMode https://download3.vmware.com/software/vmw-tools/nested-esxi/Nested_ESXi7.0u1_Appliance_Template_v1.ova vi://*protected email*@vc-mng.lab.ihumster.ru/LAB/host/MNG/
But here I got the same problem. Deciding to check if I have problems with vCenter and the Internet, I launched Deploy OVF from vSphere Client using your link to Nested_ESXi7.0u1_Appliance_Template_v1.ova from the example above.
Deploy completed successfully.
It turns out that something was changed in the API and it was not reflected in the documentation? I really need help with this issue.
Thank you!
The example SHA1 hash is/was only valid as of the original blog post which was back in 2020 🙂 Since then, the TLS certificate has been replaced, so you'll need to follow the instructions on obtaining the new SHA1 for VMware site. So everything is behaving as expected
William, I wrote that I replaced the thumbprint with the current one. This does not work! Try it yourself
I _did_ try it, hence reply and it works as expected
You can get me work example?
OMG. I forgot about httpClient in Firewall on hosts. Sorry.