A pretty common task for vSphere Administrators is to upload or download content from a vSphere Datastore which usually contains ISOs and floppy images. You can initiate the file transfer using the vSphere Web/C# Client, however this process can be quite tedious when having to manually upload several ISOs. Instead, you will probably want to automate this process and and there are a couple of ways in which you can accomplish this. One option, is to go directly to an ESXi host and upload your files but this is not ideal when you have vCenter Server to centrally manage your infrastructure. The second option is to go through vCenter Server, but depending on the implementation, you can potentially add unnecessary load to the vCenter Server if implemented incorrectly.
Let me explain this further with two diagrams and you can decide which implementation you prefer?
In this first implementation, I directly access the file management API which leverages a simple HTTP GET/PUT operation to upload files to a vSphere Datastore. What I found out while transferring the data was that the data actually traverses through the vCenter Server and then onto the ESXi host before writing to the vSphere Datastore. This of course made the data transfer very inefficient not to mention additional bandwidth and load being added to vCenter Server.
I created a sample vSphere SDK for Perl script that demonstrates this inefficent transfer called inefficent-upload-files-to-datastore.pl
Here is sample execution of the script which accepts the name of the vSphere Datacenter, vSphere Datastore, the source file to transfer and the destination path of where the file will be uploaded to:
./inefficent-upload.pl --config ~/vmware-dev/.vcenter55-1 --datacenter Datacenter --datastore vsanDatastore --sourcefile /Volumes/Storage/Images/CentOS-6.4-x86_64-netinstall.iso --destfile ISO/CentOS-6.4-x86_64-netinstall.iso
After talking to some folks about this problem, I learned about a more efficient method as shown in the diagram below.
As you can see, we can still initiate the transfer using the vCenter Server, but the actual data transfer is than sent to one of the ESXi hosts that has access to the vSphere Datastore. To accomplish this, we need to use the AcquireGenericServiceTicket() method which is part of the sessionManager. Using this method, we can request a ticket for a one time HTTP request to connect directly to an ESXi. To upload a file, the request must include the method which in this case will be a PUT operation and the local URL to an ESXi host that has access to the vSphere Datastore.
Here is an example of a URL: https://vesxi55-1.primp-industries.com/folder/ISO/CentOS-6.4-x86_64-netinstall.iso?dcPath=ha-datacenter&dsName=vsanDatastore
- ESXi IP Address/Hostname - In the script, I select the first ESXi host that has access to the vSphere Datastore
- vSphere Datastore Directory - Directory into which the contents of the file will be placed in. In this example, we just have one top-level directory called ISO which must already exist
- Destination file name - The name of the file that should appear in the vSphere Datastore
- Datacenter Name - This should always be ha-datacenter when connecting directly to an ESXi host
- vSphere Datastore - The name of the vSphere Datastore
To demonstrate this functionality, I have created a vSphere SDK for Perl script called efficent-upload-files-to-datastore.pl which accepts the name of the vSPhere Datastore along with the source and destination of where the file will be placed:
./upload-files-to-datastore.pl --config ~/vmware-dev/.vcenter55-1 --datastore vsanDatastore --sourcefile /Volumes/Storage/Images/CentOS-6.4-x86_64-netinstall.iso --destfile ISO/CentOS-6.4-x86_64-netinstall.iso
Hopefully after looking at these two implementations, you will also agree that the second option is the best! One last thing that I would like to point out is that even though we are talking about transferring files to a vSphere Datastore, this method can also be used to efficiently transfer other supported files to an ESXi hosts through vCenter Server as described in this blog article.
This seems like a great time to point out a common practice I use in client environments: a filer volume that offers simultaneous NFS & SMB access. That allows the admin to completely bypass the vSphere environment by using his/her workstation to manage the file store using native tools while the connected ESXi hosts have access for use by guests. An additional admin perk is that the datastore name (eg, "ISOs") makes it readily apparent when a guest is using a file on the datastore.
Though the tip you provide is an excellent one, in my example I was using VSAN datastore so this would not be possible 🙂
I use FileZilla, connect directly to host (ssh enabled) and upload everything very fast.
Good idea with NFS and SMB, but it'll not help if you have VMFS.
Definitely another option. Though the solution listed above does not require SSH which not all customers have enabled and definitely would not work if you're using VMFS or VSAN.
Thanks for the information. This is exactly the problem that i was trying to solve.
But i wanted some help regarding the use of the AcquireGenericServiceTicket() method. While i was able to download the file for the ESX host version 5.5 but it failed for the versions 5.1 and 4.x. Is this method specific to any host version or we need to enable anything on the esx host for this to work.
In the vSphere API documentation, the API has been available sine 5.0, so it would not work with 4.x
Thanks for the reply.
So it should work on all the 5.x hosts. But it seems that this api is not working on the 5.1 hosts. It always fails when trying to acquire the ticket.
Below is the error that i get from the logs.:
"A specified parameter was not correct. url"
I have verified that the url i am using is correct. Are we required to configure any settings on the ESX host for this to work?
No configurations are required on the ESXi host, as long as the VC can communicate to ESXi host through port 443, that is all that is needed. I don't have a 5.1 infrastructure to test on. If you have SNS, you can file an SR to see if GSS can see if there is a known issue w/5.1
Have you had any luck getting this working with 5.1? I have access to a 5.1 and 5.5 setup. With 5.5, I was able to get this working well (after granting my user the Host->Configuration->System Management privilege). However, when I try against my 5.1 setup, I get the same error, even after fixing the permissions.
Thanks for posting this. This is going to help me get around a nasty situation I just ran into... vCenter isn't ignoring disconnected hosts when selecting a host to transfer through, but the disconnected state causes the file upload to fail. Which also leads to a possible improvement to your efficient script.
The mount status for a host doesn't change when a host is disconnected (and likely when its not responding/down). It'd probably be worth verifying the host's runtime.connectionState is good when selecting a host.
I noticed Filezilla worked perfects. I had issues with WinSCCP also where in FileZilla it takes 5 minutes and in the VCenter GUI and WinSCP it was still increasing higher above 2 hours. Simply put FileZilla was a star
Yep Filezilla over SCP is a LOT faster.
Looks like you are using an octet-stream http put - currently doing the same here. I am intermitantly having problems putting large files on some of my testing environments. It is probobly due to wacky network trafficin these environemnts. When I ask the http server for OPTIONS, the http server chooses not to list the options. Do the http puts accept encoding or chunking? Chunking would be sweet.
Hi,
That seems useful, but I see no explanation on how to create that config file ? (for --config)
I'm guessing it has to do with setting up the API, which seems hard enough to install on non-rpm systems, but I can't find an easy example.
Hi, i have a very low bandwidth, 1mb and takes 10hrs to copy iso to datastore on remote esxi.
what is the efficient way to transfer data to remote esxi. USb ? or ..
Very interesting article, thanks William ! Is there a difference on the way vCenter access these datastores comparing vSphere 5.5 C# client and vSphere 6.5 Web Client ? On one of my lab with 2 vCenters being in country A and the ESXi host being in country B :
- When connecting to the vCenter 5.5 through the vSphere C# client from a VM hosted on my ESXi host in country B, transferring a file from the said VM to a local datastore seems to bypass vCenter Control + Data and go directly from ESXi host to the datastore. The transfer is then very fast as being local I guess (between the ESXi host and the storage array)
- When connecting to the vCenter 6.5 through the vSphere Web client from a VM hosted on my ESXi host in country B, transferring a file from the said VM to a local datastore seems to go through the vCenter, then come back to the ESXi host, then to the datastore. The transfer is then very slow, going back and forth between the vCenter in country A and the ESXi host being in country B.
Is this a normal behaviour ? If so, is there a way, in vSphere 6.5 Web Client to avoid going back and forth between vCenter and ESXi host ? Or should I use the provided script and/or WinSCP for example ?
Thanks !
I need to upload ISO files from a python3 script to a vmware datastore.
I know how to deal with HTTP connections but how do I translate Vim:: calls?
BTW: it would be very cool if I could use REST calls.
I found a solution, https://vc/folder is the key. Now I just have to find out if I can use the auth-token from my REST connection. At the moment I use 2 separate connections.
Hello, a little late to the party here.
How about moving a file to a content library?