ESXi – Page 139

How to efficiently transfer files to Datastore in vCenter using the vSphere API?

06.18.2014 by William Lam // 19 Comments

A pretty common task for vSphere Administrators is to upload or download content from a vSphere Datastore which usually contains ISOs and floppy images. You can initiate the file transfer using the vSphere Web/C# Client, however this process can be quite tedious when having to manually upload several ISOs. Instead, you will probably want to automate this process and and there are a couple of ways in which you can accomplish this. One option, is to go directly to an ESXi host and upload your files but this is not ideal when you have vCenter Server to centrally manage your infrastructure. The second option is to go through vCenter Server, but depending on the implementation, you can potentially add unnecessary load to the vCenter Server if implemented incorrectly.

Let me explain this further with two diagrams and you can decide which implementation you prefer?

In this first implementation, I directly access the file management API which leverages a simple HTTP GET/PUT operation to upload files to a vSphere Datastore. What I found out while transferring the data was that the data actually traverses through the vCenter Server and then onto the ESXi host before writing to the vSphere Datastore. This of course made the data transfer very inefficient not to mention additional bandwidth and load being added to vCenter Server.

I created a sample vSphere SDK for Perl script that demonstrates this inefficent transfer called inefficent-upload-files-to-datastore.pl

Here is sample execution of the script which accepts the name of the vSphere Datacenter, vSphere Datastore, the source file to transfer and the destination path of where the file will be uploaded to:

./inefficent-upload.pl --config ~/vmware-dev/.vcenter55-1 --datacenter Datacenter --datastore vsanDatastore --sourcefile /Volumes/Storage/Images/CentOS-6.4-x86_64-netinstall.iso --destfile ISO/CentOS-6.4-x86_64-netinstall.iso

After talking to some folks about this problem, I learned about a more efficient method as shown in the diagram below.

As you can see, we can still initiate the transfer using the vCenter Server, but the actual data transfer is than sent to one of the ESXi hosts that has access to the vSphere Datastore. To accomplish this, we need to use the AcquireGenericServiceTicket() method which is part of the sessionManager. Using this method, we can request a ticket for a one time HTTP request to connect directly to an ESXi. To upload a file, the request must include the method which in this case will be a PUT operation and the local URL to an ESXi host that has access to the vSphere Datastore.

Here is an example of a URL: https://vesxi55-1.primp-industries.com/folder/ISO/CentOS-6.4-x86_64-netinstall.iso?dcPath=ha-datacenter&dsName=vsanDatastore

ESXi IP Address/Hostname - In the script, I select the first ESXi host that has access to the vSphere Datastore
vSphere Datastore Directory - Directory into which the contents of the file will be placed in. In this example, we just have one top-level directory called ISO which must already exist
Destination file name - The name of the file that should appear in the vSphere Datastore
Datacenter Name - This should always be ha-datacenter when connecting directly to an ESXi host
vSphere Datastore - The name of the vSphere Datastore

To demonstrate this functionality, I have created a vSphere SDK for Perl script called efficent-upload-files-to-datastore.pl which accepts the name of the vSPhere Datastore along with the source and destination of where the file will be placed:

./upload-files-to-datastore.pl --config ~/vmware-dev/.vcenter55-1 --datastore vsanDatastore --sourcefile /Volumes/Storage/Images/CentOS-6.4-x86_64-netinstall.iso --destfile ISO/CentOS-6.4-x86_64-netinstall.iso

Hopefully after looking at these two implementations, you will also agree that the second option is the best! One last thing that I would like to point out is that even though we are talking about transferring files to a vSphere Datastore, this method can also be used to efficiently transfer other supported files to an ESXi hosts through vCenter Server as described in this blog article.

An alternate way to inject OVF properties when deploying virtual appliances directly onto ESXi

06.16.2014 by William Lam // 13 Comments

I recently published an article demonstrating how to inject OVF properties into the VCSA and other virtual appliances when deploying directly onto ESXi using an unreleased version of ovftool (4.0). A fellow reader by the name of VirtualJMills, as he is known on Twitter left an interesting comment using an alternate solution which I thought was actually pretty clever!

[Read more...]

Two coredump partitions in ESXi 5.5?

06.12.2014 by William Lam // 8 Comments

A couple of days back I had to re-install ESXi on a physical host for some troubleshooting purposes and while looking at the partitions on the disks using ESXCLI, I noticed the fresh ESXi installation had created two coredump partitions.

I was quite surprised to see two, since normally you would just have one configured. I even asked a colleague if he had ever see this before and he had not, so I wanted to double check that there was in fact two coredump partitions being created which I verified by using partedUtil.

As you can see from the screenshot above, there are definitely two coredump partitions. I took a look at our vSphere documentation, but did not find any mention of this. I decided to look internally and found that this is actually a new behavior that was introduced in ESXi 5.5. From what I can tell, the second coredump partition which is 2.5GB was created to ensure that there was sufficient space to handle ESXi hosts configured with a huge amount of memory (up to 4TB) if a coredump were to occur. This new coredump partition is only created on a fresh ESXi install, for upgrade scenarios the original partition structure is preserved. I suspect even on the fresh install, the original coredump partition was kept for potential backwards compatibility.

This definitely made sense given the reason. I guess this actually raises another interesting point from an operational point of view that though upgrades may be preferred, there are also good reasons to perform a fresh install over an upgrade. In this case, to ensure we do not break past requirements/assumptions, we could not just automatically expand or create a larger coredump partition to adhere to new requirements. This is actually not the first instance of this, here are two additional examples in which a fresh installation would have potentially yielded a more optimal environment: