vSphere 6.0 Update 1

Quick Tip - iPerf now available on ESXi

03.15.2016 by William Lam // 25 Comments

The other day I was looking to get a baseline of the built-in ethernet adapter of my recently upgraded vSphere home lab running on the Intel NUC. I decided to use iPerf for my testing which is a commonly used command-line tool to help measure network performance. I also found a couple of articles from well known VMware community members: Erik Bussink and Raphael Schitz on this topic as well which were quite helpful. Erik's article here outlines how to run the iPerf Client/Server using a pair of Virtual Machines running on top of two ESXi hosts. Although the overhead of the VMs should be negligible, I was looking for a way to benchmark the ESXi hosts directly. Raphael's article here looked promising as he found a way to create a custom iPerf VIB which can run directly on ESXi.

I was about to download the custom VIB and I had just remembered that the VSAN Health Check plugin in the vSphere Web Client also provides some proactive network performance tests to be run on your environment. I was curious on what tool was being leveraged for this capability and in doing a quick search on the ESXi filesystem, I found that it was actually iPerf. The iPerf binary is located in /usr/lib/vmware/vsan/bin/iperf and looks to have been bundled as part of ESXi starting with the vSphere 6.0 release from what I can tell.

UPDATE (10/19/23) - As of ESXi 8.x, you may run into following error when attempting to start running iperf:

iperf3: running in appDom(30): ipAddr = ::, port = 5201: Access denied by vmkernel access control policy

To workaround this, you can run the following command:

esxcli system secpolicy domain set -n appDom -l disabled

and re-enable (use "enforcing" as the value) once you have completed your iperf test both on the server and client ESXi host.

UPDATE (09/20/22) - As of ESXi 7.0 Update 3 20036589 (possibly earlier) and later, you no longer need to make a copy of the iperf3 utility. You can simply run it from /usr/lib/vmware/vsan/bin/iperf3 and you also do NOT have to lower the security by changing ESXi Advanced Setting execInstalledOnly to FALSE

UPDATE (10/02/18) - It looks like iPerf3 is now back in both ESXi 6.5 Update 2 as well as the upcoming ESXi 6.7 Update 1 release. You can find the iPerf binary under /usr/lib/vmware/vsan/bin/iperf3

One interesting thing that I found when trying to run iPerf in "server" mode is that you would always get the following error:

bind failed: Operation not permitted

The only way I found to fix this issue was to basically copy the iPerf binary to another file like iperf3.copy which it would then allow me to start iPerf in "server" mode. You can do so by running the following command on the ESXi Shell:

cp /usr/lib/vmware/vsan/bin/iperf3 /usr/lib/vmware/vsan/bin/iperf3.copy

Running iPerf in "Client" mode works as expected and the copy is only needed when running in "server" mode. To perform the test, I used both my Apple Mac Mini and the Intel NUC which had ESXi running with no VMs.

I ran the iPerf "Server" on the Intel NUC by running the following command:

/usr/lib/vmware/vsan/bin/iperf3.copy -s -B [IPERF-SERVER-IP]

Note: If you have multiple network interfaces, you can specify which interface to use with the -B option and passing the IP Address of that interface.

I ran the iPerf "Client" on the Mac Mini by running the following command and specifying the address of the iPerf "Server":

/usr/lib/vmware/vsan/bin/iperf3 -n 800M -c [IPERF-SERVER]

I also disabled the ESXi firewall before running the test, which you can do by running the following command:

esxcli network firewall set --enabled false

Here is a screenshot of my iPerf test running between my Mac Mini and Intel NUC. Hopefully this will come in handy for anyone needing to run some basic network performance tests between two ESXi hosts without having to setup additional VMs.

Deploying Nested ESXi is even easier now with the ESXi Virtual Appliance

12.14.2015 by William Lam // 92 Comments

Several months back I had built an ESXi Virtual Appliance that allows anyone to quickly stand up a fully functional Nested ESXi VM which includes guest customization such as networking, NTP, syslog, passwords, etc. The virtual appliance was initially built for my own personal use as I found myself constantly rebuilding my lab environment for evaluating and breaking new VMware software. I figured if this was useful for myself, it probably could benefit others at VMware and I posted the details internally on our Socialcast forum. Since then, I have received numerous stories on how helpful the ESXi Virtual Appliance has been for both our Field and Engineering for setting up demos, POCs, evaluations, etc.

[Read more...]

Automating post-configurations for both PSC & VCSA 6.0u1 using appliancesh

11.23.2015 by William Lam // 4 Comments

In vSphere 6.0, we introduced a new command-line option to allow you to automate both the deployment and upgrade of a vCenter Server Appliance (VCSA) and Platform Services Controller (PSC) using a simple JSON configuration file. This has been a very popular request from customers and one that I have been asking for some time now and was glad to see it was finally made available with the VCSA. One thing that was still missing from an Automation standpoint was being able to some basic post-configurations after the initial deployment. Common operations such as adding additional user accounts, configuring SNMP for monitoring or adding proxy server were available but had to be done interactively and manually.

In vSphere 6.0 Update 1, an enhancement was made to the appliancesh interface which will now allow customers to automate the post-configurations of either a VCSA or PSC by simply re-directing a series of appliancesh commands within a file using SSH. Although SSH may not be ideal for all customers and having a programmatic interface via an API is ultimately where we want to get to; This at least allows customers to automate the end-to-end deployment of both the VCSA and PSC as well as covering any additional post-configurations that might be required to stand up a vSphere environment.

To make use of this feature, you simply create a file that contains the list of appliancesh commands that you wish to run on either the VCSA and/or PSC. Here is an example configuration called psc.config (you can name it anything you want):

access.shell.set --enabled false
access.ssh.set --enabled false
ntp.server.add --servers "0.pool.ntp.org,1.pool.ntp.org"
timesync.set --mode NTP
services.restart --name ntp
proxy.set --protocol https --server proxy.primp-industries.com
localaccounts.user.add --email *protected email* --role operator --fullname 'William Lam' --username lamw --password 'VMware1!'
snmp.set --communities public --targets 192.168.1.160@161/public
snmp.enable

Once you have saved the configuration file, you simply SSH to either your VCSA or PSC and re-direct the configuration file by running the following command:

ssh *protected email* < psc.config

Once authenticated, the series of appliancesh commands will be executed and then you will be automatically logged off as seen in the screenshot below.

If you have any feedback in this particular area, please leave a comment as I know both PM/Engineering are interested in hearing your thoughts and what you might want to see in the future in terms of post-configuration of the VCSA and PSC.