WilliamLam.com

Quick Tip - lldpnetmap, a handy utility to map pNic to pSwitch on ESXi

05.20.2014 by William Lam // 8 Comments

Last week while attending VMware's R&D Innovation Offsite (RADIO), I ran into Christian Dickmann, who as many of you know works on the VSAN team. During our discussion, he mentioned a nifty little utility called lldpnetmap that he had used recently. This utility is found within the ESXi Shell and provides a quick and easy way to display the mapping between an ESXi hosts physical network interface to the physical switch they are connected to using LLDP (Link Layer Discovery Protocol). This is similar to what Cisco's proprietary discovery protocol (CDP) provides, but only details about the physical switch.

CDP has been supported with vSphere Standard Switches for quite sometime now, but LLDP support was only added recently with the introduction of the vSphere Distributed Switch. Chris Wahl has a great article here on why you should enable either CDP/LLDP and the benefits you get with it. For customers who are running non-Cisco switches, lldpnetmap is a great way to quickly figure out which physical switch your ESXi hosts are connected to, especially useful during troubleshooting where every minute counts.

There are actually two ways in which you can run the lldpnetmap utility. The first method is by running it within the ESXi Shell using the following command:

lldpnetmap

The command takes about about 30-60 seconds to run and if successful, you should see the name of the physical network switch and the vmnic (pNIC) that they are connected to.

Here is a screenshot of what that output looks like:

The second method is actually how Christian had been using the command which is through RVC. Using the vsan.lldpnetmap command, you can specify an individual ESXi host or an entire vSphere Cluster. Even though the command is under the VSAN namespace, you do not need to have VSAN enabled to use the command.

Here is a screenshot of what that output looks like:

Note: If you do not see any output, you are most likely connected to a Cisco switch or to a non-managed switch that does not support LLDP.

This is one utility I will be sure to remember the next time I need to troubleshoot a networking issue. Thanks for sharing this handy tidbit Christian!

How to run the VSAN Observer in "collection" mode in the background?

05.18.2014 by William Lam // 1 Comment

The VSAN Observer is a very powerful tool that allows you to get in-depth performance analysis of your VSAN environment. One of the really useful feature is the ability to run the VSAN Observer in "collection" mode by using the --generate-html-bundle option. Something that I have noticed when running the VSAN Observer in collection mode is that you not close the current SSH session, else the collection will stop. I have even tried running the VSAN Observer using RVC's not very well known "script" feature and then back-grounding the process, but after a minute or so the collection also just stops.

The only workaround that I have found is by using Screen, a full-screen windows session manager usually found on most Linux/UNIX and Mac OS X systems. Having used Screen in the past life as a Systems Administrator, I can say it is an extremely useful tool when needing to perform long running tasks and not have to worry about your SSH session being disconnected. You can start a session, disconnect and then re-connect at a later time to monitor the progress.

If you are on a Mac, then Screen should have already been installed. Below are the steps to run the VSAN Observer in the VCSA using Screen:

Step 1 - Start screen and give the session a name such as "VSAN-Observer" for example:

screen -S VSAN-Observer

Step 2 - SSH to your VCSA and login to RVC and start the VSAN Observer using the collection mode as you normally would. For step by step instructions, check out Rawlinson Rivera's article here on setting up the VSAN Observer.

Step 3 - Once the VSAN Observer is running, enter the following key combinations which will detach your Screen session:

Crtl+a d

Step 4 - To list the available Screen sessions, you can run the following command:

screen -list

Step 5 - To re-attach to your Screen session, you will need to specify the session name. In our example, it was called VSAN-Observer:

screen -r VSAN-Observer

An alternative to Step 2, instead of running the VSAN Observer interactively, I actually prefer to run the VSAN Observer using RVC's script option. It is just less typing for me and makes it easy to collect stats across multiple VSAN environments

To do so, you will need to create a script file that contains the following:

# William Lam
# www.virtuallyghetto.com
# RVC script for running VSAN Observer

datacenter_name = "VSAN-Datacenter"
cluster_name = "VSAN-Cluster"
vsan_html_output_directory = "/storage/core"
vsan_observer_runtime = "1"

# Do not edit beyond here #

puts "Enabling VSAN Observer collection for: #{cluster_name} ..."
rvc_exec("vsan.observer --run-webserver --force --generate-html-bundle #{vsan_html_output_directory} --max-runtime #{vsan_observer_runtime} /localhost/#{datacenter_name}/computers/#{cluster_name}")

The RVC script option actually accepts a Ruby script to execute and if we take a look at the script, we are just passing some arguments to the vsan.observer command.

To use the RVC script instead of interactively logging in, you can run the following command:

rvc -s [SCRIPT-NAME] [USERNAME:PASSWORD]@localhost

I think a nice feature enhancement to the VSAN Observer is the ability to automatically background the collection process without having to rely on the existing SSH connection, perhaps this is something Christian may consider for a future update to RVC 🙂 In the meantime, this is a pretty decent work around

Configuring a "Whitelist" for VM advanced settings in vCloud Director

05.10.2014 by William Lam // 3 Comments

The ability to preserve a Virtual Machine's "virtual hardware personality" during an export has been around since the release of vSphere and vCloud Director 5.1 also known as a Lossless OVF Export. However, when it comes time to import that Virtual Machine back into your environment, will it be a "lossless" import? For vSphere, the answer is yes as long as you click on the "Accept extra configuration options" during the import of your OVF/OVA in the vSphere Web Client.

For vCloud Director however, the answer is sadly no. If you try to import an OVF/OVA that contains additional VM advanced settings, vCloud Director will automatically strip away these parameters before importing the Virtual Machine. As annoying as this is for end users, there is a very good reason why this happens.

In a public cloud you do not know who your users are and whether they have malicious intentions. Since the "Extra Configuration" of a Virtual Machine can not be validated, the safest way to guard against this type of a security risk is by sanitizing the content prior to upload.

Although I have known about this default behavior in vCloud Director for some time now, it was only in a recent meeting with some of our internal folks managing our OneCloud environment did I realize how annoying this behavior was for "private" clouds as well. I was looking to import an existing OVF that contains several VM Advanced settings, but realized this would not be supported due to the sanitization of the OVF.

The other issue is that vApps within OneCloud are also exported/re-imported into other vCloud Director environments. Even if our Cloud Administrator were to make a one time exception and manually add these these VM Advanced Settings, they would automatically be stripped away the next time they were re-imported into another environment. After our meeting, I thought there had to be a better way! (at least I hope)

Disclaimer: This may or may not be officially supported by VMware, please use at your own risk.

In doing a quick bit of research, it turns out this is actually a pretty elegant solution that allows you to create a "Whitelist" of trusted VM advanced settings that would be accepted as part of the import process in vCloud Director. To configure this "Whitelist", you will need to add a new category called ExtraConfigWhitelist into the vCloud Director Database for each and every VM advanced setting that you wish to allow.

To demonstrate this functionality, I have the latest vCloud Director appliance (I believe this should also work on 5.1 as well) running and I would like to add the following VM advanced settings to the whitelist:

scsi0:0.virtualSSD
scsi0:1.virtualSSD
scsi0:2.virtualSSD
disk.enableUUID

Before starting, you will need to SSH to our vCloud Director instance and stop the VCD service by running the following command:

/etc/init.d/vmware-vcloud stop

Next, you will need to switch over to the "oracle" user by running the following command:

su - oracle

sqlplus vcloud/VCloud

Once you have logged in, you will need to insert the following rows into the "config" table of VCD Database:

INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'scsi0:0.virtualSSD', 'true', 0);
INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'scsi0:1.virtualSSD', 'true', 0);
INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'scsi0:2.virtualSSD', 'true', 0);
INSERT INTO config (config_id, cat, name, value, sortorder) VALUES (seq_config.NextVal, 'ExtraConfigWhitelist', 'disk.enableUUID', 'true', 0);

Note: The syntax used here is for an Oracle database. You may need to tweak it if you are using MS SQL Server.

Finally, for the changes to go live you just need to start up your VCD instance:

/etc/init.d/vmware-vcloud start

Once the VCD service has returned, you can now import any OVF/OVA that contains a list of approved VM advanced settings and VCD will ensure these settings are preserved as part of the import. I hope this solution is something our OneCloud team will consider as it solves my immediate problem but I also think this can greatly benefit the overall use base, especially those building out content.

Note: Even though this "whitelist" option is available, I would still recommend you carefully determine which VM advanced setting is required for your environment and minimize the size of the list as much as possible from an operational perspective, especially when it comes to new installation or upgrades.