WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Hardware Options
    • Hardware Reviews
    • Lab Deployment Scripts
    • Nested Virtualization
    • Homelab Podcasts
  • VMware Nostalgia
  • Apple

How To Enable Nested ESXi Using VXLAN In vSphere & vCloud Director

05.06.2013 by William Lam // 9 Comments

Recently I had received several inquiries asking on how to configure nested ESXi (Nested Virtualization) to function in a VXLAN environment. I have written several articles in the past on configuring nested ESXi in a regular vSphere and vCloud Director environment, but with the use of a VXLAN backed network, there are a few additional steps that are required. These steps include additional configurations of the vCloud Network & Security Manager (previously known as vShield Manager) which ensures that both the required promiscuous mode and forged transmits are automatically enabled for the VXLAN virtual wires (vWires) as they are managed exclusive by the vCNS Manager.

In this article, I will walk you through the configurations that is required when using VXLAN in both a vSphere only environment as well as a vCloud Director environment. If you would like to learn more about how VXLAN works, be sure to check out the multi-part VXLAN series (Part 1/Part 2) by Venky Deshpande.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Configurations for VXLAN in vSphere Environment

Step 1 - Deploy vCNS Manager and configure it to point to your vCenter Server (do not enable or prepare VXLAN, this must be done after the configurations)

Step 2 - You will need to identify the VDS MoRef ID in your vCenter Server which will be used in the next step. Since the configuration is applied at the VDS level, you may want to consider having a separate VDS serving Nested Virtualization traffic since both promiscuous mode & forged transmits will automatically be enabled for all vWires. To locate the VDS MoRef ID, login to the vSphere Web Client and select the summary view for the VDS.

The VDS MoRef ID will be towards the end of the URL link and it should start with dvs-X where X is some arbitrary number. Record this value down for the next step

Step 3 - Download the enablePromForVDS.sh shell script which will be used to prepare the VDS within the vCNS Manager. The script basically performs a POST to the REST API to the vCNS Manager using cURL and it accepts three input parameters: vCNS Manager IP Address/Hostname, VDS MoRef ID and VDS MTU. The username/password is hard coded in the script to use the default which is admin/default. If you have modified the default password like any good admin, you will want to change the password before running the script. If you take a look at the request body, you will notice only promiscuous mode is enabled to true, but this will also automatically enable forged transmits as well.

In my lab enviroment, I have the vCNS Manager IP to be 172.30.0.196, VDS MoRef ID to be dvs-13 and VDS MTU to be 9000. So the syntax to run the script would be:

./enablePromForVDS.sh 172.30.0.196 dvs-13 9000

Here is a screenshot of executing the script, you should see a response back with 200 to indicate successful execution of the script.

Step 4 - Now, we will proceed with the VXLAN preparation. Start off by logging into the vCNS Manager and selecting the vSphere Datacenter which you wish to enable VXLAN. On the right you should see a tab called "Network Virtualization" go ahead and click on that and then click on the sub-tab called "Preparation". Click on edit and then select the vSphere Cluster and proceed through the wizard based on your environment configuration.

Step 5 - Once the VXLAN preparation has completed, click on the "Segment ID" and configure that based on your environment.

Step 6 - Next, click on "Network Scopes" and you will create a network scope and specify the set of vSphere Clusters the VXLAN network will span.

Step 7 - Lastly, click on "Networks" and this is where you will create your vWires and ensure it the proper network scope is selected.

Step 8 - To confirm that everything has been configured properly. We now log back into our vSphere Web Client and heading over to the VDS settings page. You should now see a new vWire portgroup that is created, if we take a look at it's settings we should see that both promiscuous mode and forged transmits is enabled.

You are now done with the VXLAN configurations in the vCNS Manager and can proceed to the regular instructions for enabling Nested ESXi for vSphere.

Note: If you have already prepared VXLAN in your environment, you can still configure the above without having to un-prepare your VXLAN configurations. You just need to login to the vCNS Manager via the REST API and perform a DELETE on the VDS switch (Please refer to page 153 of the vCNS API Programming Guide) which will just delete the mapping from vCNS but will not destroy any of your VDS configuration. Once that is done, you will be able to use the script to configure the VDS with the proper settings.

Configurations for VXLAN in vCloud Director Environment

A VXLAN network pool is automatically created for you when using vCloud Director 5.1, so the steps for preparing Nested Virtualization for vCloud Director is extremely simple compared to the vSphere only environment.

Note: VXLAN is only supported in vCloud Director 5.1, for previous versions you have the choice of using a VCD-NI or vSphere backed network and the configurations for that can be found here.

Step 1 - Please follow the steps 1-5 from above in the vSphere only environment and then you are done. If you would like a more detailed walk through for configuring VXLAN for a vCloud Director environment, check out this article by Rawlinson Rivera who takes you through the process step by step.

Step 2 - Proceed to the regular instructions for enabling Nested ESXi for vCloud Director.

Step 3 - Lastly, you will go through the vCloud Director setup which is to attach your vCenter Server & vCNS Manager, create a Provider VDC, create an Organization and assign resources to your Organization VDC and ensure that the OrgVDC is consuming the VXLAN network pool that is automatically created for you when you create the Provider VDC. Once that is done, when you deploy your vApp, you will see a vWire that automatically created for you. If we login to the vSphere Web Client and go to the VDS settings, you will see the vWire has both promiscuous mode and forged transmits automatically enabled.

Additional Resources:

  • Nested Virtualization Resources

Categories // Automation, Nested Virtualization, NSX Tags // nested, vcloud director 5.1, vcloud networking and security, vcns, vhv, vSphere 5.1, VXLAN

Dude, Where's My vCenter Server 5.1 Components Installed At?

04.03.2013 by William Lam // 5 Comments

You would be surprised at the number of times I have heard this question get asked and this is not regarding the installation path but the specific server a given vCenter Server 5.1 component is installed on. I am just wondering if people are somehow miss-placing their infrastructure? I would hope that most organizations have some type of CMDB (Configuration Management Database) even if it is just a spreadsheet or at a minimum a memorable hostname. In any case, this question is only relevant for those of you who decided to separate out the vCenter SSO (Single Sign-On) Server, vSphere Web Client, Inventory Service and the vCenter Server and are now wondering where a given component is installed at.
To begin, you will need to know at a minimum where your vCenter Server is installed at. If you do not know that, then you should take the walk of shame and install this utility (be-careful with port scanning tools, as it may not be allowed by your Security Operations team). Go to the advanced settings of your vCenter Server and look up one of the following settings:
  • config.vpxd.sso.sts.uri
  • config.vpxd.sso.groupcheck.uri
  • config.vpxd.sso.admin.uri

All three of these settings should contain the same hostname or IP Address which is the location of where your SSO Server is installed. You can also find this information by looking at the vCenter Server configuration file located in the following location:

Windows vCenter Server: C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
vCenter Server Appliance: /etc/vmware-vpx/vpxd.cfg

Next, you will need to login directly to your vCenter Server (RDP or SSH) depending on the version you are using. Using the hostname or IP Address of our vCenter SSO Server, we will now connect to the Lookup Service which is installed alongside the vCenter SSO Server. This service will provide us with the location of all services registered to vCenter SSO and we will be able to identify the location of the remainder vCenter Server components.

For Windows vCenter Server, make sure you have the JAVA_HOME environmental variable set to C:\Program Files\VMware\Infrastructure\jre and open up a command prompt and run the following (subsitute in the hostname or IP Address of your vCenter SSO Server):

vSphere 5.5

"C:\Program Files\VMware\Infrastructure\VMware\CIS\vmware-sso\ssolscli.cmd" listServices https://winvc.primp-industries.com:7444/lookupservice/sdk

vSphere 5.1

"C:\Program Files\VMware\Infrastructure\SSOServer\ssolscli\ssolscli.cmd" listServices https://winvc.primp-industries.com:7444/lookupservice/sdk

We can take a look at the serviceName which describes the specific vCenter Server component such as the vSphere Web Client or Log Browser and endpoints property will tell you which server it is installed on.

For vCenter Server Appliance, there is a similar command by running the following:

/usr/lib/vmware-sso/bin/vi_regtool listServices https://172.30.0.186:7444/lookupservice/sdk

The only vCenter Server component that we have not found is the Inventory Service. To find the server where this component is installed, we just need to look at the vCenter Server Extensions and and we can simply open up a web browser and connect to the following URL (substitute in your vCenter Server address):

https://vcsa.primp-industries.com/mob/?moid=ExtensionManager&doPath=extensionList[%22com.vmware.vim.inventoryservice%22].healthInfo

Hopefully at this point you are able to figure out where all your vCenter Server 5.1 components are installed at and you are also documenting all this information in your CMDB or spreadsheet 🙂

Categories // Automation Tags // inventory service, lookupservice, sso, VCSA, vcva, vSphere 5.1, vSphere 5.5, vsphere web client

Automate vCenter Orchestrator Configuration Backups

03.29.2013 by William Lam // Leave a Comment

Last year I wrote an article on how to quickly configure a new vCenter Orchestrator 5.1 appliance which automatically goes through the necessary steps of configuring your vCO appliance and enabling the vCenter Server plugin and associating it with your vCenter Server. These steps are usually performed manually, but when you are looking at deploying multiple vCO instances or even quickly spinning up vCO appliance for testing, this will definitely help speed up your deployment.
Something that I did not consider after completing the vCO setup was backups. Fortunately, this was something that was shared with me recently from a customer who had this exact workflow on backing up their vCO configuration after their initial deployment. This may not be a very well known feature, but vCO provides a very simple mechanism to export your vCO configurations and allows you to restore the configuration in case of a miss-configuration or even deploying a similar configuration to another vCO instance.
Using the same HTTP request trick, to export the vCO configuration you would need to make a request to the following URL:

https://${VCO_IP_ADDRESS}:8283/config_general/ExportConfig_export.action

Similar to the vCO UI, the backup will be stored on the vCO appliance itself and the path will be provided back to you in the message response. To help demonstrate this, I created a simple shell script called backupVCO51.sh which is similar to the setup script in my previous blog article. You can easily take the few lines of code and integrate that with the setup script.

Here is a screenshot of running the backup script:

From the output we can see where the backup configuration is stored on the vCO appliance and you can easily copy the backup to an external system using SCP.

Whether or not you are automating your vCO setup, you should definitely consider performing periodic backups of your vCO configuration, especially before making any changes to your vCO Server.

Categories // Uncategorized Tags // appliance, vcenter orchestrator, vCO, vSphere 5.1

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • …
  • 18
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Ultimate Lab Resource for VCF 9.0 06/25/2025
  • VMware Cloud Foundation (VCF) on ASUS NUC 15 Pro (Cyber Canyon) 06/25/2025
  • VMware Cloud Foundation (VCF) on Minisforum MS-A2 06/25/2025
  • VCF 9.0 Offline Depot using Synology 06/25/2025
  • Deploying VCF 9.0 on a single ESXi host? 06/24/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025