Awhile back I discovered an undocumented flag called "esxvm" in the SQL statements of the new vCloud Director 1.5 installer that suggested the possibility of deploying nested ESXi hosts in vCD. However, after further investigation the flag only enables the automated deployment of an ESXi 5 parameter (vhv.allow) which is required to run nested ESXi 4.x/5.x hosts as part of preparing a new ESXi 5 hosts in vCloud Director. There was still a missing piece to the puzzle to enable this functionality within vCloud Director user interface.
The answer eventually came from attending a recent session at VMworld 2011 in Las Vegas CIM1436 - Virtual SE Lab (vSEL) Building the VMware Hybrid Cloud by Ford Donald of VMware. I will not go into detail about what vSEL is, if you would like more information take a look at this blog post The Demo Cloud at VMworld Copenhagen or check out Ford's VMworld presentation online. In one of Ford's slides, he describes the necessary steps to enable nested ESXi called ESX_VM mode in vCloud Director which actually consists of two parts:
- Enable nested virtualization and 64-bit vVM support in vSphere 5
- Enable special mode in vCloud Director called ESX_VM to allow for vSphere 4 and 5 hosts as valid guestOS types
There are also some additional steps that are required after enabling ESX_VM mode:
- Preparing or re-preparing ESXi 5 hosts
- Allowing for Promiscuous Mode in vCD-NI or VLAN-backed Network Pool
********************* DISCLAIMER *********************
This is not a supported configuration by VMware and this can disappear at any time, use at your own risk
Note: I will assume the reader has a good understanding of how to install/configure vCloud Director and how it works. I will not be going into any details in configuring or installing vCD, you can find plenty of resources on the web including here, here, here and here. I will also assume you understand how to configure vCD-NI and VLAN-backed network pools in vCloud Director and how they work.
The first part is to enable nested virtualization (nested ESXi) support within the ESXi 5 hosts when they're being prepared by vCloud Director by updating the following SQL statement as noted in my earlier blog post Cool Undocumented Features in vCloud Director 1.5:
UPDATE config SET value='true' WHERE name='extension.esxvm.enabled';
The second part is to update the vCloud Director database to add support for both vSphere 4 and 5 hosts as valid guestOS types:
INSERT INTO guest_osfamily (family,family_id) VALUES ('VMware ESX/ESXi',6);
INSERT INTO guest_os_type (guestos_id,display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version, supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto, is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version) VALUES (seq_config.NextVal,'ESXi 4.x', 'vmkernelGuest', 6, 1, 1, 8, 3072, 7,1, 1, 4, 8, 0, 0, 0, 0, 107, 40);
INSERT INTO guest_os_type (guestos_id,display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version, supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto, is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version) VALUES (seq_config.NextVal, 'ESXi 5.x', 'vmkernel5Guest', 6, 1, 1, 8, 3072, 7,1, 1, 4, 8, 0, 0, 0, 0, 107, 50);
To apply these SQL statements to your vCloud Director 1.5 database, you will need to login to either to your Oracle or SQL Server database and manually execute these statements using the account that you originally created.
Here is an example of executing the SQL statements on an Oracle Express 11g database (Oracle Express is not officially supported by VMware):
As you can see, we need we first create a new guest_osfamily type called "VMware ESX/ESXi" and we need to also provide a unique family_id, which from a default installation of vCloud Director 1.5, the next available value will be 6. Next, we need to create the two new guestos_type "ESXi 4.x" and "ESXi 5.x" and again we need to provide a unique guestos_id which from a default installation of vCloud Director 1.5, the next available values will be 81 and 82. If any errors are thrown regarding a constraint being violated, then the ids may already have been used, you can always query to see what the next value is or select a new id.
Once you have executed the SQL statements, you will need to restart the vCloud Director Cell for the changes to take effect and if you already have prepared ESXi 5 hosts, you will need to re-prepare the hosts.
If you prefer not to manually do this, you can take a look at my blog post Automating vCloud Director 1.5 & Oracle DB Installation which has been updated to allow you to enable ESX_VM mode with your vCloud Director 1.5 installation. There is a new flag in the vcd.rsp file called ENABLE_NESTED_ESX which can be toggled to true/false which will automatically perform the SQL statements as part of the post-installation of vCloud Director 1.5 and restart the vCD Cell for you.
Here is a screenshot if you decide to enable this flag:
Finally, the last configuration tweak is to enable both promiscuous mode and forged transmit in either your vCD-NI or VLAN-backed Network Pool which is a requirement to run nested ESXi hosts. You locate the name of your network pool to identify distributed portgroup.
Next you can either use the vCD API or login to your vCenter Server and enable the promiscuous mode for that specific distributed portgroup.
UPDATE: Thanks to @DasNing - You can also enable promiscuous mode by executing the following SQL query: UPDATE network_pool SET promiscuous_mode='1' WHERE name=';
We are finally done with all the configurations!
If you successfully completed the above, when you go and create a new virtual machine in vCloud Director, you should now have a new Operation System Family called "VMware ESX/ESXi"
Within this new OS family, you can now provision a new ESXi 4.x or ESXi 5.x guestOS
Here is an example of my own vGhettoPod which includes vMA5 and vESXi 5 host which I can use to perform various types of testing in my home lab.
Now you can create your own ghetto vSEL cloud using VMware vSphere 5, vCloud Director 1.5 and vShield 5!
On MS SQL i am getting this error... any ideas? I did some googling and see how to turn on INSER_Identity but it doesnt help
Msg 544, Level 16, State 1, Line 1
Cannot insert explicit value for identity column in table 'guest_os_type' when IDENTITY_INSERT is set to OFF.
If you remove the ID's from the insert you can avoid the errors, the guest_osfamily didn't error for me so I just had to do this:
INSERT INTO guest_os_type (display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version, supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto, is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version) VALUES ('ESXi 4.x', 'vmkernelGuest', 6, 1, 1, 8, 3072, 7,1, 1, 4, 8, 0, 0, 0, 0, 107, 40);
INSERT INTO guest_os_type (display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version, supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto, is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version) VALUES ('ESXi 5.x', 'vmkernel5Guest', 6, 1, 1, 8, 3072, 7,1, 1, 4, 8, 0, 0, 0, 0, 107, 50);
You can also set the Network Pool so that all portgroup's that get created are automatically set to promiscuous mode. Note that this affects ALL portgroups created in the pool.
UPDATE network_pool SET promiscuous_mode='1' WHERE name='
Hey man
i was trying to do it with vcloud 5.1
i set the network pool Lab-PvDC-VXLAN-NP to promiscuos_mode 1
and the sql query worked but
every newly created portgroup is still on reject mode
any tips
regards
shay hyams
The SQL query is not applicable for VXLAN networks as the management of the VXLAN NP is controlled by the vCNS. I'll look at updating this information as it requires a slightly separate step.
thanks man
i'll appriciate it 🙂
Hey wiiliam,
another thing that came up to my mind...
if i'll use a vlan backed NP than the SQL update qeury is applicable.
but...i dont want to use vlan's so...is it possible to create a blocking task that will change the vlan ID to none after that the vApp portgroup creation ?
just wondering
by the way,
found a nice article that allow this modification thrugh the vcenter settings
works nicely
http://geekafterfive.com/2013/03/14/setting-default-vdswitch-security-for-vcloud-with-powercli/
Just what we needed. 🙂 But we are seeing another "issues" with Nested ESXi hosts.
Pinging from the VMKernel on vESXi out through the vSwitch on the vESXi causes a dup response. If i ping another vESXi on the same host and net, it'll create a 4x response of the package.
We are seeing this on 2 seperate vCloud Installations.
@Jakobwill,
This is expected if you have prom mode enabled, you will see duplicate packets.
Hi William,
DUP! packets only happens when running vESX on VCDNI backup network pool. No DUP! messages when running vlan-backed or PG-backed. This error is confirmed by VMware and is to be fixed in an upcoming patch release
@DasNing,
Thanks for the tip
Hi guys! On my vDS MAC Address Change/Forged Transmit is also rejected by default. These also need to be enabled to run a nested lab. This guy is also reporting on this: http://grokee.com/nested-networking-woes-solved/.
Of course you can manually change it, but a database tweak would be better. Cannot find a "mac address change" or "forged transmit" field in DB. Any suggestions?
And....The family_id was 4 and not 6 in my case. So I had to make a slight change to the long SQL insert statements...
So I'm running into an error when I run the insert statements for setting the guest OS's. The insert into guest_os_type yields this error message: The multi-part identifier "seq_config.NextVal" could not be bound. SQL 2008 R2 SP2
I'm not much of a SQL guy so any help or advice would be greatly appreciated. Thanks!
- Mike
Check out the results of this SQL query on the vCloud Director v5.5.1 database.
SELECT [vCD].[dbo].[guest_os_type].*
FROM [vCD].[dbo].[guest_os_type]
WHERE [vCD].[dbo].[guest_os_type].[family_id] = 3
AND [vCD].[dbo].[guest_os_type].[guestos_id] > 99;
Version 4, 5 and 6, yes I said 6 !!
I found that this is added by default with vCloud 5.5 so if you upgrade from 5.1 to 5.5 and have made this change you will have duplicate entries in the guest_os_type table. Remove the duplicates and everything goes back to working as expected.
I made all the necessary changes, however the GUI does not reflect the changes exactly. I now see the VMware radio button, however when I select it, the OS drop-down is populated with all the values that were originally in the "Other" category of OS Family. When I select the Other radio button, the OS drop-down is empty. No where do I see the VMware ESXi as an OS.
I looked at the database and made sure that the new family_id was unique and corresponded to the correct guestOS.
Any ideas?
Why does your vCloud Director interface look different than mine? Are you using Air? I'm using 5.5.3 and the interface looks different.
It's because the screen shots are of the really old vCloud Director v1.5