Several months back I had built an ESXi Virtual Appliance that allows anyone to quickly stand up a fully functional Nested ESXi VM which includes guest customization such as networking, NTP, syslog, passwords, etc. The virtual appliance was initially built for my own personal use as I found myself constantly rebuilding my lab environment for evaluating and breaking new VMware software. I figured if this was useful for myself, it probably could benefit others at VMware and I posted the details internally on our Socialcast forum. Since then, I have received numerous stories on how helpful the ESXi Virtual Appliance has been for both our Field and Engineering for setting up demos, POCs, evaluations, etc.
Most recently, I was contacted by Massimo Re Ferre' (crazy Mainframe guy ;)) who works over in our Cloud Native Apps team and was working on a pretty cool project with Photon Controller which was recently open sourced. He was interested in leveraging the ESXi Virtual Appliance along with using VMware AppCatalyst to make it super simple for anyone to try out the Photon Controller in their own environment. Over the last couple of weeks, I have been working closely with Massimo on incorporating on his requirements for the Photon Controller POC back into my ESXi Virtual Appliance. My original goal for the appliance was to keep it generic so that it could cater to multiple use cases and Photon Controller POC was just another neat solution that could be built on top of.
I just found out today that new Photon Controller POC has just been released and you can find more details in the links below:
- Photon Controller main page
- Getting started guide for deploying Photon Controller on OS X
- Getting started guide for deploying Photon Controller on Windows
- Photon Controller Google Group
As part of the release, the ESXi Virtual Appliance is also made available which I thought was pretty cool! 😀 I highly recommend you check out the awesome work done by Massimo if you want to play with Photon Controller. This is a really easy way of getting started with Photon Controller and giving it a spin in your own environment.
Since the ESXi Virtual Appliance is now available externally, I wanted to share a few details about the appliance for those who might be interested in checking it out. As I mentioned earlier, the goal of the ESXi Virtual Appliance was to be generic and to be used as a build block that could enable different use cases such as spinning up a quick vSphere lab using it and the VCSA or putting together a fully functional VSAN lab in literally a couple of minutes (at the very bottom, I have a couple of PowerCLI scripts to demonstrate this). You could deploy 3 instances of the appliance to get a basic 3 Node VSAN Cluster or you could scale up to 64 Node VSAN Cluster all within just minutes. The limit is truly your imagination.
The appliance contains a pre-installed GA release of ESXi 6.0 Update 1. There are 11 OVF properties that are available for customizing the Nested ESXi VM which are shown in the table below. Once powered on, the default 60 day evaluation will start counting down as if you had manually installed ESXi yourself. In addition, the OVA also contains several optimizations for running Nested ESXi including the Mac Learn dvFilter params as well as other configurations for quickly setting up a VSAN environment which are also described below. I have also built the appliance to be easily consume in all VMware based environments including vSphere, vCloud Air, Fusion, Workstation, Player & AppCatalyst.
UPDATE (05/10/17) - Updated VA to latest ESXi 6.0u3 & 6.5d (vSAN 6.6)
UPDATE (05/09/17) - The ESXi 5.5u3 has been decommissioned due to its limited use.
ESXi 5.5 Virtual Appliance download link(Decommissioned)
UPDATE (11/18/16) - ESXi 6.5 Virtual Appliance has been released and you can find the details here.
UPDATE (04/07/16) - Minor bug fix with refreshing auto-generated SSL Certificates, latest version is v5 for 6.0u2 VA and v2 for 5.5u3 VA
UPDATE (03/18/16) - I updated the ESXi 6.0 VA using vSphere 6.0 Update 2. It is now back to vHW10 for backwards compat and includes an All-Flash configuration + 2 VMXNET3 adapters and ready for VSAN 6.2 😀
UPDATE (03/01/16) - I have also created a new ESXi 5.5 VA using vSphere 5.5 Update 3b
| OVF Property | Description | Type |
|---|---|---|
| guestinfo.hostname | FQDN of the ESXi host | string |
| guestinfo.ipaddress | IP Address | string |
| guestinfo.vlan | VLAN ID | string |
| guestinfo.netmask | Netmask | string |
| guestinfo.gateway | Gateway | string |
| guestinfo.dns | DNS Server | string |
| guestinfo.domain | DNS Domain | string |
| guestinfo.ntp | NTP Server | string |
| guestinfo.ssh | Whether or not SSH is enabled | boolean |
| guestinfo.syslog | Syslog Server | string |
| guestinfo.password | Root password for ESXi host | string |
| guestinfo.createvmfs | Whether to automatically creates a VMFS datastore (datastore1) on largest VMDK | boolean |
The ESXi 6.x Virtual Appliance includes the following configuration:
- ESXi 6.0 Update 2
- GuestType: ESXi 5.x (backwards compat)
- vHW 10
- 2 vCPU
- 6GB vMEM
- 2 x vmxnet3 vNIC
- 1 x 2GB HDD (ESXi Installation)
- 1 x 4GB SSD (for use w/VSAN, empty by default)
- 1 x 8GB SSD (for use w/VSAN, empty by default)
- VHV added (more info here)
- dvFilter Mac Learn VMX params added (more info here)
- disk.enableUUID VMX param added
- VSAN traffic tagged on vmk0
- Disabled VSAN device monitoring for home labs (more info here)
The ESXi 5.x Virtual Appliance includes the following configuration:
- ESXi 5.5 Update 3b
- GuestType: ESXi 5.x
- vHW 10
- 2 vCPU
- 6GB vMEM
- 2 x vmxnet3 vNIC
- 1 x 2GB HDD (ESXi Installation)
- 1 x 4GB SSD (for use w/VSAN, empty by default)
- 1 x 8GB HDD (for use w/VSAN, empty by default)
- VHV added (more info here)
- dvFilter Mac Learn VMX params added (more info here)
- disk.enableUUID VMX param added
- VSAN traffic tagged on vmk0
- Disabled VSAN device monitoring for home labs (more info here)
- ESXi Embedded Host Client (more info here)
If you do not wish to use VSAN, there is an OVF property that allows you to specify whether or not a default VMFS datastore is created. You can increase the capacity of any of the disks after deployment (if you wish for the automatically VMFS creation, you will need to expand the disk prior to powering on the VM). Below are the different methods in which you can deploy the ESXi Virtual Appliance which includes vSphere, vCloud Air, Fusion, Workstation, Player & AppCatalyst. The idea is that you can easily setup Nested ESXi on any VMware based hypervisor and be up and running in just minutes!
Option 1 - Deploy to vSphere environment w/vCenter Server or vCloud Air
Download the OVA (or you can just paste the URL link) and import that into your vCenter Server using the vSphere Web Client. Make sure to accept the "Extra Configuration" when prompted and then fill out the 11 OVF properties which will allow you to customize the Nested ESXi VM to your environment. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM. Once powered on, the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM. Below are a couple of screenshots
Option 2 - Deploy to ESXi
Download the OVA and import it into ESXi using either the vSphere C# Client but do not power it on. Since ESXi does not support OVF properties, you will need to add the following guestinfo*. properties as shown in "Option 3" below by using the VM Advanced Settings UI.
If you prefer NOT to mess around with manually adding these VM Advanced Settings which can also be automated using the vSphere API or PowerCLI, one additional method which CAN make use of the OVF properties is by using ovftool which is a CLI to import the OVA and using the --injectOvfEnv option that was added in ovftool version 4.x You can find more details in this blog post here.
Option 3 - Deploy to VMware Workstation, Fusion or Player
Download the OVA and import it into Fusion/Workstation but do not power it on. You will then need to edit the VMX file and add the following guestinfo.* properties as shown below since Fusion/Workstation do not support OVF properties. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM. Once you have saved the changes, you can then power on the Nested ESXi VM and the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM.
guestinfo.hostname = "vsan-1.primp-industries.com" guestinfo.ipaddress = "172.16.78.90" guestinfo.netmask = "255.255.255.0" guestinfo.gateway = "172.16.78.1" guestinfo.dns = "172.16.78.1" guestinfo.domain = "primp-industries.com" guestinfo.ntp = "172.16.78.1" guestinfo.ssh = "True" guestinfo.syslog = "192.168.1.100" guestinfo.password = "VMware1!" guestinfo.createvmfs = "False"
Option 4 - Deploy to VMware AppCatalyst
Download the the OVA and import that into AppCatalyst using ovftool but do not power it on. You will then need to edit the VMX file and add the following guestinfo.* properties as in the above example for Workstation/Fusion in addition to the following params listed below which are required for Nested ESXi to run in AppCatalyst. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM.
guestos = "vmkernel6" virtualhw.version = "11" svga.vgaOnly = "true"
You will also need to run the following command to allow promiscuous mode in AppCatalyst since there's no UI to prompt (this is only required once):
touch "/Library/Preferences/VMware AppCatalyst/promiscAuthorized"
Once you have saved the changes, you can then power on the Nested ESXi VM and the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM.
Option 5 - Deploy to vSphere using PowerCLI
I have also created two very simple PowerCLI scripts which demonstrates how you can easily deploy N-number of these Nested ESXi VMs and in fact, you can setup a fully functional VSAN Cluster in just under 5 minutes! You can find the two scripts below:
The first script will perform the deployment of the Nested ESXi VM, but you will first need to convert the OVA to an OVF because the Get-OvfConfiguration cmdlet does not support OVA. To properly convert the OVA, you will need ovftool and then run the following command:
ovftool.exe --allowAllExtraConfig --skipManifestCheck Nested_ESXi_Appliance.ova Nested_ESXi_Appliance.ovf
The second script will go ahead and add the deployed Nested ESXi VMs to a vSphere Cluster and then enable VSAN on the vSphere Cluster.
Awesome work William! As you mentioned, I use your VAPP on a regular basis. Have you thought about adding support for: Choose amount of ESXi Hosts to deploy and number of VMDKs for Flash and Capacity layer? Is this even possible?
Being able to specify the number of instances would be possible with ScaleOutSection if the OVF 2.0 specification, but that's not currently supported today. You can easily do this with some Automation (there's even a script that I've attached to this blog post) that does this and literally deploy 3 or 64 VSAN Nodes in literally within minutes. The latter cases are more specific and as mentioned in the article, I wanted to keep a generic ESXi Appliance so it could cater to variety of use cases. You can always deploy & then add additional VDMKs based on your requirements, that's the easy part and I'm just providing a base 🙂
Looks awesome, William! I am downloading and deploying it to vSphere now! In your experience, have you tried adding additional vNICs to nested ESX hosts for segregation of traffic (vm, mgt, vSAN, vMotion)? Any tips / tricks? Or do you just use the one vNic?
Sure, you can add additional vNICs. This is no different than any other VMs 🙂 Definitely recommend using vmxnet3 and I've done this to do variety of things like supporting various networking configurations.
I have been trying to add a couple of more vNICs by copying and modifying the first two. The problem is that the new vNICs show up as disconnected. I'm guessing that part of the problem is knowing what values to use for some of the instance specific parameters such as: AddressOnParent, InstanceID, pciSlotNumber, ethernetX.pciSlotNumber for vNIC N+1.
How did you generate/add the vNICs?
Best news I've seen all month - this is a real boon for those of us regularly redeploying labs to test stuff. Thanks for your efforts with this!
Great stuff. What about hot plug/remove vcpu/mem on nested ESXi? Any info on that?
I hope you know that hotplug is based on the guestOS supporting the feature, not all OSes support it. There's a VMware KB that shows which OSes are supported. ESXi running in a physical host doesn't support hot add of memory/CPU, so it wouldn't be supported when running in a VM.
I was actually pointing to something else. Windows 2016 will support hot plug/remove. So we will be able to deploy nested Windows 2016 hypervisors under ESXi (and sooner than later under MS hypervisor) and dynamically give them resources on demand...not a case with VMware.
Thanks a lot William. Looks very interesting, I will give it a try. Especially after my ovftool confusion that I described in https://pixelschatten.net/ovftool-conversion-confusion/. Thanks again for being that responsive on Twitter and taking a look. You rock!
I also would like to know how to create these OVF/OVA files myself. Like you did with the ESXi virtual appliance. I would like to better understand the tech behind it, learn and be able to use it internally with some of our systems. Makes deployment much easier. Do you have some pointers to the right docs for me?
Thanks again, much appreciated!
I would recommend taking a look at VMware Studio, which is authoring tool for OVFs & there's a great tech paper that describes quite a bit of the concepts. Also have a look at my OVF resources http://www.virtuallyghetto.com/ovf
Creating an OVF can be as simple as building it in vSphere and then exporting. Feel free to inspect the ones I've built to get an idea of how things fit together.
This is great!
FYI, I did Option 2 above, and configured guestinfo.domain to "lab.local" yet I still get "primp-industries.com, lab.local" for my Customer DNS Suffixes. Just nitpicking, so nice and easy to deploy, thank you!
Given the likely popularity of your download link
https://bintray.com/artifact/download/photon-controller/public/UserWorkflow/Nested_ESXi_Appliance.ova
I wonder if bintray knows what they're in for 😉
Ah darn, it looks like when using ESXCLI to add DNS Domain, it's an additive versus replacement of the value. It looks like it picked up my local build's domain. I'll have to fix this internally and if they decide to push out another build or allow me to update, you should see that fix.
Should be good publicity for them 😉
Wow! Nice, I will definitely check it out this weekend. So much work saved 🙂
If we deploy the appliance and add NICs, etc, how could we modify the ova properties or inject new ones? Assuming we make the changes and just export it, the 11 original properties would still be valid, right?
The OVA properties are only applicable on initial setup. You can always add/change the vHW after its been setup. For things like VMDKs, you can actually hotadd and just do a rescan at the ESXi layer. Once configured, you can make whatever changes and if you export it, it'll retain the settings/configurations. Of course you won't be able to change the OVF properties after that, as I said its only applicable on initial deployment
Gotcha. What did you use to insert the properties?
Thank you William for sharing it.
William,
Thanks again for the hard work.
Might want to mention if you add a NIC you must also add the dvfilter commands for that NIC as well. Also, when adding a NIC cannot select VMXNET3? Had to edit the vmx file to switch e1000 to vmxnet3?
Thanks
Tom Miller
Also - NONE of the ovf answer file worked?
I have found the same issue. The guestinfo* settings work, so I cannot login to the host to fix. The ESXi 6.0 ovf hung and never completed the boot sequence.
Hi,
Great tool!
Is there a way to deploy this with vRa? It would be great to be able to create "on-demand" nested esxi for testing purpose.
Thanks
Sylvain
Sure. I know of a few customers who were already doing this before this virtual appliance. This is just like another VM, so it would be possible to drive it through vRA automation, however I'm not familiar enough with that product to give you the exact steps. You may want to post in the VMTN community forums for any tips or guidances but to me, this is just another VM that would sit in the vRA Catalog and made available for deployment.
Thanks William
Actually, I managed to add a template created with the OVA and make it available through the vRA Catalog.
Problem is: I don't know how to pass parameters to change the OVF properties (guestinfo*. properties).
I'm just a beginer in vRA so if you're not familiar with the process either, I'll try asking in the VMTN community forums.
Hi Sylvain, did you figured this out, cause I'm trying to do the same thing, however with vRealize Orchestrator.
Hello William,
This page is great thank you for all you have done, new to using ESX and trying to get this up and running with a second nic on my workstation.
I have before I turned on the vm, added the second nic in the hardware settings section and assigned it to the proper network.
QUESTION: Do I need to have two entries in the VMX file for both networks?
Thank you,
Hey William,
Love your articles and posts. They've seriously helped me out. Keep it up!
I don't really know how to contact you but I'm in need of some expertise.
I'm trying to correlate the Portgroup number with the dvswitch names from the command line, but having a lot of troubles figuring it out.
1.) Here is how I was able to get the Portgroups:
[root@esxi3:~] esxcli network vswitch dvs vmware list | grep DVPortgroup | sort | uniq
DVPortgroup ID: dvportgroup-3970
DVPortgroup ID: dvportgroup-3971
DVPortgroup ID: dvportgroup-3972
DVPortgroup ID: dvportgroup-3981
2.) Here is how I was able to get the PortgroupNames
[root@esxi3:~] vim-cmd hostsvc/net/dvs_info
....
.....
portgroupName = (string) [
"dvSwitch-vSAN-NV-DVUplinks-3969",
"dv-VSAN",
"dv-vMotion-1",
"dv-VM-DEV-10.188.0.0-1"
],
......
.....
However, "dvSwitch-vSAN-NV-DVUplinks-3969" is not really a valid dropdown option when I'm trying do a deploy from the interface.
3.) This what I get for Standard VSwitch
[root@esxi3:~] esxcli network vswitch standard list
vSwitch0
Name: vSwitch0
Class: etherswitch
Num Ports: 10752
Used Ports: 4
Configured Ports: 128
MTU: 1500
CDP Status: listen
Beacon Enabled: false
Beacon Interval: 1
Beacon Threshold: 3
Beacon Required By:
Uplinks: vmnic0
Portgroups: Management Network
vSwitch1
Name: vSwitch1
Class: etherswitch
Num Ports: 10752
Used Ports: 4
Configured Ports: 128
MTU: 1500
CDP Status: listen
Beacon Enabled: false
Beacon Interval: 1
Beacon Threshold: 3
Beacon Required By:
Uplinks: vmnic1
Portgroups: PERF_DATA_META
4.) Another Vswitch and DVSwitch listing
[root@esxi3:~] esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 10752 4 128 1500 vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 0 1 vmnic0
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 10752 4 128 1500 vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
PERF_DATA_META 3962 1 vmnic1
DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
dvSwitch-vSAN-NVAN 10752 49 512 1500 vmnic5,vmnic4
DVPort ID In Use Client
22 1 vmnic4
23 1 vmnic5
14 1 vmk1
10 1 vmk2
100 1 dvtest-ops-tan.eth0
.... .. ...............
(The names and prompts have been modified in my paste for security, but i changed them consistently)
What I'm trying to find out is to find a correlated list like this:
DVPortgroup ID: dvportgroup-3970 "dv-vMotion-1",
DVPortgroup ID: dvportgroup-3971 "dv-VSAN",
....................... ........................ ....................
Where I know what 'dvportgroup.....' in my VMX file is in terms of its correlated name.
I feel like and think that I've explored all the Command Line Options and Outputs (i think). Any clues, or better yet answers would be most appreciate.
Cheers,
Zubin
William,
This post is awesome. Now I have a question.
My usage for the OVA is to develop other OVA, and keep the Nested ESX OVA in the workflow.
So basically we keep the OVA in our artifactory server, then:
* our maven process deploys it with ovftool, ( we need to add few tweaks here. See later note)
* packer creates an OVA inside this Nested ESX.
* maven creates the OVA using ovftool
* we destroy the virtual ESX
This way in 7 years we can reproduce our build with the exact same version of ESX and we do not need to keep it on.
Now the note I refereed above, is that packer requires some configuration on the ESX.
- Enable GuestIPHack
- adding one extra local admin ( not completely necessary but we do it)
- adding rsh keys ( via /etc/rc.local.d/local.sh)
- opening VNC inbound ports (via: /etc/rc.local.d/local.sh and /etc/vmware/firewall/service.xml)
So once the OVA is configured, I close it and I use ovftool to export it again, but anytime I deploy it it has the IP settings of the original, and it disregards all the -prop:guestinfo.XXX information
I think I am missing how to reset the vami parameters before I export my customized Nested ESX.
You use Vmware Studio, but it cannot be that complicated, just to reset the vami properties and create another OVA.
Any help would be appreciated.
I found that you hvae installed this:
[root@machine esxcli software vib list | grep Ghetto
nested-esxi-customization 5.0.0-1.0.0 virtuallyGhetto VMwareCertified 2015-12-24
I suppose that is what I am looking for.. now let's find how it works!!
I am answering all my questions. 🙂
You do it on /etc/rc.local.d/098* scripts.
Now I need to figure out why addinging a new static IP does not work.
I found it, sorry for the long reply, now I can create my customized OVA
# esxcli system settings advanced set -o /UserVars/vGhettoSetup -i 0
William,
I am trying to get one more funtionality on this Netsted ESX and I see that you have this file /etc/rc.local.d/098.vghetto_config
but you do not allow dhcp option, what it good for a lot of environments, mainly when we cna deploy with ovftool with options --powerON -X:waitForIP, and get the IP of the ESX, so we can work on it.
I have modified the file this way:
# diff 098.vghetto_config 098.vghetto_config.DST
70,82c70,77
< if ip == 'dhcp':
< os.system("esxcfg-init --set-boot-progress-text \"Applying DHCP Network Configuration ...\"")
< os.system("esxcli network ip interface ipv4 set -i vmk0 -t none")
< os.system("esxcli network ip interface ipv4 set -i vmk0 -t dhcp -P 1")
< else:
< os.system("esxcfg-init --set-boot-progress-text \"Applying Static Network Configurations ...\"")
< os.system("esxcli network ip interface ipv4 set -i vmk0 -I " + ip + " -N " + netmask + " -t static")
< os.system("esxcli network ip route ipv4 add -n default -g " + gateway)
< os.system("esxcfg-init --set-boot-progress-text \"Configuring hostname ...\"")
< os.system("esxcli system hostname set -f " + hostname)
< os.system("esxcfg-init --set-boot-progress-text \"Configuring DNS ...\"")
< os.system("esxcli network ip dns server add -s " + dns)
os.system("esxcfg-init --set-boot-progress-text \"Applying Network Configurations ...\"")
> os.system("esxcli network ip interface ipv4 set -i vmk0 -I " + ip + " -N " + netmask + " -t static")
> os.system("esxcli network ip route ipv4 add -n default -g " + gateway)
> os.system("esxcfg-init --set-boot-progress-text \"Configuring hostname ...\"")
> os.system("esxcli system hostname set -f " + hostname)
> os.system("esxcfg-init --set-boot-progress-text \"Configuring DNS ...\"")
> os.system("esxcli network ip dns server add -s " + dns)
> os.system("esxcli network ip dns search add -d " + domain)
But, it looks like this file is one of those that is not backed up in the boot bank, so at boot time it reverts to the saved version without my changes.
I also run /sbin/auto-backup.sh with no luck
My question is:
where can I make these changes to /etc/rc.local.d/098.vghetto_config is changed at boot time?
I could eventually create a /etc/rc.local.d/.#098.vghetto_config file, but I wonder how you manage to put it there.
Maybe I find the answer myself later, but if you have a chance I'd appreciate it.
Hi William,
First of all, love your work man, keep it up!
Would you consider making this release or the next of Nested ESXi 6 appliance = vHW10 like the vcsa 6 appliance is? Makes it easier to deploy on vSphere 5.5.
HI William
awesome post, i'll use this a lot
i think there was a esxi6 nested appliance hardware version 10 up here before?
the one above is hw v11
any chance i could get the hw v10 one again? i deleted my old one unfortunately!
thx
Yea, I had a few people ask about this and I plan to build a new one using vHW10 once vSphere 6.0 Update 2 has GA'ed. If you need right away, you can take the existing OVA and convert it to OVF using ovftool and then manually changing the vHW requirement from 11 to 10 which will continue to work.
William,
We also had a need for a V10 and we actually created our own OVF, but even when I put the config parameters for guestinfo in they don't show up when I do the get-ovfconfiguration?
PowerCLI C:\TEMP\esx> Get-OvfConfiguration .\esx.ovf
=========================
OvfConfiguration: esx.ovf
Properties:
-----------
NetworkMapping
In addition to this I attempted to convert to VMX and then back to OVF to set to hardware 10, but it set the networking to bridged and broke it :/
Now that vSphere 6.0 Update 2 has officially GA'ed, I'll be updating my 6.x Nested ESXi Appliance to that version. I'm hoping I'll get some time this week to do that. So stay tune for that
Gotcha, thanks I was hoping I could just get it working, but at this point I guess I should just cut my losses and wait for the export. Any idea why it changed the networking to bridged??
Thanks, William! I'll keep an eye out for the download link to come active again.
In case you didn't see the update, the link has been re-activated and the 6.x link now points to latest 6.0u2 Nested ESXi VA
Hello, trying to deploy the nested_esxi_appliance_v6.0.ova on my esxi 6.0 free, i get a "The task was canceled by a user" with no errors...
I don't understand why !
Thank-you for this! Saves me a lot of time that I've been wasting with my nested labs!!
Thank you for great job!
But anyway, after deploying version 5.5 appliance, my 64-bit nested VMs didn't work.
Had to add "vhv.enabled="true"" manually to each nested ESXi.
Could you please re-check it?
Trying to download the Nested ESXi 6 U2 OVA and the link doesn't appear to be working. I'd love a shot at downloading it so that I can start playing around with Photon Controller!
Thanks for all your hard work for the community, I know I certainly appreciate learning from your efforts!
I have the same issue here 🙁 William may I ask you please to check the links?
Links should be working now, perhaps there was an issue w/bintray earlier.
FYI - The latest version for 6.0u2 is v5 and 5.5u3 is v2 (there's a version associated when you try to import using either vSphere Web/C# Client), so make sure you're pulling the latest
Thank you! Really appreciate your commitment!
Thanks again, I was able to get the latest file!
Hello,
Is there a way to apply VLAN for ESXi, or does anyone know how to identify whats the exact VLAN id key in properties
like: "guestinfo.vlan" and apply the value for it
Hi folks,
BYI
After deploying my vSAN lab (W2012R2+VMwareWS12+vCenter) I've got an alarms on my vSAN. There was a MTU mismatch between W2012R12 (MTU 1514) and esxi6.0U2 (MTU1500)
Wii
Hi
Thanks for your amazing OVA.
I Install 2 Nested ESXi from ISO and 2 from the Nested 6u2 OVA.
When I install NSX VIB, it fail on thos 2 from OVA, may be something with the dvFilter.
Thanks for your work
Hi,
Sorry not from your OVA but from me. On the OVA ESXi I had the wrong DNS and the VIB Installation need a DNS to resolve vCenter
Hi William,
I have had an interesting issue when configuring VSAN using add_esxi_vsan_appliance_to_cluster.ps1 script.
VSAN was set to automatically provision disks, but it couldn't find any storage devices and create a storage group. I thought it might be because of no license was assigned to the cluster (in VSAN 6.2, the configuration page shows the message that the disk auto provisioning requires the appropriate license to be assigned to the cluster). I added VSAN Enterprise license to the system. However, it didn't help to proceed with the disk configuration and went with disk error -1 message when I setup it manually.
Any advise where I should look for?
Thanks,
Question from a newbie here... If I want to add VM's to a nested ESXi, will I need to add additional hardware, CPU, RAM, disk space to the nested ESXi or how will it now to use the external hardware... I'm trying to deploy an OVA to the Nested ESXi you created but I get not enogh disk space on the Local DS.
Links are both dead. Looking for the 5.5 - link says forbidden!
None of the links work. Can you please fix.
Folks - I've just been made aware that we'll be moving away from Bintray and hence why the links are dead. I need to find a new hosting source, so hopefully I'll have an update later this week.
William, Any chance you can upload the OVA's to a private cloud that I am willing to host for anyone to access while a permanent solution is found?
Hi Dizzy,
I was actually planning to use Dropbox for the short term, will share an update once I have those links up. Thanks for the offer or if anyone else has perm solution that would like to share, I'm all ears as well 🙂
I've just updated the download links w/Dropbox URLs for time being.
Excellent William, any know issue when trying to install NSX VIBs in the nested ESXi ?
I've got na nested esxi lab and it works perfectly.
Great work William, thank you for it. So I can do a lot of tests before going productive,
Great Work. It is really helpful.
Can you please help me on the below issue ?
I have set guestinfo.createvmfs= True, this works only if i go with default disk capacity.
VMFS datastore1 is not created If we reconfigure HDD 1 as 200 GB before poweron while doing initial deployment
I resolved it. If we increase the HDD 3 instead of HDD1 , VMFS is created successfully
Hello
I build a lab using 4 ESXi 6.0 Virtual Appliance, finally the capacity of the VSAN is just 7,47Go the virtual disks was well viewed
Coud you explain why i had just 7,47Go
Thank you,
Hi William,
Great work. A quick note on the deploy script. First, the script may fail to deploy because the manifest cannot be verified. Conversion of the OVA to OVF using the "--skipManifestGeneration" allows the OVF to deploy successfully.
Second, in your deploy script, I noticed the $iprange variable has an IP Address of 192.168.1.1. This will cause the DNS lookup function to fail as the IP address it will attempt to find is 192.168.1.1.x. In my case, the lookup tried 10.10.1.1.100-107. I updated the $iprange to 10.10.1 and all worked well.
Jim
Oh Jim, Should have seen your comment first ,, been tried and figured out that Manifest generation.. took me like 1+ hour..Anyways thanks for Mentioning.
I know this post is very old but I've only just discovered the v6 appliance.
I have it deployed fine ontop of a simple exi5.5 setup (on real tin).
I have the vswitches set to accept for all three categories but I still can not ping the vmkp of the esxiv6 appliance so unable to connect it to a v6 vcsa I have installed on the same v5.5 environment.
Is there something I am missing
(I have nested esxi5.5 (not appliance version) working fine that I use to test esxi patching.
William - great work -- I deployed the v6.5 OVA ---only issue I have is I cannot ping its IP address - in VM's settings it says OS is not supported ? Let me know what I am missing - all nics are connected online - my lab environment is v6.0 U2 - what changes are needed to the VMX file if any ?
Hi, its nice article… I have my nested esxi. Now, if I want my nested esxi to have ip from dhcp server rather than 192.168.1.190, how can I do that? Thanks in advance
How to add virtual machine to this nested esxi?
Awesome job William! I've been using the 6.0 version for a while and it works great! I came back here today to get the 5.5 version (looking at testing some upgrade paths) and noticed you had decommissioned it yesterday (just my luck and timing). Would it still be possible to get a copy of 5.5?
Thanks
John
Hi William,
can we access the over esxi build for the .ova? I was hoping to get a 6.0.u2 version
Any chance you would be willing to share the code used for the VIB to create the appliance?
Hi,
Note: Pardon me. I am a novice in VMWare ESXi
I am trying to run the ESXi in EVE-NG as per the following URL
https://interestingtraffic.nl/2017/03/21/running-esxi-nodes-in-eve-ng/
In which in the bottom section of the author has mentioned that, to enable the VM's to start in a nested environment, advised to add the following
"add vmx.allowNested = TRUE to your VM's configuration (in the Web Client under Edit Settings > VM Options > Advanced > Edit Configuration > Add Parameter"
It is also been mentioned that, to make this default for all VM's this can be added to the /etc/vmware/config. When I checked the config, the config was different and I was bit reluctant to configure. While I was researching through i found esx.conf is having lot of parameters and I think this parameter can be added here. But I am not sure how do I do that.
Can someone help me out in adding this parameter
Thank you
PAM
Hi William, your Nested ESXi OVA works great.. and it really helped me in setting up a lab environment.
Would it be possible to share the configuration changes that has been done in the OVA.
I want to know what I was missing with configuration.
Regards
Ravi
For some reason the login for the esxi 6.5 of login name root and password is left blank does not work. is there another way to login?
VMware1!
I wasn't able to deploy the 6.0 Nested ESX using ovftool on Linux. The deployment options weren't recognized.
Can someone give an example?
I keep on getting the following error even after a new deploy with no modifications, "Call "PropertyCollector.RetrieveContents" for object "ha-property-collector" on ESXi "172.16.1.101" failed." Any ideas?
I'm having trouble using this for the lab but using a VSAN datastore. Any idea?
Take a look at https://www.williamlam.com/2013/11/how-to-run-nested-esxi-on-top-of-vsan.html
Thanks William, yes I've tried that but it still did not work. Moving is to a normal datastore works fine.
That's usually 99.9% of the time 🙂 I'd recommend you double check and ensure you've applied the setting to the physical ESXi host and NOT the Nested ESXi instances (which is a common mistake many folks make)
On the physical host... now I cannot do it... we have non PROD VM using the VSAN datastore.
Thanks again ... I don't have a choice but use normal DS.
Weird thing is my colleague was able do it without this setting. will try to dig some more.
Hi William,
I've tried your latest ESXi 6.5 nested VM aind got it working for basic VM's etc. on my existing host with ESXi_Learnswitch installed and configured. However when enabling the cluster for NSX the install/setup works but logical switch connectivity is broken. The LS test shows random results, with some host-host pings working OK at 1600MTU , but as soon as I try VM to VM pings, the LS test shows failures. Pinging the vxlan stack with size 1572 packets between hosts always works, however, so the problem is not MTU in the dvs or physical switch, but somewhere else.
NSX traceflow shows packets stopping at the ESXi host VTEP vNIC, with no further flow or errors.
I have promiscuous mode and forged xmit set on the esxi vm's portgroup connections.
VM's on a LS within one cluster sometimes talk, but VM's on physical cluster pinging to nested Cluster, on the same Transport zone, never have been able to ping.
Is this an interaction between esxi_learnswitch and the dv-filter in your ovf ? or are these additional things needed for NSX to work. I see some info on setting promiscuous mode required on vxlan, but it is old info.
Any chance summarizing what is needed for current version of your OVF, esxi 6.5 , and NSX 4.3. I'm finding so many back-references and nothing definitive on nesting NSX hosts on current software.
Lee
This post seems to describe accurately what I'm seeing in my environment:
https://vlenzker.net/2016/04/nsx-and-nested-esxi-environments-caveats-layer-2-troubleshooting/
Haven't tried it out yet.
Lee
As well as this link:
https://telecomoccasionally.wordpress.com/2016/03/10/from-the-dept-of-the-knowledge-arcane-nsx-v-with-nested-esxi/
I'll try disabling vxlan on physical host.
So this appears to be a bug ( in un-supported nested ESXi ) that is causing lots of trouble for people with NSX labs.
Lee
G'day Will!
Fantastic work here, very useful!
A few notes/questions:
- Whilst the OVF configuration allows a specification for the host syslog target, it doesn't enable the syslog firewall ruleset and thus all of the syslog traffic is blocked from ever arriving at the syslog target until we explicitly go and enable that ruleset. Is it possible to include the enabling of the "syslog" ruleset as part of the deployment?
- What is the correct method for specifying more than 1 DNS/NTP server in the OVF configuration? I have tried comma separated but the values are ignored altogether if I do that.
Cheers!
Kane.
Hi Kane,
Thanks for the feedback.
1) Hm, I don't recall this was explicitly needed after several releases back as the FW should automatically be handled during the "reload" operation. To be honest, this is the first time its been reported. Happy to add that in a future update of the Appliance
2) There's only support for single DNS/NTP to keep the setup simple. You can always handle this via post-deployment for further customizations