WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud
  • Tanzu
    • Application Modernization
    • Tanzu services
    • Tanzu Community Edition
    • Tanzu Kubernetes Grid
    • vSphere with Tanzu
  • Home Lab
  • Nested Virtualization
  • Apple
You are here: Home / ESXi / Deploying Nested ESXi is even easier now with the ESXi Virtual Appliance

Deploying Nested ESXi is even easier now with the ESXi Virtual Appliance

12.14.2015 by William Lam // 92 Comments

Several months back I had built an ESXi Virtual Appliance that allows anyone to quickly stand up a fully functional Nested ESXi VM which includes guest customization such as networking, NTP, syslog, passwords, etc. The virtual appliance was initially built for my own personal use as I found myself constantly rebuilding my lab environment for evaluating and breaking new VMware software. I figured if this was useful for myself, it probably could benefit others at VMware and I posted the details internally on our Socialcast forum. Since then, I have received numerous stories on how helpful the ESXi Virtual Appliance has been for both our Field and Engineering for setting up demos, POCs, evaluations, etc.

Most recently, I was contacted by Massimo Re Ferre' (crazy Mainframe guy ;)) who works over in our Cloud Native Apps team and was working on a pretty cool project with Photon Controller which was recently open sourced. He was interested in leveraging the ESXi Virtual Appliance along with using VMware AppCatalyst to make it super simple for anyone to try out the Photon Controller in their own environment. Over the last couple of weeks, I have been working closely with Massimo on incorporating on his requirements for the Photon Controller POC back into my ESXi Virtual Appliance. My original goal for the appliance was to keep it generic so that it could cater to multiple use cases and Photon Controller POC was just another neat solution that could be built on top of.

I just found out today that new Photon Controller POC has just been released and you can find more details in the links below:

  • Photon Controller main page
  • Getting started guide for deploying Photon Controller on OS X
  • Getting started guide for deploying Photon Controller on Windows
  • Photon Controller Google Group

As part of the release, the ESXi Virtual Appliance is also made available which I thought was pretty cool! 😀 I highly recommend you check out the awesome work done by Massimo if you want to play with Photon Controller. This is a really easy way of getting started with Photon Controller and giving it a spin in your own environment.

Since the ESXi Virtual Appliance is now available externally, I wanted to share a few details about the appliance for those who might be interested in checking it out. As I mentioned earlier, the goal of the ESXi Virtual Appliance was to be generic and to be used as a build block that could enable different use cases such as spinning up a quick vSphere lab using it and the VCSA or putting together a fully functional VSAN lab in literally a couple of minutes (at the very bottom, I have a couple of PowerCLI scripts to demonstrate this). You could deploy 3 instances of the appliance to get a basic 3 Node VSAN Cluster or you could scale up to 64 Node VSAN Cluster all within just minutes. The limit is truly your imagination.

The appliance contains a pre-installed GA release of ESXi 6.0 Update 1. There are 11 OVF properties that are available for customizing the Nested ESXi VM which are shown in the table below. Once powered on, the default 60 day evaluation will start counting down as if you had manually installed ESXi yourself. In addition, the OVA also contains several optimizations for running Nested ESXi including the Mac Learn dvFilter params as well as other configurations for quickly setting up a VSAN environment which are also described below. I have also built the appliance to be easily consume in all VMware based environments including vSphere, vCloud Air, Fusion, Workstation, Player & AppCatalyst.

UPDATE (05/10/17) - Updated VA to latest ESXi 6.0u3 & 6.5d (vSAN 6.6)

  • ESXi 6.0 Update 3 Virtual Appliance download link
  • ESXi 6.5d Virtual Appliance download link

UPDATE (05/09/17) - The ESXi 5.5u3 has been decommissioned due to its limited use.

  • ESXi 5.5 Virtual Appliance download link (Decommissioned)

UPDATE (11/18/16) - ESXi 6.5 Virtual Appliance has been released and you can find the details here.

UPDATE (04/07/16) - Minor bug fix with refreshing auto-generated SSL Certificates, latest version is v5 for 6.0u2 VA and v2 for 5.5u3 VA

UPDATE (03/18/16) - I updated the ESXi 6.0 VA using vSphere 6.0 Update 2. It is now back to vHW10 for backwards compat and includes an All-Flash configuration + 2 VMXNET3 adapters and ready for VSAN 6.2 😀

UPDATE (03/01/16) - I have also created a new ESXi 5.5 VA using vSphere 5.5 Update 3b

OVF Property Description Type
guestinfo.hostname FQDN of the ESXi host string
guestinfo.ipaddress IP Address string
guestinfo.vlan VLAN ID string
guestinfo.netmask Netmask string
guestinfo.gateway Gateway string
guestinfo.dns DNS Server string
guestinfo.domain DNS Domain string
guestinfo.ntp NTP Server string
guestinfo.ssh  Whether or not SSH is enabled boolean
guestinfo.syslog Syslog Server string
guestinfo.password Root password for ESXi host string
guestinfo.createvmfs Whether to automatically creates a VMFS datastore (datastore1) on largest VMDK boolean

The ESXi 6.x Virtual Appliance includes the following configuration:

  • ESXi 6.0 Update 2
  • GuestType: ESXi 5.x (backwards compat)
  • vHW 10
  • 2 vCPU
  • 6GB vMEM
  • 2 x vmxnet3 vNIC
  • 1 x 2GB HDD (ESXi Installation)
  • 1 x 4GB SSD (for use w/VSAN, empty by default)
  • 1 x 8GB SSD (for use w/VSAN, empty by default)
  • VHV added (more info here)
  • dvFilter Mac Learn VMX params added (more info here)
  • disk.enableUUID VMX param added
  • VSAN traffic tagged on vmk0
  • Disabled VSAN device monitoring for home labs (more info here)

The ESXi 5.x Virtual Appliance includes the following configuration:

  • ESXi 5.5 Update 3b
  • GuestType: ESXi 5.x
  • vHW 10
  • 2 vCPU
  • 6GB vMEM
  • 2 x vmxnet3 vNIC
  • 1 x 2GB HDD (ESXi Installation)
  • 1 x 4GB SSD (for use w/VSAN, empty by default)
  • 1 x 8GB HDD (for use w/VSAN, empty by default)
  • VHV added (more info here)
  • dvFilter Mac Learn VMX params added (more info here)
  • disk.enableUUID VMX param added
  • VSAN traffic tagged on vmk0
  • Disabled VSAN device monitoring for home labs (more info here)
  • ESXi Embedded Host Client (more info here)

If you do not wish to use VSAN, there is an OVF property that allows you to specify whether or not a default VMFS datastore is created. You can increase the capacity of any of the disks after deployment (if you wish for the automatically VMFS creation, you will need to expand the disk prior to powering on the VM). Below are the different methods in which you can deploy the ESXi Virtual Appliance which includes vSphere, vCloud Air, Fusion, Workstation, Player & AppCatalyst. The idea is that you can easily setup Nested ESXi on any VMware based hypervisor and be up and running in just minutes!

Option 1 - Deploy to vSphere environment w/vCenter Server or vCloud Air

Download the OVA (or you can just paste the URL link) and import that into your vCenter Server using the vSphere Web Client. Make sure to accept the "Extra Configuration" when prompted and then fill out the 11 OVF properties which will allow you to customize the Nested ESXi VM to your environment. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM. Once powered on, the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM. Below are a couple of screenshots

Screen Shot 2015-12-10 at 1.35.02 PM
Screen Shot 2015-12-10 at 1.36.04 PM
Screen Shot 2015-12-10 at 1.38.51 PM

Option 2 - Deploy to ESXi

Download the OVA and import it into ESXi using either the vSphere C# Client but do not power it on. Since ESXi does not support OVF properties, you will need to add the following guestinfo*. properties as shown in "Option 3" below by using the VM Advanced Settings UI.

Screen Shot 2015-12-11 at 6.48.51 AM
If you prefer NOT to mess around with manually adding these VM Advanced Settings which can also be automated using the vSphere API or PowerCLI, one additional method which CAN make use of the OVF properties is by using ovftool which is a CLI to import the OVA and using the --injectOvfEnv option that was added in ovftool version 4.x You can find more details in this blog post here.

Option 3 - Deploy to VMware Workstation, Fusion or Player

Download the OVA and import it into Fusion/Workstation but do not power it on. You will then need to edit the VMX file and add the following guestinfo.* properties as shown below since Fusion/Workstation do not support OVF properties. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM. Once you have saved the changes, you can then power on the Nested ESXi VM and the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM.

guestinfo.hostname = "vsan-1.primp-industries.com"
guestinfo.ipaddress = "172.16.78.90"
guestinfo.netmask = "255.255.255.0"
guestinfo.gateway = "172.16.78.1"
guestinfo.dns = "172.16.78.1"
guestinfo.domain = "primp-industries.com"
guestinfo.ntp = "172.16.78.1"
guestinfo.ssh = "True"
guestinfo.syslog = "192.168.1.100"
guestinfo.password = "VMware1!"
guestinfo.createvmfs = "False"

Option 4 - Deploy to VMware AppCatalyst

Download the the OVA and import that into AppCatalyst using ovftool but do not power it on. You will then need to edit the VMX file and add the following guestinfo.* properties as in the above example for Workstation/Fusion in addition to the following params listed below which are required for Nested ESXi to run in AppCatalyst. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM.

guestos = "vmkernel6"
virtualhw.version = "11"
svga.vgaOnly = "true"

You will also need to run the following command to allow promiscuous mode in AppCatalyst since there's no UI to prompt (this is only required once):

touch "/Library/Preferences/VMware AppCatalyst/promiscAuthorized"

Once you have saved the changes, you can then power on the Nested ESXi VM and the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM.

Option 5 - Deploy to vSphere using PowerCLI

I have also created two very simple PowerCLI scripts which demonstrates how you can easily deploy N-number of these Nested ESXi VMs and in fact, you can setup a fully functional VSAN Cluster in just under 5 minutes! You can find the two scripts below:

  • deploy_esxi_vsan_appliance.ps1
  • add_esxi_vsan_appliance_to_cluster.ps1

The first script will perform the deployment of the Nested ESXi VM, but you will first need to convert the OVA to an OVF because the Get-OvfConfiguration cmdlet does not support OVA. To properly convert the OVA, you will need ovftool and then run the following command:

ovftool.exe --allowAllExtraConfig --skipManifestCheck Nested_ESXi_Appliance.ova Nested_ESXi_Appliance.ovf

The second script will go ahead and add the deployed Nested ESXi VMs to a vSphere Cluster and then enable VSAN on the vSphere Cluster.

More from my site

  • Virtual NVMe and Nested ESXi 6.5?
  • Nested ESXi Enhancements in vSphere 6.5
  • VM serial logging to the rescue for capturing Nested ESXi PSOD
  • vSphere 6.0 Update 2 hints at Nested ESXi support for Paravirtual SCSI (PVSCSI) in the future
  • VMware Tools for Nested ESXi updated to v1.2

Categories // ESXi, Fusion, Home Lab, Nested Virtualization, Not Supported, vSphere, vSphere 6.0, vSphere 6.5, Workstation Tags // esxi, nested, nested virtualization, ova, vSphere 6.0 Update 1, vSphere 6.5

Comments

  1. rmesterheide says

    12/14/2015 at 7:32 am

    Awesome work William! As you mentioned, I use your VAPP on a regular basis. Have you thought about adding support for: Choose amount of ESXi Hosts to deploy and number of VMDKs for Flash and Capacity layer? Is this even possible?

    Reply
    • William Lam says

      12/14/2015 at 10:41 am

      Being able to specify the number of instances would be possible with ScaleOutSection if the OVF 2.0 specification, but that's not currently supported today. You can easily do this with some Automation (there's even a script that I've attached to this blog post) that does this and literally deploy 3 or 64 VSAN Nodes in literally within minutes. The latter cases are more specific and as mentioned in the article, I wanted to keep a generic ESXi Appliance so it could cater to variety of use cases. You can always deploy & then add additional VDMKs based on your requirements, that's the easy part and I'm just providing a base 🙂

      Reply
  2. Tom says

    12/14/2015 at 7:35 am

    Looks awesome, William! I am downloading and deploying it to vSphere now! In your experience, have you tried adding additional vNICs to nested ESX hosts for segregation of traffic (vm, mgt, vSAN, vMotion)? Any tips / tricks? Or do you just use the one vNic?

    Reply
    • William Lam says

      12/14/2015 at 10:37 am

      Sure, you can add additional vNICs. This is no different than any other VMs 🙂 Definitely recommend using vmxnet3 and I've done this to do variety of things like supporting various networking configurations.

      Reply
      • aenagy says

        07/17/2017 at 5:59 am

        I have been trying to add a couple of more vNICs by copying and modifying the first two. The problem is that the new vNICs show up as disconnected. I'm guessing that part of the problem is knowing what values to use for some of the instance specific parameters such as: AddressOnParent, InstanceID, pciSlotNumber, ethernetX.pciSlotNumber for vNIC N+1.

        How did you generate/add the vNICs?

        Reply
  3. Kev Johnson (@kev_johnson) says

    12/14/2015 at 11:33 am

    Best news I've seen all month - this is a real boon for those of us regularly redeploying labs to test stuff. Thanks for your efforts with this!

    Reply
  4. Matjaž Antloga (@BalkanCloud) says

    12/14/2015 at 1:25 pm

    Great stuff. What about hot plug/remove vcpu/mem on nested ESXi? Any info on that?

    Reply
    • William Lam says

      12/14/2015 at 2:43 pm

      I hope you know that hotplug is based on the guestOS supporting the feature, not all OSes support it. There's a VMware KB that shows which OSes are supported. ESXi running in a physical host doesn't support hot add of memory/CPU, so it wouldn't be supported when running in a VM.

      Reply
      • Matjaž Antloga (@BalkanCloud) says

        12/15/2015 at 12:27 am

        I was actually pointing to something else. Windows 2016 will support hot plug/remove. So we will be able to deploy nested Windows 2016 hypervisors under ESXi (and sooner than later under MS hypervisor) and dynamically give them resources on demand...not a case with VMware.

        Reply
  5. Jason says

    12/14/2015 at 5:02 pm

    Thanks a lot William. Looks very interesting, I will give it a try. Especially after my ovftool confusion that I described in https://pixelschatten.net/ovftool-conversion-confusion/. Thanks again for being that responsive on Twitter and taking a look. You rock!

    I also would like to know how to create these OVF/OVA files myself. Like you did with the ESXi virtual appliance. I would like to better understand the tech behind it, learn and be able to use it internally with some of our systems. Makes deployment much easier. Do you have some pointers to the right docs for me?

    Thanks again, much appreciated!

    Reply
    • William Lam says

      12/14/2015 at 5:37 pm

      I would recommend taking a look at VMware Studio, which is authoring tool for OVFs & there's a great tech paper that describes quite a bit of the concepts. Also have a look at my OVF resources http://www.virtuallyghetto.com/ovf

      Creating an OVF can be as simple as building it in vSphere and then exporting. Feel free to inspect the ones I've built to get an idea of how things fit together.

      Reply
  6. paulbraren says

    12/14/2015 at 5:06 pm

    This is great!
    FYI, I did Option 2 above, and configured guestinfo.domain to "lab.local" yet I still get "primp-industries.com, lab.local" for my Customer DNS Suffixes. Just nitpicking, so nice and easy to deploy, thank you!

    Given the likely popularity of your download link
    https://bintray.com/artifact/download/photon-controller/public/UserWorkflow/Nested_ESXi_Appliance.ova
    I wonder if bintray knows what they're in for 😉

    Reply
    • William Lam says

      12/14/2015 at 5:39 pm

      Ah darn, it looks like when using ESXCLI to add DNS Domain, it's an additive versus replacement of the value. It looks like it picked up my local build's domain. I'll have to fix this internally and if they decide to push out another build or allow me to update, you should see that fix.

      Should be good publicity for them 😉

      Reply
  7. Dudu says

    12/14/2015 at 9:38 pm

    Wow! Nice, I will definitely check it out this weekend. So much work saved 🙂

    Reply
  8. Brian says

    12/15/2015 at 1:49 pm

    If we deploy the appliance and add NICs, etc, how could we modify the ova properties or inject new ones? Assuming we make the changes and just export it, the 11 original properties would still be valid, right?

    Reply
    • William Lam says

      12/15/2015 at 3:29 pm

      The OVA properties are only applicable on initial setup. You can always add/change the vHW after its been setup. For things like VMDKs, you can actually hotadd and just do a rescan at the ESXi layer. Once configured, you can make whatever changes and if you export it, it'll retain the settings/configurations. Of course you won't be able to change the OVF properties after that, as I said its only applicable on initial deployment

      Reply
      • Brian says

        12/16/2015 at 12:48 pm

        Gotcha. What did you use to insert the properties?

        Reply
  9. Davoud Teimouri says

    12/17/2015 at 1:09 am

    Thank you William for sharing it.

    Reply
  10. tom miller says

    12/17/2015 at 11:36 am

    William,
    Thanks again for the hard work.
    Might want to mention if you add a NIC you must also add the dvfilter commands for that NIC as well. Also, when adding a NIC cannot select VMXNET3? Had to edit the vmx file to switch e1000 to vmxnet3?

    Thanks
    Tom Miller

    Reply
    • tom miller says

      12/17/2015 at 11:52 am

      Also - NONE of the ovf answer file worked?

      Reply
      • Matt Murphy says

        07/07/2016 at 6:21 am

        I have found the same issue. The guestinfo* settings work, so I cannot login to the host to fix. The ESXi 6.0 ovf hung and never completed the boot sequence.

        Reply
  11. Sylvain says

    12/29/2015 at 7:48 am

    Hi,
    Great tool!
    Is there a way to deploy this with vRa? It would be great to be able to create "on-demand" nested esxi for testing purpose.
    Thanks
    Sylvain

    Reply
    • William Lam says

      01/02/2016 at 6:57 am

      Sure. I know of a few customers who were already doing this before this virtual appliance. This is just like another VM, so it would be possible to drive it through vRA automation, however I'm not familiar enough with that product to give you the exact steps. You may want to post in the VMTN community forums for any tips or guidances but to me, this is just another VM that would sit in the vRA Catalog and made available for deployment.

      Reply
      • Sylvain says

        01/05/2016 at 9:10 am

        Thanks William
        Actually, I managed to add a template created with the OVA and make it available through the vRA Catalog.
        Problem is: I don't know how to pass parameters to change the OVF properties (guestinfo*. properties).
        I'm just a beginer in vRA so if you're not familiar with the process either, I'll try asking in the VMTN community forums.

        Reply
        • blackburn83 says

          04/21/2016 at 12:27 am

          Hi Sylvain, did you figured this out, cause I'm trying to do the same thing, however with vRealize Orchestrator.

          Reply
  12. David says

    01/15/2016 at 10:13 am

    Hello William,

    This page is great thank you for all you have done, new to using ESX and trying to get this up and running with a second nic on my workstation.

    I have before I turned on the vm, added the second nic in the hardware settings section and assigned it to the proper network.

    QUESTION: Do I need to have two entries in the VMX file for both networks?

    Thank you,

    Reply
  13. Zubin Parihar says

    02/06/2016 at 9:26 am

    Hey William,

    Love your articles and posts. They've seriously helped me out. Keep it up!

    I don't really know how to contact you but I'm in need of some expertise.

    I'm trying to correlate the Portgroup number with the dvswitch names from the command line, but having a lot of troubles figuring it out.

    1.) Here is how I was able to get the Portgroups:

    [[email protected]:~] esxcli network vswitch dvs vmware list | grep DVPortgroup | sort | uniq
    DVPortgroup ID: dvportgroup-3970
    DVPortgroup ID: dvportgroup-3971
    DVPortgroup ID: dvportgroup-3972
    DVPortgroup ID: dvportgroup-3981

    2.) Here is how I was able to get the PortgroupNames

    [[email protected]:~] vim-cmd hostsvc/net/dvs_info

    ....
    .....
    portgroupName = (string) [
    "dvSwitch-vSAN-NV-DVUplinks-3969",
    "dv-VSAN",
    "dv-vMotion-1",
    "dv-VM-DEV-10.188.0.0-1"
    ],
    ......
    .....

    However, "dvSwitch-vSAN-NV-DVUplinks-3969" is not really a valid dropdown option when I'm trying do a deploy from the interface.

    3.) This what I get for Standard VSwitch

    [[email protected]:~] esxcli network vswitch standard list
    vSwitch0
    Name: vSwitch0
    Class: etherswitch
    Num Ports: 10752
    Used Ports: 4
    Configured Ports: 128
    MTU: 1500
    CDP Status: listen
    Beacon Enabled: false
    Beacon Interval: 1
    Beacon Threshold: 3
    Beacon Required By:
    Uplinks: vmnic0
    Portgroups: Management Network

    vSwitch1
    Name: vSwitch1
    Class: etherswitch
    Num Ports: 10752
    Used Ports: 4
    Configured Ports: 128
    MTU: 1500
    CDP Status: listen
    Beacon Enabled: false
    Beacon Interval: 1
    Beacon Threshold: 3
    Beacon Required By:
    Uplinks: vmnic1
    Portgroups: PERF_DATA_META

    4.) Another Vswitch and DVSwitch listing

    [[email protected]:~] esxcfg-vswitch -l
    Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
    vSwitch0 10752 4 128 1500 vmnic0

    PortGroup Name VLAN ID Used Ports Uplinks
    Management Network 0 1 vmnic0

    Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
    vSwitch1 10752 4 128 1500 vmnic1

    PortGroup Name VLAN ID Used Ports Uplinks
    PERF_DATA_META 3962 1 vmnic1

    DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
    dvSwitch-vSAN-NVAN 10752 49 512 1500 vmnic5,vmnic4

    DVPort ID In Use Client
    22 1 vmnic4
    23 1 vmnic5
    14 1 vmk1
    10 1 vmk2
    100 1 dvtest-ops-tan.eth0
    .... .. ...............

    (The names and prompts have been modified in my paste for security, but i changed them consistently)

    What I'm trying to find out is to find a correlated list like this:

    DVPortgroup ID: dvportgroup-3970 "dv-vMotion-1",
    DVPortgroup ID: dvportgroup-3971 "dv-VSAN",
    ....................... ........................ ....................

    Where I know what 'dvportgroup.....' in my VMX file is in terms of its correlated name.

    I feel like and think that I've explored all the Command Line Options and Outputs (i think). Any clues, or better yet answers would be most appreciate.

    Cheers,

    Zubin

    Reply
  14. Francisco Manso says

    02/24/2016 at 4:17 pm

    William,

    This post is awesome. Now I have a question.

    My usage for the OVA is to develop other OVA, and keep the Nested ESX OVA in the workflow.

    So basically we keep the OVA in our artifactory server, then:
    * our maven process deploys it with ovftool, ( we need to add few tweaks here. See later note)
    * packer creates an OVA inside this Nested ESX.
    * maven creates the OVA using ovftool
    * we destroy the virtual ESX

    This way in 7 years we can reproduce our build with the exact same version of ESX and we do not need to keep it on.

    Now the note I refereed above, is that packer requires some configuration on the ESX.
    - Enable GuestIPHack
    - adding one extra local admin ( not completely necessary but we do it)
    - adding rsh keys ( via /etc/rc.local.d/local.sh)
    - opening VNC inbound ports (via: /etc/rc.local.d/local.sh and /etc/vmware/firewall/service.xml)

    So once the OVA is configured, I close it and I use ovftool to export it again, but anytime I deploy it it has the IP settings of the original, and it disregards all the -prop:guestinfo.XXX information

    I think I am missing how to reset the vami parameters before I export my customized Nested ESX.
    You use Vmware Studio, but it cannot be that complicated, just to reset the vami properties and create another OVA.

    Any help would be appreciated.

    Reply
    • Francisco Manso says

      02/24/2016 at 5:14 pm

      I found that you hvae installed this:

      [[email protected] esxcli software vib list | grep Ghetto
      nested-esxi-customization 5.0.0-1.0.0 virtuallyGhetto VMwareCertified 2015-12-24

      I suppose that is what I am looking for.. now let's find how it works!!

      Reply
    • Francisco Manso says

      02/24/2016 at 5:37 pm

      I am answering all my questions. 🙂

      You do it on /etc/rc.local.d/098* scripts.

      Now I need to figure out why addinging a new static IP does not work.

      Reply
    • Francisco Manso says

      02/24/2016 at 5:46 pm

      I found it, sorry for the long reply, now I can create my customized OVA

      # esxcli system settings advanced set -o /UserVars/vGhettoSetup -i 0

      Reply
  15. Francisco Manso says

    02/26/2016 at 11:09 am

    William,
    I am trying to get one more funtionality on this Netsted ESX and I see that you have this file /etc/rc.local.d/098.vghetto_config

    but you do not allow dhcp option, what it good for a lot of environments, mainly when we cna deploy with ovftool with options --powerON -X:waitForIP, and get the IP of the ESX, so we can work on it.

    I have modified the file this way:

    # diff 098.vghetto_config 098.vghetto_config.DST
    70,82c70,77
    < if ip == 'dhcp':
    < os.system("esxcfg-init --set-boot-progress-text \"Applying DHCP Network Configuration ...\"")
    < os.system("esxcli network ip interface ipv4 set -i vmk0 -t none")
    < os.system("esxcli network ip interface ipv4 set -i vmk0 -t dhcp -P 1")
    < else:
    < os.system("esxcfg-init --set-boot-progress-text \"Applying Static Network Configurations ...\"")
    < os.system("esxcli network ip interface ipv4 set -i vmk0 -I " + ip + " -N " + netmask + " -t static")
    < os.system("esxcli network ip route ipv4 add -n default -g " + gateway)
    < os.system("esxcfg-init --set-boot-progress-text \"Configuring hostname ...\"")
    < os.system("esxcli system hostname set -f " + hostname)
    < os.system("esxcfg-init --set-boot-progress-text \"Configuring DNS ...\"")
    < os.system("esxcli network ip dns server add -s " + dns)
    os.system("esxcfg-init --set-boot-progress-text \"Applying Network Configurations ...\"")
    > os.system("esxcli network ip interface ipv4 set -i vmk0 -I " + ip + " -N " + netmask + " -t static")
    > os.system("esxcli network ip route ipv4 add -n default -g " + gateway)
    > os.system("esxcfg-init --set-boot-progress-text \"Configuring hostname ...\"")
    > os.system("esxcli system hostname set -f " + hostname)
    > os.system("esxcfg-init --set-boot-progress-text \"Configuring DNS ...\"")
    > os.system("esxcli network ip dns server add -s " + dns)
    > os.system("esxcli network ip dns search add -d " + domain)

    But, it looks like this file is one of those that is not backed up in the boot bank, so at boot time it reverts to the saved version without my changes.
    I also run /sbin/auto-backup.sh with no luck

    My question is:
    where can I make these changes to /etc/rc.local.d/098.vghetto_config is changed at boot time?

    I could eventually create a /etc/rc.local.d/.#098.vghetto_config file, but I wonder how you manage to put it there.

    Maybe I find the answer myself later, but if you have a chance I'd appreciate it.

    Reply
  16. Mattias says

    03/04/2016 at 11:17 am

    Hi William,

    First of all, love your work man, keep it up!
    Would you consider making this release or the next of Nested ESXi 6 appliance = vHW10 like the vcsa 6 appliance is? Makes it easier to deploy on vSphere 5.5.

    Reply
  17. Sullybags says

    03/08/2016 at 6:01 am

    HI William

    awesome post, i'll use this a lot

    i think there was a esxi6 nested appliance hardware version 10 up here before?

    the one above is hw v11

    any chance i could get the hw v10 one again? i deleted my old one unfortunately!

    thx

    Reply
    • William Lam says

      03/08/2016 at 8:10 am

      Yea, I had a few people ask about this and I plan to build a new one using vHW10 once vSphere 6.0 Update 2 has GA'ed. If you need right away, you can take the existing OVA and convert it to OVF using ovftool and then manually changing the vHW requirement from 11 to 10 which will continue to work.

      Reply
  18. Mark Jones says

    03/16/2016 at 7:13 am

    William,
    We also had a need for a V10 and we actually created our own OVF, but even when I put the config parameters for guestinfo in they don't show up when I do the get-ovfconfiguration?

    PowerCLI C:\TEMP\esx> Get-OvfConfiguration .\esx.ovf

    =========================
    OvfConfiguration: esx.ovf

    Properties:
    -----------
    NetworkMapping

    Reply
    • Mark Jones says

      03/16/2016 at 10:37 am

      In addition to this I attempted to convert to VMX and then back to OVF to set to hardware 10, but it set the networking to bridged and broke it :/

      Reply
      • William Lam says

        03/16/2016 at 10:58 am

        Now that vSphere 6.0 Update 2 has officially GA'ed, I'll be updating my 6.x Nested ESXi Appliance to that version. I'm hoping I'll get some time this week to do that. So stay tune for that

        Reply
        • Mark Jones says

          03/16/2016 at 11:52 am

          Gotcha, thanks I was hoping I could just get it working, but at this point I guess I should just cut my losses and wait for the export. Any idea why it changed the networking to bridged??

          Reply
        • Matt says

          03/17/2016 at 12:53 pm

          Thanks, William! I'll keep an eye out for the download link to come active again.

          Reply
          • William Lam says

            03/21/2016 at 6:27 am

            In case you didn't see the update, the link has been re-activated and the 6.x link now points to latest 6.0u2 Nested ESXi VA

  19. AdesfireAdesfire says

    03/16/2016 at 10:13 am

    Hello, trying to deploy the nested_esxi_appliance_v6.0.ova on my esxi 6.0 free, i get a "The task was canceled by a user" with no errors...
    I don't understand why !

    Reply
  20. Tim Dressel says

    03/17/2016 at 11:42 am

    Thank-you for this! Saves me a lot of time that I've been wasting with my nested labs!!

    Reply
  21. Ildar Mukhametzyanov says

    04/03/2016 at 6:27 am

    Thank you for great job!

    But anyway, after deploying version 5.5 appliance, my 64-bit nested VMs didn't work.
    Had to add "vhv.enabled="true"" manually to each nested ESXi.

    Could you please re-check it?

    Reply
  22. Dustin Mitton says

    04/07/2016 at 12:42 pm

    Trying to download the Nested ESXi 6 U2 OVA and the link doesn't appear to be working. I'd love a shot at downloading it so that I can start playing around with Photon Controller!

    Thanks for all your hard work for the community, I know I certainly appreciate learning from your efforts!

    Reply
    • Calin Chiorean (@yotis) says

      04/07/2016 at 6:16 pm

      I have the same issue here 🙁 William may I ask you please to check the links?

      Reply
      • William Lam says

        04/07/2016 at 6:46 pm

        Links should be working now, perhaps there was an issue w/bintray earlier.

        FYI - The latest version for 6.0u2 is v5 and 5.5u3 is v2 (there's a version associated when you try to import using either vSphere Web/C# Client), so make sure you're pulling the latest

        Reply
        • Calin Chiorean (@yotis) says

          04/07/2016 at 10:20 pm

          Thank you! Really appreciate your commitment!

          Reply
        • Dustin Mitton says

          04/11/2016 at 8:10 am

          Thanks again, I was able to get the latest file!

          Reply
  23. Johny says

    04/13/2016 at 6:12 am

    Hello,

    Is there a way to apply VLAN for ESXi, or does anyone know how to identify whats the exact VLAN id key in properties
    like: "guestinfo.vlan" and apply the value for it

    Reply
  24. Wii says

    04/21/2016 at 2:14 pm

    Hi folks,

    BYI
    After deploying my vSAN lab (W2012R2+VMwareWS12+vCenter) I've got an alarms on my vSAN. There was a MTU mismatch between W2012R12 (MTU 1514) and esxi6.0U2 (MTU1500)

    Wii

    Reply
  25. Fouad says

    04/25/2016 at 8:03 am

    Hi
    Thanks for your amazing OVA.

    I Install 2 Nested ESXi from ISO and 2 from the Nested 6u2 OVA.

    When I install NSX VIB, it fail on thos 2 from OVA, may be something with the dvFilter.

    Thanks for your work

    Reply
    • Fouad says

      04/25/2016 at 8:17 am

      Hi,
      Sorry not from your OVA but from me. On the OVA ESXi I had the wrong DNS and the VIB Installation need a DNS to resolve vCenter

      Reply
  26. Roman says

    07/03/2016 at 10:19 pm

    Hi William,

    I have had an interesting issue when configuring VSAN using add_esxi_vsan_appliance_to_cluster.ps1 script.

    VSAN was set to automatically provision disks, but it couldn't find any storage devices and create a storage group. I thought it might be because of no license was assigned to the cluster (in VSAN 6.2, the configuration page shows the message that the disk auto provisioning requires the appropriate license to be assigned to the cluster). I added VSAN Enterprise license to the system. However, it didn't help to proceed with the disk configuration and went with disk error -1 message when I setup it manually.

    Any advise where I should look for?

    Thanks,

    Reply
  27. David Valenzuela Jr says

    07/07/2016 at 2:10 pm

    Question from a newbie here... If I want to add VM's to a nested ESXi, will I need to add additional hardware, CPU, RAM, disk space to the nested ESXi or how will it now to use the external hardware... I'm trying to deploy an OVA to the Nested ESXi you created but I get not enogh disk space on the Local DS.

    Reply
  28. JenBell says

    07/09/2016 at 2:02 pm

    Links are both dead. Looking for the 5.5 - link says forbidden!

    Reply
  29. Dizzy says

    07/11/2016 at 8:46 am

    None of the links work. Can you please fix.

    Reply
  30. William Lam says

    07/11/2016 at 9:32 am

    Folks - I've just been made aware that we'll be moving away from Bintray and hence why the links are dead. I need to find a new hosting source, so hopefully I'll have an update later this week.

    Reply
    • Dizzy says

      07/11/2016 at 9:43 am

      William, Any chance you can upload the OVA's to a private cloud that I am willing to host for anyone to access while a permanent solution is found?

      Reply
      • William Lam says

        07/11/2016 at 10:14 am

        Hi Dizzy,

        I was actually planning to use Dropbox for the short term, will share an update once I have those links up. Thanks for the offer or if anyone else has perm solution that would like to share, I'm all ears as well 🙂

        Reply
        • William Lam says

          07/11/2016 at 10:19 am

          I've just updated the download links w/Dropbox URLs for time being.

          Reply
  31. Brian Lee Foster says

    08/18/2016 at 4:52 pm

    Excellent William, any know issue when trying to install NSX VIBs in the nested ESXi ?

    Reply
    • Rob says

      08/18/2016 at 11:57 pm

      I've got na nested esxi lab and it works perfectly.

      Reply
  32. Detlef says

    09/30/2016 at 6:23 am

    Great work William, thank you for it. So I can do a lot of tests before going productive,

    Reply
  33. SANKARA NARAYANAN M says

    10/06/2016 at 11:54 am

    Great Work. It is really helpful.

    Can you please help me on the below issue ?

    I have set guestinfo.createvmfs= True, this works only if i go with default disk capacity.

    VMFS datastore1 is not created If we reconfigure HDD 1 as 200 GB before poweron while doing initial deployment

    Reply
    • SANKARA NARAYANAN M says

      10/06/2016 at 12:45 pm

      I resolved it. If we increase the HDD 3 instead of HDD1 , VMFS is created successfully

      Reply
  34. missdream says

    11/22/2016 at 8:41 am

    Hello
    I build a lab using 4 ESXi 6.0 Virtual Appliance, finally the capacity of the VSAN is just 7,47Go the virtual disks was well viewed

    Coud you explain why i had just 7,47Go

    Thank you,

    Reply
  35. Jim says

    11/30/2016 at 11:45 am

    Hi William,

    Great work. A quick note on the deploy script. First, the script may fail to deploy because the manifest cannot be verified. Conversion of the OVA to OVF using the "--skipManifestGeneration" allows the OVF to deploy successfully.

    Second, in your deploy script, I noticed the $iprange variable has an IP Address of 192.168.1.1. This will cause the DNS lookup function to fail as the IP address it will attempt to find is 192.168.1.1.x. In my case, the lookup tried 10.10.1.1.100-107. I updated the $iprange to 10.10.1 and all worked well.

    Jim

    Reply
    • Geek in LA (@in4sometech) says

      12/18/2016 at 10:41 pm

      Oh Jim, Should have seen your comment first ,, been tried and figured out that Manifest generation.. took me like 1+ hour..Anyways thanks for Mentioning.

      Reply
  36. Richard says

    04/07/2017 at 8:08 am

    I know this post is very old but I've only just discovered the v6 appliance.
    I have it deployed fine ontop of a simple exi5.5 setup (on real tin).
    I have the vswitches set to accept for all three categories but I still can not ping the vmkp of the esxiv6 appliance so unable to connect it to a v6 vcsa I have installed on the same v5.5 environment.

    Is there something I am missing
    (I have nested esxi5.5 (not appliance version) working fine that I use to test esxi patching.

    Reply
  37. Roger says

    04/12/2017 at 6:40 am

    William - great work -- I deployed the v6.5 OVA ---only issue I have is I cannot ping its IP address - in VM's settings it says OS is not supported ? Let me know what I am missing - all nics are connected online - my lab environment is v6.0 U2 - what changes are needed to the VMX file if any ?

    Reply
  38. Rashmi says

    04/19/2017 at 9:42 am

    Hi, its nice article… I have my nested esxi. Now, if I want my nested esxi to have ip from dhcp server rather than 192.168.1.190, how can I do that? Thanks in advance

    Reply
  39. Rashmi says

    04/19/2017 at 3:42 pm

    How to add virtual machine to this nested esxi?

    Reply
  40. John Hays says

    05/10/2017 at 11:04 am

    Awesome job William! I've been using the 6.0 version for a while and it works great! I came back here today to get the 5.5 version (looking at testing some upgrade paths) and noticed you had decommissioned it yesterday (just my luck and timing). Would it still be possible to get a copy of 5.5?

    Thanks

    John

    Reply
  41. Dave OSullivan says

    06/16/2017 at 3:36 am

    Hi William,

    can we access the over esxi build for the .ova? I was hoping to get a 6.0.u2 version

    Reply
  42. Corey Dickson says

    07/28/2017 at 2:47 pm

    Any chance you would be willing to share the code used for the VIB to create the appliance?

    Reply
  43. PAM says

    08/26/2017 at 12:48 am

    Hi,

    Note: Pardon me. I am a novice in VMWare ESXi

    I am trying to run the ESXi in EVE-NG as per the following URL

    https://interestingtraffic.nl/2017/03/21/running-esxi-nodes-in-eve-ng/

    In which in the bottom section of the author has mentioned that, to enable the VM's to start in a nested environment, advised to add the following

    "add vmx.allowNested = TRUE to your VM's configuration (in the Web Client under Edit Settings > VM Options > Advanced > Edit Configuration > Add Parameter"

    It is also been mentioned that, to make this default for all VM's this can be added to the /etc/vmware/config. When I checked the config, the config was different and I was bit reluctant to configure. While I was researching through i found esx.conf is having lot of parameters and I think this parameter can be added here. But I am not sure how do I do that.

    Can someone help me out in adding this parameter

    Thank you

    PAM

    Reply
  44. Ravi says

    09/01/2017 at 8:00 am

    Hi William, your Nested ESXi OVA works great.. and it really helped me in setting up a lab environment.
    Would it be possible to share the configuration changes that has been done in the OVA.
    I want to know what I was missing with configuration.
    Regards
    Ravi

    Reply
  45. Larry Whitfield says

    09/07/2017 at 3:37 am

    For some reason the login for the esxi 6.5 of login name root and password is left blank does not work. is there another way to login?

    Reply
    • Bruno Filipe says

      02/16/2019 at 5:01 am

      VMware1!

      Reply
  46. Pierre Rajha says

    09/28/2017 at 1:48 pm

    I wasn't able to deploy the 6.0 Nested ESX using ovftool on Linux. The deployment options weren't recognized.
    Can someone give an example?

    Reply
  47. tkrn says

    10/10/2017 at 11:20 am

    I keep on getting the following error even after a new deploy with no modifications, "Call "PropertyCollector.RetrieveContents" for object "ha-property-collector" on ESXi "172.16.1.101" failed." Any ideas?

    Reply
  48. mheloy says

    04/09/2018 at 9:05 pm

    I'm having trouble using this for the lab but using a VSAN datastore. Any idea?

    Reply
    • William Lam says

      04/10/2018 at 7:27 am

      Take a look at https://www.williamlam.com/2013/11/how-to-run-nested-esxi-on-top-of-vsan.html

      Reply
      • mheloy says

        04/10/2018 at 12:16 pm

        Thanks William, yes I've tried that but it still did not work. Moving is to a normal datastore works fine.

        Reply
        • William Lam says

          04/10/2018 at 12:38 pm

          That's usually 99.9% of the time 🙂 I'd recommend you double check and ensure you've applied the setting to the physical ESXi host and NOT the Nested ESXi instances (which is a common mistake many folks make)

          Reply
          • mheloy says

            04/10/2018 at 5:06 pm

            On the physical host... now I cannot do it... we have non PROD VM using the VSAN datastore.
            Thanks again ... I don't have a choice but use normal DS.
            Weird thing is my colleague was able do it without this setting. will try to dig some more.

  49. leemarzke says

    08/22/2018 at 10:05 am

    Hi William,

    I've tried your latest ESXi 6.5 nested VM aind got it working for basic VM's etc. on my existing host with ESXi_Learnswitch installed and configured. However when enabling the cluster for NSX the install/setup works but logical switch connectivity is broken. The LS test shows random results, with some host-host pings working OK at 1600MTU , but as soon as I try VM to VM pings, the LS test shows failures. Pinging the vxlan stack with size 1572 packets between hosts always works, however, so the problem is not MTU in the dvs or physical switch, but somewhere else.

    NSX traceflow shows packets stopping at the ESXi host VTEP vNIC, with no further flow or errors.
    I have promiscuous mode and forged xmit set on the esxi vm's portgroup connections.

    VM's on a LS within one cluster sometimes talk, but VM's on physical cluster pinging to nested Cluster, on the same Transport zone, never have been able to ping.

    Is this an interaction between esxi_learnswitch and the dv-filter in your ovf ? or are these additional things needed for NSX to work. I see some info on setting promiscuous mode required on vxlan, but it is old info.

    Any chance summarizing what is needed for current version of your OVF, esxi 6.5 , and NSX 4.3. I'm finding so many back-references and nothing definitive on nesting NSX hosts on current software.

    Lee

    Reply
    • leemarzke says

      08/23/2018 at 10:02 am

      This post seems to describe accurately what I'm seeing in my environment:
      https://vlenzker.net/2016/04/nsx-and-nested-esxi-environments-caveats-layer-2-troubleshooting/

      Haven't tried it out yet.
      Lee

      Reply
    • leemarzke says

      08/23/2018 at 10:12 am

      As well as this link:
      https://telecomoccasionally.wordpress.com/2016/03/10/from-the-dept-of-the-knowledge-arcane-nsx-v-with-nested-esxi/

      I'll try disabling vxlan on physical host.

      So this appears to be a bug ( in un-supported nested ESXi ) that is causing lots of trouble for people with NSX labs.
      Lee

      Reply
  50. Kane says

    08/31/2021 at 12:15 am

    G'day Will!

    Fantastic work here, very useful!

    A few notes/questions:

    - Whilst the OVF configuration allows a specification for the host syslog target, it doesn't enable the syslog firewall ruleset and thus all of the syslog traffic is blocked from ever arriving at the syslog target until we explicitly go and enable that ruleset. Is it possible to include the enabling of the "syslog" ruleset as part of the deployment?

    - What is the correct method for specifying more than 1 DNS/NTP server in the OVF configuration? I have tried comma separated but the values are ignored altogether if I do that.

    Cheers!
    Kane.

    Reply
    • William Lam says

      08/31/2021 at 6:00 am

      Hi Kane,

      Thanks for the feedback.

      1) Hm, I don't recall this was explicitly needed after several releases back as the FW should automatically be handled during the "reload" operation. To be honest, this is the first time its been reported. Happy to add that in a future update of the Appliance
      2) There's only support for single DNS/NTP to keep the setup simple. You can always handle this via post-deployment for further customizations

      Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Search

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Infrastructure Business Group (CIBG) at VMware. He focuses on Cloud Native technologies, Automation, Integration and Operation for the VMware Cloud based Software Defined Datacenters (SDDC)

Connect

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Recent

  • How to enable passthrough for USB Network Adapters claimed by ESXi CDCE Driver? 03/30/2023
  • Self-Contained & Automated VMware Cloud Foundation (VCF) deployment using new VLC Holodeck Toolkit 03/29/2023
  • ESXi configstorecli enhancement in vSphere 8.0 Update 1 03/28/2023
  • ESXi on Intel NUC 13 Pro (Arena Canyon) 03/27/2023
  • Quick Tip - Enabling ESXi Coredumps to be stored on USB 03/26/2023

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2023

 

Loading Comments...