WilliamLam.com

Search Results for: nested esxi

Changing the default size of the ESX-OSData volume in ESXi 7.0

by // 29 Comments

In ESXi 7.0, a new partition scheme was introduced which also brings along a new set of storage requirements. These changes are explained in the official documentation here and the following VMware KB 77009 also contains some additional info which can be helpful. Storage changes are not easy but this was necessary to not only better support some of the current capabilities but more importantly, it setups the foundation for future ESXi capabilities.

The biggest change to the partition layout is the consolidation of VMware Tools Locker, Core Dump and Scratch partitions into a new ESX-OSData volume (based on VMFS-L). This new volume can vary in size (up to 138GB) depending on a number of factors including the current ESXi boot media (USB SD-Card, Local Disk) but also the size of the device itself, which is explained in the official documentation.

From some of the comments on Twitter, Reddit and the direct inquiries that I have received, this new behavior seems to be most impactful to smaller homelabs where a fresh install of ESXi 7.0 has been performed. Folks have shared that their ESX-OSData volume has taken up 120GB which can be quite significant if you have a smaller disk which can be quite common. I normally install ESXi on a USB device and I also use vSAN, which has a different behavior and I have also not upgraded my physical ESXi host (E200-8D) to 7.0 yet.

I performed a fresh installation of ESXi 7.0 (running as Nested ESXi VM) that was configured with 1TB of storage and here is what the filesystem layout now looks:


We can see that the ESX-OSData volume takes up ~119.75GB, which is not too bad for 1TB volume but I can understand this may not be ideal if you have something smaller such as 250GB to 512GB disk. Due to the size of the local device, the boot options mentioned in the KB would not be helpful and I was curious myself if this ESX-OSData volume size could be configurable. In doing some research it looks like the size of the ESX-OSData can be specified using the following ESXi boot option (SHIFT+O during the initial boot) called autoPartitionOSDataSize

UPDATE (12/17/20) - Official support for specifying the size of ESX-OSData has been added to the release of ESXi 7.0 Update 1c with a new ESXi kernel boot option called systemMediaSize which takes one of four values:

  • min = 25GB
  • small = 55GB
  • default = 138GB (default behavior)
  • max = Consumes all available space

If you do not require or have 138GB for the ESX-OSData, you can override the default behavior by appending this option with the specified value (e.g. systemMediaSize=min). It is worth noting that by using this setting, the smallest ESX-OSData volume you can configure is 25GB. For homelabs or environment which require less than this, you would have to use the unsupported autoPartitionOSDataSize parameter , which is not officially supported as mentioned below.

Disclaimer: This may not be officially supported by VMware as it deviates from the system defaults and can have other unintended behaviors. Use at your own risk.

[Read more...]

How to prevent physical CD-ROM from ejecting after installing or upgrading ESXi?

by // Leave a Comment

While catching up on some news over the weekend, I had noticed a VMware Reddit thread asking a pretty interesting question on how to prevent the physical CD-ROM tray from ejecting after installing or upgrading ESXi? This behavior occurs whether you are using a physical CD-ROM media or a "Virtual" ISO image via an out-of-band interface like an iDRAC or iLO. If you are automating the installation or upgrade using Auto Deploy or network installation such as Kickstart, this is not a problem.

However, I was a bit surprised to hear that this was still a pain point in 2019, as many of the new servers in market do not even include an option for CD-ROM. Some of the suggestions really brought me back to the early 2000's including physically taping up the CD-DROM tray, which I have definitely seen customers doing but this is not a scalable solution and it requires a visit to the datacenter. 

One easy solution that I had suggested was to take advantage of ESXi's scripted installation capability also known as Kickstart and use the supported ESXi --noeject option after reboot. Since the install/upgrade was being done manually, the added benefit of this solution is that you can now have it automated 🙂 The other nice thing about this option is that you can specify the kickstart using the default ESXi ISO or you can take it a step further and embed the Kickstart with a custom ESXi ISO.

[Read more...]

ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

by // 23 Comments

The ESXi MAC Learn dvFilter Fling was released a little over two years ago and it has become a must have when it comes to running our ESXi Hypervisor within a VM, also referred to as Nested ESXi. The reason this Fling has become such a popular hit amongst our customers and partners is that it greatly improves the performance when “Promiscuous Mode” is enabled on a Virtual or Distributed Virtual Portgroup, which is a requirement for using Nested ESXi. Although this Fling works great, there are a couple of limitations with this solution today. The first of which is called out in the original Fling release notes, that once a MAC Address has been learned, it never ages out which is not ideal for long running Nested ESXi environments that generates a large amount of new MAC Addresses. The second, is the lack of vMotion support where the learned MAC Address table is not transfered to the destination ESXi host and must be re-learned.

To help address both of these limitations, the folks over in the Network and Security Business Unit (NSBU) have been working hard to improve upon the existing solution and have developed a new native MAC Learning VMkernel module called the Learnswitch. This new Learnswitch not only helps improves Nested ESXi workloads but it can also potentially benefit other workloads such as Nested Containers or other 3rd Party network inspection software. One immediate difference from the previous MAC Learn dvFilter solution is that rather than operating on the Network IO Chain, the filtering is now performed within the outer virtual switch layer itself which will provide some additional performance gains. The other added benefit from an internal VMware standpoint is that the Learnswitch is now vmkapi compatible, which means we will have a better backwards compatible story for supporting old releases of ESXi. One downside to this new solution compared to the previous one is that because the dvFilter operated below the virtual switch layer, it could support both a Virtual Standard Switch as well as the Distributed Virtual Switch. With the new Learnswitch, a Distributed Virtual Switch will be required. If you currently do not meet the requirements of the new Learnswitch, you can continue using the dvFilter, but it is recommended that you do not mix both on a single system but you can definitely make use of both solutions across different ESXi hosts depending on the constraints of your environment.

Here are some of the new capabilities provided by the new Learnswitch module:

  • Overlay Network based that learning and filtering are done in Etherswitch forwarding check
  • MAC Address learning is based on VLAN ID or VXLAN ID on uplink and leaf port
  • Packet is filtered on uplink and leaf port if the MAC is learned on a different port
  • MAC Address table size is 32k per system
  • MAC Address aging support with default aging time of 5 minutes and configurable
  • Unknown unicast packet is flooded by default and configurable to drop
  • vMotion support that the MAC table learned on the port is transferred to destination host and RARP packet is sent
  • Standalone VMkernel module available as a VIB
  • net-learnswitch CLI to display MAC Address table, configuration and stats

[Read more...]