In ESXi 7.0, a new partition scheme was introduced which also brings along a new set of storage requirements. These changes are explained in the official documentation here and the following VMware KB 77009 also contains some additional info which can be helpful. Storage changes are not easy but this was necessary to not only better support some of the current capabilities but more importantly, it setups the foundation for future ESXi capabilities.
The biggest change to the partition layout is the consolidation of VMware Tools Locker, Core Dump and Scratch partitions into a new ESX-OSData volume (based on VMFS-L). This new volume can vary in size (up to 138GB) depending on a number of factors including the current ESXi boot media (USB SD-Card, Local Disk) but also the size of the device itself, which is explained in the official documentation.
From some of the comments on Twitter, Reddit and the direct inquiries that I have received, this new behavior seems to be most impactful to smaller homelabs where a fresh install of ESXi 7.0 has been performed. Folks have shared that their ESX-OSData volume has taken up 120GB which can be quite significant if you have a smaller disk which can be quite common. I normally install ESXi on a USB device and I also use vSAN, which has a different behavior and I have also not upgraded my physical ESXi host (E200-8D) to 7.0 yet.
I performed a fresh installation of ESXi 7.0 (running as Nested ESXi VM) that was configured with 1TB of storage and here is what the filesystem layout now looks:
We can see that the ESX-OSData volume takes up ~119.75GB, which is not too bad for 1TB volume but I can understand this may not be ideal if you have something smaller such as 250GB to 512GB disk. Due to the size of the local device, the boot options mentioned in the KB would not be helpful and I was curious myself if this ESX-OSData volume size could be configurable. In doing some research it looks like the size of the ESX-OSData can be specified using the following ESXi boot option (SHIFT+O during the initial boot) called autoPartitionOSDataSize
UPDATE (12/17/20) - Official support for specifying the size of ESX-OSData has been added to the release of ESXi 7.0 Update 1c with a new ESXi kernel boot option called systemMediaSize which takes one of four values:
- min = 25GB
- small = 55GB
- default = 138GB (default behavior)
- max = Consumes all available space
If you do not require or have 138GB for the ESX-OSData, you can override the default behavior by appending this option with the specified value (e.g. systemMediaSize=min). It is worth noting that by using this setting, the smallest ESX-OSData volume you can configure is 25GB. For homelabs or environment which require less than this, you would have to use the unsupported autoPartitionOSDataSize parameter , which is not officially supported as mentioned below.
Disclaimer: This may not be officially supported by VMware as it deviates from the system defaults and can have other unintended behaviors. Use at your own risk.
I performed another fresh installation of ESXi 7.0 but now passing in autoPartitionOSDataSize=4096 (MB) we can now see our ESX OS-Data volume is no longer using 120GB like before.
You should ensure that size your ESX-OSData greater than 4GB, to ensure that coredump files can be created as you can see my example, 50% has already been used up. Since this new volume will store some pretty important files, you should really give yourself a buffer if you decided to deviate from the system defaults that has been selected.
Interestingly, I also ran another experiment where I had upgraded from ESXi 6.7 Update 3 to ESXi 7.0 and here is the partition layout before the upgrade
and here is the partition layout after the upgrade and for the exact same 1TB disk which was completely empty. I suspect the size of ESX-OSData was due to my selection of preserving the existing VMFS volume.
William - thanks for posting this. When you say "performed fresh install using autoPartitionOSDataSize" - did you set this value as part of the "install" directive line in the kickstart, or as a separate standalone config option?
As mentioned in the blog post, this is an ESXi boot option. You can either append it interactively (SHIFT+O) when the installer boots up OR if you're doing it via Kickstart, append it on kernelopt line just like you would for any other ESXi boot option
I'm upgrading from 6.7 to 8 on a SAN booted host. The upgrade stops and I get "disk device does not support OSDATA". Any idea why this is happening?
I have a question, I am attempting to upgrade ESXi 6.7 to ESXI 7 and there is an error im getting that says this "The boot disk has a size of 1024MB, the minimum requirement of the upgrade image is 3814MB." We contacted VMware but they said it's an HP issue. An idea's what needs to be done to fix this?
William. A explation about new storage requerimientos when installing in usb/SD would be great. There are only a few info about running only with usb and a deprecated mode and that you should redirect scratch like before but I cannot find a explation about the deprecated mode and what is the impact.
Thanks a lot for clarifing things. Yet another reason why it proves valuable to follow your blog!
With the minimum recommended size of 32GB for HDD installations, and taking the two 4GB bootbank partitions into account, "autoPartitionOSDataSize=24576" (24GB) should be a good compromise for physical test/lab environments with limited disk sapce.
Hi William . I noticed when using your vSphere 7 ova nested ESXi template that i get a warning about the core dump missing and needs to be configured. And i have not found a way yet. Is this related to this blog.
Not directly related to this blog post but its a new warning message in case you don't have a coredump configured. See https://www.williamlam.com/2020/05/quick-tip-suppress-new-core-dump-warning-in-esxi-7-0.html on how to toggle it off, I don't in case folks want to set one up afterwards.
Just an added note, this boot parameter does *not* work if you're installing to a USB device.
Hi, William
Thanks for your information. I tried to install ESXi 7.0 on local SSD drive and the capacity size is around 128GB. After finishing the setup, then I don't see that the local drive is in datastore. The reason why I don't see is that the VMimage or data is stored and it can't be used?
Hi willian, can you do a new fresh install esxi 7.0 to a 128GB or 256GB sd card or usb without needing additional local disk or ramdisk? I read storage requeriments article but is confusing as it says for sd or usb you always need an additional local disk or it will run in deprecated mode.
Hi William,
Yesterday I decided to reinstall my homelab with version 7.0.1c and i wanted to re-use (as i did before) the autoPartitionOSDataSize parameter.
Turns out that in the latest version this parameters is not only unsupported but it doesnt work anymore. No problem here so i decided to use the systemMediasize parameter as you mentioned. (this worked!)
Turns out that the information on this blog is not exactly accurate.
You say is 'small,medium, default and max but actually its min,small and max
See: https://kb.vmware.com/s/article/81166
The size on mentioned in the Vmware article is incorrect. Min means 25gb and not 33gb, small was 55gb and not 69gb. (i tried it in my lab)
Hope this helps someone
Regards
systemMediaSize values are incorrect, on ESXi 7.0U1c build 17325551 small created a 55 GB OSDATA partition
Thanks Erik/Tomi, article has been fixed. Funny enough, these were values provided by Engr 🙂
What do you guys recommend if your ESXi 7.0 is already using a 120GB OSData partition (from an 480GB SSD) and there is no way to reach server physically, due COVID-19 all company is at home office work. There is any way to resize this OSData partition without doing an mess with ESXi? After OSData resize, I'de expect to be easy to increase current datastore size ;-0
I'm not sure if resize is possible but I'd recommend reaching out to VMware Support to see if they have guidance. If not, then you'd have to reinstall
This is probably a dumb question, but if the ESXi-7.0U1c-17325551 has already been installed, is there a way to resize /vmfs/volumes/OSDATA partition, or new install is the only way?
I performed an ESXi 6.5 to 7 upgrade recently on a system with a USB 8Gb boot drive. Before and after df -h output for reference...
Before:
vfat 249.7M 181.7M 68.0M 73% /vmfs/volumes/6c4216ef-ca8b741d-ba5d-286e5759f5bd
vfat 285.8M 197.5M 88.4M 69% /vmfs/volumes/5c061168-03b422c9-4bbb-a0369fa12d00
vfat 249.7M 182.1M 67.6M 73% /vmfs/volumes/75fe4ee0-b048970b-ce03-69181b388790
After:
VMFS-L 6.2G 1.6G 4.6G 26% /vmfs/volumes/LOCKER-60451967-dafb0a00-1172-48df3705e578
vfat 499.7M 194.6M 305.1M 39% /vmfs/volumes/BOOTBANK1
vfat 499.7M 194.4M 305.4M 39% /vmfs/volumes/BOOTBANK2
William: I've run into a problem upgrading my vSAN nodes from 7.0 to 7.0U2.
I have ESXi installed on a 32GB Samsung USB key. Each node as its USB, one 256GB M.2 cache drive and one 2TB M.2 capacity drive. Each node has a Thunderbolt 3 port and an OWC Thunderbolt 3 to 10G adapter hanging off the back.
I tried upgrading one node tonight by booting off a USB containing 7.0U2 and directing it to the Samsung USB boot drive. I selected (x) Upgrade after it scanned the device.
The Installer then came back with:
----- An unexpected error occurred -----
See logs for details
OSError: No System VMFS-L partition found
Is there a way I can get around this?
Thank you!
Hi all. Struggling to make this work on USB drives I came up with the following:
Setup a bootable ESXi 7 and datastore on the same 16GB USB drive
1. insall ESXi on a 4GB USB drive
2. create an image of it using imageUSB
3. burn image to 16GB USB drive (this leave approx 12GB of unallocated space), make sure it boots ESXi correctly
4. attach the 16GB drive to a regular linux distro (e.g. a live boot disk) and use gdisk to create a GPT partition with fs type fb00 (this is equivalent to GUID AA31E02A-400F-11DB-9590-000C2911D1B8
for VMWARE vmfs); write the partition table (this will normally NOT destroy the existing partitions)
5. boot ESXi from the 16GB USB drive, connect over SSH to it
6. identify the 16GB device name and the vmfs partition created previously with
ls /dev/disks -l
and
partedUtil getptbl /dev/disks/
to display its partition table
7. create a vmfs filesystem with
vmkfstools -C vmfs6 -S usb /dev/disks/ vmkfstools -C vmfs6 -S usb /dev/disks/
it is then visible as datastore 'usb' in ESXi Web GUI (with ESXi 7 the 12GB of unallocated space end up as datastore with approx. 9.5 GB free space)
8. we now have a bootable ESXi 7 and datastore on a single USB drive providing a kind of portable virtualization home lab; I haven't tested it on other machines than the one used to set it up but I assume it should work provided the hardware it supported by ESXi.
The same procedure applies of course for bigger USB drives to have a larger datastore. The trick is really to install ESXi first on a small drive (ESXi 7 requires at least 3.72 GB) to prevent it from claiming the whole disk space as the boot option autoPartitionOSDataSize has no effect on USB drives.
Hello,
I am no expert but somehow Esxi 7.02a allowed me to "Delete" the VMFS-L partition that it creates by default on the USB upon which ESXi was installed.
Initially with no interference installation it did not allowed me to do anything with the USB and my experience was similar to that of everyone else. However, later when I reinstalled the ESXi with added Kernel options, it did installed with consuming 26+ Gb (for VMFS-L partition) of total 29+ Gb available space on a 32 GB SanDisk USB drive (USB 3.2 Gen-A running on USB 2.0 Port), the ESXi web console allowed me to delete the 26+ Gb partition from Storage> Local USB Drive> Action> Edit Partitions> Select VMFS-L partition> Delete Partition> Save> Reboot.
Now I am wondering if ESXi does need VMFS-L partition for some reason and how do I recreate them, as I am not finding the option in the web console.
After deleting the VMFS-L partition can you mount VMware Tools on a Guest Machine?
Great solution, works fine, I have two 240GB M2 disks and it left 208GB available.
I just tried the autoPartitionOSDataSize=4096 and in a second boot the systemMediaSize=min append it interactively (SHIFT+O) when the installer boots and also via Kickstart, append it on kernelopt line but it does not work in a ESXi7.0.3c default installation.
So I made another fresh ESXi 7.0.3c installation appending the systemMediaSize=min it interactively (SHIFT+O) when the installer boots and works like a charm!
Enviroment:
Apple MBP mid 2012, Core i7, 16 GB RAM and 1TB SSD
Catalina 10.15.7
VMWare Fusion 12.1.2
I did the same but it has no effect on an update....
When upgrading from 6.7 to 7.0c I lost my datastore and that partition was created with 100% of the disk, I tried to mount the partition and recover... but I can only see the VM's folders no data. :/
Thanks, worked like a charm upon adding when fresh installing, but not worked on existing installation.
quick one: i just tested ESXi 8: the _systemMediaSize=small_ overwrites the _autoPartitionOSDataSize=4096_ parameter. you cannot use both parameters, remove _systemMediaSize_ if you want to use _autoPartitionOSDataSize_
with the latest 8.0.1 U1 the systemMediaSize=min resulted in 55GB partition, not 25.
What is the size of your disk? I've just tested this with 8.0 Update 1 w/164GB disk and values look simliar to what I've shared (also in documentation)
min for me was 23.9GB and small was 55.9GB