Since the release of VMware Cloud Foundation (VCF) 9.0, you may have seen a few blogs from me 😅

To help folks digest all the resources for deploying and running VMware Cloud Foundation (VCF) 9.0 in a lab environment, I have put this blog post together to summarize the various considerations and/or tweaks that will allow you to get the full experience of VCF 9.0 for non-production/commercial usage.

Make sure you to bookmark this page as I will continue to add new updates, I still have a number of backlog items to share ... but for now that will have to wait as I am in need of some PTO, which starts tomorrow! 😎

ESXi CPU Support

The following CPU generations listed below will no longer be supported with vSphere 9

• Intel Broadwell-DE, Broadwell-EP
• Intel Kabylake-S
• Intel Skylake-D, Skylake-S,Skylake-SP, Skylake-W

The implication is that the ESXi 9.0 installer by default will prevent a user from installing ESXi on a system that contains an unsupported CPU to ensure users get the best out of the box experience.

With that said, there is a workaround for those that wish to forgo official support from Broadcom or use hardware for lab/testing purposes, you can add the following ESXi kernel option (SHIFT+O):

allowLegacyCPU=true

which will turn the error message into a warning message and allow the option to install ESXi, knowing that the CPU is NOT officially supported and accept any risks in doing so.

As of the ESXi 8.0 Update 2 release, CPU processors must support the XSAVE instruction and this workaround will NOT work if you do not meet this absolute minimum requirement.

ESXi I/O Device Support

Please refer to KB 391170 - Deprecated devices in ESXi 9.0 and implications for support as well as the Broadcom Compatibility Guide (BCG).

For those requiring USB-based Networking, you can use the USB Network Native Driver for ESXi 9.0 Fling

ESXi Hardware Support

For a major vSphere release, it can be challenging to understand whether your existing hardware will continue to work or if you need to upgrade.

During the development of VCF 9.0, I had put together a survey that would crowdsource from users who have successfully installed ESXi 9.0, so that it can be a reference for users in the community. We almost have 50 submissions, which you can view the results at: https://lamw.github.io/community-lab-hardware-for-esxi9/

If you would like to contribute your own system that has NOT already been submitted, you can submit your system using this quick form: https://forms.gle/dXbzpTz4JpwP8bXr7

AMD Ryzen

• If you plan on using the NVMe Tiering feature on an AMD Ryzen processor, make sure to apply the workaround HERE for proper functionality
• Check out this automation HERE to help rollout the required workaround to successfully run NSX Edges on an AMD Ryzen platform

Nested ESXi

• The IDE Controller is no longer supported, see this blog post HERE for its implication
• The NVMe Tiering feature is not interoperable with Nested Virtualization, including Nested ESXi. See this blog post HERE for more details

VCF Resources

VCF 9.0 supports both Simple (Single-node) or High Availability (Three-node) deployments. The Simple deployment will allow users to experience the full capabilities of VCF 9.0 with a minimal number of component nodes compared to VCF 5.x where you would need to have three NSX Manager as an example. Below is a table of the resources that is required to deploy VCF 9.0 out of the box

To better understand the hardware implications with respect to minimal resources needed to run VCF 9.0 in a lab environment, please see this blog post HERE for more details.

VCF Hardware Options

• Running VCF on Minisforum MS-A2
• Running VCF on ASUS NUC 15 Pro
• Running VCF on GMKtec K11

VCF Installer

• Configure VCF Installer to use an Offline Depot with just HTTP, see this blog post HERE for more details
• Easily serve VCF Offline Depot using HTTP or HTTPS without requiring full blown web server, see this blog post HERE for more details
• If you have a Synology, you can easily setup an VCF Offline Depot, see this blog post HERE for more details
• Disable 10GbE NIC Pre-Check for initial VCF Fleet/Domain deployment, see this blog post HERE for more details
  • When deploying a new VCF Workload Domain from VCF Operations, a different 10GbE pre-check is enabled and there is no way to currently disable that pre-check which is same behavior as VCF 5.x. You will need to have an ESXi host that has 10GbE, if you are using Nested ESXi VM, the VMXNET3 driver will automatically show 10GbE and that would allow you to easily deploy VCF Workload Domain
• Different ESXi host vendor workaround, see this blog post HERE for more details
• Required number of ESXi Hosts:
  • vSAN OSA/ESA - Minimum 3 x hosts (previously it was 4 with VCF 5.x)
  • NFS - Minimum of 2 x hosts
  • Can a single ESXi host be used? See this blog post HERE for more details
• Unlike the Cloud Builder appliance in VCF 5.x, which was only used for the initial deployment of VCF, the VCF 9.0 Installer can either just be the installer for VCF 9.0 deployment or it can transition to become the SDDC Manager appliance as part of deploying VCF 9.0. The benefit of having the VCF Installer become SDDC Manager is that it already contains the downloaded binaries and can speed up your deployment as it does not have to re-download the binaries from your original VCF Software Depot. If you are using the JSON API to deploy VCF, you can add the following "useExistingDeployment": true which it will simply re-use the VCF Installer appliance and you may want to use DNS name that reflects the final personality rather than just the initial installation
• While I love the simplicity of the VCF Installer UI, I prefer to not have to type a whole bunch or go through wizards and a huge enhancement to the installer compared to Cloud Builder is that I can now export the deployment specification, which allows you to easily source control, templatize or deploy more advanced configuration and best of all, the VCF Installer UI has option that you can feed the JSON deployment specification, so while you can automate using the VCF Installer API, its even easier to use the UI and just upload the JSON and click next to validate and deploy. Here is a reference VCF 9 deployment specification for deploying the "Basic" mode where each component is using a single appliance.

{
  "sddcId": "vcf-m01",
  "vcfInstanceName": "William VCF 9 Instance",
  "workflowType": "VCF",
  "version": "9.0.0.0",
  "ceipEnabled": true,
  "fipsEnabled": true,
  "skipEsxThumbprintValidation": true,
  "skipGatewayPingValidation": true,
  "sddcManagerSpec": {
    "rootUserCredentials": {
      "username": "root",
      "password": "VMware1!VMware1!"
    },
    "secondUserCredentials": {
      "username": "vcf",
      "password": "VMware1!VMware1!"
    },
    "hostname": "sddcm01",
    "useExistingDeployment": true,
    "rootPassword": "VMware1!VMware1!",
    "sshPassword": "VMware1!VMware1!",
    "localUserPassword": "VMware1!VMware1!"
  },
  "dnsSpec": {
    "nameservers": [
      "192.168.30.29"
    ],
    "subdomain": "vcf.lab"
  },
  "ntpServers": [
    "104.167.215.195"
  ],
  "vcenterSpec": {
    "vcenterHostname": "vc01",
    "rootVcenterPassword": "VMware1!VMware1!",
    "vmSize": "tiny",
    "storageSize": "",
    "adminUserSsoPassword": "VMware1!VMware1!",
    "ssoDomain": "vsphere.local",
    "useExistingDeployment": false
  },
  "hostSpecs": [
    {
      "hostname": "esx01",
      "credentials": {
        "username": "root",
        "password": "VMware1!"
      }
    },
    {
      "hostname": "esx02",
      "credentials": {
        "username": "root",
        "password": "VMware1!"
      }
    }
  ],
  "clusterSpec": {
    "clusterName": "VCF-Mgmt-Cluster",
    "datacenterName": "VCF-Datacenter",
    "clusterEvcMode": "",
    "clusterImageEnabled": true
  },
  "datastoreSpec": {
    "vsanSpec": {
      "failuresToTolerate": 0,
      "vsanDedup": false,
      "esaConfig": {
        "enabled": true
      },
      "datastoreName": "vsanDatastore"
    }
  },
  "nsxtSpec": {
    "nsxtManagerSize": "medium",
    "nsxtManagers": [
      {
        "hostname": "nsx01a"
      }
    ],
    "vipFqdn": "nsx01",
    "useExistingDeployment": false,
    "nsxtAdminPassword": "VMware1!VMware1!",
    "nsxtAuditPassword": "VMware1!VMware1!",
    "rootNsxtManagerPassword": "VMware1!VMware1!",
    "skipNsxOverlayOverManagementNetwork": true,
    "ipAddressPoolSpec": {
      "name": "tep01",
      "description": "ESXi Host Overlay TEP IP Pool",
      "subnets": [
        {
          "cidr": "172.60.0.0/24",
          "ipAddressPoolRanges": [
            {
              "start": "172.60.0.150",
              "end": "172.60.0.160"
            }
          ],
          "gateway": "172.60.0.1"
        }
      ]
    },
    "transportVlanId": "60"
  },
  "vcfOperationsSpec": {
    "nodes": [
      {
        "hostname": "vcf01",
        "rootUserPassword": "VMware1!VMware1!",
        "type": "master"
      }
    ],
    "adminUserPassword": "VMware1!VMware1!",
    "applianceSize": "small",
    "useExistingDeployment": false,
    "loadBalancerFqdn": ""
  },
  "vcfOperationsFleetManagementSpec": {
    "hostname": "opsfm01",
    "rootUserPassword": "VMware1!VMware1!",
    "adminUserPassword": "VMware1!VMware1!",
    "useExistingDeployment": false
  },
  "vcfOperationsCollectorSpec": {
    "hostname": "opsproxy01",
    "applicationSize": "small",
    "rootUserPassword": "VMware1!VMware1!",
    "useExistingDeployment": false
  },
  "vcfAutomationSpec": {
    "hostname": "auto01",
    "adminUserPassword": "VMware1!VMware1!",
    "ipPool": [
      "172.30.0.31",
      "172.30.0.32"
    ],
    "nodePrefix": "auto01",
    "internalClusterCidr": "198.18.0.0/15",
    "useExistingDeployment": false
  },
  "networkSpecs": [
    {
      "networkType": "MANAGEMENT",
      "subnet": "172.30.0.0/24",
      "gateway": "172.30.0.1",
      "subnetMask": null,
      "includeIpAddress": null,
      "includeIpAddressRanges": null,
      "vlanId": "0",
      "mtu": "1500",
      "teamingPolicy": "loadbalance_loadbased",
      "activeUplinks": [
        "uplink1"
      ],
      "standbyUplinks": [],
      "portGroupKey": "DVPG_FOR_MANAGEMENT"
    },
    {
      "networkType": "VM_MANAGEMENT",
      "subnet": "172.30.0.0/24",
      "gateway": "172.30.0.1",
      "subnetMask": null,
      "includeIpAddress": null,
      "includeIpAddressRanges": null,
      "vlanId": "0",
      "mtu": "1500",
      "teamingPolicy": "loadbalance_loadbased",
      "activeUplinks": [
        "uplink1"
      ],
      "standbyUplinks": [],
      "portGroupKey": "DVPG_FOR_VM_MANAGEMENT"
    },
    {
      "networkType": "VMOTION",
      "subnet": "172.40.0.0/24",
      "gateway": "172.40.0.1",
      "subnetMask": null,
      "includeIpAddress": null,
      "includeIpAddressRanges": [
        {
          "startIpAddress": "172.40.0.150",
          "endIpAddress": "172.40.0.160"
        }
      ],
      "vlanId": "40",
      "mtu": "8900",
      "teamingPolicy": "loadbalance_loadbased",
      "activeUplinks": [
        "uplink1"
      ],
      "standbyUplinks": [],
      "portGroupKey": "DVPG_FOR_VMOTION"
    },
    {
      "networkType": "VSAN",
      "subnet": "172.50.0.0/24",
      "gateway": "172.50.0.1",
      "subnetMask": null,
      "includeIpAddress": null,
      "includeIpAddressRanges": [
        {
          "startIpAddress": "172.50.0.150",
          "endIpAddress": "172.50.0.160"
        }
      ],
      "vlanId": "50",
      "mtu": "8900",
      "teamingPolicy": "loadbalance_loadbased",
      "activeUplinks": [
        "uplink1"
      ],
      "standbyUplinks": [],
      "portGroupKey": "DVPG_FOR_VSAN"
    }
  ],
  "dvsSpecs": [
    {
      "dvsName": "sddc1-cl01-vds01",
      "networks": [
        "MANAGEMENT",
        "VM_MANAGEMENT",
        "VMOTION",
        "VSAN"
      ],
      "mtu": "8900",
      "nsxtSwitchConfig": {
        "transportZones": [
          {
            "transportType": "OVERLAY",
            "name": "vcf-overlay-TZ"
          },
          {
            "transportType": "VLAN",
            "name": "vcf-vlan-TZ"
          }
        ],
        "hostSwitchOperationalMode": "STANDARD"
      },
      "vmnicsToUplinks": [
        {
          "uplink": "uplink1",
          "id": "vmnic0"
        }
      ],
      "nsxTeamings": [
        {
          "standByUplinks": [],
          "policy": "LOADBALANCE_SRCID",
          "activeUplinks": [
            "uplink1"
          ]
        }
      ],
      "lagSpecs": null,
      "vmnics": [
        "vmnic0"
      ]
    }
  ]
}

vSphere

• When deploying VCF using the VCF Installer UI, you will be required to confirm the SSL Thumbprint for your ESXi hosts. When using the JSON method via UI or API, if you do not include the thumbprint values, the VCF Installer will fail the validation, which is by design. If you prefer not retrieve the thumbprint and you trust your environment, then you can add the following: "skipEsxThumbprintValidation" = true to the deployment JSON specification to by pass the validation.

• By default the VCF Installer will perform an ICMP ping to validate all gateway including ESXi Management, vSAN and vMotion are accessible before allowing you to proceed and will validation if the gateway is not reachable. For certain deployments where you might place everything on a single flat network, these gateway entries may not actually be pingable and you can bypass the check by adding the following: "skipGatewayPingValidation" = true to the deployment JSON specification to by pass the validation.
• When installing ESXi, you must ensure that you have ESX-OSData running on a persistent device (e.g. SSD) and not on USB or you will run into issues as explained in this blog post HERE
• If your physical ESXi host only has 2 x NVMe devices, one method is to share a single NVMe device so you can have persisted ESX-OSData, NVMe Tiering and VMFS partition while the remainder NVMe device can be used for vSAN ESA. See this blog post HERE for more details

Storage

• If you are using vSAN ESA w/hardware that is NOT on the BCG/HCL, you will need to apply a workaround, see this blog post HERE for more details
• If you are using NFS, you will need to have a separate network (VLAN) from the ESXi Management network configured
  • Crate a separate portgroup and create an NFS netstack via ESXCLI before creating your dedicated VMkernel interface to access your NFS storage
    • esxcli network ip netstack add -N NFS
  • If your storage network is non-routable from your ESXi Management network, you will also need to add a default gateway
    • esxcli network ip route ipv4 add --gateway 172.50.0.1 --network default -N NFS

Networking

• With VCF 9.0, you can now use a system that only has a single pNIC, where as in VCF 5.x, a minimum of 2 x pNIC was a hard requirement which also limited the type of consumer hardware you might use for deploying VCF
• NSX Manager is automatically configured to take backups every hour, which are then sent over to SDDC Manager. For lab purposes, you may want to reduce the frequency on when backups are taken as it can quickly use up quite a bit of space in the SDDC Manager. Thank you to Dimitri from the VCF Technical Marketing team for sharing this nice tidbit with me!

• Enhanced Data Path (EDP) switch mode is the default in VCF 9.0, check to see if your NIC is supported, see this blog post HERE for more details
• If you do not have a physical NIC on your ESXi host that supports EDP and you are deploying using JSON method, you will need to add the hostSwitchOperationalMode property under the nsxtSwitchConfig section to use non-EDP host switch with the value of STANDARD as shown in JSON snippet below:

  "nsxtSwitchConfig": {
    "transportZones": [
      {
        "transportType": "OVERLAY",
        "name": "vcf-overlay-TZ"
      },
      {
        "transportType": "VLAN",
        "name": "vcf-vlan-TZ"
      }
    ],
    "hostSwitchOperationalMode": "STANDARD"
  }

VCF SSO

• Play with the new VCF Sign-On (SSO) feature by using a free and self-hosted IdP, see this blog post HERE for more details