Quick Tip - Improved NVMe Tiering Device Health Monitoring in VCF 9.1

Unfortunately, one of my consumer-grade NVMe SSD devices that I had configured for NVMe Tiering (Memory Tiering) in my VMware Cloud Foundation (VCF) 9.1.1 environment had recently degraded. While vCenter Server had raised an alarm stating “NVMe Memory Tiering device is not healthy.”, I had dismissed it since the alarm did not provide any additional details about the underlying issue.

In VCF 9.1, NVMe Tiering usage can be conveniently viewed inn the vSphere UI under the Memory tile for either a selected ESX host or vSphere Cluster, which summarizes both Tier 0 (DRAM) and Tier 1 (NVMe Tiering) usage.

When it comes to understanding the actual health of an NVMe device used for NVMe Tiering, I was not sure whether looking at the SMART data was still the recommended approach or if there was a better way?

After speaking with one of the engineers who works on NVMe Tiering, I came to learn that we have made some nice improvements in VCF 9.1, where only the applicable SMART metrics that are relevant to NVMe Tiering are now summarized in a new NVMe device health report. The report can be viewed for a specific ESX host by navigating to Monitor > Memory Tiering, as shown in the screenshot below.

While capturing the NVMe Tiering usage screenshot, I also realized there is a Health Overview link that takes you directly to the ESX host NVMe device health report, which certainly would have been useful to know when I first saw the vCenter Server alarm! 😅

For those interested in programmatically retrieving this health report, we can use ESXCLI either running locally in ESX Shell or remotely and the output is returned as JSON response:

esxcli system health report get -r vmw.memTierHealth

We can also retrieve the NVMe Tiering device health report using PowerCLI and the following example will collect this information and display it for all ESX hosts within a vSphere Cluster:

$vSphereClusterWithNVMeEnabledHosts = "VCF-Mgmt-Cluster"
$vmhosts = Get-Cluster -Name $vSphereClusterWithNVMeEnabledHosts | Get-VMHost

$results = foreach ($vmhost in $vmhosts) {
    $esxcli = Get-EsxCli -VMHost $vmhost -V2
    
    # Execute via hashtable parameter
    $response = $esxcli.system.health.report.get.Invoke(@{
        'reportnames' = @('vmw.memTierHealth')
    })
    
    # Extract payload (handle object or JSON text return)
    $rawResult = if ($response.result) { $response.result } else { $response }
    $data = if ($rawResult -is [string]) { $rawResult | ConvertFrom-Json } else { $rawResult }
    
    # Parse report structure
    $tierHealth = $data.'vmw.memTierHealth'
    $deviceData = $tierHealth.unstructured[0]
    
    [PSCustomObject]@{
        VMHost            = $vmhost.Name
        Device            = $deviceData.model
        DeviceState       = $deviceData.device_state
        CapacityGiB       = $deviceData.capacity_in_GiBs
        TierSizeGiB       = $deviceData.tier_size_in_GiBs
        TierUsedGiB       = $deviceData.tier_used_in_GiBs
        SpareRemainingPct = $deviceData.device_smart_stats.available_spare
        SubsystemDegraded = $deviceData.device_smart_stats.subsystem_reliability_degraded
    }
}

# Display results table
$results | Format-Table -AutoSize

Here is a screenshot of what the output could look like across ESX hosts have NVMe Tiering enabled:

The simple and important metric to keep an eye on is Available Spare (SpareRemainingPct), which represents the remaining spare capacity as a percentage out of 100%. As you can see, all three of my NVMe devices are currently at 100%, and once this value reaches the 10% threshold, you should start planning to replace the device.

Unfortunately, I had already replaced my degraded NVMe device before learning about this new NVMe Tiering device health report. Hopefully, you now know exactly where to look for a quick and easy way to determine the health of an NVMe device being used for NVMe Tiering and whether it may need to be replaced.