WilliamLam.com

Quick Tip - ESXCLI CSV --format-param options

by // Leave a Comment

When using ESXCLI, the output is formatted using a "default" formatter based on the type of data being displayed. However, you can easily modify the output by using one of the three supported formatters: xml, csv and keyvalue or even leverage some internal ones mentioned by Steve Jin here. When working with some of the ESXCLI 'storage' namespaces, such as listing all the devices on an ESXi host, the output can be quite verbose as seen in the example below:

~ # esxcli storage core device list
t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL336304Z6600TGN__
   Display Name: Local ATA Disk (t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL336304Z6600TGN__)
   Has Settable Display Name: true
   Size: 572325
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL336304Z6600TGN__
   Vendor: ATA
   Model: INTEL SSDSC2BB60
   Revision: D201
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.01000000004254574c3333363330345a3636303054474e2020494e54454c20
   Is Local SAS Device: false
   Is Boot USB Device: false
   No of outstanding IOs with competing worlds: 32

t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL318301JL600TGN__
   Display Name: Local ATA Disk (t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL318301JL600TGN__)
   Has Settable Display Name: true
   Size: 572325
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL318301JL600TGN__
   Vendor: ATA
   Model: INTEL SSDSC2BB60
   Revision: D201
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.01000000004254574c3331383330314a4c36303054474e2020494e54454c20
   Is Local SAS Device: false
   Is Boot USB Device: false
   No of outstanding IOs with competing worlds: 32

Usually for such a command, you are interested in a couple of specific properties and I bet you are probably spend a good amount of time scrolling up and down, I know I do. One useful option that is not very well documented (will be filing a bug for this) is the --format-param options which goes in-conjunction with the csv formatter. I always forget the syntax and can never find it when I Google for it so I am documenting this for myself but I think this would also be useful for others to know about.

The --format-param option allows you to specify specific property fields you care about. If we use the our ESXCLI example above, what I really care about are the following for each device:

  • Display Name
  • Is Local
  • Is SSD

Using the following command, we can then extract only those fields we care about:

~ # esxcli --formatter=csv --format-param=fields="Display Name,Model, Is Local,Is SSD" storage core device list
DisplayName,Model,IsLocal,IsSSD,
Local ATA Disk (t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL336304Z6600TGN__),INTEL SSDSC2BB60,true,true,
Local ATA Disk (t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL318301JL600TGN__),INTEL SSDSC2BB60,true,true,
Local ATA Disk (t10.ATA_____WDC_WD4000FYYZ2D01UL1B0_______________________WD2DWMC130199689),WDC WD4000FYYZ-0,true,false,
Local ATA Disk (t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL3183002H600TGN__),INTEL SSDSC2BB60,true,true,
Local ATA Disk (t10.ATA_____INTEL_SSDSC2BB600G4_____________________BTWL336304XL600TGN__),INTEL SSDSC2BB60,true,true,

If we now look at our output, we can easily see that we have 5 devices on our ESXi host and I can quickly see the Display Name of our device, whether it is a local device seen by ESXi and if it is an SSD. I find this filtering mechanism especially handy during troubleshooting or when you need to quickly identify a device for configuration.

Quick Tip - Don't always assume your local HDs will be claimed correctly

by // Leave a Comment

For whatever reason I woke up super early this morning around 4am PST and since I could not go back to sleep, I figure I might as well finish re-building one of my lab environments. I was trying to create a VSAN Disk Group using ESXCLI, but ran into the following error: Stamping non-local disks requires a pre-created vsan cluster If I run the following ESXCLI command, we can clearly see the Hitachi disks that I have are not being recognized as a local device:

esxcli storage core device list

hard-disk-claim-rules-0
I already know that some disks may not be recognized as a local device which VSAN requires and this is nothing new, in fact Duncan Epping even shows you how to get around this in this article here. Seeing that it was so early in the US, I figure Duncan was probably awake and wanted to run a few things by him. It was while he was looking around in the system did he notice a stranger issue.
hard-disk-claim-rules-1
It turns out my local Hitachi disks were being claimed by the VMW_SATP_DEFAULT_AA (ALUA) SATP plugin which was kind of strange as I would have expected to be claimed by VMW_SATP_LOCAL instead. I decided to take a look at the SATP claim rules to see why this was the case by running the following ESXCLI command:

esxcli storage nmp satp rule list

It turns out, for identifying Hitachi storage devices, the SATP rules is quite generic and keys off of the vendor name only and hence the assignment of the ALUA SATP plugin is choosen.

VMW_SATP_DEFAULT_AA          HITACHI                                                                   system      inq_data[128]={0x44 0x46 0x30 0x30}  VMW_PSP_RR
VMW_SATP_DEFAULT_AA          HITACHI                                                                   system

In my opinion this can be problematic as the plugin can potentially cause strange IO or pathing behaviors as it expects one thing but is getting something completely different. I suspect this plugin was probably meant for an Hitachi storage array like an HDS instead of a local device. Once I was able to narrow this down, I just needed to create a new rule that would override the system defaults.

To verify this, I created a new rule based on the vendor name just for testing purposes which will have priority over the default system rules. When creating custom SATP rules, you can filter on a variety of attributes such as the model, transport, controller, target, adapter, etc. Depending on your use case, you may want to be generic or quite specific. The following ESXCLI command will create the new rule and assign it the VMW_SATP_LOCAL plugin:

esxcli storage nmp satp rule add -V HITACHI -P VMW_PSP_FIXED -s VMW_SATP_LOCAL

hard-disk-claim-rules-3
Once that rule has been added, I can now finish my original task which is to mark my Hitachi drives as "local" and reclaim these devices by running the following ESXCLI commands:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -o "enable_local" -d [DEVICE]
esxcli storage core claiming reclaim -d [DEVICE]

hard-disk-claim-rules-4
If we now take a look at our device, we can see that are rule was activated and if we perform the following ESXCLI command, we can now see the device is "local":

esxcli storage core device list

hard-disk-claim-rules-5

The lesson here, do not assume your local devices and potentially even remote devices will be claimed correctly. There maybe vendor best practices that differ from the out out box rules and you should always do your due diligence to either verify yourself or work with your vendors to confirm the configurations.

Additional steps required to completely disable VSAN on ESXi host

by // 11 Comments

Something that I had noticed while working with VSAN in my lab is that when you disable VSAN on your vSphere Cluster, the disks that were used for VSAN in each of the ESXi hosts were no longer available for use afterwards. If you want to use one of the disks for creating a regular VMFS volume or even use it for for vSphere Flash Read Cache, the disks would not show up as an available device. The reason this is occurring is that the disks still contains a VSAN partition and this is not automatically removed when disabling VSAN.

You can view the partition details by using the partedUtil and specifying the "getptbl" option and the device.

Now I could use partedUtil to clear the partition, but there is actually a nice ESXCLI command that can be used to remove the disks used in a VSAN disk group and this will automatically clear the VSAN partition. The ESXCLI command is:

esxcli vsan storage remove -s [SSD-DEVICE-ID]

When I tried to run the command, I was surprised to get the following error message:

Unable to remove device: Can not destroy disk group for SSD naa.6000c29c581358c23dcd2ca6284eec79 : storage auto claim mode is enabled

It turns out when you use "Automatic" claiming mode when enabling VSAN on your vSphere Cluster, that configuration is left enabled on the ESXi host even when disabling VSAN. This then prevents you from destroying the disk group. So there is an extra step required if you choose automatic mode and you will need to run the following ESXCLI command to disable it:

esxcli vsan storage automode set --enabled false

If you are not sure, you can always perform a "get" operation to check whether automatic claim mode is enabled. Once that has been disabled, you will now be able to destroy the diskgroup by running the original command above:

The remove operation only requires the SSD device front-ending the VSAN disk group and you can identify the SSD by running "esxcli vsan storage list". I did find it odd that disabling VSAN in your vSphere Cluster did not completely disable the automatic mode on the ESXi host and I have already filed a bug request to get that fix.