For whatever reason I woke up super early this morning around 4am PST and since I could not go back to sleep, I figure I might as well finish re-building one of my lab environments. I was trying to create a VSAN Disk Group using ESXCLI, but ran into the following error: Stamping non-local disks requires a pre-created vsan cluster If I run the following ESXCLI command, we can clearly see the Hitachi disks that I have are not being recognized as a local device:
esxcli storage core device list
I already know that some disks may not be recognized as a local device which VSAN requires and this is nothing new, in fact Duncan Epping even shows you how to get around this in this article here. Seeing that it was so early in the US, I figure Duncan was probably awake and wanted to run a few things by him. It was while he was looking around in the system did he notice a stranger issue.
It turns out my local Hitachi disks were being claimed by the VMW_SATP_DEFAULT_AA (ALUA) SATP plugin which was kind of strange as I would have expected to be claimed by VMW_SATP_LOCAL instead. I decided to take a look at the SATP claim rules to see why this was the case by running the following ESXCLI command:
esxcli storage nmp satp rule list
It turns out, for identifying Hitachi storage devices, the SATP rules is quite generic and keys off of the vendor name only and hence the assignment of the ALUA SATP plugin is choosen.
VMW_SATP_DEFAULT_AA HITACHI system inq_data[128]={0x44 0x46 0x30 0x30} VMW_PSP_RR
VMW_SATP_DEFAULT_AA HITACHI system
In my opinion this can be problematic as the plugin can potentially cause strange IO or pathing behaviors as it expects one thing but is getting something completely different. I suspect this plugin was probably meant for an Hitachi storage array like an HDS instead of a local device. Once I was able to narrow this down, I just needed to create a new rule that would override the system defaults.
To verify this, I created a new rule based on the vendor name just for testing purposes which will have priority over the default system rules. When creating custom SATP rules, you can filter on a variety of attributes such as the model, transport, controller, target, adapter, etc. Depending on your use case, you may want to be generic or quite specific. The following ESXCLI command will create the new rule and assign it the VMW_SATP_LOCAL plugin:
esxcli storage nmp satp rule add -V HITACHI -P VMW_PSP_FIXED -s VMW_SATP_LOCAL
Once that rule has been added, I can now finish my original task which is to mark my Hitachi drives as "local" and reclaim these devices by running the following ESXCLI commands:
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -o "enable_local" -d [DEVICE]
esxcli storage core claiming reclaim -d [DEVICE]
If we now take a look at our device, we can see that are rule was activated and if we perform the following ESXCLI command, we can now see the device is "local":
esxcli storage core device list
The lesson here, do not assume your local devices and potentially even remote devices will be claimed correctly. There maybe vendor best practices that differ from the out out box rules and you should always do your due diligence to either verify yourself or work with your vendors to confirm the configurations.
