How to replace some of ESXi Kickstart automation with new configstorecli commands?

I had received a question a couple of weeks back from a customer who was already automating their ESXi installation using ESXi Kickstart, also known as ESXi Scripted Installation but they had ran into an issue when migrating the exact same automation to the latest ESXi 7.0 releases.

The method the customer was using to manage their ESXi password policies, which was by updating the /etc/pam.d/passwd file, no longer function as expected and this was a result of the introduction of the ESXi ConfigStore, which I have written about here.

As mentioned in the article, the goal of the ESXi ConfigStore is the following:

The goal of the ConfigStore, initially introduced in ESXi 7.0 Update 1, is to centrally manage all configurations for an ESXi host instead of relying on different methods including a variety of configuration files.

The solution that I had provided to the customer was to replace their current method of configuring ESXi password policies with the configstorecli to maintain their ESXi Kickstart automation. While typing up the details for the customer, I realized it may not be very intuitive in how you go about updating some of the old methods of configuring ESXi with the new configstorecli. Even with the password policy example above, it actually took me a few minutes to figure out the correct configstorecli commands and syntax.

In vSphere 8, VMware has released a Tech Preview for the new vSphere Configuration Profiles, which will be the eventual replacement for vSphere Host Profiles and this is part of the overall vSphere Lifecycle Management (vLCM) solution for both image and configuration management going forward for ESXi. The reason I bring up is that the vSphere Configuration Profiles actually leverages the ESXi ConfigStorage behind the scenes and I found it useful in helping me understand where specific configuration settings were stored.

While the configstorecli does have a schema operation which you can use to see the initial top level configuration using the following command:

configstorecli schema get

but to actually see the configuration file schema, you then need to explore further and potentially iterate through the each of the different components, groups and and keys to find what you are looking for.

Using our ESXi password policy example, we eventually find that it is located under esx->security->settings and schema settings can then be accessed by using the following command:

configstorecli schema get -c esx -g security -k settings

While this might be a very basic example, I certainly had to play around a bit before I could locate settings and for other more complex settings, it certainly may not be intuitive.

By utilizing a vSphere 8.0 Cluster that has been enabled with vLCM and attaching a Nested ESXi VM, I was able to take advantage of a few cool features of vSphere Configuration Profiles to help me understand the configuration structure that I can then use with configstorecli.

Option 1 -  Use the Compliance view within the vSphere Configuration Profiles. To do so, you simply make the desired change in the Nested ESXi Host (using Nested ESXi, also means you can explore without worry about affecting a real ESXi host and then apply the outcome to your actual setup). We will stick with the same password policy example and then select the vSphere Cluster containing the ESXi host and navigate to Configure->Desired State->Configuration->Compliance and you can then scroll down the different settings that differ from the desired state.

We can see that the ESXi password policy settings is actually located in /profile/esx/security/settings which respectively maps to component, group and key in the configstorecli.

Option 2 - Use the Settings view within the vSphere Configuration Profiles to export either the full host schema or host configuration. To do so, you simply make the desired change in the Nested ESXi Host (using Nested ESXi, also means you can explore without worry about affecting a real ESXi host and then apply the outcome to your actual setup). We will stick with the same password policy example and then select the vSphere Cluster containing the ESXi host and navigate to Configure->Desired State->Configuration->Settings and then click on the three dots button, which will present two options: Extract from reference host or Export configuration schema. I personally recommend the former, since you can easily identify the setting change you made rather than just looking at the raw schema content.

I have collapsed the rest of the JSON document and as you can see, the outcome is the same as Option 1, the ESXi password policy settings is actually located in /profile/esx/security/settings which respectively maps to component, group and key in the configstorecli. One additional benefit to using Option 2 is that you are also able to see the actual configurable parameters and their respective values, which is also what we need to make our desired changes.

Now that we have identified the correct component, group and key for managing ESXi password policies, we can now use the configstorecli to apply our desired configuration. For file manipulation using configstorecli, you can either replace all configurations using the "set" operation or you can change a subset of the configurations by using the "update" operation.

One way to use configstorecli to extract all the configurable parameters is by simply retrieving the current configuration by running the following command:

configstorecli config current get -c esx -g security -k settings

If you have a brand new, un-configugred ESXi host (especially if you are using the Nested ESXi VM), you will find that no output is shown and the reason is that the configuration has not differed from the system default values as shown in the screenshot below.

Instead, we can extract all the configurable parameters is by retrieving the default configuration by running the following command:

configstorecli config default get -c esx -g security -k settings

If we wish to replace all configurations, then we just need to save the JSON output to a file, make the desired edits and then apply the changes by running the following commands:

configstorecli config default get -c esx -g security -k settings > current.json
configstorecli config current set -c esx -g security -k settings -j current.json

If we wish to replace subset of configurations, then we can save or create the desired JSON file, make the desired edits and then apply the changes by running the following commands:

configstorecli config default get -c esx -g security -k settings > current.json
configstorecli config current update -c esx -g security -k settings -infile current.json

Notice that the contents of current.json file only contains a single entry, so make sure the file that you create/update is a valid JSON file. If you are unsure, you can run it through JSON linting tool such as https://jsonlint.com/