cmsso-util

How to split vCenter Servers configured in an Enhanced Linked Mode (ELM)?

03.16.2017 by William Lam // 22 Comments

An interesting question that came up on the VMTN forum the other day (thanks to Andreas Peetz for sharing via Twitter) was how to split two vCenter Servers configured in an Enhanced Linked Mode (ELM)? Due to an organization changes in the customers environment, they needed to separate out their two vCenter Servers and run them independently of each other. Although this may sound like an rare event, I have actually seen this use case come up several times now which maybe from a business unit restructuring, spinning out or selling off company assets which then requires the customer to split their existing vCenter Servers that is configured with ELM.

Below is a diagram depicting an example where the original source environment (left) which is composed of two vCenter Servers and two external Platform Services Controller (PSC) configured in an ELM and the desired destination environment (right) which are two separate vCenter Server instances no longer configured in ELM.

The solution to this problem is actually pretty straight forward and leverages the existing vCenter Server and/or Platform Services Controller (PSC) "decommission" workflow. Rather than decommissioning the nodes, we are just simply keeping them around. Below are the instructions on how to achieve this outcome.

UPDATE (05/31/22) - I was recently made aware of the following VMware KB 2106736 article that provides official guidance for splitting/unregistering your vCenter Server from ELM. This should be followed as the officially supported method

UPDATE (01/28/19) - As of vSphere 6.7 Update 1, splitting an Enhanced Linked Mode (ELM) configuration is now supported by using the repointing workflow provided by the enhanced cmsso-util tool.

Disclaimer: Although this solution uses an existing supported workflow, this particular use case has not been tested by VMware. As such, this would not be officially supported by VMware until the appropriate testing has been done by our Engineering teams. One potential option in the short term if you are looking for support from VMware is to file an RPQ request through your VMware account team.

[Read more...]

How to remotely run appliancesh & other shell commands on VCSA w/o requiring SSH?

02.25.2016 by William Lam // 9 Comments

In vSphere 6.0 Update 1, the vCenter Server Appliance (VCSA) has received a significant enhancement to its Virtual Machine Management Interface also known as VAMI for short. As the name suggests, this interface provides basic configuration, monitoring and management capabilities for the Virtual Appliance which can be consumed through either a UI using a web browser or from the appliancesh CLI running within the VCSA Shell.

When talking to customers, they love the fact that the VCSA is harden out of the box and things like SSH are disabled by default. However, one challenge today is that if you need to access the appliancesh interface, SSH still must be enabled or direct console access would be required which is not ideal from an automation as well as from a security standpoint. Although things like SNMP can be configured on the VCSA to help alleviate some of these challenges, it does not solve the problem of having programmatic and remote management access.

VMware Engineering is aware of this request and is working on exposing the VAMI capabilities as an API in a future release of vSphere. In the mean time, not all hope is lost and there is still a solution which does not require you to give up security to be able to operate and manage your VCSA. We can do so by leveraging one of my all time favorite features of the vSphere Platform which is the Guest Operations API which allows you perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a really handy feature for troubleshooting and can even extend into application level provisioning using a single API. I can not stress enough on how cool and underutilized this feature is and it still comes as a surprise when I tell customers that this is actually possible.

Customers can consume the Guest Operations API by consuming it through one of our many supported vSphere SDKs as I have shown here or you can also consume it through PowerCLI using the Invoke-VMSCript cmdlet. To demonstrate the power of the Guest Operations API with the VCSA, I will completely disable all remote access to the VCSA which includes Local Login, Bash Shell and SSH as shown in the screenshot below.

Here is an example of running a simple "echo" command using the vSphere SDK for Perl:

Note: You will notice that there is no output and that is because the standard output must be re-directed to a file and then downloaded back to your client. The PowerCLI's Invoke-VMScript does handle this for you and will return any standand output to the console. For more complex commands, I would recommend creating a script that contains the command and just running the script itself which you can then log locally or into a file.

Here is an example of running the "appliancesh" command using the Invoke-VMScript cmdlet:

Invoke-VMScript -ScriptText "echo 'VMware1!' | appliancesh help pi list
" -vm VCSA-No-SSH -GuestUser root -GuestPassword VMware1!

Here is an example of running the "cmsso-util" command using the Invoke-VMScript cmdlet:

Invoke-VMScript -ScriptText "export VMWARE_VAPI_HOME=/usr/lib/vmware-vapi
export VMWARE_RUN_FIRSTBOOTS=/bin/run-firstboot-scripts
export VMWARE_DATA_DIR=/storage
export VMWARE_INSTALL_PARAMETER=/bin/install-parameter
export VMWARE_LOG_DIR=/var/log
export VMWARE_OPENSSL_BIN=/usr/bin/openssl
export VMWARE_TOMCAT=/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
export VMWARE_RUNTIME_DATA_DIR=/var
export VMWARE_PYTHON_PATH=/usr/lib/vmware/site-packages
export VMWARE_TMP_DIR=/var/tmp/vmware
export VMWARE_PERFCHARTS_COMPONENT=perfcharts
export VMWARE_PYTHON_MODULES_HOME=/usr/lib/vmware/site-packages/cis
export VMWARE_JAVA_WRAPPER=/bin/heapsize_wrapper.sh
export VMWARE_COMMON_JARS=/usr/lib/vmware/common-jars
export VMWARE_TCROOT=/opt/vmware/vfabric-tc-server-standard
export VMWARE_PYTHON_BIN=/opt/vmware/bin/python
export VMWARE_CLOUDVM_RAM_SIZE=/usr/sbin/cloudvm-ram-size
export VMWARE_VAPI_CFG_DIR=/etc/vmware/vmware-vapi
export VMWARE_CFG_DIR=/etc/vmware
cmsso-util --help
" -vm VCSA-No-SSH -GuestUser root -GuestPassword VMware1!

Note: The reason the additional "export" commands are required is that certain commands may rely on certain environmental variables to be setup. In the case of the cmsso-util command, there are several VMware environmental variables it uses. I decided to just export them all but you can selectively figure out which ones are truly needed.

As you can see from the examples above, I was able to successfully run both shell commands as well as the appliancesh without requiring SSH and even local login! This methods works whether you are connected to vCenter Server or ESXi host from vSphere API perspective.

UPDATE (06/06/19) - Example joining the VCSA to Active Directory using domainjoin-cli

Invoke-VMScript -ScriptText "echo 'VMware1!' | /opt/likewise/bin/domainjoin-cli join vmware.corp administrator
" -vm VCSA -GuestUser root -GuestPassword VMware1!

How to automatically repoint & failover VCSA to another replicated Platform Services Controller (PSC)?

12.18.2015 by William Lam // 30 Comments

For those of you who read my previous article (if you have not read it, please do so before proceeding forward), at the very end I showed off a screenshot of a script that I had created for the vCenter Server Appliance (VCSA) which automatically monitors the health of the primary Platform Services Controller (PSC) it is connected to and in the event of a failure, it would automatically repoint and failover to another healthy PSC. The way it accomplishes this is by first deploying two externally replicated PSC's and then associating the VCSA with just the first PSC which we will call our primary/preferred PSC node. Both PSC's are in an Active/Active configuration using a multi-master replication and any changes made in SSO on psc-01 (as shown in the diagram) will automatically be replicated to psc-02.

From a vCenter Server's point of view, it is only get requests serviced by a single PSC, which is psc-01 as shown in the diagram above. Within the VCSA, there is a script which runs a cronjob that will periodically check psc-01's connectivity by performing a simple GET operation on the /websso endpoint. If it is unable to connect, the script will retry for a certain number of times before declaring that the primary/preferred PSC node is no longer available. At this point, the script will automatically re-point the VCSA to the secondary PSC and in a couple of minutes, any users who might have tried to login to the vSphere Web Client will be able login and this happens transparently behind the scenes without any manual interaction. For users that have already logged in to vCenter Server, those sessions will continue to work unless they have timed out, in which case you would need to log back in.

The script is configurable in terms of the number of times to check the PSC for connectivity as well as the amount of time to wait in between each check. In addition, if you have an SMTP server configured on the vCenter Server, you can also specify an email address which the script can send a notification after the failover and alert administrators to the failed PSC node. Although this example is specific to the VCSA, a similar script could be developed on a Windows platform using the same core foundation.

Disclaimer: This script is not officially supported by VMware, it is intended as an example of what can be done with the cmsso-util utility. Use at your own risk.

To setup a similar configuration, you will need to perform the following:

Step 1 - Deploy two External PSCs that are replicated with each other. Ensure you select the "Join an SSO domain in an existing vCenter 6.0 platform services controller" option to setup replication and ensure you are joining the same SSO Site.

Step 2 - Deploy your VCSA and when asked to specify the PSC to connect to, specify the primary/preferred PSC node you had deployed earlier. In my example, this would be psc-01 as seen in the screenshot below.

Step 3 - Download the checkPSCHealth.sh script which can be found here.

Step 4 - SCP the script to your VCSA and store it under /root and then set it to be executable by running the following command:

chmod +x /root/checkPSCHealth.sh

Step 5 - Next, you will need to edit the script and adjust the following variables listed below. They should all be self explanatory and if you do not have an SMTP server setup, you can leave the EMAIL_ADDRESS variable blank.

PRIMARY_PSC - IP/Hostname of Primary PSC
SECONDARY_PSC- IP/Hostname of Secondary PSC (must already be replicating with Primary PSC)
NUMBER_CHECKS- Number of times to check PSC connectivity before failing over (default: 3)
SLEEP_TIME - Number of seconds to wait in between checks (default: 30)
EMAIL_ADDRESS- Email when failover occurs

Step 6 - Lastly, we need to setup a scheduled job using cron. To do so you can run the following command:

crontab -e

Copy the following snippet as shown below into the crontab of the root user account. The first half just covers all the default paths and the expected libraries to perform the operation. I found that without having these paths, you will run into issues calling into the cmsso-util and I figured it was easier to take all the VMware paths from running the env command and just making it available. The very last last line is actually what setups the scheduling and in the example below, it will automatically run the script every 5 minutes. You can setup even more complex rules on how to run the script, for more info, take a look here.

PATH=/sbin:/usr/sbin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/java/jre-vmware/bin:/opt/vmware/bin

SHELL=/bin/bash
VMWARE_VAPI_HOME=/usr/lib/vmware-vapi
VMWARE_RUN_FIRSTBOOTS=/bin/run-firstboot-scripts
VMWARE_DATA_DIR=/storage
VMWARE_INSTALL_PARAMETER=/bin/install-parameter
VMWARE_LOG_DIR=/var/log
VMWARE_OPENSSL_BIN=/usr/bin/openssl
VMWARE_TOMCAT=/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
VMWARE_RUNTIME_DATA_DIR=/var
VMWARE_PYTHON_PATH=/usr/lib/vmware/site-packages
VMWARE_TMP_DIR=/var/tmp/vmware
VMWARE_PERFCHARTS_COMPONENT=perfcharts
VMWARE_PYTHON_MODULES_HOME=/usr/lib/vmware/site-packages/cis
VMWARE_JAVA_WRAPPER=/bin/heapsize_wrapper.sh
VMWARE_COMMON_JARS=/usr/lib/vmware/common-jars
VMWARE_TCROOT=/opt/vmware/vfabric-tc-server-standard
VMWARE_PYTHON_BIN=/opt/vmware/bin/python
VMWARE_CLOUDVM_RAM_SIZE=/usr/sbin/cloudvm-ram-size
VMWARE_VAPI_CFG_DIR=/etc/vmware/vmware-vapi
VMWARE_CFG_DIR=/etc/vmware
VMWARE_JAVA_HOME=/usr/java/jre-vmware

*/5 * * * * /root/checkPSCHealth.sh

Step 7 - Finally, you will probably want to test the script to ensure it is doing what you expect. The easiest way to do this is by disconnecting the vNIC on psc-01 and depending on how you have configured the script, in a short amount of time it should automatically start the failover. All operations are automatically logged to the system logs which you can find under /var/log/messages.log and I have also tagged the log entries with a prefix of vGhetto-PSC-HEALTH-CHECK, so you can easily filter out those message in Syslog as seen in the screenshot below.

If a failover occurs, the script will also log additional output to /root/psc-failover.log which can be used to troubleshoot in the case a failover was attempted but failed. To ensure that the script does not try to failover again, it creates an empty file under /root/ran-psc-failover which the script checks at the beginning before proceeding. Once you have verified the script is doing what you expect, you will probably want to manually fail back the VCSA to the original PSC node and then remove the /root/ran-psc-failover file else the script will not run when it is schedule to.

As mentioned earlier, though this is specifically for the the VCSA, you can build a similar solution on a Windows system using Windows Task Scheduler and the scripting language of your choice. I, of course highly recommend customers to take a look at the VCSA for its simplicity in management and deployment, but perhaps thats just my bias 🙂