In vSphere 6.0 Update 1, the vCenter Server Appliance (VCSA) has received a significant enhancement to its Virtual Machine Management Interface also known as VAMI for short. As the name suggests, this interface provides basic configuration, monitoring and management capabilities for the Virtual Appliance which can be consumed through either a UI using a web browser or from the appliancesh CLI running within the VCSA Shell.
When talking to customers, they love the fact that the VCSA is harden out of the box and things like SSH are disabled by default. However, one challenge today is that if you need to access the appliancesh interface, SSH still must be enabled or direct console access would be required which is not ideal from an automation as well as from a security standpoint. Although things like SNMP can be configured on the VCSA to help alleviate some of these challenges, it does not solve the problem of having programmatic and remote management access.
VMware Engineering is aware of this request and is working on exposing the VAMI capabilities as an API in a future release of vSphere. In the mean time, not all hope is lost and there is still a solution which does not require you to give up security to be able to operate and manage your VCSA. We can do so by leveraging one of my all time favorite features of the vSphere Platform which is the Guest Operations API which allows you perform guest operations (running commands, transferring files, etc) directly within the guestOS as if you were logged in. Valid guest credentials are still required and once authenticated, the operations are then proxied through VMware Tools. Networking is not even required which makes this a really handy feature for troubleshooting and can even extend into application level provisioning using a single API. I can not stress enough on how cool and underutilized this feature is and it still comes as a surprise when I tell customers that this is actually possible.
Customers can consume the Guest Operations API by consuming it through one of our many supported vSphere SDKs as I have shown here or you can also consume it through PowerCLI using the Invoke-VMSCript cmdlet. To demonstrate the power of the Guest Operations API with the VCSA, I will completely disable all remote access to the VCSA which includes Local Login, Bash Shell and SSH as shown in the screenshot below.
Here is an example of running a simple "echo" command using the vSphere SDK for Perl:
Note: You will notice that there is no output and that is because the standard output must be re-directed to a file and then downloaded back to your client. The PowerCLI's Invoke-VMScript does handle this for you and will return any standand output to the console. For more complex commands, I would recommend creating a script that contains the command and just running the script itself which you can then log locally or into a file.
Here is an example of running the "appliancesh" command using the Invoke-VMScript cmdlet:
Invoke-VMScript -ScriptText "echo 'VMware1!' | appliancesh help pi list " -vm VCSA-No-SSH -GuestUser root -GuestPassword VMware1!
Here is an example of running the "cmsso-util" command using the Invoke-VMScript cmdlet:
Invoke-VMScript -ScriptText "export VMWARE_VAPI_HOME=/usr/lib/vmware-vapi export VMWARE_RUN_FIRSTBOOTS=/bin/run-firstboot-scripts export VMWARE_DATA_DIR=/storage export VMWARE_INSTALL_PARAMETER=/bin/install-parameter export VMWARE_LOG_DIR=/var/log export VMWARE_OPENSSL_BIN=/usr/bin/openssl export VMWARE_TOMCAT=/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE export VMWARE_RUNTIME_DATA_DIR=/var export VMWARE_PYTHON_PATH=/usr/lib/vmware/site-packages export VMWARE_TMP_DIR=/var/tmp/vmware export VMWARE_PERFCHARTS_COMPONENT=perfcharts export VMWARE_PYTHON_MODULES_HOME=/usr/lib/vmware/site-packages/cis export VMWARE_JAVA_WRAPPER=/bin/heapsize_wrapper.sh export VMWARE_COMMON_JARS=/usr/lib/vmware/common-jars export VMWARE_TCROOT=/opt/vmware/vfabric-tc-server-standard export VMWARE_PYTHON_BIN=/opt/vmware/bin/python export VMWARE_CLOUDVM_RAM_SIZE=/usr/sbin/cloudvm-ram-size export VMWARE_VAPI_CFG_DIR=/etc/vmware/vmware-vapi export VMWARE_CFG_DIR=/etc/vmware cmsso-util --help " -vm VCSA-No-SSH -GuestUser root -GuestPassword VMware1!
Note: The reason the additional "export" commands are required is that certain commands may rely on certain environmental variables to be setup. In the case of the cmsso-util command, there are several VMware environmental variables it uses. I decided to just export them all but you can selectively figure out which ones are truly needed.
As you can see from the examples above, I was able to successfully run both shell commands as well as the appliancesh without requiring SSH and even local login! This methods works whether you are connected to vCenter Server or ESXi host from vSphere API perspective.
UPDATE (06/06/19) - Example joining the VCSA to Active Directory using domainjoin-cli
Invoke-VMScript -ScriptText "echo 'VMware1!' | /opt/likewise/bin/domainjoin-cli join vmware.corp administrator
" -vm VCSA -GuestUser root -GuestPassword VMware1!
But can you do this with govc? Searches docs...
Blake,
Yes, since the Guest Operations API is part of the vSphere API and there's a vSphere SDK for Go
William,
Do you know if there exists a cli for altering database eventlog retention from default (long) to short 2d? How to discover it?
I want to deploy OVF with ssh enabled ( for root) by default.
Please suggest which property to use.
<Property ovf:key
Hi William,
Thanks for your blog!
I have a question though: what is the server you are connected on to perform the Invoke-VMScript against your VCSA?
I'm using a Docker PowerCLI to connect to my VCSA 6.7 (Connect-VIServer) but then I can't perform this command on the VCSA itself.
Is there any way to create script to check all the services, health status as showing in VAMI?
William - your postings have saved my bacon ... many times ... for that, much gratitude!
I'm currently working on a project to automate deployment of VCSA. So far, I'm finding govc to be the best, lightest weight tool to get the job done - embedded in a lightweight container. I am successfully doing the initial deploy of VCSA via govc, but it appears there's a secondary "stage 2" configuration that needs completing.
To date, I have not been able to find any references to handle the "stage 2" or "firstboot config" commands to get to a fully functioning VCSA (embedded, standalone - for now).
If you know of any resources - I would be grateful for the pointers. Google-foo has failed and after a full week of playing with various automation tools (vcsa-deploy, ovftool, powercli, govc, etc.) - I really hope I can get over this last hurdle with govc alone. The container bloat to get a functioning vcsa-deploy / ovftool combo working is ... obnoxious.
Thank you.
Hi Shane,
Thanks for the comment and glad to hear the blog has been useful 🙂
For Stage 2, you can check out https://www.williamlam.com/2016/10/how-to-deploy-the-vcenter-server-appliance-vcsa-6-5-running-on-vmware-fusion-workstation.html which gives you the required params to fully automate deployment and configuration
William - thank you for the pointer. This was enough to (eventually) get me over the hump on this - and I now have vCenter 6.7.0u3 deploying successfully via 'govc'. The 'govc' command and environment I'm using is:
GOVC_NETWORK=VM Network
GOVC_VER=v0.21.0
GOVC_RESOURCE_POOL=*/Resources
GOVC_INSECURE=1
GOVC_DATASTORE=datastore1
# connect/auth to ESXi instance to deploy vCenter to
GOVC_URL=root:[email protected]
govc import.ova -options=/tmp/template.json http://10.10.10.10/files/images/vcsa.ova
I have copied the JSON (cleaned) that is referenced here (/tmp/template.json), to the pastebin:
https://pastebin.com/deDpFKfk
It's entirely plausible that some of the Key/Value pairs are gratuitous at this point ... but it took a lot of iterations to get that working smoothly.
Thank you immensely for your fast response!
~~shane
P.S. For reference, I'm automating this in our Digital Rebar Platform solution for extremely large scale environments (10,000s of systems).
Hi William! Great post, it looks really handy, but I seem to be missing something very basic. I would simply like to view services running on the vcenter but can't seem to get it working:
Invoke-VMScript -ScriptText "echo 'myp@ass1' | service-control --status" -VM "VCNUC.lebrine.local" -GuestUser root -GuestPassword myp@ass1
ScriptOutput
-----------------------------------------------------------------------------------------------------------------------| Traceback (most recent call last):
| File "/usr/bin/service-control", line 12, in
| sys.path.extend(os.environ['VMWARE_PYTHON_PATH'].split(':'))
| File "/usr/lib/python3.7/os.py", line 679, in __getitem__
| raise KeyError(key) from None
| KeyError: 'VMWARE_PYTHON_PATH'
|
-----------------------------------------------------------------------------------------------------------------------
Any idea what I'm doing wrong?
See post which mentions various env variables which are typically set and giveaway is in error message where it can’t access env var 🙂
Very cool, so this worked:
Invoke-VMScript -ScriptText "export VMWARE_VAPI_HOME=/usr/lib/vmware-vapi
export VMWARE_RUN_FIRSTBOOTS=/bin/run-firstboot-scripts
export VMWARE_DATA_DIR=/storage
export VMWARE_INSTALL_PARAMETER=/bin/install-parameter
export VMWARE_LOG_DIR=/var/log
export VMWARE_OPENSSL_BIN=/usr/bin/openssl
export VMWARE_TOMCAT=/opt/vmware/vfabric-tc-server-standard/tomcat-7.0.55.A.RELEASE
export VMWARE_RUNTIME_DATA_DIR=/var
export VMWARE_PYTHON_PATH=/usr/lib/vmware/site-packages
export VMWARE_TMP_DIR=/var/tmp/vmware
export VMWARE_PERFCHARTS_COMPONENT=perfcharts
export VMWARE_PYTHON_MODULES_HOME=/usr/lib/vmware/site-packages/cis
export VMWARE_JAVA_WRAPPER=/bin/heapsize_wrapper.sh
export VMWARE_COMMON_JARS=/usr/lib/vmware/common-jars
export VMWARE_TCROOT=/opt/vmware/vfabric-tc-server-standard
export VMWARE_PYTHON_BIN=/opt/vmware/bin/python
export VMWARE_CLOUDVM_RAM_SIZE=/usr/sbin/cloudvm-ram-size
export VMWARE_VAPI_CFG_DIR=/etc/vmware/vmware-vapi
export VMWARE_CFG_DIR=/etc/vmware
service-control --status
" -vm myvcenter1 -GuestUser root -GuestPassword myp@ass!
I guess I'm just not understanding what this line from the other example is actually doing:
Invoke-VMScript -ScriptText "echo 'VMware1!' | appliancesh help pi list
" -vm VCSA-No-SSH -GuestUser root -GuestPassword VMware1!
Why is the root password echo'd? In addition to being entered after -guestpassword ?
-Guestpassword only refers to credentials to perform the Guest Operations using vSphere API, it has NOTHING to do with actual guest command you're running. If you were to just run "echo", then no additional password is needed but if you're running the "appliancesh" utility from shell, you need to login and hence the extra password is needed there. So context matters 🙂
If you want to better understand how these things wok, go back to first principles by manually performing the operations to really understand what is happening. At the end of the day, you need to put whatever is needed in the script section as if you were doing it manually.