vCenter Server – Page 16

A preview of native syslog support in VCSA 6.0

03.30.2015 by William Lam // 29 Comments

Proper logging of VMware hosts, services and application logs are becoming more and more critical these days and their usage goes beyond just troubleshooting. In many of our customer environments, extended log retention is often mandatory to satisfy auditing and compliance requirements. Support for remote syslog has been around in ESXi for quite some time and has included several enhancements over the years, however logging for vCenter Server itself has not changed much over the years. Historically, vCenter Server started out as a Windows application and outside of standard filesystem logging there is also Microsoft Event Logs which was not really all that useful. With the release of the vCenter Server Appliance (VCSA), syslog support became more attainable, at least without additional 3rd party tools.

I can even remember when I was an administrator, I had to get creative on how to forward vCenter Server logs to a remote syslog server which I had blogged about back in 2012. Though the solution works, it was not ideal especially when you are running several dozen to several hundred vCenter Server instances like many of our customers do today. When I had discovered that there was a Common Logging initiative within VMware for vSphere 6.0, I was pretty excited and I can only guess that this also put a big smile on many of our GSS folks faces 😉

As you can imagine this was no small undertaking, especially with the organic growth of services and applications within vCenter Server. The goal was not only to support native remote syslog but to also standardize on the location, rotation, retention of all the logs and most importantly providing a consistent time stamp of events so that an administrator or 3rd party tool can easily correlate operations across multiple VMware log files. Though complete native syslog support in vCenter Server is not 100% ready just yet, much of the plumbing and foundation has already been finished and in fact you can see some of this in the latest release VCSA 6.0.

With VCSA 6.0, there is partial support for native remote syslog which is configurable through the VMware Syslog Service under the new vCenter Server System Configuration found within the vSphere Web Client.

There are four settings that you will need to configure:

Common Log Level - * (everything), info, notice, warn, error, crit, alert & emerg
Host - Hostname/IP Address of a *single* remote syslog server
Port - Port of the remote syslog server (514 for UDP & 1514 for TCP is already opened on the VCSA firewall)
Protocol - Supports tcp, udp & tls

A restart is not required when configuring the syslog service and logs will automatically be forwarded to the remote syslog server which is quite nice. You can also view the health status of the syslog service and its connectivity to the remote syslog server by clicking onto the "Summary" view as seen in the screenshot below. For more information about the new syslog service, check out the official documentation here.

So what exactly does partial syslog support really mean? What logs are being forwarded to a syslog server when the syslog service is enabled?

There are currently two major sets of logs that are forwarded to a remote syslog server when the new syslog service is configured:

All logs from ESXi hosts that are connected to the vCenter Server will be forwarded
A partial set of vCenter Server services (details in table below) will be forwarded

Service Name	Service Description	Service Log Location
applmgmt-audit	Appliance Management	/var/log/vmware/applmgmt/applmgmt-audit/applmgmt-audit-syslog.log
audispd	Audit Event Dispatcher	/var/log/audit/audispd/audispd-syslog.log
auditd	Audit System	/var/log/audit/auditd/auditd-syslog.log
rbd	Auto Deploy	/var/log/vmware/rbd/rbd-syslog.log
vmafdd	VMware Authentication Framework	/var/log/vmware/vmafdd/vmafdd-syslog.log
vmcad	VMware Certificate Service	/var/log/vmware/vmcad/vmcad-syslog.log
vmdird	VMware Directory Service	/var/log/vmware/vmdird/vmdird-syslog.log
watchdog-rhttpproxy	Watchdog for Reverse HTTP Proxy service	/var/log/vmware/rhttpproxy/watchdog-rhttpproxy/watchdog-rhttpproxy-syslog.log
watchdog-syslog	Watchdog for Syslog service	/var/log/vmware/syslog/watchdog-syslog/watchdog-syslog-syslog.log
watchdog-vmware-vpostgres	Watchdog for vPostgres DB service	/var/log/vmware/vpostgres/watchdog-vmware-vpostgres/watchdog-vmware-vpostgres-syslog.log
watchdog-vpxd	Watchdog for vCenter Server service	/var/log/vmware/vpxd/watchdog-vpxd/watchdog-vpxd-syslog.log
watchdog-vws	Watchdog for vCenter Web Services service	/var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log

Note: The information above was extracted from /etc/vmware-syslog/custom-file-location.conf

Here is a screenshot of my vRealize Log Insight instance ingesting the logs that have been forwarded over from my VCSA 6.0:

Although not all the vCenter Server services have been integrated into this new native syslog mechanism, you can see where things headed and hopefully in the not too distant future we will have full native syslog support for all application and system logs found withint vCenter Server. One thing that I really do like is that I can go to one single location to configure my remote syslog server and automatically receive all logs from the ESXi hosts being managed by that vCenter Server and forwarded to the configured syslog server. This definitely makes it operationally friendly so that you have one less thing to configure when provisioning new ESXi hosts.

One limitation that I found when configuring your remove syslog server is that there is no way to reset the values to NULL and the UI also limits the number of remote syslog server to just one, even though you can specify multiple targets. One way to get around this UI limitation is by editing the underlying configuration file which is located in /etc/vmware-syslog/syslog.conf

Here is an example of what the syslog.conf looks like for the above configuration:

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you wish to add a second or even third syslog server, you simply just need to duplicate the existing line and update the hostname or IP Address of your syslog server.

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
*.info @log2.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you are manually editing the syslog.conf, you will need to restart the syslog service by running the following command for the changes to take effect:

/etc/init.d/vmware-syslog restart

Some of you might say this is great and all, but one of the most important log files which is the vCenter Server log (vpxd.log) is not being being forwarded. How useful is this really to me? I know I definitely asked that question 🙂 Though not ideal, there is a small configuration change you can apply to easily get vpxd.log to also forward to a remote syslog server using the new syslog service.

You will need to change the vCenter Server advanced setting "config.log.outputToSyslog" property (can also be done using vSphere API) from false to true as seen in the screenshot below.

The above assumes you have already configured the syslog service and for this change to go into effect, you will need to restart the vCenter Server service. This can be done using the System Configuration and under the vCenter Server Service, by just right clicking and selecting "Restart".

If we now look at our vRealize Log Insight instance or whatever syslog server you are using, you should now see entries from the vpx.log being forwarded:

You can also perform this change from the command-line by editing the vCenter Server configuration file at /etc/vmware-vpx/vpxd.cfg and modifying <outputToSyslog>true</outputToSyslog>

Once you have saved the changes, you will need to restart the vCenter Server by running the following command:

/etc/init.d/vmware-vpxd restart

For those of you who are considering vSphere 6.0 and using the VCSA, this is something I definitely recommend checking out to help simplify the management of both your logs for vCenter Server and your ESXi hosts. I know the VMware Engineering team is working hard on making native syslog support even easier in the future and I look forward to the complete solution hopefully in the near future.

Ultimate automation guide to deploying VCSA 6.0 Part 4: vCenter Server Management Node

03.10.2015 by William Lam // 11 Comments

In this last and final article, I will share alternative methods of deploying vCenter Server management node using the VCSA 6.0 appliance. Take a look at the various deployment methods below and their respective instructions for more details. If you are deploying using one of the scripts below, you will need to extract the contents of the VCSA ISO. If you are deploying to Workstation/Fusion, you will need to extract the VCSA ISO and add the .ova extension to the following file VMware-VCSA-all-6.0.0-2562643->vcsa->vmware-vcsa before deploying.

Disclaimer: Though these alternative deployment options work, they are however not officially supported by VMware. Please use at your own risk.

Deploying to an existing vCenter Server using ovftool (shell script)

I have created a shell script called deploy_vcsa6_mgmt_to_vc.sh which requires using ovftool 4.1 (included in the VCSA ISO) to specify the appropriate OVF "guestinfo" properties for a vCenter Server Management Node deployment. You will need to edit the script and modify several variables based on your environment.

Here is an example of executing the script:

Deploying to an ESXi host using ovftool (shell script)

I have created a shell script called deploy_vcsa6_mgmt_to_esxi.sh which requires using ovftool 4.0 or greater to specify the appropriate OVF "guestinfo" properties for a vCenter Server Management Node deployment. You will need to edit the script and modify several variables based on your environment. The behavior of this script is similar to the one above, except you are deploying directly to an ESXi host.

Deploying to an existing vCenter Server using ovftool (PowerCLI)

I have created a PowerCLI script called Deployment-VCSA-Mgmt.ps1 which uses ovftool and specifies the appropriate OVF "guestinfo" properties for a vCenter Server Management Node deployment. You will need to edit the script and modify several variables based on your environment.

Deploying to VMware Fusion & Workstation

To properly deploy the new VCSA 6.0, the proper OVF properties MUST be set prior to the booting of the VM. Since VMware Fusion and Workstation do not support OVF properties, you will need to manually deploy the VCSA, but not power it on. Once the deployment has finished, you will need to add the following entries to the VCSA's VMX file and replace it with your environment settings. Once you have saved your changes, you can then power on the VM and the configurations will then be read into the VM for initial setup.

guestinfo.cis.deployment.node.type = "management"
guestinfo.cis.system.vm0.hostname = "192.168.1.50"
guestinfo.cis.vmdir.domain-name = "vghetto.local"
guestinfo.cis.vmdir.site-name = "vghetto"
guestinfo.cis.vmdir.password = "VMware1!"
guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.addr = "192.168.1.64"
guestinfo.cis.appliance.net.pnid = "192.168.1.64"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.1"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.root.passwd = "VMware1!"
guestinfo.cis.appliance.ssh.enabled = "true"
guestinfo.cis.appliance.ntp.servers = "0.pool.ntp.org"

For more information, you can take a look at this article here.

Deploying using new supported scripted install (bonus)

As mentioned earlier, there is also a new scripted installer included inside of the VMware-VCSA ISO under /vcsa-cli-installer which supports Windows, Mac OS X and Linux, but must be connected directly to an ESXi host. There are several templates that are also included within the /vcsa-cli-installer/templates. I thought as a bonus I would also share the template I have been using to deploy replicated PSC instances using a static IP Address which some of you may find useful.

{
    "__comments":
    [
        "William Lam - www.virtuallyghetto.com",
        "Example VCSA 6.0 vCenter Server Management Node Deployment w/Static IP Address"
    ],

    "deployment":
    {
        "esx.hostname":"192.168.1.200",
        "esx.datastore":"mini-local-datastore-1",
        "esx.username":"root",
        "esx.password":"vmware123",
        "deployment.option":"tiny",
        "deployment.network":"VM Network",
        "deployment.option":"management-tiny",
        "appliance.name":"vcsa-mgmt-node",
        "appliance.thin.disk.mode":true
    },

    "vcsa":
    {
        "system":
        {
            "root.password":"VMware1!",
            "ssh.enable":true,
            "ntp.servers":"0.pool.ntp.org",
            "platform.service.controller":"192.168.1.50"
        },

        "sso":
        {
            "password":"VMware1!",
            "domain-name":"vghetto.local",
            "site-name":"virtuallyGhetto"
        },

        "networking":
        {
            "ip.family":"ipv4",
            "mode":"static",
            "ip":"192.168.1.52",
            "prefix":"24",
            "gateway":"192.168.1.1",
            "dns.servers":"192.168.1.1",
            "system.name":"192.168.1.52"
        }
    }
}

The use the scripted installer, you just need to change into the appropriate OS platform directory (win32,mac or lin64) and there should be a binary called vcsa-deploy. To use this template, you just need to save the JSON to a file and then specify that as the first argument to vcsa-deploy utility.

Here is an example of deploying a PSC using the vcsa-deploy scripted installer.

Dynamic memory resizing for vCenter Server 6.0

02.23.2015 by William Lam // 32 Comments

In previous releases of vSphere, scaling up resources such as storage or memory for vCenter Server was a huge pain-point for our customers. Before the various vCenter Server services can consume the new resources, some additional manual steps were required. Though this type of an operation is usually infrequent, there is still an operational overhead which can potentially lead to increased downtime of your vCenter Server.

For example, increasing storage capacity for the VCSA was an offline operation that required adding an additional disk and then copying the existing content to the new disk which can be quite error prone and lead to a significant amount of downtime. In vSphere 6.0, the VCSA now uses LVM which provides the ability for online storage capacity increase without any downtime to vCenter Server. Increasing memory was also challenging because you had to manually adjust several configuration files that manages the JVM heap settings for various vCenter Server services as described in this VMware KB. Having complex workflows to perform basic resource expansion can increase risk of errors, especially when the process is foreign to those performing it for the very first time.

To help solve this problem, in vSphere 6.0 vCenter Server (Windows & VCSA) now includes a built-in dynamic memory reconfiguration process that automatically runs at boot up. This process includes a dynamic algorithm that inspects the current amount of CPU, Memory and Storage that is available to determine the appropriate size to configure the vCenter Server. This means that if you no longer have to tweak individual JVM settings for the various services within vCenter Server, this will happen automatically by analyzing the resources that are available and then calculating the configuration based on the supported maximums for vCenter Server.

Note: In vSphere 6.0, there are additional services going beyond just the core vCenter Server, vSphere Web Client, vCenter SSO and Inventory Services.

The dynamic memory algorithm is configured to understand the minimal amount of resources for running a vCenter Server and is bounded between a "Tiny" configuration which is 2vCPU and 8GB memory and a "Large" configuration which is 16vCPU and 32GB memory. This is important to note because if you try to configure the vCenter Server with less memory than the minimal supported, though the algorithim will dynamically distribute the available memory to the various resources, it could lead to performance degradation as the different services may not be receiving the amount of memory they require to run. YMMV if you decide to reduce the supported amount of memory but the algorithm will distribute what's available.

The process which does all the magic is a utility called cloudvm-ram-size and there are several useful options to be aware of. To view the current memory assignment for the various vCenter Server services including the OS, you can run the following command on the VCSA as an example:

cloudvm-ram-size -l

From the screenshot above, we can see a very simple break down of the current memory assignment for a "Tiny" deployment which has 8GB of memory.

To show that the dynamic memory algorithm is in fact running when more memory is added, the example below is of a VCSA that was initially configured with 8GB of memory. I then capture the running configuration and then shut down the vCenter Server and increased its memory to 10GB. I then power on the VCSA and capture the running state and you can see differences in the screenshot below.

Another useful command to be aware is being able to see the current memory usage for all services. You can do this by running the following command:

cloudvm-ram-size -S

As you can see the dynamic memory algorithm is a very much welcome feature for vCenter Server and will greatly simplify the operational tasks when needing to scale up or down resources such as CPU and Memory. I know this is definitely one of the enhancements I have been waiting for and I am glad to see it here in the new vSphere 6.0 release! As of right now, a system reboot is required but who knows maybe in the future we can increase memory while the VCSA is still running and simply reloading the services ...