WilliamLam.com

Quick Tip - Monitoring ESXi remote syslog forwarding

by // 3 Comments

When an ESXi host is unable to forward its logs to a remote syslog server, a VMkernel Observation (VOB) is automatically raised by the host and it can be used to proactively alert administrators, which has been possible since ESXi 5.0 .... per this blog post from 2012 after some Googling! ??

While I was pretty confident the behavior described above still holds true for our latest ESXi 7.x and 8.x releases, I wanted to be sure before responding back to a colleague. I deployed the latest ESXi 7.0 Update 3q and ESXi 8.0 Update 3b and after configuring syslog forwarding, I disabled the NIC on my Aria Operations for Logs to simulate a network disconnect and I saw the following log entry in /var/log/vobd.log

2024-09-28T21:12:00.298Z: [UserLevelCorrelator] 7452916537us: [esx.problem.vmsyslogd.remote.failure] The host "192.168.30.62:514" has become unreachable. Remote logging to this host has stopped.

By default, ESXi will attempt to retry the remote syslog connection after the configured timeout (default 180 seconds), which is a relatively new configuration option that is available with ESXCLI (esxcli system syslog config set --default-timeout XX).

[Read more...]

Quick Tip - vCenter Server Advanced Settings Reference

by // 1 Comment

Simliar to my ESXi Advanced and Kernel Settings reference, I was recently asked about creating one for vCenter Server to capture all the default out of the box advanced settings.

With some automation in place, I deployed all major releases of the vCenter Server Appliance (VCSA) from 6.7 to 8.0 which is available in this repo: https://github.com/lamw/vc-advanced-settings


For those interested, here is the one-liner PowerCLI code to retrieve all vCenter Server advanced settings:

Get-AdvancedSetting -Entity ($global:DefaultVIServer)

Useful vSphere Automation techniques for assisting with CrowdStrike remediation

by // 9 Comments

By now, you have probably heard about or have directly been impacted by the recent CrowdStrike software update to Microsoft Windows system causing an unprecedented global outage. I know IT administrators are working around the clock to remediate thousands if not tens of thousands of Windows systems, the current recommended remediation process from CrowdStrike is definitely painful since it requires users to go into Windows safe mode to remove the offending file. To further complex things, most organizations enable Microsoft Bitlocker, which adds additional step to the already painful manual remediation process as you now have to locate your recovery keys before you login to apply the fix.

Within hours of the CrowdStrike news, I already saw a number of inquiries from our customers and field asking if there were any automated solutions or scripts that could aide in their remediation as asking any organization to manually remediate is a non-starter with the scale of deployments for most Enterprises. While getting up to speed on the remediation steps and thinking about how our vSphere platform can help users automate, what is typically a manual task, I had a few ideas that folks might find useful.

Disclaimer: The scripts provided in this article are meant as examples, please test and adapt them based on your own environment as these have not been tested in any official capacity and the behaviors may vary from environment to environment. Please use at your own risk.

[Read more...]