WilliamLam.com

Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm

by // 6 Comments

I was just cleaning up one of my development labs and found that one of my VCSA (vCenter Server Appliance) which I had configured with vSphere Syslog Collector was no longer capturing logs for several of my ESXi hosts. After looking at some of the ESXi logs, I realized that I had rebooted the VCSA at one point and that caused an interruption in syslog forwarding and then knew immediately that I just needed to reload the syslog configuration via ESXCLI as noted in this VMware KB to restore log forwarding.

After restoring my syslog configurations, I had remembered a neat little trick I learned from one of the VMware TAMs about creating a vCenter Alarm to alert you when an ESXi host is no longer able to reach a remote syslog server. I thought this might be very handy alarm to have in your vCenter Server in case you hit a similar issue or having some connectivity issues with your syslog servers. By default, there is not an event on syslog connectivity but you can create a vCenter Alarm based on an eventId which shows up as "esx.problem.vmsyslogd.remote.failure" in both /var/log/hostd.log as well as /var/log/vobd.log.

Now that we know the eventId, we just need to create a vCenter Alarm which will notify us when it has a connectivity issue with it's configured syslog server.

Step 1 - Create a new alarm, in this example I am calling it "Syslog Connection Error" and you will need to specify the Alarm Type as "Host" and monitor for a specific event.

Step 2 - Next, click on Triggers and we will go ahead and paste in our eventId which is "esx.problem.vmsyslogd.remote.failure"

Step 3 - Lastly, you can configure an Action, if you wish to send an SNMP trap, run a command or send an email notification. In this example, we are just going to generate a regular vCenter Alarm event, so go ahead and just click OK to save the alarm.

To test the alarm, I just disabled the syslog-collector on the VCSA using "service syslog-collector stop" and you should see an alarm generate for any ESXi hosts forwarding it's logs to that particular syslog server.

So now when your ESXi hosts can not reach it's syslog server, you will automatically be notified and can look into the problem immediately. Now having an alarm is great ... but you might be wondering what about the need to reload the syslog configuration on all your ESXi hosts to restore syslog forwarding? This can definitely be a challenge/annoying, especially if the syslog server's connectivity is returned after some amount of time and you have hundreds of hosts.

Well luckily, you no longer have to worry about this, with the latest ESXi 5.0 patch03 that was just released, this problem has been addressed and ESXi syslog daemon will automatically start forwarding logs once connectivity has been restored to the syslog server. It is still definitely recommended that you have more than one syslog server in your environment and that they are properly being monitored. Also, do not forget with ESXi 5.0 you can now configure more than one remote syslog server, for more details take a look at this article here.

Note: After applying the patch, you will no longer be able to generate an alarm based on the eventId for syslog when using UDP. You will see something like "Hostd [290D5B90 verbose 'SoapAdapter'] Responded to service state request" in the hostd.log. The alarm will only be valid if you're using TCP or SSL protocol for syslog which have not been patched with latest p03.

If you are looking for a quick way to reload your syslog configurations, you can easily write a simple for loop to reload your ESXi hosts using the remote ESXCLI:

Here is another example using PowerCLI in-conjunction with ESXCLI:

That's so cool! Running ESXi 5.0 & 5.1 on Apple Mac Mini

by // 39 Comments

Those of you who follow me on twitter should know that I recently got my hands on an Apple Mac Mini Server (Thanks Randy K.) and are probably thinking I would install Apple OSX on the Mini. Nope! I am actually running vSphere ESXi 5.0 or 5.1 on the Mac Mini!

UPDATE (12/21): Yo no longer need to create a customized ESXi ISO for Apple Mac Mini, the necessary drivers are now included in the new ESXi 5.0 Update 2 and will work out of the box. Please take a look at this article for more details.

UPDATE (01/18): To run ESXi 5.1 on Mac Mini 5,1 or 5,3 please follow the same steps outlined below but for an ESXi 5.1 ISO image. If you are trying to run ESXi 5.0 or 5.1 on the new 2012 Apple Mac Mini 6,2 please refer to this blog post for the instructions.

Disclaimer: This is not officially supported by VMware. Use at your own risk.

Note: I did not have a spare monitor at home and luckily the Mac Mini has a DVI output which I was able to connect to my 46" TV. Nothing like ESXi on the big screen 🙂

Even though this is not officially supported by VMware, it is still a very cool solution and the Mac Mini is great form factor for a vSphere home lab. I also want to mention that this was only possible with the research from the folks over at Paraguin Consulting who initially blogged about the process needed to get ESXi 5 running on a Mac Mini. I would highly recommend you check out their blog post which provides additional details as well as a step by step installation guide including screenshots for each step.

I did not have access to an Apple Super Drive which is what the Paraguin folks used in their installation guide, as they thought formatting a USB key would have taken too long ... and who has a CD burner these days? 😉 There is also an additional step that is needed to get network connectivity which requires the user to manually install a network driver on the ESXi host.

I decided to go down the route of using a USB key to perform the installation and using a spare 1GB USB key, I created a custom ESXi installation that included the network driver which allows for network connectivity during and after the installation.

What you will need:

Step 1 - Extract the offline bundle "tg3-3.120h.v50.2-offline_bundle-547149.zip" from Broadcom zip file

Step 2 - Use Image Builder to add the Broadcom driver and create a custom ESXi 5 ISO (steps taken from this VMware KB article)

# Add the ESXi 5.0 Update 1 Offline Bundle

Add-EsxSoftwareDepot "C:\VMware-ESXi-5.0.0-623860-depot.zip"

# Add the Broadcom Offline VIB

Add-EsxSoftwareDepot "C:\tg3-3.120h.v50.2-offline_bundle-547149.zip"

# Create a new Image Profile (in example, I'm using the full version w/tools)

New-EsxImageProfile -CloneProfile "ESXi-5.0.0-623860-standard" -name "ESXi50u1-Custom"

# Add the broadcom driver to our Image Profile

Add-EsxSoftwarePackage -ImageProfile "ESXi50u1-Custom" -SoftwarePackage "net-tg3"

# Create an ISO from our custom Image Profile

Export-EsxImageProfile -ImageProfile "ESXi50u1-Custom" -ExportToISO -filepath C:\VMware-ESXi-5.0u1-Custom.iso

Step 3 - Plug in a USB key into your system and run UNetbootin which will take the custom ISO we just created and make it bootable on the USB key

Step 4 - Finally, power on your Mac Mini and plug in the USB key. You can either hold the "Alt/Option" key while the system is booting and select the EFI volume OR just plug the USB key and the ESXi installer should automatically start up.

After a few minutes, you should now have ESXi 5 running on your Mac Mini, here is my setup:

Note: If you wish to add a custom kickstart file so it automatically installs and configures the host, take a look at this blog post for the details. If you want the Mac Mini to automatically boot from USB without having to press the "Alt/Option" key after installation, you can set the default boot device by following this article here.

Since you are running ESXi on Apple hardware, you can also create Mac OSX Virtual Machines (10.5 Server and 10.6 Server) and with the latest release of vSphere 5.0 Update 1, OSX 10.7 (Lion) is now officially supported. To install OSX 10.7 as a VM, make sure you follow the instructions here.

Here is a screenshot of running OSX Lion 64bit as a VM running on ESXi 5 running on a Mac Mini 😀

If you are looking to refresh your vSphere home lab, definitely consider looking at the Mac Mini, which can get up to a max of 16GB of memory. Also if you are wondering if other people are doing this, I would recommend you check out Christopher Well's vSamarai blog here which details his experience using the Mac Mini as well as voting for his VMworld CFP which will also include topic about running ESXi on Mac Mini.

I want to thank Randy K. again for lending me the Mac Mini, you rock dude! Hopefully this will not be the last article about the Mac Mini 😉

Additional Resources:

How to Deploy an OVF/OVA in the ESXi Shell

by // 54 Comments

I recently answered, what I thought was pretty straight forward question on the VMTN forums about whether it was possible to to deploy an OVA directly onto an ESXi host without leveraging remote tools such as the vSphere Client or the ovftool. The response that I provided was, no it was not possible to deploy an OVF/OVA within the ESXi Shell and recommended the user to take a look at the vSphere Client or the ovftool.

For whatever reason, my brain decided to ponder about this specific question over the weekend (even though I had answered dozen or so questions earlier in the week) and came up an idea that could make this work. As many of you know, I am a big fan of the ovftool and I have written several articles about the tool such as here and here. I wanted to see if I could get the ovftool to run in the ESXi Shell as all the necessary libraries and required packages are all self contained within /usr/lib/vmware-ovftool directory. If this works, it would allow a user to deploy a VM from an OVF or OVA format within the ESXi Shell and would not require a remote system which is great for kickstart deployments or ISO installations. As you probably have guessed, I was able to get this to work 🙂

Disclaimer: This is not officially supported by VMware, please test this in a lab before deploying on production systems.

Before you begin, you will need to get the ovftool installed on an existing Linux system, you can use vMA for your convenience. Next, you will need to use the scp command to copy the entire /usr/lib/vmware-ovftool directory onto an ESXi host. Ensure you place the contents on either a shared or local datastore as the size of the ovftool content is quite large (~119 MB).

In this example, I am scp'ing the ovftool directory to a local VMFS datastore (/vmfs/volumes/datastore1)

scp -r /usr/lib/vmware-ovftool/ root@vesxi50-7:/vmfs/volumes/datastore1

Once you have successfully copied the ovftool directory over to your ESXi host, you will need to make a small tweak to the file located in /vmfs/volumes/datastore1/vmware-ovftool/ovftool (shell script that calls the ovftool binary). You will need to modify the the first line using the vi editor from #!/bin/bash to #!/bin/sh as ESXi does not recognize the bash shell. You are now ready to copy an OVF or OVA to your ESXi host which should also reside within a shared or local datastore.

In this example, I uploaded a SLES OVF to the same datastore which contains the ovftool as seen below from the datastore browser:

Let's go ahead and perform a simple probe operation on the OVF we just uploaded to ensure that ovftool is working as expected. To do so, you just need to specify the full path to the ovftool as well as the full path to either your OVF or OVA file.

Note: The ovftool does take slightly longer to run in the ESXi Shell compared to a regular system with the ovftool installed.

Now that we have confirmed the ovftool is working, let's go ahead and deploy from the OVF image. Even though we are running the ovftool locally in the ESXi Shell, you will still need to specify the credentials to your ESXi host during deployment as ovftool was not designed for this use case.

Note: You must specify both the username and password in the ovftool command line, as the password prompt does not function properly in the ESXi Shell and you will see a looping of  "*" characters on the screen.

If you are familiar with the ovftool, you know you can specify an OVF/OVA from both a local resource as well as remote location such as a web server. Here is another example of deploying an OVF from a remote web server:

We can see that is pretty easy to deploy an OVF or OVA from within the ESXi Shell, but what about unattended installations such as ESXi kickstart? Yep, we can do that too! The easiest way is to compress the vmware-ovftool directory using tar command and then download it remotely using the wget command during the %firstboot stanza. I would also recommend placing your OVF/OVA images on a remote web server as well for centralize management and deployment.

Here is the sample code snippet that can be used in your kickstart:

# download ovftool tar to local storage
wget http://air.primp-industries.com/vmware-ovftool.tar.gz -O /vmfs/volumes/datastore1/vmware-ovftool.tar.gz

# extract ovftool content to /vmfs/volumes/datastore1
tar -xzvf /vmfs/volumes/datastore1/vmware-ovftool.tar.gz -C /vmfs/volumes/datastore1/

# deploy OVF from remote HTTP source
/vmfs/volumes/datastore1/vmware-ovftool/ovftool -dm=thin -ds=datastore1 "--net:access333=VM Network" "http://air.primp-industries.com/SLES-VM/SLES-VM.ovf" "vi://root:[email protected]"

# power on VM
vim-cmd vmsvc/power.on $(vim-cmd vmsvc/getallvms | grep "SLES-VM" | awk '{print $1}')

As you can see, virtually anything is possible ... even if you thought it was not earlier 🙂