WilliamLam.com

How to Send vCenter Alarm Notification to Growl

by // 2 Comments

This tweet from Jason Nash and @PunchingClouds says it all and here it is!

I did some research this afternoon and stumbled upon this article Nagios: Notifications via Growl and leveraging the Net::Growl Perl module, I was able to forward alarms generated from a vCenter server to a system that was running Growl.

Software Requirements:

Step 1 -  Install Grow and configure a password under the "Security" tab and ensure you "Allow network notification"

Step 2 - Install vSphere SDK for Perl on your vCenter server. You may also need to update the PATH variable with Perl bin directory (e.g. C:\Program Files\VMware\VMware vSphere CLI\Perl\bin)

Step 3 - Install Net::Growl Perl module using ppm (Perl Package Manager) which is part of ActiveState Perl with the installation of vSphere SDK for Perl. This will require your vCenter server have internet access to ActiveState Perl site, if you can not get this access, you can install this locally on your system and extract the Growl.pm and copy it to your vCenter server C:\Program Files\VMware\VMware vSphere CLI\Perl\site\lib\Net

Step 4 - Copy the Perl script from here and store it somewhere on your vCenter server, make sure it has the .pl extension. In this example, I named it growl.pl

Step 5 - To verify that Growl Perl script works and can communicate to the system with Growl install, you can manually test it by running the following command:

growl.pl -H william.primp-industries.com -p vmware -a custom -t Alert -m "hello william" -s 1

You will need to change -H to the hostname or IP Address of the system with Growl installed and of course the password you had setup. You should see a notification of the message you had just sent.

Step 6 - Create a batch script which will call the growl.pl script and store it somewhere on your vCenter server. Here is what the script (sendGrowl.bat) looks like, you can modify it to fit your requirements.

:: http://www.virtuallyghetto.com/
:: Custom vCenter Alarm script to generate growl notifications

set GROWL_SERVER=william.primp-industries.com
set GROWL_PASSWORD=vmware
set GROWL_SCRIPT_PATH="C:\Documents and Settings\primp.PRIMP-IND\Desktop\growl.pl"
set PATH="%PATH%;C:\Program Files (x86)\VMware\VMware vSphere CLI\Perl\site\bin"

%GROWL_SCRIPT_PATH% -H %GROWL_SERVER% -p %GROWL_PASSWORD% -a %COMPUTERNAME% -t Alert -m "%VMWARE_ALARM_EVENTDESCRIPTION%" -s 1

Note: If you would like to get a list of other default VMware alarm variables, run the "SET" command and output it to a file to get more details on various variables that can be accessed.

Step 7 - Create a new or update an existing vCenter alarm and under "Actions", specify "Run a command" option and provide the full path to the sendGrowl.bat

Step 8 - For testing purposes, I created a new alarm that would trigger upon an ESX(i) host going in/out of maintenance mode and you can see from the "Tasks and Events", our script is triggered on the vCenter server

and now for the finale, you should see a notification from Growl on your system and since we enable the "sticky" parameter, the notification will stay on your screen until you click on it. You can see that in the script example, I set the message to the event and application is registered as the name of the vCenter server, which allows you to have multiple vCenter forward you notifications.

So there you have it, forwarding vCenter alarms to Growl.

Note: Once a vCenter alarm has been triggered, the script will not fire off again until the original alarm has been reset to green. This behavior probably is okay for majority of the events one would want to monitor, but if you want it to continuously alert you, you will need to fiddle with a way to reset the alarm on the vCenter server.

UPDATE:  Thanks to Richard Cardona for reminding me, but this can also be implemented on the new VCVA (vCenter Server Virtual Appliance) in vSphere 5. Here are the instructions on setting it up

Step 1 - Install Grow and configure a password under the "Security" tab and ensure you "Allow network notification" on the system that is receiving the Growl notifications

Step 2 - To install Net::Growl, we'll be using cpan which requires 2 modules that are not installed by default on the SLES VCVA. Using the Tips and Tricks for vMA 5 (running SLES as well), we'll go ahead and setup zypper package manager for VCVA to install the two required packages: make and yaml

zypper --gpg-auto-import-keys ar http://download.opensuse.org/distribution/11.1/repo/oss/ 11.1
zypper --gpg-auto-import-keys ar http://download.opensuse.org/update/11.1/ Update-11.1
zypper refresh
zypper in make
zypper in perl-YAML

Step 3 - You will use cpan to install Net::Growl

perl -MCPAN -e shell

Step 4 - Once you are inside the cpan shell, type the following to install Net::Growl

install Net::Growl

Step 5 - Copy the Perl script from here and store it somewhere on your vCenter server (e.g. /root), make sure it has the .pl extension and has execute permission. In this example, I named it growl.pl

Step 6 - To verify that Growl Perl script works and can communicate to the system with Growl install, you can manually test it by running the following command:

vcenter50-2:~ # ./growl.pl -H william.primp-industries.com -p vmware -a custom -t Alert -m "hello william" -s 1

Step 7 - Create a shell script which will call the growl.pl script and store it somewhere on your vCenter server (e.g. /root). Here is what the script (sendGrowl.sh) looks like, you can modify it to fit your requirements.

Step 8 - Create a new or update an existing vCenter alarm and under "Actions", specify "Run a command" option and provide the full path to the sendGrowl.sh

How to Persist Configuration Changes in ESXi 4.x/5.x Part 2

by // 5 Comments

Continuing from part 1 of How to Persist Configuration Changes in ESXi 4.x/5.x Part 1, here is another method which I prefer when trying to persist configuration changes in ESXi. When ESXi boots up, it loads it's filesystem into memory which the modules to be loaded up are determined by the configuration found in /bootbank/boot.cfg and /altbootbank/boot.cfg for the two respective partitions (primary / backup).

UPDATE: You can now persist configuration files such as firewall rules and others using the new VIB Author Fling, please take a look at this article for more details.

Here is an example of ESXi 4.x default boot.cfg:

~ # cat /bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz
build=4.1.0-348481
updated=1
bootstate=0

Here is an example of ESXi 5.x default boot.cfg:

~ # cat /bootbank/boot.cfg
bootstate=0
kernel=tboot.b00
title=Loading VMware ESXi
kernelopt=no-auto-partition
modules=b.b00 --- useropts.gz --- k.b00 --- a.b00 --- ata-pata.v00 --- ata-pata.v01 --- ata-pata.v02 --- ata-pata.v03 --- ata-pata.v04 --- ata-pata.v05 --- ata-pata.v06 --- ata-pata.v07 --- block-cc.v00 --- ehci-ehc.v00 --- s.v00 --- ima-qla4.v00 --- ipmi-ipm.v00 --- ipmi-ipm.v01 --- ipmi-ipm.v02 --- misc-cni.v00 --- misc-dri.v00 --- net-be2n.v00 --- net-bnx2.v00 --- net-bnx2.v01 --- net-cnic.v00 --- net-e100.v00 --- net-e100.v01 --- net-enic.v00 --- net-forc.v00 --- net-igb.v00 --- net-ixgb.v00 --- net-nx-n.v00 --- net-r816.v00 --- net-r816.v01 --- net-s2io.v00 --- net-sky2.v00 --- net-tg3.v00 --- ohci-usb.v00 --- sata-ahc.v00 --- sata-ata.v00 --- sata-sat.v00 --- sata-sat.v01 --- sata-sat.v02 --- sata-sat.v03 --- scsi-aac.v00 --- scsi-adp.v00 --- scsi-aic.v00 --- scsi-bnx.v00 --- scsi-fni.v00 --- scsi-hps.v00 --- scsi-ips.v00 --- scsi-lpf.v00 --- scsi-meg.v00 --- scsi-meg.v01 --- scsi-meg.v02 --- scsi-mpt.v00 --- scsi-mpt.v01 --- scsi-mpt.v02 --- scsi-qla.v00 --- scsi-qla.v01 --- uhci-usb.v00 --- imgdb.tgz --- state.tgz
build=5.0.0-441354
updated=1

As we learned from the previous article, the cron'd /sbin/auto-backup.sh generates a local.tgz which is then converted to state.tgz which contains all files automatically backed up by VMware. This file is loaded up along with other modules as part of the boot process. Understanding this, allows us to take advantage of this feature for persisting our own configuration files.

Here is an example use case for creating .ssh directory for SSH keys and persisting a script
(ghettoVCB.sh) in /bin for an ESXi 4.x host:

Step 1 - Re-create the modified directory structure and files in a temporary local path which will then be tarred and gzip. An example would be the following:

Actual change:
/.ssh/authorized_keys
/bin/ghettoVCB.sh

Temporary local directory structure of change:
/tmp/.ssh/authorized_keys
/tmp/bin/ghettoVCB.sh

Note: It is very important to ensure that the modified files get the stickybit permission set. As noted in the last article that upon a change, the visorFS will automatically create a special file to denote it for backup but also it allows the file to be writable at some later point for custom files being added.

Step 2 - You will use the tar utility to tar/gzip the contents in a file with extension .tgz. One thing to note, the file name including the extension must not exceed 12 characters. In our example, we made two changes and re-created the local structure under /tmp. We will need to change into /tmp directory and tar up the contents by using the following command:

/tmp # tar -czvf ghetto.tgz .ssh/ bin/
.ssh/
.ssh/authorized_keys
bin/
bin/ghettoVCB.sh

Now you should have the contents of .ssh and bin within your *.tgz file. To verify and view the contents, you can run the following command:

/tmp # tar -tzvf ghetto.tgz
drw------- 0/0 0 2011-08-10 04:10:11 .ssh/
-rw------T 0/0 399 2011-08-10 04:10:11 .ssh/authorized_keys
drwxr-xr-x 0/0 0 2011-08-10 04:10:24 bin/
-rwxr-xr-t 0/0 46973 2011-08-10 04:10:24 bin/ghettoVCB.sh

Step 3 - Next we need to copy the *.tgz file into /bootbank and modify the /bootbank/boot.cfg to ensure our new module is included in the boot up process

cp /tmp/ghetto.tgz /bootbank

Here is what the modified ESXi 4.x boot.cfg should look like after:

kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- ghetto.tgz
build=4.1.0-348481
updated=1
bootstate=0

The next time you reboot the system, you will automatically have your .ssh directory containing your SSH keys and the ghettoVCB script under /bin directory.

Now this is great for an inline modification, but what about creating custom configuration files and including that as part of a default kickstart installation? What about something like custom firewall rules in ESXi 5? In the following example, we'll include a custom firewall rule called "virtuallyGhetto.xml" which will be stored in /etc/vmware/firewall when the contents of the module is extracted.

Step 1 - We of course need to create the XML file containing the firewall rule and create the directory structure in which it will be unloaded to.

/tmp/etc/vmware/firewall/virtuallyGhetto.xml

Step 2 - Next we'll tar up the local contents

[root@air tmp]# tar -zcvf ghetto.tgz etc/
etc/
etc/vmware/
etc/vmware/firewall/
etc/vmware/firewall/virtuallyGhetto.xml

Step 3 - This new package will need to be stored on you installation server which will be reachable via http using wget and as part of the %firstboot stanza in your ESXi 5 kickstart. It will download the *.tgz file and append the entry in /bootbank/boot.cfg configuration file. Here are the entries that should go into your kickstart:

# Adding custom files to boot argument (e.g. custom firewall rules,etc)

wget http://air.primp-industries.com/esxi5/ghetto.tgz -O /bootbank/ghetto.tgz
sed -e '/modules=/s/$/ --- ghetto.tgz/' -i /bootbank/boot.cfg

Note: If you add custom files that are located under /etc and you have the stickybit enabled on your file, changes made will persist upon the next reboot by either manually running /sbin/auto-backup.sh or letting it run via cron. If you add custom files that are not located under /etc, any change you make must be periodically updated in your custom *.tgz file else the next reboot, the original file will be loaded.

How to Create and Modify vgz (vmtar) Files on ESXi 3.x/4.x

by // 6 Comments

There were several questions today on the VMTN community forums with regards to manipulating .vgz files in ESXi, also known as vmtar files. Due to the sparse amount of information on the web, I wanted to document some of the common operations that can be performed on the vmtar files. I will not be going over the use cases for manipulating or creating custom vmtar files, but here is one use case.

UPDATE (10/16/18) - For ESXi 6.5+, please use the following commands and the example below is using the s.v00 file:

Decompress file:

gunzip < s.v00 > s.v00.xz
xz --single-stream --decompress < s.v00.xz > s.v00.vtar
vmtar -v -x s.v00.vtar -o s.v00.tar
tar -xvf s.v00.tar

Compress file:

tar -cvf s.v00-new.tar bin/ etc/ lib/ lib64/ opt/ usr/ var/
vmtar -v -c s.v00.tar -o s.v00.vtar
xz --single-stream --compress < s.v00.vtar > s.v00.xz
xz --single-stream --compress < s.v00.vtar > s.v00

You can find some of these vmtar files with .vgz extension in the ESXi installation iso, here are a few highlighted in red:

To operate on existing vmtar files, you will need access to an ESXi host via ESXi Shell and using the /sbin/vmtar utility.

Usage: vmtar {[-x vtar/vgz-file] [-c tar/tgz-file] [-v] -o destination} | -t < vtar/vgz-file

In this example, we will copy the install.vgz to an ESXi host to perform some operations.

To list the contents of a vmtar file, you will need to use the -t option:

To extract the contents of vmtar file, you will need to use the -x and -o option:

vmtar -x install.vgz -o install.tar

Note: The output will be a standard tar file which will then need to be extracted before getting to the actual contents

To extract the tar file, we will be using the tar utility:

Let's say we made a change to one of the files and now we would like to re-create the vmtar file, we will first need to tar up the contents by using the tar utility again:

To verify the contents were all tarred up, we can view the contents by using the following command:

tar -tf install.tar

Now we will create the vmtar file using the vmtar utility:

vmtar -c install.tar -o install.vgz

We can confirm the contents by using vmtar -t option once again:

vmtar -t < install.vgz

If you decide to create your own custom vmtar files and want to verify the file layout, you can use vmkramdisk to assist you. Using vdf command, make note of the number of tardisks that have been mounted up.

Also make note of the filesystem layout by performing an "ls" on / (slash):

Now let's say you wanted to create a directory called virtuallyGhetto with a file in that directory called foobar and you wanted it to be mounted up under /

Here are the steps to perform the above:

Do you notice anything different? How about performing an "ls" on / (slash) again?

To umount the vmtar disk, you would use the following command:

vmkramdisk -u virtuallyGhetto.vgz