WilliamLam.com

How to automate & cron vi-fastpass scripts on vMA 4.1

by // 3 Comments

vMA's vi-fastpass and vifptarget utility is meant to be used interactively on the command line, this requires the user to set a particular target (ESX,ESXi or vCenter host) and then execute either a vCLI and/or vSphere SDK for Perl script. If you want to stay in the context of a particular server (e.g without having to manually specify --server parameter), you would need to manually change the target. There have been two common set of questions that have re-appeared several times on the VMTN forums:

1) How to automate a particular script and leverage vi-fastpass without having to change the context for each ESX or ESXi host

2) How to automate a particular script and leverage vi-fastpass via cron without having to provide username/password 

    Before we dive into the solutions, the key in using vi-fastpass in a non-interactive fashion is how vifptarget is called from a script. When you call the vifptarget, it is actually an alias to source the following shell script located in /opt/vmware/vma/bin/vifptarget. You can see this by using the "which" utility on a particular command.

    This alias is defined in /etc/bashrc

    You can even redefine the alias name to whatever you like, so long as it's aliasing the actual vifptarget script. Properly sourcing vifptarget script is the key in solving Question 1. If you look at /etc/bashrc, you will also notice that LD_LIBRARY_PATH environmental variable for shared libraries is also defined, this will be the key in solving Question 2 along with properly sourcing vifptarget. You will see these referenced in the example scripts. 

    There are two solutions for Question 1:

    Solution A:

    If you are leveraging vCenter Server and the ESX(i) hosts are being managed by vCenter, then you just need to add vCenter to vi-fastpass and run a simple for loop across the hosts.

    Example:
    2 x ESXi 4.1 hosts being managed by vCenter

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vCenter and reads in a file containing the name of the ESX(i) hosts:

    example1.sh

    Download: example1.sh

    Solution B:

    In this solution, vCenter is not available or you each ESX(i) host being managed by vMA's vi-fastpass. The script will be very similar, but instead of setting vCenter as the target, you will be setting an ESX(i) host. What is interesting that may not be well known is if you are managing more than one ESX(i) vi-fastpass target, you only need to set the target once and you can perform the same operation on all other host. The caveat being, if you want to switch out of the default context of your target, you need to specify --server parameter. This script will actually do it for you behind to scenes.

    Example:
    2 x ESXi 4.1 hosts being managed by vMA

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

    • ntp1.ucsb.edu
    • ntp2.ucsb.edu

    Here is a shell script that performs the configuration on the two ESX(i) host being managed by vMA and reads in a file containing the name of the ESX(i) hosts:

     example2.sh

    Download: example2.sh

    After executing either SolutionA or SolutionB, you now should have the configurations applied to your ESX(i) host as you would if you manually ran it on each host. This is just an example, you can easily substitute multiple esxcfg-* and/or vSphere SDK for Perl scripts. Here is also another example script for a user on the VMTN forums on configuring a vSwitch and adding portgroups.

    Solution for Question 2:

    Now that we know how to automate a particular operation using vi-fastpass in a script, the next logical question is how do we automate it using a cronjob. I will not go into details of how cron works, take a look at the link for more details. One thing I will mention is that cron does not have all the same PATH environmental variables defined as you normally would expect. You should always use full path to scripts and binaries and any shared library modules that is required to execute a particular utility. What this means for leveraging vi-fastpass via a cronjob is that you need to specify the LD_LIBRARY_PATH. In solving Question 1, all we had to do was properly source the vifptarget and the reason we did not have to specify the shared library path is that it was already defined as a default for vi-admin user. This is not the case for cron and you will need to specify that in either the script you are executing or within the crontab.

    Example:

    2 x ESXi 4.1

    • esx4-4.primp-industries.com
    • esx4-5.primp-industries.com

    Execute vmware-cmd -l to extract virtual machines on both ESX(i) host and store the output that to a file called /tmp/vms every 5 minutes. Here is what the cron entry should look like for example3.sh

    example3.sh

    Download: example3.sh

    After 5 minutes, you should now see a new file under /tmp/vms that contains all virtual machines registered on your ESX(i) host. Again, you can easily apply this to any other vCLI and/or vSphere SDK for Perl script.

    Here another example that gets asked quite often in scheduling the popular VMware vSphere Health Check Script, here is the crontab entry for automatically running the healthcheck script every hour against your vCenter server.

    example4.sh

    Download: example4.sh

    Again, the key is to make sure you specify the full path to not only the script you would like to run but also if there is any type of output, that you specify the full path to the output.

    How to increase capacity of /var/log on vMA 4.1

    by // 4 Comments

    This question pops up from time to time on the VMTN community forums on how to increase the size of /var/log on vMA. By default, the /var/log partition is configured to ~500MB, which is a decent size for system logs, but one of the features of vMA is vilogger which adds a syslog capability for your ESX and ESXi hosts. This allows you to ship all your host logs over to vMA, the problem is the logs are also stored in /var/log along with vMA's system logs. If you are managing several dozen hosts, you can easily fill up this partition using the default configured size.

    With vMA 4.0, it was pretty trivial to increase/resize partitions within vMA using LVM and I had written a quick guide on how to do this - How to increase/resize vMA Disks. With the release of vMA 4.1, VMware kind of went backwards and decided not to leverage LVM and lost the benefits with using a volume manager. Resizing the partitions can still be accomplished but it requires a few additional steps and tools. I will show you two methods in which you can increase your /var/log partition, primarily to accommodate larger number of hosts to utilize the vilogger functionality.

    Option1: Resize /var/log using gparted 

    Here is what a default vMA disk layout looks like and note the size of /var/log

    First, you will want to shutdown your vMA host, you may also want to backup any scripts/configs you may have on your vMA host in case you make a mistake and can not recover. Next, you will need to increase the existing virtual disk, by default it is configured for 5GB. In this example, I will increase the disk capacity to 10GB bringing the total to 15GB. You will also need to download a gparted live CD ISO, in this example I am using gparted 0.3.4-11, it is what I have available on my datastore.

    Next, you will want to attach the gparted ISO to vMA, you probably will need to add a CD-ROM drive since vMA does not come with one by default. Once you power vMA on, you will need to hit ESC to and select boot from CD-ROM. You should be able to just accept all the defaults and once gparted is loaded, you should see the following screen or something similar depending on the version of gparted you are using.

    As you can see, it is the partition layout of your vMA host including the unallocated space, which in this case is 10GB. At a high level, we need to resize the partitions inside the extended partition first before we can resize /var/log partition. First you will select/highlight the "extended (/dev/sd4)" partition which is colored in baby blue color and you will move the right arrow and extend it all the way to the right and then click "Resize/Move" button.

    Next, you will select/highlight "/ (dev/sd5)" partition which is colored in blue. Here we want to keep / (root) the same default size of 3.39GB, so we are just going to shift the entire box from the left to far right. You just need to bring the cursor on top of the dark blue section and drag it to the right. Make sure you do not move the left or right arrows, else you will be changing the size of root which is not what we want. Once you have it like the picture below, click on the "Resize/Move" button.

    Now you will select/highlight "extended (/dev/sd4)" partition again and you will move the left arrow and drag it towards the right like the picture below and again click on the "Resize/Move" button once you are done.

    At this point, your screen should look like the following, if it does not for what ever reason you can "undo" all the changes, since none of these changes take affect until you apply them. 

    Now, we will finally increase the size of "/var/log (/dev/sd3)" and consume the full amount of space that we have allocated.

    Once you are done, we are now ready to apply these changes, the "apply" button is at the top.

    After this, it can take up to several minutes depending on your allocation but if everything went well, you should see this success screen at the end and you just need to reboot the system to see the new changes.

    One your vMA host is up, you now should be able to run "df -h" again and you should see that /var/log has now increased with the additional space we assigned it.

    Option2: Add a new virtual disk and reconfigure vilogger syslog location

    If you did not want to go through the process in option 1 and just want to add another virtual disk and dedicate that for vilogger syslog, you can actually change the path of the logs in /etc/vmware/vMA/vMA.conf. You can control not only the vilogger syslog but also vi-fastpass logs and vilogger daemon logs, but we will only change the "vMALogCollector" which is basically the vilogger syslog component.

    I will assume you already have a disk added to vMA and you have created the appropriate mount point (If you are unsure how, do a search online). In this example, I have a 15GB partition in /vmasyslog that I will use as the new home for all my vilogger syslog files.

    After you have made the edit to vMA.conf, you will need to restart vilogger daemon and you will need to use sudo and the service command.

    Now, you can enable vilogger for your ESX and ESXi host and you should see all logs being redirected to this new partition instead of the default /var/log/vmware/*

    Updated vSphere Security Hardening Report Script for vSphere 4.1

    by // 5 Comments

    VMware released earlier this week the first draft copy of the vSphere 4.1 Security Hardening Guide which provides several changes to the vSphere 4.0 version released last year. Unfortunately there was no change list provided and you have to manually go through both documents to get the differences. Luckily I did the heavy lifting for you and here are the changes in 4.1 version:

    Edit: It looks like Charu of VMware has already posted a "diff" of the 4.0 and 4.1 version here.

    Added Checks (14):

    • VSH07 (Enterprise) - Check for privilege re-assignment after vCenter Server restarts
    • VSH10 (Enterprise) - Clean up log files after failed installations of vCenter Server
    • VUM06 (Enterprise) - Do not use default self-signed certificates
    • VMX23 (Enterprise) - Use secure protocols for virtual serial port access
    • VMX24 (DMZ) - Disable certain unexposed features
    • VMX56 (Enteprise) - Restrict access to VMsafe network APIs
    • HIN02 (Enterprise) - Keep ESX/ESXi system properly patched
    • HCM05 (DMZ) - Disable Welcome web page
    • HMT12 (Enterprise) - Prevent unintended use of VMsafe network APIs
    • HMT15 (Enterprise) - Audit for loading of unauthorized kernel modules (ESXi only)
    • HMT20 (DMZ) - Ensure that vpxuser auto-password change meets policy
    • HMT21 (DMZ) - Ensure that vpxuser password meets length policy
    • HCN05 (SSLF) - Disable DCUI to prevent all local administrative control
    • HCN06 (Enterprise) - Disable Tech Support Mode unless needed for diagnostics and break-fix

    Removed Checks (10):

    • VMX03 (Enterprise) - Disable copy/paste to remote console
    • VMX51 (Enterprise) - Restrict access to VMsafe CPU/memory APIs
    • VMX54 (Enterprise) - Restrict access to VMsafe network APIs
    • HCM04 (Enterprise) - Ensure that ESX is configured to encrypt all sessions
    • HMT10 (Enterprise) - Prevent unintended use of VMsafe CPU/memory APIs
    • HMT11 (Enterprise) - Prevent unintended use of VMsafe network APIs
    • HCN01 (Enterprise) - Ensure that only authorized users have access to the DCUI
    • HCN03 (Enterprise) - Avoid adding the root user to local groups
    • HCN04 (SSLF) - Disable tech support mode
    • COP06 (DMZ) - Ensure that vpxuser auto-password change in vCenter meets policy

    Note: Some of the removed checks may have been replaced with newer and updated information and shows up in the added checks.

    To help with your vSphere validation, here is the latest version of the vSphere Security Hardening Report script 1.5 script. There have been a few enhancements to the script which only validates a check based on whether it it is applicable to classic ESX or ESXi, which in the past it would display "N/A". There is also some further validation of the service endpoints for /, /ui, and /mob that may also help reduce manual verification where applicable. You can also join the new vSphere Security Hardening Report VMTN Group for new updates, bug report and discussions.

    Here is an updated sample report based on vSphere 4.1:
    vmwarevSphereSecurityHardeningReport-SAMPLE.html

    One other thing I noticed while going through both the 4.0 and 4.1 security guide is the numbers for the code are all over the place, there are sometimes huge gaps that are unexplained (e.g. VSH6, VSH7 ... VSH10)