WilliamLam.com

How to increase capacity of /var/log on vMA 4.1

by // 4 Comments

This question pops up from time to time on the VMTN community forums on how to increase the size of /var/log on vMA. By default, the /var/log partition is configured to ~500MB, which is a decent size for system logs, but one of the features of vMA is vilogger which adds a syslog capability for your ESX and ESXi hosts. This allows you to ship all your host logs over to vMA, the problem is the logs are also stored in /var/log along with vMA's system logs. If you are managing several dozen hosts, you can easily fill up this partition using the default configured size.

With vMA 4.0, it was pretty trivial to increase/resize partitions within vMA using LVM and I had written a quick guide on how to do this - How to increase/resize vMA Disks. With the release of vMA 4.1, VMware kind of went backwards and decided not to leverage LVM and lost the benefits with using a volume manager. Resizing the partitions can still be accomplished but it requires a few additional steps and tools. I will show you two methods in which you can increase your /var/log partition, primarily to accommodate larger number of hosts to utilize the vilogger functionality.

Option1: Resize /var/log using gparted 

Here is what a default vMA disk layout looks like and note the size of /var/log

First, you will want to shutdown your vMA host, you may also want to backup any scripts/configs you may have on your vMA host in case you make a mistake and can not recover. Next, you will need to increase the existing virtual disk, by default it is configured for 5GB. In this example, I will increase the disk capacity to 10GB bringing the total to 15GB. You will also need to download a gparted live CD ISO, in this example I am using gparted 0.3.4-11, it is what I have available on my datastore.

Next, you will want to attach the gparted ISO to vMA, you probably will need to add a CD-ROM drive since vMA does not come with one by default. Once you power vMA on, you will need to hit ESC to and select boot from CD-ROM. You should be able to just accept all the defaults and once gparted is loaded, you should see the following screen or something similar depending on the version of gparted you are using.

As you can see, it is the partition layout of your vMA host including the unallocated space, which in this case is 10GB. At a high level, we need to resize the partitions inside the extended partition first before we can resize /var/log partition. First you will select/highlight the "extended (/dev/sd4)" partition which is colored in baby blue color and you will move the right arrow and extend it all the way to the right and then click "Resize/Move" button.

Next, you will select/highlight "/ (dev/sd5)" partition which is colored in blue. Here we want to keep / (root) the same default size of 3.39GB, so we are just going to shift the entire box from the left to far right. You just need to bring the cursor on top of the dark blue section and drag it to the right. Make sure you do not move the left or right arrows, else you will be changing the size of root which is not what we want. Once you have it like the picture below, click on the "Resize/Move" button.

Now you will select/highlight "extended (/dev/sd4)" partition again and you will move the left arrow and drag it towards the right like the picture below and again click on the "Resize/Move" button once you are done.

At this point, your screen should look like the following, if it does not for what ever reason you can "undo" all the changes, since none of these changes take affect until you apply them. 

Now, we will finally increase the size of "/var/log (/dev/sd3)" and consume the full amount of space that we have allocated.

Once you are done, we are now ready to apply these changes, the "apply" button is at the top.

After this, it can take up to several minutes depending on your allocation but if everything went well, you should see this success screen at the end and you just need to reboot the system to see the new changes.

One your vMA host is up, you now should be able to run "df -h" again and you should see that /var/log has now increased with the additional space we assigned it.

Option2: Add a new virtual disk and reconfigure vilogger syslog location

If you did not want to go through the process in option 1 and just want to add another virtual disk and dedicate that for vilogger syslog, you can actually change the path of the logs in /etc/vmware/vMA/vMA.conf. You can control not only the vilogger syslog but also vi-fastpass logs and vilogger daemon logs, but we will only change the "vMALogCollector" which is basically the vilogger syslog component.

I will assume you already have a disk added to vMA and you have created the appropriate mount point (If you are unsure how, do a search online). In this example, I have a 15GB partition in /vmasyslog that I will use as the new home for all my vilogger syslog files.

After you have made the edit to vMA.conf, you will need to restart vilogger daemon and you will need to use sudo and the service command.

Now, you can enable vilogger for your ESX and ESXi host and you should see all logs being redirected to this new partition instead of the default /var/log/vmware/*

Updated vSphere Security Hardening Report Script for vSphere 4.1

by // 5 Comments

VMware released earlier this week the first draft copy of the vSphere 4.1 Security Hardening Guide which provides several changes to the vSphere 4.0 version released last year. Unfortunately there was no change list provided and you have to manually go through both documents to get the differences. Luckily I did the heavy lifting for you and here are the changes in 4.1 version:

Edit: It looks like Charu of VMware has already posted a "diff" of the 4.0 and 4.1 version here.

Added Checks (14):

  • VSH07 (Enterprise) - Check for privilege re-assignment after vCenter Server restarts
  • VSH10 (Enterprise) - Clean up log files after failed installations of vCenter Server
  • VUM06 (Enterprise) - Do not use default self-signed certificates
  • VMX23 (Enterprise) - Use secure protocols for virtual serial port access
  • VMX24 (DMZ) - Disable certain unexposed features
  • VMX56 (Enteprise) - Restrict access to VMsafe network APIs
  • HIN02 (Enterprise) - Keep ESX/ESXi system properly patched
  • HCM05 (DMZ) - Disable Welcome web page
  • HMT12 (Enterprise) - Prevent unintended use of VMsafe network APIs
  • HMT15 (Enterprise) - Audit for loading of unauthorized kernel modules (ESXi only)
  • HMT20 (DMZ) - Ensure that vpxuser auto-password change meets policy
  • HMT21 (DMZ) - Ensure that vpxuser password meets length policy
  • HCN05 (SSLF) - Disable DCUI to prevent all local administrative control
  • HCN06 (Enterprise) - Disable Tech Support Mode unless needed for diagnostics and break-fix

Removed Checks (10):

  • VMX03 (Enterprise) - Disable copy/paste to remote console
  • VMX51 (Enterprise) - Restrict access to VMsafe CPU/memory APIs
  • VMX54 (Enterprise) - Restrict access to VMsafe network APIs
  • HCM04 (Enterprise) - Ensure that ESX is configured to encrypt all sessions
  • HMT10 (Enterprise) - Prevent unintended use of VMsafe CPU/memory APIs
  • HMT11 (Enterprise) - Prevent unintended use of VMsafe network APIs
  • HCN01 (Enterprise) - Ensure that only authorized users have access to the DCUI
  • HCN03 (Enterprise) - Avoid adding the root user to local groups
  • HCN04 (SSLF) - Disable tech support mode
  • COP06 (DMZ) - Ensure that vpxuser auto-password change in vCenter meets policy

Note: Some of the removed checks may have been replaced with newer and updated information and shows up in the added checks.

To help with your vSphere validation, here is the latest version of the vSphere Security Hardening Report script 1.5 script. There have been a few enhancements to the script which only validates a check based on whether it it is applicable to classic ESX or ESXi, which in the past it would display "N/A". There is also some further validation of the service endpoints for /, /ui, and /mob that may also help reduce manual verification where applicable. You can also join the new vSphere Security Hardening Report VMTN Group for new updates, bug report and discussions.

Here is an updated sample report based on vSphere 4.1:
vmwarevSphereSecurityHardeningReport-SAMPLE.html

One other thing I noticed while going through both the 4.0 and 4.1 security guide is the numbers for the code are all over the place, there are sometimes huge gaps that are unexplained (e.g. VSH6, VSH7 ... VSH10)

Ghetto Groups

by // 1 Comment

Back in December, VMware upgraded their VMTN (VMware Technology Network) forum software Jive and introduced a completely new layout of the forums that would hopefully enhance the user experience. Though it brought many new features, it also brought on several new issues. The one bug that affected me was the incorrect conversion of the ghettoVCB document, because the conversion was unsuccessful it was decided to be left alone until the issue could be resolved. If you visited the document, it would display a "Forbidden" error message. Unfortunately due to the time it took to resolve, even Google cache started to get stale and stopped serving the cached contents.

Luckily, with the help of Alex Maier (VMTN Community Manager) and her team, she was able to get the ball rolling and got the fix tested and rolled out to production. The ghettoVCB document is once again alive and hopefully in no time it will be returned as the first search result on various search engines.

Going through the pain of receiving dozen of emails, private messages, tweets, etc. per week regarding the issue, I came to realize that VMTN document itself was not the right medium to host both content and user discussions. As it stands today, there are over 1,100+ comments which is pretty significant and managing and keeping up with the conversations is a pretty daunting task. I enjoy the feedback that community provides and the collaboration that takes place and I realize that this can be solved by using the new Groups feature.

To be honest, I did not spend much time looking at Groups when the VMTN software was upgraded, but now that the ghettoVCB document has been fixed, I realized this would fit this need perfectly. With the help of categories users can now post their feedback, discussions/issues and feature request and it can be easier consumed by both new users and myself. Starting today, I will have the following groups based on the top 5 most popular and active scripts:

I have also disabled any new comments on these VMTN documents and will ask that all new comments be re-directed to respective VMware Groups. I'm currently working with Alex to see if there is an easy way to convert the existing comments into a document and attached that as a download to help minimize the complexity of the document. In the worse case, the comments will be left alone as read-only as I think the discussion that currently exists are invaluable. All other VMTN documents that I maintain in the vGhetto Repository will continue to use comments and depending on how well the groups go, I may migrate those over as well.

I hope these new groups will be beneficial for everyone and I am looking forward to the collaboration. Thanks for your support and please help spread the word!