WilliamLam.com

How to Automate the Upgrade of Classic ESX 4.x to ESXi 5

by // 3 Comments

In prior releases of ESXi, there was not a supported method of upgrading from classic ESX to ESXi, you had to perform a clean installation. With the release vSphere 5, ESXi is the only available option and providing a supported and easy method for migrating to ESXi will be very helpful for users. There are currently three options of migrating/upgrading from ESX 4.x to ESXi 5. Going forward in the future, two additional methods will be available for upgrading ESXi 5.x to subsequent update/patch releases.

Here is a table of the supported ESXi 5 upgrade options:

Upgrade Method Upgrade from ESX or ESXi 4.x to ESXi 5.0 Upgrade or Patch from ESXi 5.0 to ESXi 5.n
vSphere Update Manager yes yes
Interactive upgrade from CD, DVD, or USB drive yes yes
Scripted upgrade yes yes
vSphere Auto Deploy no yes
esxcli no yes

The first two options should be pretty straight forward and I won't go into any details, but if you are interested, check out Ivo Beeren's post here. If you decide to use VUM to perform you upgrade, make sure you check out this post about lopsided bootbanks before doing so. The 3rd option is a new feature in a kickstart installation and you now can specify two additional types of installation:

  • upgrade - Tries to perform an upgrade from ESX(i) 4.x to ESXi 5.x
  • installorupgrade - Tries to perform an upgrade from ESX(i) 4.x to ESXi 5.x, if it fails, it will perform a clean installation

In addition to the new installation types, there are two new options that can be specified:

  • --deletecosvmdk - If the system is being upgraded from ESX, remove the directory that contains the old Service Console VMDK file, cos.vmdk, to reclaim unused space in the VMFS datastore
  • --forcemigrate - If the host contains customizations, such as third-party VIBS or drivers, that are not included in the installer .ISO, the installer exits with an error describing the problem. The forcemigrate option overrides the error and forces the upgrade

Here is an example of kickstart specifically for upgrading from ESX 4.x to ESXi 5:

Note: One thing I noticed from the upgrade is that even if you specify a new root password, the current password is still preserved. Virtual machines located on local VMFS volumes will also be preserved as long as you do not use the --overwritevmfs option

You will also know that an ESXi 5 host was upgraded from ESX 4.x when you login to ESXi Shell, a motd will display a message.

As you can see you have several options of upgrading both ESX(i) 4.x to ESXi 5, though if you have a choice between an upgrade and reinstall, my personal preference would still be a clean installation via kickstart or host profiles.

How to Persist Configuration Changes in ESXi 4.x/5.x Part 2

by // 5 Comments

Continuing from part 1 of How to Persist Configuration Changes in ESXi 4.x/5.x Part 1, here is another method which I prefer when trying to persist configuration changes in ESXi. When ESXi boots up, it loads it's filesystem into memory which the modules to be loaded up are determined by the configuration found in /bootbank/boot.cfg and /altbootbank/boot.cfg for the two respective partitions (primary / backup).

UPDATE: You can now persist configuration files such as firewall rules and others using the new VIB Author Fling, please take a look at this article for more details.

Here is an example of ESXi 4.x default boot.cfg:

~ # cat /bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz
build=4.1.0-348481
updated=1
bootstate=0

Here is an example of ESXi 5.x default boot.cfg:

~ # cat /bootbank/boot.cfg
bootstate=0
kernel=tboot.b00
title=Loading VMware ESXi
kernelopt=no-auto-partition
modules=b.b00 --- useropts.gz --- k.b00 --- a.b00 --- ata-pata.v00 --- ata-pata.v01 --- ata-pata.v02 --- ata-pata.v03 --- ata-pata.v04 --- ata-pata.v05 --- ata-pata.v06 --- ata-pata.v07 --- block-cc.v00 --- ehci-ehc.v00 --- s.v00 --- ima-qla4.v00 --- ipmi-ipm.v00 --- ipmi-ipm.v01 --- ipmi-ipm.v02 --- misc-cni.v00 --- misc-dri.v00 --- net-be2n.v00 --- net-bnx2.v00 --- net-bnx2.v01 --- net-cnic.v00 --- net-e100.v00 --- net-e100.v01 --- net-enic.v00 --- net-forc.v00 --- net-igb.v00 --- net-ixgb.v00 --- net-nx-n.v00 --- net-r816.v00 --- net-r816.v01 --- net-s2io.v00 --- net-sky2.v00 --- net-tg3.v00 --- ohci-usb.v00 --- sata-ahc.v00 --- sata-ata.v00 --- sata-sat.v00 --- sata-sat.v01 --- sata-sat.v02 --- sata-sat.v03 --- scsi-aac.v00 --- scsi-adp.v00 --- scsi-aic.v00 --- scsi-bnx.v00 --- scsi-fni.v00 --- scsi-hps.v00 --- scsi-ips.v00 --- scsi-lpf.v00 --- scsi-meg.v00 --- scsi-meg.v01 --- scsi-meg.v02 --- scsi-mpt.v00 --- scsi-mpt.v01 --- scsi-mpt.v02 --- scsi-qla.v00 --- scsi-qla.v01 --- uhci-usb.v00 --- imgdb.tgz --- state.tgz
build=5.0.0-441354
updated=1

As we learned from the previous article, the cron'd /sbin/auto-backup.sh generates a local.tgz which is then converted to state.tgz which contains all files automatically backed up by VMware. This file is loaded up along with other modules as part of the boot process. Understanding this, allows us to take advantage of this feature for persisting our own configuration files.

Here is an example use case for creating .ssh directory for SSH keys and persisting a script
(ghettoVCB.sh) in /bin for an ESXi 4.x host:

Step 1 - Re-create the modified directory structure and files in a temporary local path which will then be tarred and gzip. An example would be the following:

Actual change:
/.ssh/authorized_keys
/bin/ghettoVCB.sh

Temporary local directory structure of change:
/tmp/.ssh/authorized_keys
/tmp/bin/ghettoVCB.sh

Note: It is very important to ensure that the modified files get the stickybit permission set. As noted in the last article that upon a change, the visorFS will automatically create a special file to denote it for backup but also it allows the file to be writable at some later point for custom files being added.

Step 2 - You will use the tar utility to tar/gzip the contents in a file with extension .tgz. One thing to note, the file name including the extension must not exceed 12 characters. In our example, we made two changes and re-created the local structure under /tmp. We will need to change into /tmp directory and tar up the contents by using the following command:

/tmp # tar -czvf ghetto.tgz .ssh/ bin/
.ssh/
.ssh/authorized_keys
bin/
bin/ghettoVCB.sh

Now you should have the contents of .ssh and bin within your *.tgz file. To verify and view the contents, you can run the following command:

/tmp # tar -tzvf ghetto.tgz
drw------- 0/0 0 2011-08-10 04:10:11 .ssh/
-rw------T 0/0 399 2011-08-10 04:10:11 .ssh/authorized_keys
drwxr-xr-x 0/0 0 2011-08-10 04:10:24 bin/
-rwxr-xr-t 0/0 46973 2011-08-10 04:10:24 bin/ghettoVCB.sh

Step 3 - Next we need to copy the *.tgz file into /bootbank and modify the /bootbank/boot.cfg to ensure our new module is included in the boot up process

cp /tmp/ghetto.tgz /bootbank

Here is what the modified ESXi 4.x boot.cfg should look like after:

kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- ghetto.tgz
build=4.1.0-348481
updated=1
bootstate=0

The next time you reboot the system, you will automatically have your .ssh directory containing your SSH keys and the ghettoVCB script under /bin directory.

Now this is great for an inline modification, but what about creating custom configuration files and including that as part of a default kickstart installation? What about something like custom firewall rules in ESXi 5? In the following example, we'll include a custom firewall rule called "virtuallyGhetto.xml" which will be stored in /etc/vmware/firewall when the contents of the module is extracted.

Step 1 - We of course need to create the XML file containing the firewall rule and create the directory structure in which it will be unloaded to.

/tmp/etc/vmware/firewall/virtuallyGhetto.xml

Step 2 - Next we'll tar up the local contents

[root@air tmp]# tar -zcvf ghetto.tgz etc/
etc/
etc/vmware/
etc/vmware/firewall/
etc/vmware/firewall/virtuallyGhetto.xml

Step 3 - This new package will need to be stored on you installation server which will be reachable via http using wget and as part of the %firstboot stanza in your ESXi 5 kickstart. It will download the *.tgz file and append the entry in /bootbank/boot.cfg configuration file. Here are the entries that should go into your kickstart:

# Adding custom files to boot argument (e.g. custom firewall rules,etc)

wget http://air.primp-industries.com/esxi5/ghetto.tgz -O /bootbank/ghetto.tgz
sed -e '/modules=/s/$/ --- ghetto.tgz/' -i /bootbank/boot.cfg

Note: If you add custom files that are located under /etc and you have the stickybit enabled on your file, changes made will persist upon the next reboot by either manually running /sbin/auto-backup.sh or letting it run via cron. If you add custom files that are not located under /etc, any change you make must be periodically updated in your custom *.tgz file else the next reboot, the original file will be loaded.

How to Create and Modify vgz (vmtar) Files on ESXi 3.x/4.x

by // 6 Comments

There were several questions today on the VMTN community forums with regards to manipulating .vgz files in ESXi, also known as vmtar files. Due to the sparse amount of information on the web, I wanted to document some of the common operations that can be performed on the vmtar files. I will not be going over the use cases for manipulating or creating custom vmtar files, but here is one use case.

UPDATE (10/16/18) - For ESXi 6.5+, please use the following commands and the example below is using the s.v00 file:

Decompress file:

gunzip < s.v00 > s.v00.xz
xz --single-stream --decompress < s.v00.xz > s.v00.vtar
vmtar -v -x s.v00.vtar -o s.v00.tar
tar -xvf s.v00.tar

Compress file:

tar -cvf s.v00-new.tar bin/ etc/ lib/ lib64/ opt/ usr/ var/
vmtar -v -c s.v00.tar -o s.v00.vtar
xz --single-stream --compress < s.v00.vtar > s.v00.xz
xz --single-stream --compress < s.v00.vtar > s.v00

You can find some of these vmtar files with .vgz extension in the ESXi installation iso, here are a few highlighted in red:

To operate on existing vmtar files, you will need access to an ESXi host via ESXi Shell and using the /sbin/vmtar utility.

Usage: vmtar {[-x vtar/vgz-file] [-c tar/tgz-file] [-v] -o destination} | -t < vtar/vgz-file

In this example, we will copy the install.vgz to an ESXi host to perform some operations.

To list the contents of a vmtar file, you will need to use the -t option:

To extract the contents of vmtar file, you will need to use the -x and -o option:

vmtar -x install.vgz -o install.tar

Note: The output will be a standard tar file which will then need to be extracted before getting to the actual contents

To extract the tar file, we will be using the tar utility:

Let's say we made a change to one of the files and now we would like to re-create the vmtar file, we will first need to tar up the contents by using the tar utility again:

To verify the contents were all tarred up, we can view the contents by using the following command:

tar -tf install.tar

Now we will create the vmtar file using the vmtar utility:

vmtar -c install.tar -o install.vgz

We can confirm the contents by using vmtar -t option once again:

vmtar -t < install.vgz

If you decide to create your own custom vmtar files and want to verify the file layout, you can use vmkramdisk to assist you. Using vdf command, make note of the number of tardisks that have been mounted up.

Also make note of the filesystem layout by performing an "ls" on / (slash):

Now let's say you wanted to create a directory called virtuallyGhetto with a file in that directory called foobar and you wanted it to be mounted up under /

Here are the steps to perform the above:

Do you notice anything different? How about performing an "ls" on / (slash) again?

To umount the vmtar disk, you would use the following command:

vmkramdisk -u virtuallyGhetto.vgz