WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How to compile Busybox for ESXi ... kind of Part 2

02.15.2011 by William Lam // Leave a Comment

Continuing from part1 of How to compile Busybox for ESXi ... kind of Part 1 we found some challenges in compiling VMware's Busybox version found in ESXi OSS Source Code. In this article, we will go through process of compiling the latest version of Busybox which is currently 1.18.3 to run on ESXi.

Again, before getting started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!! 

The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.

Step 1

Download busybox-1.18.3.tar.bz2 from the Busybox's website and SCP it to your build system.

Step 2

You will also need to install the following packages, you can do so using yum if you are using CentOS or RHEL and have a proper repository configured. You can use the following:

yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64

Step 3

Extract the contents of busybox-1.18.3.tar.bz2 using the following command:

tar -xvjpf busybox-1.18.3.tar.bz2

Step 4

You will now change into the busybox-1.18.3 directory and from here you have a few ways of building Busybox. To get a list of build options, use the following command:
make help

You can customize the build of Busybox by enabling and disable specific applets to be compiled. If you want to build Busybox with no applets (not really useful), you can run the following command:

make allnoconfig
make

Once the compilation is complete, you will now have a Busybox binary in which you can run but it does nothing useful:

You can also perform the exact opposite by enabling all options which can be called using the "allnoconfig" or "defconfig", you can run the following command:

make defconfig
make

Once the compilation is complete, you now have Busybox binary which has all the available applets by default:

Enabling all applets, the Busybox binary still comes out to be less than 900k. If you wanted to create a custom Busybox with specific applets, you can manually edit the .config file which is where the features are either enabled or disabled. This can get very tedious, luckily there is a make option which allows for an interactive menu on selecting the applets you would like to include in the build.

Step 5

To enable the interactive menu, you will run the following command:

make menuconfig

From here, you will be able to configure the Busybox settings and the various applet types by their functions such as networking or mail utilities. One feature that is actually disabled in the VMware's version of Busybox is the support for large files, if you tried to tar up a file that was larger than 2GB and then extract the file, you will notice you get an error regarding the file size. The reason is VMware decided to not enable this feature in their Busybox build.

Step 6

Once you are done selecting or de-selecting all your Busybox settings and applets, you will need to save your configuration which is stored in the .config.

Step 7

Once you have exited from the interactive Busybox menu, you are now ready to build your custom version of Busybox, you will run the following command:

make

Step 8

If everything went well, you now should have a busybox binary in your current working directory. Here is an example of my custom Busybox image:

If you would like to customize the version information found when just running busybox command without any arguments, you can edit the Makefile and append your custom text to EXTRAVERSION variable.

Now you just need to re-run the "make" command and your new busybox binary will include the additional version information.

Now before you jump and start creating your own Busybox binary, I must throw in another caveat that re-iterates the title of this post. Just because you have Busybox built with all these applets, it does not mean it will run 100% on ESXi, the reason being is there are specific dependencies that the non-VMware Busybox applets may rely on which are just not available with ESXi. This is one of the reasons why VMware did not just enable all applets from Busybox to begin with, your mileage will vary depending on the particular command you have enabled.

Here is an example of my custom Busybox applet "crontab" using the "-l" flag which lists any active cron entries for the particular user:

It runs just fine on ESXi as you can see, but let's try using the "-e" flag which if you are familiar with using crontab you will know that is the option to edit your cron entries:

Ooops, what's this? It looks like this particular operation of crontab relies on vfork which is not implemented. I have seen other similar errors with other applets relying on particular dependencies in the environment that just does not exist with ESXi. I have not tested every single applet and some may work 100% while others may be partially functional, you will need to test and verify for yourself.

If you feel adventurous, go ahead and download Busybox and start playing with it. I would highly recommend that you test this in your vSphere development environment before trying this on a production host. I would also recommend that you do NOT replace the default Busybox binary found on ESX(i) as you can run into some serious issues. Since the Busybox binary is self-contained, you can store it in /tmp or somewhere more permanent such as a local or remote VMFS datastore and renaming the file will also remove any confusion.

If you find other utilities that you feel that should be included in the default Busybox applet and do not want to resort to something like Poor man's traceroute for ESXi, be sure to submit your feedback to either your VMware rep or file a feature request.

If you are interesting to learning more about Busybox, check out their FAQ page and this page for more details.

Again, another warning:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!

Categories // Uncategorized Tags // busybox, ESXi 4.1

How to compile Busybox for ESXi ... kind of Part 1

02.14.2011 by William Lam // 1 Comment

I had initially researched this topic a few months back and decided not to write about it since it did not yield the results I was expecting. Some recent twitter conversations with Carter Shanklin and Didier Pironet this morning motivated me to dig up this content and share it with the community.

Before I get started, a word of caution:
!!!! THIS IS NOT SUPPORTED BY VMWARE - PLEASE USE AT YOUR OWN RISK !!!!

As you all know, ESXi does not have a Service Console like the classic ESX, but it does have a limited console based on Busybox, specifically v1.9.1 which is still the current version even with the latest release vSphere 4.1 Update 1. Busybox is heavlily customized by VMware and you will find a limited set of POSIX-like utilities, it is no where feature complete to your traditional POSIX environment and there is good reason for this.

If you have ever downloaded vSphere Hypervisor (formally known as ESXi), you may have noticed an "Open Source" tab at the top. This page contains some of the open source licensing information that VMware has used and incorporated into their products, most of these are just text files listing the various packages. One file that is of interest is the "OSS Source Code for ESX, ESXi and ESXi Embedded 4" which actually includes some of the open source RPMs and packages used to build Busybox for both classic ESX and ESXi (Yes, classic ESX also includes a busybox binary).

One interesting thing to note, is the date of this package, it has not been updated for almost two years. I am not sure if this is done intentionally, but I doubt that would be the case. It is also odd that VMware is still using an old release of Busybox (1.9.1), the latest release is at 1.18.3 and there have been quite a few updates between 1.9.1 and the current release. There has been speculation and rumors in the past that VMware choose a particular version of Busybox to get around the licensing of Busybox with ESXi. I do not know if this is true, but I do know, there have been quite a few bug fixes since incarnation of ESXi and you may hit some of those if you venture into Busybox console also known as Tech Support Mode (formally known as Unsupported Console)

The build environment that I used is running the latest version of CentOS 5.5 64bit which can be downloaded here.

Step 1

Download VMware-esx-public-source-4.0-162945.tar.gz (565MB) from from VMware's website and SCP it to your build system

Step 2

You will also need to install the following packages, you can do so using yum if you are using CentOS or RHEL and have a proper repository configured. You can use the following:

yum install -y gcc flex bison texinfo ncurses-devel libselinux-devel.x86_64 pam-devel.x86_64

Step3

Create a temporarily directory and move the tar.gz file into the directory and then extract the contents using the following command:

tar -zxvf VMware-esx-public-source-4.0-162945.tar.gz

You now should have a bunch of *.rpm files that have been extracted. There are two interesting RPM packages that we are interested that have "busybox" in the name, one of classic ESX (vmware-esx-busybox-1.9.1-1.624.vmw.src.rpm) and one for ESXi (vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm)

Step 4

You will need to create the following directory before installing the ESXi RPM package. Use the following command:

mkdir -p /usr/src/redhat/SOURCES

Next, you will install the vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm by using the following command:

rpm -ivh vmware-visor-busybox-1.9.1-1.654.vmw.src.rpm

Note: Don't worry about the warnings regarding the user/groups that do not exists, these are most likely a default from VMware's environment that expects the mts user and group.

Step5

Now you will change into the /usr/src/redhat/SOURCES directory and you should see three files:

  • busybox-1.9.1.tar.bz2 (Busybox v1.9.1)
  • config-vmware-visor (Busybox config file from VMware)
  • vmware.patch (VMware patch for building Busybox)

You will now extract the contents of the Busybox archive by using the following command:

tar -xvjpf busybox-1.9.1.tar.bz2

Next you will copy the vmware.patch into the busybox-1.9.1 directory by using the "cp" command and then change busybox-1.9.1 directory. If you do a "ls" you should see the following files:

Step 6

We are now going to apply the patch provided by VMware which will update some of the C source code found in default Busybox 1.9.1 along with setting up a build environment. We go through a dryrun process to see what files will be patched, use the following command:

patch --dry-run -p1 < vmware.patch

To apply the patch, you will run the following command:

patch -p1 < vmware.patch

After the patch has been applied, there are two interesting files that will be created in addition to the source code updates:

  • README.vmware (Pretty well documented set of changes + how to build Busybox)
  • vmware-env.sh (environment script that is sourced to build Busybox)
Step7

Now we are ready to build Busbox, the exact instructions are specified towards the end of the README.vmware file

Basically you will be executing the following commands:

VMWARE_TARGET=visor
source vmware-env.sh
cp ../config-vmware-visor .config
make oldconfig
busybox-build

Now if you executed the above, you will find that you quickly run into some build errors.

This is where the "kind of" part of the blog post comes in. If you recall, we sourced the vmware-env.sh script just prior to building, this is a script that VMware wrote to quickly setup the build environment for compiling Busybox for either ESX or ESXi.

For this particular build and release, they are using specific versions of GCC, Binutils and PAM. You might ask why not substitute these with the versions found on system? I have actually gone through that, but ran into other compliation issues which maybe due to fact that GCC and other components are much newer and not compatible with older versions of Busybox. I have also tried to match the build environment by using buildroot but continue to hit other compilation issues. The only way I can see building this particular patched version of Busybox is to get a build environment that matches what VMware uses.

In Part2, I will go into details on building your own Busybox binary for ESXi which I promise will be much more fruitful than this. Stay tuned ...

Update: It looks like there is a slightly more recent version of ESX(i) OSS released on 07/13/2010, for whatever reason, it was filed under VMware View section. The source RPM for ESXi still looks like the same one released back in 2009, but there some other packages that differ from the one found on vSphere Hypervisor page.

Categories // Uncategorized Tags // busybox, ESXi 4.1

Another way to enable management traffic on ESXi

02.09.2011 by William Lam // 3 Comments

Here is another way in you can enable the management traffic type on a VMkernel interface in ESXi without having to resort to using the vSphere API, this especially useful when automating a kickstart installation.

When you enable a specific vmkX interface (esxcfg-vmknic -l) to allow for the management traffic type, there is an entry that is made in the /etc/vmware/hostd/hostsvc.xml file. The specific interface is denoted by a unique nic id which starts off at 0000 and is incremented by one for additional VMkernel interfaces that are added.

If you add a second VMkernel interface called vmk1 and you wanted to also enable it for management traffic, the file would look like the following:

For the changes to take effect, you will need to restart hostd agent by running /etc/init.d/hostd restart. You will also need refresh the network sub-system by using vim-cmd hostsvc/net/refresh which will refresh the vSphere Client view else you will have to do it manually before you can see the update.

This is probably overkill, but I decided to write a simple script in which you can pass the VMkernel interface name and the script will update hostsvc.xml file with the proper nic id/etc. Here is an example for enabling management traffic for vmk1:

Download: enableMgmt.sh

If you would like to integrate this into your ESXi kickstart, you can easily do so based on the number of VMkernel interfaces you will be creating during the installation. You can add the following into your %post section which uses a here document to overwrite the existing hostsvc.xml with the expected VMkernel interfaces that should have the management traffic type enabled.

HOSTSVC_FILE=/etc/vmware/hostd/hostsvc.xml

cat > ${HOSTSVC_FILE} << __CREATE_HOST_SVC__


vmk0
vmk1
vmk2

normal

on
on
on


__CREATE_HOST_SVC__

This is a cleaner alternative than using python and connecting to the vSphere API locally on an ESXi host which is described in my blog article Automating ESXi 4.1 Kickstart Tips & Tricks Tip #7

Categories // Uncategorized Tags // ESXi 4.1, management interface

  • « Previous Page
  • 1
  • …
  • 8
  • 9
  • 10
  • 11
  • 12
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025