WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How should bulk operations such as adding ESXi hosts work?

06.04.2015 by William Lam // 4 Comments

I had a meeting yesterday with a couple of Engineers to provide some feedback on a new project they were working on and they were interested in getting feedback on their  workflows but also my overall impression of the user experience (UX). One area that we had spent some time covering was on adding new resources such as an ESXi host which is performed using either a UI/API/CLI. In the case of adding an ESXi host, this was done sequentially and a user would need to specify multiple parameters like the hostname, IP Address, credentials, etc. and perform this task N-number of times until you had added all of the ESXi hosts that you wanted. Many of you are probably already familiar with this workflow as it exists in many of the current VMware products today when requiring to add a resource like an ESXi host.

I was wondering why this was the case and why not support a bulk add operation? This was not a new idea and in fact I and many others have asked about this in the past on why our own vSphere Web Client does not support bulk operations for some of the more common operations such as adding an ESXi host or provisioning a VM? In fact, I was actually reminded of this topic again this morning when I needed to mount an NFS Volume across multiple ESXi hosts and one of the new capabilities that was introduction in the vSphere Web Client over the vSphere C# Client was being able to bulk add an NFS Volume across multiple hosts.

Bulk support for adding NFS volumes across multiple ESXi host, still love this feature! Simple, yet powerful & useful pic.twitter.com/TZ7KeG52f6

— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) June 4, 2015

Not only is this a useful capability to have but when you need to perform the same operation multiple times, there is also an increase of risk for making a mistake. Sure, if you mount the wrong NFS volume, you would have it wrong across all the hosts but this is no different if you were to Automate it from an API/CLI. I am sure some of you are probably saying, a simple for loop would do the trick and though I agree that would be the easiest way. However, it would be really nice that some of these operations would actually have native support for bulk operations from an API standpoint, that way this is not limited to just a UI-only feature which it is today. Even in the case of a simple for-loop, you are still executing the individual operation N-number of times versus being able to pass in a list and perform a single request and then rely on the backend implementation to properly handle the operation which could be done in parallel or sequentially depending on the type of operation.

Thinking about this a bit more, it may not be as simple as just adding bulk operation support into an API but there are also other things to consider. What happens when the resource you are adding has varying configurations, a classic example of this is adding multiple ESXi hosts. If they all have the same username and password, then you can provide the credentials once and then a list of ESXi hosts. What happens when they differ or in the worse case scenario, they all have different passwords? What happens when only a subset of the hosts are added, do you continue or fail all together?

I am actually curious to hear what others think about this and what your expectations would be? Both from a UI standpoint as well as from an Automation standpoint which includes API/CLI. Would a simple JSON/YAML configuration file that contains all hosts and credentials be acceptable with the option of having common global configurations and ability to override for individual entries? Would love to hear your thoughts on this and what you would be expecting? Perhaps this is something I can also share back with the Engineering teams as this is something I have been wanting to see for quite some time, especially when it comes to common operations like adding ESXi hosts and provisioning VMs.

Categories // Automation, vSphere Web Client Tags // add host, bulk operation, ESXi

ghettoVCB VIB & offline bundle for ESXi

05.28.2015 by William Lam // 63 Comments

It is still amazing to see that the number of contributions and suggestions from the community continues to grow for my free and simple VM backup solution called ghettoVCB. I created ghettoVCB almost 8 years ago which now has over 1.2 million views, pretty insane if you ask me! Although I am quite busy these days which includes a new born, I still try to find time to update the script as time permits. A couple of weeks back I received an email from one of my readers who came across ghettoVCB and was quite happy with the free solution. He also had some feedback asking why I did not provide an installable VIB for ghettoVCB?

A totally valid question and the answer was quite simple. When I had first created ghettoVCB back in the classic ESX 3.x days, the concept of a VIB had not existed yet. With the release of ESXi 5.0, the idea of the VIB was introduced but it was only recently in 2012 did VMware publish a method for customers to create custom VIBs for ESXi using the VIB Author Fling. I do have to admit at one point I did think about providing a VIB for ghettoVCB, but I guess I never went through with it for whatever reason. Looking back now, this was a no-brainer to provide a simplified user experience and not to mention the benefit of having ghettoVCB installed as a VIB is that it will automatically persist on ESXi after reboots which was a challenge for new users to ESXI.

So without further ado, here is ghettoVCB provided in either a VIB or offline bundle form:

  • vghetto-ghettoVCB.vib
  • vghetto-ghettoVCB-offline-bundle.zip

To install the ghettoVCB VIB, you just need to download the VIB and run the following ESXCLI command and specifying the full path to the VIB:

esxcli software vib install -v /vghetto-ghettoVCB.vib -f

Once installed, you will find all ghettoVCB configuration files located in:

/etc/ghettovcb/ghettoVCB.conf
/etc/ghettovcb/ghettoVCB-restore_vm_restore_configuration_template
/etc/ghettovcb/ghettoVCB-vm_backup_configuration_template

Both ghettoVCB and ghettoVCB-restore scripts are located in:

/opt/ghettovcb/bin/ghettoVCB.sh
/opt/ghettovcb/bin/ghettoVCB-restore.sh

One additional thing I would like to point out is that you can also quickly tell which version of ghettoVCB is running by inspecting the installed VIB by using the following ESXCLI command:

esxcli software vib get -n ghettoVCB

If you look at the screenshot above, I have highlighted two important pieces of information in green. The first is the "Description" property which includes the Github commit hash of the particular revision of ghettoVCB and the "Creation Date" property which contains the date of that commit. This can be handy if you want to compare it to the latest ghettoVCB repository found on Github here. Thanks again Markus for the suggestion!

For those of you who are interested in the details for creating your own ghettoVCB VIB, the next section is specifically for you. Earlier this week I blogged about a Docker Container that I have created to help build custom ESXi VIBs and as you can see now, that was the basis for us to be able to quickly create ghettoVCB VIB based on the latest revision of the script.

Step 1 - Create a new Docker Machine following the steps outlined here.

Step 2 - Login to the Docker Machine and create a new Dockerfile which contains the following:

FROM lamw/vibauthor

# Due to https://stackoverflow.com/a/49026601
RUN rpm --rebuilddb
RUN yum clean all
RUN yum update -y nss curl libcurl;yum clean all

# Download ghettoVCB VIB build script
RUN curl -O https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/create_ghettoVCB_vib.sh && chmod +x create_ghettoVCB_vib.sh

# Run ghettoVCB VIB build script
RUN /root/create_ghettoVCB_vib.sh

CMD ["/bin/bash"]

Step 3 -  Next we need to build our new Docker Container which will use the VIB Author Container by running the following command:

docker build -t lamw/ghettovcb .

Screen Shot 2015-05-26 at 2.14.52 PMThe output will be quite verbose, but what you will be looking for is text highlighted in green as shown in the screenshot above. You should see the successful build of both the VIB and offline bundle as well as Docker Container showing a successful build.

Step 4 - After a successful build of our Docker Container, we can now launch the container by running the following command:

docker run --rm -it lamw/ghettovcb

Screen Shot 2015-05-26 at 2.16.58 PM
Once logged into the Docker Container, you will see the generated VIB and the offline bundle for ghettoVCB as shown in the screenshot above.

If you wish to copy the VIB and offline bundle out of the Docker Container into the Docker Host, you can use Docker Volumes. I found this useful thread over on Stack overflow which I have modified to include the copying of the ghettoVCB VIB and offline bundle out to Docker Host by running the following command:

docker run -i -v ${PWD}/artifacts:/artifacts lamw/ghettovcb sh << COMMANDS
cp vghetto-ghettoVCB* /artifacts
COMMANDS

Finally, to copy the ghettoVCB VIB from the Docker Host to your desktop, we first need to identify the IP Address given to our Docker Machine by running the following command:

docker-machine ip osxdock

Currently, Docker Machine does not include a simple "scp" command so we will need to use regular scp command and specify the private SSH keys which you can find by running "docker-machine inspect [NAME-OF-DOCKER-HOST]" and connecting to our Docker Host to copy the ghettoVCB VIB by running the following command:

scp -i /Users/lamw/.docker/machine/machines/osxdock/id_rsa [email protected]:artifacts/vghetto-ghettoVCB.vib .

Categories // Automation, Docker, ESXi, Fusion Tags // container, Docker, docker-machine, ESXi, ghettoVCB, ghettovcb-restore, vib, vib author

A Docker Container for building custom ESXi VIBs

05.26.2015 by William Lam // 16 Comments

I recently had a need to create a custom ESXi VIB using the VIB Author Fling for a project that I was working on. As part of the project's deliverables, I wanted to also provide an ESXi VIB which would need to be built against any new updates for the project. Given this would be an infrequent operation, I thought why not use a Docker Container for this operation? I could just spin up a Docker Container on-demand and not have to worry about managing a local VM for just running this particular task.

With that I have created a VIB Author Docker Container which can be used to author custom ESXi VIBs. I have also made this container available on the Docker Registry for others to use which you can find more details here: https://registry.hub.docker.com/u/lamw/vibauthor/

If you already have a Docker host running, you can pull down the VIB Author Docker Container by jumping to Step 5 in the instructions below. If you do not and you are running Mac OS X like I am, you can follow the instructions below using Docker Machine and VMware Fusion to try out my VIB Author Docker Container.

Step 1 - Install the Docker client by running the following command:

brew install docker

Step 2 - Download and install Docker Machine by running the following commands:

curl -L https://github.com/docker/machine/releases/download/v0.2.0/docker-machine_darwin-amd64 > /usr/local/bin/docker-machine
chmod +x /usr/local/bin/docker-machine

Step 3 - Create Docker Machine using the VMware Fusion driver by running the following command:

docker-machine create --driver vmwarefusion osxdock --vmwarefusion-memory-size 1024
eval "$(docker-machine env osxdock)"

docker-container-vib-author-esxi-vib-0
Note: Thanks to Omer Kushmaro for his blog post here on how to quickly get started with Docker Machine with VMware Fusion

Step 4 - Once the Docker Machine is booted up, we can now connect to it using SSH by running the following command:

docker-machine ssh osxdock

docker-container-vib-author-esxi-vib-3
At this point, we are now logged into our Docker Machine which has both the Docker client/server running and we are now ready to pull down the VIB Author container from the Docker registry.

Step 5 - To pull down the VIB Author Docker Container that I have built, run the following command within the Docker Machine:

docker pull lamw/vibauthor

docker-container-vib-author-esxi-vib-1
Step 6 - Once the Docker Container has been successfully downloaded, you can now run the VIB Author Container by running the following command:

docker run --rm -it lamw/vibauthor

docker-container-vib-author-esxi-vib-2
Once logged into the VIB Author Container, you confirm that the VIB Author Fling has been installed by running the "vibauthor" command as shown in the screenshot above. In the next blog post, I will go through an example of building a custom ESXi VIB using the VIB Author Container as well as transferring the outputted files from the Docker host back onto your desktop. Stay tuned!

Categories // Apple, Docker, ESXi, Fusion Tags // container, Docker, docker-machine, ESXi, vib, vib author

  • « Previous Page
  • 1
  • …
  • 34
  • 35
  • 36
  • 37
  • 38
  • …
  • 61
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025

 

Loading Comments...