VMware Edge Cloud Orchestrator (VECO) was announced at VMware Explore US last week and one of the components that it is built on is using something called Project Keswick.
@alanrenouf and Elliot now on stage to talk about Project Keswick which VECO is built on #VMwareExplore pic.twitter.com/v9r29qDLIX
— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) August 23, 2023
In fact, at VMware Explore, Project Keswick was officially launched and is now available for customers to try out! I was fortunate enough to attend the Project Keswick technical deep dive session in person, but if you were not able to make the session or you were not at VMware Explore, the session recording is already available online for free to watch #VIB2169LV Scaling and Deploying ESXi at the Edge with Desired State Management and GitOps.
During the presentation, one of the slides mentioned that deploying a Keswick hosts takes ~30min, which is pretty darn fast if you think about all the configuration steps that is required to setup an ESXi host and register that with their cloud service, Keswick is automating all of that and more!
Project Keswick only takes 30 minutes to setup! Super fast which it automated a ton of manual steps including automatically connecting to cloud service. It’s also free to try right now! #VMwareExplore pic.twitter.com/1XDmZfzQ2h
— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) August 23, 2023
If you want to learn more about Project Keswick, you can check out their official documentation HERE. I figured I would put their claim to the test and here is your first look at using Project Keswick!
For my setup, I am using an Intel NUC 13 Pro that is configured with 64GB of memory and has a single M.2 NVME. For full Keswick hardware requirements, please refer to the official documentation HERE.
Step 1 - Register at https://keswick.showcase.vmware.com to enable the Keswick service on a new or existing VMware Cloud organization.
Step 2 - Download the Keswick Image, which is available in three different formats:
- ISO - Use this image if you will be booting from a physical/virtual CD-ROM including the use of out-of-band management like BMC/iDRAC/iLO
- OVA - If you do not have physical hardware to try out Project Keswick, you can use the OVA which is a Nested ESXi VM that contains the Keswick image and you can deploy that to vSphere, Workstation or Fusion for testing purposes.
- IMG - Use this image if you will be booting from a USB device
Note: I accidentally made the mistake of assuming the ISO image could also be used to create a bootable USB device as I typically use Unetbootin, which is a multi-platform utility for creating bootable live-CD on USB. If you use the ISO image, the ESXi installer is expecting the embedded ESXi Kickstart to be loaded from CD-ROM rather than USB, which is what the USB image is configured with. If you are familiar with ESXi Kickstart, it is a quick change to efi/boot/boot.cfg file on the USB device or you can simply download the correct image and then create the bootable USB device following directions HERE.
Step 3 - Plug in the USB device into your hardware, power it up and watch the magic of Project Keswick do its thing!
Note: The Keswick installer will automatically overwrite any existing ESXi or VMFS volumes that exists on the storage media. With that said, if one of the storage devices contains a vSAN partition, then the installer will fail. This is to be expected as there are different set of ESXi kickstart options for wiping vSAN disk. For now, you can either manually login to console and use partedutil or simply boot into live-cd to delete the partitions and then try again.
The installation itself was pretty fast, probably less than 10 minutes. You should see a splash screen like the one shown above, which might look familiar but it has been slightly modified to print out the hardware vendor, model and serial number. Make a note of this information as this is required to register your Keswick host to the cloud service.
Step 4 - Setup a Git repository that will then be associated with our Keswhick host. You have the option of using either public Git repo like Github or private repo like Gitlab. For detailed instructions, please refer to HERE.
Step 5 - Register the Git repository that was setup from the previous step. In this example, my Git repository is https://github.com/lamw/keswick-nuc13.git (it is private, which is recommended), branch name is main (you can choose anything but make sure to use that label when creating your branch) and label which will be used to associate this repository with Keswick host.
Step 6 - Register your Keswick Host to cloud service by providing the vendor, model and serial number from Step 3. If you have more than one hosts, you can also perform the registration using the CSV option. For vendor and model, if you are unable to find your specific system, you can simply add a new entry.
Once a new Keswick host has been registered, it will automatically go to the Dormant Hosts tab with a waiting status. This is expected as the Keswick service is waiting for the Keswick host to contact the Keswick service and see if there is a matching registration entry with the exact same information. This process can take up to 10minutes and I have seen it take a little longer in one case, but once it does, the status will change to pending as the host still needs to get activated which we will perform in the next step.
Note: If your Keswick host has still not transition into pending state, double check that your entry is correct. For those familiar with ESXi, you can login to DCUI remotely over SSH and simply run the dcui utility which will print out the same information, this can be useful as you can simply copy/paste to verify the string is an exact match. The default root password can be found on the USB device by examining the KS.CFG configuration file, which you can also modify if you wish to use a non-default password.
Step 7 - Finally, we just need to activate the Keswick host which involves associating the Git repository we had setup earlier in Step 5. As you can see from screenshot, in addition to the Git repository, you will also need to provide the username that has read-only access and the access token from Step 5.
After a couple of seconds, the Keswick host should now transition into the active state and is now ready for either additional host configuration and/or workload deployment via GitOps!
Host Configuration
ESXi host configurations are completely managed via GitOps, which are defined in a YAML manifests that contains the specific configurations which are expressed using the exact same syntax as vSphere Configuration Profiles.
Currently, the only available example is for configuring NTP, you will need to make a copy of this file and place that in Git repository that you had configured in Step 4. Once the configuration has been checked-in, it will be picked up by the Keswick service and automatically realized by the ESXi host when it goes and checks with Keswick service if there are any configurations that need to be applied.
Note: I found that the filename must not contain "dash" or it is not processed by the Keswick service, so ensure you are using either no spaces or "underscore".
For those interested in learning about the different configuration for an ESXi host, one option is to use the ESXi Configuration Store CLI (configstorecli) to view the configuration for an existing standard ESXi deployment.
In the NTP example, we can export from an existing system by running the following command:
configstorecli config current export -c esx -g system -o /tmp/esx_system_config.json
then we can view the contents of /tmp/esx_system_config.json to see what the key/values are.
Another example might be setting up syslog, we can export the configuration from an existing system that has all the syslog configurations by using the following command:
configstorecli config current export -c esx -g syslog -o /tmp/esx_syslog_config.json
then we can view the contents of /tmp/esx_syslog_config.json to see what the key/values are.
Note: While configstorecli works with JSON format, when constructing the Keswick host configuration files, we need to make sure the content is fully YAML compliant since it uses Kubernetes under the hood.
Workload Deployment
Similarly to ESXi host configurations, deploying workloads are also GitOps based and you simply check-in your Kubernetes YAML manifest files into the Git repository and the Keswick host will deploy them the next time it checks with the Keswick service.
The current example provided in the example repository is for setting up an Nginx "Hello World" which uses Metallb for a basic load balancer. Below are the two Kubernetes manifests which you will need to copy into your repository and make sure you update the IP pool range based on your setup in the metallb.yaml file at the very bottom.
While I really like the GitOps model with Keswick, there is currently not an easy feedback loop to see when a workload is fully realized or if there are any issues. For example, the above two manifests will setup a load balancer and then deploy the Nginx application. Since the Nginx is going to be accessed via Kubernetes Service, we need to ensure the load balancer is fully setup before an IP Address is provided for access, which is no different if you were to manually deploy this to a standard Kubernetes cluster.
For now, I found the easiest way to confirm everything was deployed correctly is by logging into ESXi host via SSH (not recommended) but from a learning perspective, this was immensely helpful to see what was happening. While looking at the some of the Keswick logs, I found an interesting command which allows me to interact with the Keswick control plane which is provided as a super tiny one-node Kubernetes cluster.
I will not bore you with the details, but using the following alias, I can now use the standard kubectl commands within ESXi Shell 😀
alias k="/bin/crx-cli exec --env 'KUBECONFIG=/mnt/volumes/volumes/kubernetes.io~empty-dir/etc-k8s/admin.conf' $(/bin/crx-cli list | grep infravisor-pod | awk '{print $1}') /infravisor/rootfs/keswick-control-plane/usr/bin/kubectl"
Using standard kubectl commands, as shown in the screenshot above, we can see the pods for both our nginx appplication and metallb are fully realized and we can then retrieve the IP Address by using:
k get svc -A
We are looking for the hello-keswick service and in this example, the IP Address is 192.168.30.240, which is part of the IP Pool we had configured in metallb.yaml and we can now access it within our browser as shown in the screenshot below!
I know the Keswick team is actively working on adding more examples and making more updates to the documentation, so definitely bookmark the official Keswick docs and/or leave any feedback you may have for those trying out Project Keswick.
Containers are not the only types of workloads that Keswick can deploy, it can also deploy Virtual Machines which uses a modified version of the VM Service Operator that is found in vSphere with Tanzu solution. The really nice thing is that the YAML manifest are exactly the same, which means easy portability or ease of use for those that have worked with the VM Service before.
Given how easy it was to setup Keswick, I think I might have gone a bit overboard as you can see below with my Keswick deployments 😛
Decided to deploy Project Keswick to just a few more hosts ... 😁 pic.twitter.com/enf7e2b30f
— William Lam (@lamw.bsky.social | @*protected email*) (@lamw) August 26, 2023
For those interested, here are the list of physical systems that I have successfully deployed Keswick to, which may also get updated in the future with more kits 😀
- Intel NUC 9 Extreme
- Intel NUC 10 Pro
- Intel NUC 11 Extreme
- Intel NUC 11 Pro
- Intel NUC 12 Pro
- Intel NUC 12 Enthusiast
- Intel NUC 12 Extreme
- Intel NUC 13 pro
- Supermicro E100-9W
- ASUS PN64-E1
- Apple Mac Mini5,3 (Requires modifying boot.cfg to include additional overrides, see post HERE)
- Lenovo P3 Tiny
- SolidRun V3000
- SimplyNUC Moonstone
- Lenovo P3 Ultra
Looks like you still need to obscure/hide the serial number.
Step 6 - Register your Keswick Host to cloud service by providing the vendor, model and serial number from Step 3.
Thanks! Just fixed