WilliamLam.com

How to configure network proxy with Tanzu Kubernetes Grid (TKG)?

by // 3 Comments

Network Proxies are commonly used by customers to provide connectivity from internal servers/services to access external networks like the Internet in a controlled and secured manner. While working on a recent network proxy enhancement for our VMware Event Broker Appliance (VEBA) Fling, I had setup a Squid server which is a popular network proxy solution.

I had noticed a couple of folks were asking about network proxy configuration for Standalone Tanzu Kubernetes Grid (TKG) and figure this might be interesting to explore, especially for my recently released TKG Demo Appliance Fling which enables folks to quickly go from zero to Kubernetes in just 30 minutes! I figured this would be another good opportunity to learn a bit more about TKG as well as Kubernetes (K8s) and I jokingly said to myself, how hard could this be!? 😉 Apparently it was not trivial and took a bit of trial/error to figure out the correct combination and below is the procedure that can be followed for both standard deployment of TKG as well as the TKG Demo Appliance Fling.

Proxy Setting configurations for TKG CLI

The TKG CLI uses KinD (Kubernetes in Docker) under the hood to setup the initial K8s bootstrap cluster to deploy the TKG Management Cluster. If you have not already downloaded KinD node image (registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2) or if you need to go through a network proxy to do so, then the following instructions can be followed to make your Docker Client aware of a network proxy.

Here is an example of the error if Docker Client can not download the image:

# docker pull registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2
Error response from daemon: Get https://registry.tkg.vmware.run/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

If you are not using a private container registry with TKG, then you also need to also ensure that the KinD Cluster can connect to your network proxy when it pulls down the required containers from the internet. Luckily, KinD can simply detect the network proxy settings of your operating system. You can either set the proxy using traditional environmental variables (http_proxy, https_proxy and no_proxy) during your use of TKG CLI or you can simply set it globally so you do not forget.

In my setup, TKG CLI is running in a Photon OS VM and global proxy settings are configured in /etc/sysconfig/proxy Proxy settings will vary across operating systems and you should check with the vendor documentation for specific instructions. The following command will set both HTTP and HTTPS proxy variables to use my proxy server and you will also want to make sure you whitelist all networks and addresses which you want to by-pass the proxy.

cat > /etc/sysconfig/proxy << EOF
PROXY_ENABLED="yes"
HTTP_PROXY="http://192.168.1.3:3128"
HTTPS_PROXY="http://192.168.1.3:3128"
NO_PROXY="localhost,192.168.1.0/24,192.168.2.0/24,registry.rainpole.io,10.2.224.4,.svc,100.64.0.0/13,100.96.0.0/11"
EOF

Note: If you are using the TKG Demo Appliance, you only need to configure the Photon OS global proxy settings. In my example, I have white listed my local 192.168.* addresses, registry.rainpole.io which is the embedded Harbor registry, 10.2.224.4 which is the internal IP Address of VMC vCenter Server, *.svc addresses which all the internal K8s services and 100.64.0.0/13 which is the CIDR range used by TKG for the Service networks and 100.96.0.0/11 which is the CIDR range used by TKG Cluster networks.

[Read more...]

Guest Customization support for Instant Clone in vSphere 7

by // 2 Comments

vSphere Instant Clone was re-architected back in vSphere 6.7 and has been enhanced to be made more powerful and flexible. These enhancements not only power solutions like VMware Horizon but it also unlocks new customer use cases including things like Instant Cloning of Nested ESXi and Apple MacOS Guests.

Although the possibilities are truly endless with Instant Clone, this also means that any customization including basic guest identity such as hostname and networking must now use an alternative workflow. For application-level customization, it is expected that customers will create and manage these custom scripts but for basic networking configuration, it would be ideal to leverage the existing and well known vSphere Guest Customization Engine.

While downloading a file from MyVmware the other day, I came across an interesting set of packages called Guest Customization Engine for Instant Clone. Upon further investigation, I came to learn that these guest packages actually enable support for native vSphere Guest Customization for Instant Clone in vSphere 7 for the following Linux guest OSes:

  • CentOS 7.4 or higher
  • RHEL 6.8 or higher
  • RHEL 7.4 or higher
  • Ubuntu 16.04
  • SUSE 11SP4
  • SUSE 12SP3 or higher

In addition, there is also new set of vSphere (SOAP) APIs that you will need to interact with to use the new Instant Clone Guest Customization feature. The GuestCustomizationManager is a new vSphere 7.0 API which includes the following three API methods:

  • AbortCustomization_Task
  • CustomizeGuest_Task
  • StartGuestNetwork_Task

If you are interested in taking advantage of the new Instant Clone Guest Customization in vSphere 7, you can refer to the official VMware documentation which has step by step instructions.

How to passthrough USB Keyboard/Mouse HID and CCID devices to VM in ESXi?

by // 66 Comments

About a month back I had received an interesting tidbit from Darius Davis (VMware Engineer) after helping a customer solve an interesting problem and Darius thought this could be a useful blog post to share. Funny enough, a couple of weeks after that conversation, a simliar issue was being faced by another customer and luckily I was able to share with them the solution and also validate the specific configuration that was needed.

The customer that Darius was helping out had two VMs running on ESXi which they wanted to configure several passthrough devices. In addition to a PCI passthrough of a GPU, they also wanted to passthrough independent USB keyboard and mouse to each individual VM. PCI passthrough to a VM is nothing new but passing through a USB keyboard/mouse also known as Human Interface Devices (HID) to a VM is generally not expected. The physical ESXi host just assumes these type of USB devices are meant for it to consume.

In addition to HID USB devices, there are also Chip Card Interface Devices (CCID) USB devices like a smart card reader which customers may also want to passthrough to a VM. The latter use case was what I ended up helping the customer out with. To passthrough HID/CCID USB devices, the following steps are required which will include changes to the ESXi host.

Step 1 - Add the following two VM Advanced Settings for all USB CCID/HID devices that you wish to enable passthrough:

HID USB Devices:

usb.generic.allowHID = "TRUE"
usb.quirks.device0 = "0xXXXX:0xYYYY allow"

CCID USB Devices:

usb.generic.allowCCID = "TRUE"
usb.quirks.device0 = "0xXXXX:0xYYYY allow"

where 0xXXXX = vendorId and 0xYYYY = deviceId (e.g 0x03f0:0x0024) which was retrieved from the previous step

To list all USB devices and get their vendor and device ID, you can use the lsusb command found within the ESXi Shell and below is an example listing out both my USB Mouse and Keyboard.

lsusb -v | grep -E '(^Bus|HID)'
Bus 001 Device 001: ID 0e0f:8003 VMware, Inc. Root Hub
Bus 002 Device 001: ID 0e0f:8003 VMware, Inc. Root Hub
Bus 001 Device 003: ID 8087:0026 Intel Corp.
Bus 001 Device 004: ID 0781:5591 SanDisk Corp. Ultra Flair
Bus 001 Device 002: ID 05fe:0011 Chic Technology Corp. Browser Mouse
    iConfiguration          4 HID Mouse
        HID Device Descriptor:
          bcdHID               1.00
Bus 001 Device 005: ID 046d:c31d Logitech, Inc. Media Keyboard K200
        HID Device Descriptor:
          bcdHID               1.10
        HID Device Descriptor:
          bcdHID               1.10

Step 2 - We need to make the USB arbitrator service aware of these USB device quirks by adding the usb.quirks.deviceN string to /etc/vmware/config file. In my example above, I want to enable passthrough for both my Mouse and Keyboard device, so the following entries would be appended:

usb.quirks.device0 = "0x05fe:0x0011 allow"
usb.quirks.device1 = "0x046d:0xc31d allow"

Step 3 - Lastly, we need to append the following string to the ESXi boot option to disable the VMkernel from claiming HID USB devices.

CONFIG./USB/quirks=0xXXXX:0xYYYY::0xffff:UQ_KBD_IGNORE

where 0xXXXX = vendorId and 0xYYYY = deviceId (e.g 0x03f0:0x0024)

The easiest way to append this to the boot option is by editing /bootbank/boot.cfg rather than manually typing this during the initial boot up (SHIFT+O)

Note1: This is not required for CCID devices or mouse devices as they are not claimed by ESXi

Note2: The USB quirks are given in sets of five parameters - vendorID:deviceID:minRevision:maxRevision:quirkName If you wish to specify multiple devices, you will need to ensure all five parameters are included. Here's an example for specifying two USB devices: CONFIG./USB/quirks=0x05fe:0x0011::0xffff:UQ_KBD_IGNORE:0x046d:0xc31d::0xffff:UQ_KBD_IGNORE

Step 4 - A system reboot of your ESXi host will be required for the changes to go into effect. Once your ESXi host is available, you can use the vSphere H5 Client or Embedded ESXi Host Client to attach the USB devices. For vCenter Server, click on "Add New Device" and select "Host USB Device" and for ESXi, click on "Add other device" and select "USB Device".