Using the new installation method for deploying OpenShift 4.5 on VMware Cloud on AWS

I recently saw a tweet from Jason Shiplett who works over on the VMware Validated Design (VVD) team (also my former team before joining VMware Cloud) who shared a new validated design for running Redhat OpenShift 4.3 on VMware Cloud Foundation. Funny enough, a couple of days ago I was just researching into deploying OpenShift running on VMware Cloud on AWS from a customer inquiry.

Timing could not have been better as RedHat just announced their OpenShift 4.5 release a few days ago as and one of the major updates is support for vSphere using their full stack automation also known as te Installer Provisioned Infrastructure (IPI) option. Previous to this, customers who wanted to deploy OpenShift on vSphere had to use the User Provisioned Infrastructure (UPI) method, which the VVD design also uses, which is much lengthier and complex when compared to the native IPI method.

For someone who has never worked with OpenShift before, this was great news and I get to try out this new deployment method on an VMware Cloud on AWS infrastructure 🙂

Pre-Requisites:

Step 1 - You will need a Linux system to perform the installation and it should have access to the vCenter Server running in VMware Cloud on AWS (VMC). In my example, I am using an Ubuntu Server 20.04 VM which is also running in the SDDC and has outbound internet connectivity.

Step 2 - Login to VMware Cloud on AWS console and create a new NSX-T network segment that is DHCP enabled. In my example, I named it openshift-network with a 192.168.3.0/24 configuration.

Step 3 - Navigate to Inventory->Groups and create the following groups and replace the CIDR networks with that of your SDDC:

Step 4 - Navigate to Security->Gateway Firewall and create and publish the following firewall configurations:

Step 5 - Create the following two DNS entries using the following format. In my example, the cluster-name is called vmc and my base DNS domain is vmware.corp

Install OpenShift:

Step 1 - Download the OpenShift Installation files for vSphere which includes the OpenShift Installer (openshift-client-linux.tar.gz), OpenShift CLI Tool (openshift-client-linux.tar.gz) and a copy of the Pull Secret to the Linux VM.

Step 2 - Extract both OpenShift Installer and CLI packages by running the following command:

tar -zxvf openshift-client-linux.tar.gz
tar -zxvf openshift-install-linux.tar.gz

Step 3 - Download the trusted root certificates from your vCenter Server which his located at https://vcenter.sddc-[a.b.c.d].vmwarevmc.com/certs/download.zip and transfer that to the Linux VM.

Step 4 - Extract the certificates from the download.zip file and import them into the Linux system. The commands below are for Ubuntu, you may need to replace the commands based on Linux distribution you are using.

unzip download.zip
cp certs/lin/* /usr/local/share/ca-certificates
update-ca-certificates

Step 5 - Run the OpenShift installer with the following command to start the configuration wizard to generate our configuration file:

./openshift-install create install-config

The wizard is pretty intuitive to use. Make sure when you are providing the hostname for the vCenter Server that you are using the FQDN as the certificate that we had imported earlier must match the hostname here. The rest of the input will be the same for all SDDC with the exception of network, API/Ingress VIP and your DNS base domain and OpenShift Cluster name and the Pull Secret.

Once you have completed the wizard, the configuration is saved to the install-config.yaml file which you can inspect, edit or save for future uses. This file is automatically deleted after performing the installation, so you may want to make a copy of the file for your own reference.

Step 6 - Next, we start the OpenShift deployment by running the following command:

./openshift-install create cluster

This process should take about 30 minutes or so. In my setup, it took 22 minutes. If everything was successfully deployed and configured, you should see the details about logging into your OpenShift Cluster along with the kubeadmin credentials. If you forget to save the credentials, the information is also stored in .openshift_install.log log file.

One thing to be aware of is that the OpenShift Installer will deploy all VMs to the vSphere Cluster level. It is recommended that you move these VMs into the Compute-ResourcePool after the deployment has completed to ensure you do not have an in-balance of resource distribution.

Step 7 - Lets verify that our DNS was properly configured by logging into the OpenShift UI. In my example, the URL to the UI is https://console-openshift-console.apps.vmc.vmware.corp and you will login using the kubeadmin credentials.

Step 8 - We can also verify the installation by logging into the OpenShift cluster using the OpenShift CLI by running the following commands:

export KUBECONFIG=/root/auth/kubeconfig
oc login
oc get nodes

If you wish to use the internal image registry that OpenShift provides, you will need to setup an external NFS volume as the vSAN File Services is currently not available in VMware Cloud on AWS which is required to support Read-Write-Many for persistent volumes. In my setup, I just created a quick NFS share running on the Ubuntu VM and once you have the NFS server address and mount point, you can follow the instructions below.

Run the following command and replace NFS path and server with your values to create the persistent volume (PV) YAML file:

cat > image-registry-pv.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: image-registry-pv
spec:
  accessModes:
    - ReadWriteMany
  capacity:
      storage: 100Gi
  nfs:
    path: /mnt/image-registry
    server: 192.168.1.25
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs01
EOF

Run the following command to create the persistent volume claim (PVC) YAML file:

cat > image-registry-pvc.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: image-registry-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeMode: Filesystem
  storageClassName: nfs01
EOF

Create both PV and PVC by running the following two commands:

kubectl apply -f image-registry-pv.yaml
kubectl -n openshift-image-registry apply -f image-registry-pvc.yaml

Verify that the PVC was successfully claimed by running the following command:

kubectl -n openshift-image-registry get pvc

Note: Since OpenShift does not make sure of the vSphere Cloud Native Storage (CNS) plugin, PVC do not show up in the vSphere UI under Container Volumes like it would when using vSphere with Kubernetes or Tanzu Kubernetes Grid (TKG).

By default, the internal registry is not configured and we will need to update it to point to our PVC. To do so, run the following command which will open up the editor:

oc edit configs.imageregistry.operator.openshift.io -o yaml

You will update the managementState and storage fields from:

to:

To check the status of the image registry, you can run the following command:

oc get clusteroperator image-registry

You have now successfully deployed and configured the latest OpenShift 4.5.x running on VMware Cloud on AWS!

For fun, I also attached my OpenShift Cluster to Tanzu Mission Control (TMC). For instructions, you can refer to this guide here.