I recently saw a tweet from Jason Shiplett who works over on the VMware Validated Design (VVD) team (also my former team before joining VMware Cloud) who shared a new validated design for running Redhat OpenShift 4.3 on VMware Cloud Foundation. Funny enough, a couple of days ago I was just researching into deploying OpenShift running on VMware Cloud on AWS from a customer inquiry.
Timing could not have been better as RedHat just announced their OpenShift 4.5 release a few days ago as and one of the major updates is support for vSphere using their full stack automation also known as te Installer Provisioned Infrastructure (IPI) option. Previous to this, customers who wanted to deploy OpenShift on vSphere had to use the User Provisioned Infrastructure (UPI) method, which the VVD design also uses, which is much lengthier and complex when compared to the native IPI method.
For someone who has never worked with OpenShift before, this was great news and I get to try out this new deployment method on an VMware Cloud on AWS infrastructure 🙂
Step 1 - You will need a Linux system to perform the installation and it should have access to the vCenter Server running in VMware Cloud on AWS (VMC). In my example, I am using an Ubuntu Server 20.04 VM which is also running in the SDDC and has outbound internet connectivity.
Step 2 - Login to VMware Cloud on AWS console and create a new NSX-T network segment that is DHCP enabled. In my example, I named it openshift-network with a 192.168.3.0/24 configuration.
|Group||Name||IP Address Members|
|Compute||SDDC Management Network||10.2.0.0/16|
Step 4 - Navigate to Security->Gateway Firewall and create and publish the following firewall configurations:
|Compute||OpenShift Network to Internet||OpenShift Network||ANY||ANY|
|Compute||OpenShift Network to SDDC Management Network||OpenShift Network||SDDC Management Network||ANY|
|Management||OpenShift Network to vCenter Server||OpenShift Network||vCenter||HTTPS|
Step 5 - Create the following two DNS entries using the following format. In my example, the cluster-name is called vmc and my base DNS domain is vmware.corp
Step 1 - Download the OpenShift Installation files for vSphere which includes the OpenShift Installer (openshift-client-linux.tar.gz), OpenShift CLI Tool (openshift-client-linux.tar.gz) and a copy of the Pull Secret to the Linux VM.
tar -zxvf openshift-client-linux.tar.gz
tar -zxvf openshift-install-linux.tar.gz
Step 3 - Download the trusted root certificates from your vCenter Server which his located at https://vcenter.sddc-[a.b.c.d].vmwarevmc.com/certs/download.zip and transfer that to the Linux VM.
Step 4 - Extract the certificates from the download.zip file and import them into the Linux system. The commands below are for Ubuntu, you may need to replace the commands based on Linux distribution you are using.
cp certs/lin/* /usr/local/share/ca-certificates
Step 5 - Run the OpenShift installer with the following command to start the configuration wizard to generate our configuration file:
./openshift-install create install-config
The wizard is pretty intuitive to use. Make sure when you are providing the hostname for the vCenter Server that you are using the FQDN as the certificate that we had imported earlier must match the hostname here. The rest of the input will be the same for all SDDC with the exception of network, API/Ingress VIP and your DNS base domain and OpenShift Cluster name and the Pull Secret.
Once you have completed the wizard, the configuration is saved to the install-config.yaml file which you can inspect, edit or save for future uses. This file is automatically deleted after performing the installation, so you may want to make a copy of the file for your own reference.
Step 6 - Next, we start the OpenShift deployment by running the following command:
./openshift-install create cluster
This process should take about 30 minutes or so. In my setup, it took 22 minutes. If everything was successfully deployed and configured, you should see the details about logging into your OpenShift Cluster along with the kubeadmin credentials. If you forget to save the credentials, the information is also stored in .openshift_install.log log file.
One thing to be aware of is that the OpenShift Installer will deploy all VMs to the vSphere Cluster level. It is recommended that you move these VMs into the Compute-ResourcePool after the deployment has completed to ensure you do not have an in-balance of resource distribution.
Step 7 - Lets verify that our DNS was properly configured by logging into the OpenShift UI. In my example, the URL to the UI is https://console-openshift-console.apps.vmc.vmware.corp and you will login using the kubeadmin credentials.
oc get nodes
If you wish to use the internal image registry that OpenShift provides, you will need to setup an external NFS volume as the vSAN File Services is currently not available in VMware Cloud on AWS which is required to support Read-Write-Many for persistent volumes. In my setup, I just created a quick NFS share running on the Ubuntu VM and once you have the NFS server address and mount point, you can follow the instructions below.
Run the following command and replace NFS path and server with your values to create the persistent volume (PV) YAML file:
cat > image-registry-pv.yaml <<EOF apiVersion: v1 kind: PersistentVolume metadata: name: image-registry-pv spec: accessModes: - ReadWriteMany capacity: storage: 100Gi nfs: path: /mnt/image-registry server: 192.168.1.25 persistentVolumeReclaimPolicy: Retain storageClassName: nfs01 EOF
Run the following command to create the persistent volume claim (PVC) YAML file:
cat > image-registry-pvc.yaml <<EOF apiVersion: v1 kind: PersistentVolumeClaim metadata: name: image-registry-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 100Gi volumeMode: Filesystem storageClassName: nfs01 EOF
Create both PV and PVC by running the following two commands:
kubectl apply -f image-registry-pv.yaml
kubectl -n openshift-image-registry apply -f image-registry-pvc.yaml
Verify that the PVC was successfully claimed by running the following command:
kubectl -n openshift-image-registry get pvc
Note: Since OpenShift does not make sure of the vSphere Cloud Native Storage (CNS) plugin, PVC do not show up in the vSphere UI under Container Volumes like it would when using vSphere with Kubernetes or Tanzu Kubernetes Grid (TKG).
By default, the internal registry is not configured and we will need to update it to point to our PVC. To do so, run the following command which will open up the editor:
oc edit configs.imageregistry.operator.openshift.io -o yaml
You will update the managementState and storage fields from:
oc get clusteroperator image-registry
You have now successfully deployed and configured the latest OpenShift 4.5.x running on VMware Cloud on AWS!