In this article, we will begin our PKS installation by deploying Pivotal Ops Manager which provides a management interface (UI/API) for Cloud/Platform Operators to manage the complete lifecycle of both BOSH and PKS from install, patch and upgrade. In addition, you can also deploy new application services using Ops Manager Tiles like adding an Enterprise-class Container Registry like VMware Harbor which can then be configured to work with PKS or any other solution that you may have deployed using Ops Manager.
As part of the the Ops Manager deployment, we will also install the BOSH Tile which is responsible for managing, provisioning, monitoring and self-healing of the VMs that make up a Kubernetes (K8S) Cluster deployment within PKS. BOSH supports a number of IaaS platforms which includes vSphere and through their Cloud Provider Interface (CPI), it will ensure the VMs are always up and in some cases, it may even delete and simply re-deploy VMs if they become unresponsive. This is all done completely automated without any interaction from the Cloud/Platform Operator which means for Developers, they can rest assure whatever SLA or High Availability options that have been defined as part of their application deployment, the underlying platform will ensure those requirements will always be met.
If you missed any of the previous articles, you can find the complete list here:
- Getting started with VMware Pivotal Container Service (PKS) Part 1: Overview
- Getting started with VMware Pivotal Container Service (PKS) Part 2: PKS Client
- Getting started with VMware Pivotal Container Service (PKS) Part 3: NSX-T
- Getting started with VMware Pivotal Container Service (PKS) Part 4: Ops Manager & BOSH
- Getting started with VMware Pivotal Container Service (PKS) Part 5: PKS Control Plane
- Getting started with VMware Pivotal Container Service (PKS) Part 6: Kubernetes Go!
- Getting started with VMware Pivotal Container Service (PKS) Part 7: Harbor
- Getting started with VMware Pivotal Container Service (PKS) Part 8: Monitoring Tool Overview
- Getting started with VMware Pivotal Container Service (PKS) Part 9: Logging
- Getting started with VMware Pivotal Container Service (PKS) Part 10: Infrastructure Monitoring
- Getting started with VMware Pivotal Container Service (PKS) Part 11: Application Monitoring
- vGhetto Automated Pivotal Container Service (PKS) Lab Deployment
Step 1 - If you have not already downloaded Ops Manager, please see Part 1 for the download URL. Deploy the Ops Manager OVA (pcf-vsphere-2.0-build*.ova) to your Management vSphere Cluster, this can be done using either the vSphere Web/H5 Client or even via Automation using OVFTool or PowerCLI. There a several OVF Properties that you will want to fill out such as the admin credentials, DNS Hostname and the desired network settings. Once the deployment has completed, go ahead and power on the VM.
Step 2 - Once Ops Manager is ready, you can connect to it using a web browser by simply providing the DNS hostname and/or IP Address of the system. You should be prompted to select the authentication type and for our setup, we will simply use the Internal Authentication option. For integrating with Active Directory or other directory services, you can do so by connecting Ops Manager to VMware Identity Manager (vIDM) which is an external identity provider. For more details, please see this blog post here.
Next, you will be prompted to create a new admin user which we will use to manage BOSH. Once you have successfully created the user, go ahead and login with the new user account.
Step 3 - Once you are logged into Ops Manager, you will see see that the BOSH Tile has already been imported but is un-configured (denoted by orange coloring) which means BOSH itself has not yet been deployed. Go ahead and click on the tile to begin the configuration.
Step 4 - This first section is for the vCenter Server information on on where the BOSH VM will be deployed to. It is important to note that which ever vCenter Server BOSH is deployed to, it must also have inventory access to the vSphere Cluster that you will use to deploy the K8S workload. This means if you have a separate vCenter Server which manages your Management and Compute Cluster, you will need to deploy BOSH onto the Compute vCenter Server. The next part is the vSphere Datacenter (must already exists) to use as well as the disk provision type.
For vSphere Datastore configuration, you have to specify one for Ephemeral disk placement as well as Persistent disk Datastore placement. These can either be the same or different, it just depends on your environment. To deploy BOSH as well as the PKS Control Plane VMs, it will go ahead and upload a Stemcell VM (basically, a VM Template that PKS uses) and it will clone from that image for both PKS Management VMs as well as base K8S VMs. If you have a separate Management and Compute Cluster which also has non-shared Datastores that you wish to use, you will need to specify them separated by a comma delimited list. This ensures that when deploying BOSH and PKS Control Plane VM, they will be stored on the Management Cluster's Datastore and for K8S VMs, they will be stored on the Compute Cluster's Datastore.
In my environment, I have the following:
- Management Cluster Datastore: himalaya-local-SATA-re4gp4T:storage
- Compute Cluster Datastore: vsanDatastore
For the networking, just leave the default to "Standard vCenter Networking", we will not touch this if you are deploying BOSH and PKS Control Plane VM using either VSS or VDS backed portgroup. The last setting towards the bottom that you would need to tweak are the names of the VM Folders to use to store PKS VMs, PKS Templates & PKS Disks. You can specify any name like you and these directories will automatically be created for you. You should make a note of what you specify for the PKS VMs folder (first option) as that will be needed when configuring the PKS Control Plane.
Step 5 - The next section is for BOSH itself, here you only need to specify an NTP Server as well as check the following three boxes:
- Enable VM Resurrector Plugin
- Enable Post Deploy Scripts
- Recreate all VMs
Step 6 - The next section is for specifying PKS's concept of an Availability Zone, which is defined at a vSphere Cluster level. These AZs will then be used by BOSH to determine where to deploy the PKS Management VMs as well as the K8S VMs. You will need to create two AZs, one for Management and one for Compute. Resource Pools are optional but for customers who have a collapsed Management and Computer Cluster, Resource Pools can be used to guarantee resources to the PKS Management VMs so that K8S VMs will not impact them during resource contention.
In my environment, I have the following configuration:
- Name: AZ-Management
- Cluster: Primp-Cluster
- Name: AZ-Compute
- Cluster: PKS-Cluster
Step 7 - In this section, we will define the networks that will be used to deploy PKS Management VMs (BOSH and PKS Control Plane) as well as the K8S Management Network which is used by the K8S Management POD. For the vSphere Network, you will simply use the name of either your VSS or VDS portgroup as shown in vCenter Server. The Reserved IP Ranges is an interesting field as it is not clear at first what is actually needed. This field is basically a blacklist of all IPs within the selected network that you do NOT want BOSH to use, which should also include the gateway address. It is recommended that you use a dedicated network for PKS Management Network so that you only have to blacklist a few IPs that may be in use such as Ops Manager. If not, this can be an operational overhead when new VMs are added to this network outside of PKS, you would need to ensure that the blacklisted IPs are updated or an IP conflict may occur. The rest of the options are pretty straight forward. Make sure you specify the Management AZ which is the vSphere Cluster that BOSH and PKS Control Plane VM will be deployed to.
Here is my configuration for PKS Management Network:
- Name: pks-mgmt-network
- vSphere Network Name: dv-vlan3251
- CIDR: 172.30.51.0/24
- Reserved IP Ranges: 172.30.51.1-172.30.51.30
- DNS: 172.30.0.100
- Gateway: 172.30.51.1
- Availability Zones: AZ-Management
Step 8 - In this section, we will define the network that will be used to deploy K8S Management Cluster VMs, which will be the Logical Switch that we had created in Part 3 during our NSX-T configuration. The settings should be pretty straight forward and just remember to check the "Service Network" box for this network definition.
Here is my configuration for K8S Management Cluster Network:
- Name: k8s-mgmt-cluster-network
- Service Network: Check box
- vSphere Network Name: K8S-Mgmt-Cluster-LS
- CIDR: 10.10.0.0/24
- Reserved IP Ranges: 10.10.0.1
- DNS: 172.30.0.100
- Gateway: 10.10.0.1
- Availability Zones: AZ-Compute
Step 9 - In this section, we are defining the AZ and networking placement settings for the PKS Management VM (BOSH and PKS Control Plane) which we had defined in earlier.
- Singleton Availability Zone: AZ-Management
- Network: pks-mgmt-network
Step 10 - For the last three sections: Security, Syslog and Resource Config you can leave the defaults for testing purposes.
Step 11 - At this point, we have completed our BOSH configurations and we are now ready to deploy BOSH. Click on the Pivotal "P" icon in the upper left hand corner of the UI to return to the main Ops Manager home page. On the right hand side, you will see a pending operation for BOSH. Go ahead and click on "Apply changes" to begin the deployment.
The installation can take some time, you can get more details by expanding the logs on the right hand side. For my deployment, it took ~18minutes to complete.
If everything was deployed and configured successfully, you should see a success message and asked to return to the main home page. You will also see a new powered on VM in your vCenter Server inventory that starts with vm-[UUID] which is the BOSH VM and IP will be the first usable IP after your blacklisted entries. Ops Manager uses vSphere Custom Attributes to add additional metadata fields to identify the various VMs it can deploy, you tell what type of VM this is by simply looking at the deployment, instance_group or job property. In this case, we can see its been noted as p-bosh.
In the next blog post, we finish up our PKS deployment by installing and configuring the PKS Control Plane VM.
On Step 4 you pick standard vCenter Networking for BOSH while the official Pivotal docs asking to use NSX- Networking -NSX-T ( https://docs.pivotal.io/runtimes/pks/1-3/vsphere-nsxt-om-config.html) . May i know what the difference these configurations make ?