If you have been following the series thus far, we have covered installing Photon Controller in Part 1 and then we learned how to create our first virtual machine using Photon Controller in Part 2. Next up, we will demonstrate how easy it is to stand up the three different Cluster Orchestration solutions that are supported on top of Photon Controller, starting with Kubernetes. Once the Cluster Orchestration solution has been setup, you can then deploy your application like you normally would through the Cluster Orchestration and behind the scenes, Photon Controller will automatically provision the necessary infrastructure to run your given application without having to know anything about the underlying resources.
If you recall from our last article, there are several default VM Flavors that are included in the Photon Controller installation. The ones that are named cluster-* are VM Flavors used for deploying the Cluster Orchestration virtual machines that have been configured to support high scale and throughput (up to 4vCPU and 8GB of memory). If you are testing this in a lab environment where you might be constrained on memory resources for your ESXi host (16GB of memory), then you actually have a few options. The first option is to create a new VM Flavor with a smaller configuration (e.g. 1vCPU/2GB memory) and then override the default VM Flavor when deploying the Cluster Orchestration. The second option which I learned from talking to Kris Thieler was that you can actually re-define the default cluster-* VM Flavors to fit your environment needs which he has documented here. To simplify our deployment, we will actually use Option 1 on just creating a new VM Flavor that we will use to override the default VM Flavor. If you have more than 16GB of memory, then you can skip Step 2.
Deploying Kubernetes Cluster
Step 1 - Download the Kubernetes VMDK from here and the Kubectl binary from here.
Step 2 - Run the following command to create our new VM Flavor override which we will call cluster-tiny-vm that is configured with 1vCPU/1GB of memory:
./photon -n flavor create --name cluster-tiny-vm --kind "vm" --cost "vm 1 COUNT,vm.flavor.cluster-other-vm 1 COUNT,vm.cpu 1 COUNT,vm.memory 1 GB,vm.cost 1 COUNT"
Step 3 - We will now upload our Kubernetes image and make note of the ID generated after the upload by running the following command:
./photon -n image create photon-kubernetes-vm-disk1.vmdk -n photon-kubernetes-vm.vmdk -i EAGER
Step 4 - Next, we also need the ID of our Photon Controller Instance deployment as it will be required in the next step by running the following command:
./photon deployment list
Step 5 - We will now enable the Kubernetes Cluster Orchestration on our Photon Controller instance by running the following command and specifying the ID of your deployment as well as the ID of the Kubernetes image from the previous two steps:
./photon -n deployment enable-cluster-type 7fd9a13d-e69e-4165-9b34-d436f4c67ea1 -k KUBERNETES -i 4332af67-2ff0-49f7-ba44-dd4140908e32
Step 6 - We can also see what Cluster Orchestration solutions have been enabled for our Photon Controller by running the following command and specifying our deployment ID:
./photon deployment show 7fd9a13d-e69e-4165-9b34-d436f4c67ea1
As you can see from the screenshot above, there is a Cluster Configuration section which provides a list of Cluster Orchestration solutions that have been enabled as well as their respective image.
Step 7 - We are now ready to spin up our Kubernetes (K8) Cluster by simply running the following command and substituting the network information from your environment. We are also going to only deploying a single K8 Slave (if you have additional resources you can spin up more or you can always re-size the cluster after it has been deployed) and lastly, we will override the default VM Flavor used by specifying -v option and providing the name of our VM Flavor called cluster-tiny-vm. You can just hit enter when prompted for the two etcd IP Addresses, the assumption is that you have DHCP running and those will automatically obtain an address.
./photon cluster create -n k8-cluster -k KUBERNETES --dns 192.168.1.1 --gateway 192.168.1.1 --netmask 255.255.255.0 --master-ip 192.168.1.55 --container-network 10.2.0.0/16 --etcd1 192.168.1.56 -s 1 -v cluster-tiny-vm
Step 8 - The process can take a few minutes and you should see a message like the one shown above which prompts you to run the cluster show command to get more details about the state of the cluster.
./photon cluster show 9b159e92-9495-49a4-af58-53ad4764f616
Exploring Kubernetes
At this point, you have now successfully deployed a fully functional K8 Cluster using Photon Controller with just a single command. We can now take explore our K8 setup a bit by using the kubectl CLI that you had downloaded earlier. For more information on how to interact with K8 Cluster using kubectl command, be sure to check out the official K8 documentation here.
To view the nodes within the K8 Cluster, you can run the following command and specifying the IP Address of the master VM provided in the previous step:
./kubectl -s 192.168.1.55:8080 get nodes
Lets now do something useful with our K8 Cluster and deploy a simple Tomcat application. We first need to download the following two configuration files that will define our application:
We then need to edit the photon-Controller-Tomcat-rc.yml file and delete the last two lines as it contains an incorrect syntax:
labels:
name: "tomcat-server"
To deploy our application, we will run the following two commands which will setup our replication controller as well as the service for our Tomcat application:
./kubectl -s 192.168.1.55:8080 create -f photon-Controller-Tomcat-rc.yml
./kubectl -s 192.168.1.55:8080 create -f photon-Controller-Tomcat-service.yml
We can then check the status of our application deloyment by running the following command:
./kubectl -s 192.168.1.55:8080 get pods
You should see a tomcat-server-* entry and the status should say "Image: tomcat is not ready on the node". You can give it a few seconds and then re-run the command until it is showing "Running" as the status which means our application has been successfully deployed by the K8 Cluster.
We can now open a browser to the IP Address of our K8 Master VM's IP, which in my environment was 192.168.1.55 and specify port 30001 which was defined in the configuration file of Tomcat application and we should see that we now have Tomcat running.
We can also easily scale up the number of replication servers for our Tomcat application by running the following command:
./kubectl -s 192.168.1.55:8080 scale --replicas=2 rc tomcat-server
You can easily scale the application back down by re-running the command and specifying a value of one. Lastly, if we want to delete our application, we can run the following two commands:
./kubectl -s 192.168.1.55:8080 delete service tomcat
./kubectl -s 192.168.1.55:8080 delete rc tomcat-server
Once we are done using using our K8 Cluster, we can tear it down by specifying the ID of the K8 Cluster found in Step 8 by running the following command which will now delete the VMs that Photon Controller had deployed:
./photon -n cluster delete 9b159e92-9495-49a4-af58-53ad4764f616
Hopefully this gave you a quick taste on how easy it is to setup a fully functional K8 Cluster using Photon Controller. In the next article, we will take a look at deploying a Mesos Cluster using Photon Controller, so stay tuned!
- Test driving VMware Photon Controller Part 1: Installation
- Test driving VMware Photon Controller Part 2: Deploying first VM
- Test driving VMware Photon Controller Part 3a: Deploying Kubernetes
- Test driving VMware Photon Controller Part 3b: Deploying Mesos
- Test driving VMware Photon Controller Part 3c: Deploying Docker Swarm
Hey Will, just FYI, I also had to remove the string "Kubernetes:" from the first line of photon-Controller-Tomcat-service.yml.
Strange, I didn't need to do anything extra on the service configuration
Everything works great following your documentation! though my lab set up is different from yours (William).
I have one Intel NUC running 6.0. On it I have 3 nested ESXi VMs (6.0 U2).
I also have a DNS VM , a Gateway VM, an Openfiler VM and the deployment VM (installer-vm.ova) running on the bare metal
Nested ESXi VM1=>management host running photon controller
Nested ESXi VM2=>Cloud host1
Nested ESXi VM3=>Cloud host2
While 'Master' and 'Slave' are running on Cloud host1, 'etcd' is running on Cloud host2
It's a steep learning curve if you're like me being new to Kubernetes. So trying to figure out all how the whole system works 🙂
thanks a ton!
Hi,
I tried all steps but unable to create kubernetes cluster or any VM on ESX 6.0 host using photon controller. Any help is really appreciated.
I have uploaded kebenetes image as below and enabled kebernetes cluster on photon controller. I tried EAGER as well.
esxcloud@esxcloud-management-vm [ ~ ]$ photon image list
Using target 'http://100.99.49.162:28080'
ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress
4b46441d-af09-4f18-b3a0-053d8547e637 photon-1.0TP2.ova READY 16777216098 ON_DEMAND 17% 100%
cb4565cf-3790-4722-b376-451d4288067c photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 17% 100%
edfd2c6f-c7e5-4099-971c-3a42a0ed149f photon-kubernetes-vm.vmdk READY 85899345920 ON_DEMAND 17% 100%
Below is the error on photon controller agent log on ESX server:
INFO [2016-10-24 08:26:33,521] [1625936:1625904:Thread-19] [image_manager.py:_move_image:292] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: _move_image: /vmfs/volumes/vvol:6090a058e019896e-6080d532b104a018/
ERROR [2016-10-24 08:26:33,522] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: Move image 027e21b7-df14-4598-93df-750031a9e179 to /vmfs/volumes/vvol:60
ERROR [2016-10-24 08:26:33,522] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: Traceback (most recent call last):
ERROR [2016-10-24 08:26:33,523] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI
ERROR [2016-10-24 08:26:33,523] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI
ERROR [2016-10-24 08:26:33,523] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI
ERROR [2016-10-24 08:26:33,524] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI
ERROR [2016-10-24 08:26:33,524] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI
ERROR [2016-10-24 08:26:33,524] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: UnsupportedFileSystem: FS type 99
I am not able to deploy any VM or cluster on the photon controller. If this is not the right forum to submit the issue, please let me know the correct forum to raise same.
Below is the photon controller log snippet:
INFO [2016-10-24 08:31:21,417] com.vmware.photon.controller.common.clients.HostClient: finished create_vm target 100.99.49.77, reservation b74c7164-297b-4eac-b007-580fd84fcd2d
INFO [2016-10-24 08:31:21,417] com.vmware.photon.controller.common.clients.HostClient: Checking CreateVmResponse(result:SYSTEM_ERROR, error:FS type 99)
INFO [2016-10-24 08:31:21,426] com.vmware.photon.controller.common.metrics.RpcMetricInterceptor: Caught exception during createVm: {}
! com.vmware.photon.controller.common.clients.exceptions.SystemErrorException: FS type 99
! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.checkCreateVmResponse(HostClient.java:1640)
! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.access$600(HostClient.java:1415)
! at com.vmware.photon.controller.common.clients.HostClient.createVm(HostClient.java:507)
! at com.vmware.photon.controller.common.metrics.RpcMetricInterceptor.invoke(RpcMetricInterceptor.java:51)
! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.createVm(VmCreateStepCmd.java:91)
! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.execute(VmCreateStepCmd.java:75)
! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58)
! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:115)
! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at java.util.concurrent.FutureTask.run(FutureTask.java:266)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
! at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-24 08:31:21,426] com.vmware.photon.controller.common.metrics.RpcMetricInterceptor: Finished call to createVm
ERROR [2016-10-24 08:31:21,426] com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd: failed creating VM 27e25bf8-f805-4f6a-8f94-99257e33c7de
! com.vmware.photon.controller.common.clients.exceptions.SystemErrorException: FS type 99
! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.checkCreateVmResponse(HostClient.java:1640)
! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.access$600(HostClient.java:1415)
! at com.vmware.photon.controller.common.clients.HostClient.createVm(HostClient.java:507)
! at com.vmware.photon.controller.common.metrics.RpcMetricInterceptor.invoke(RpcMetricInterceptor.java:51)
! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.createVm(VmCreateStepCmd.java:91)
! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.execute(VmCreateStepCmd.java:75)
! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58)
! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:115)
! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at java.util.concurrent.FutureTask.run(FutureTask.java:266)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
! at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-24 08:31:21,427] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={PATCH}, OperationId={12755}, Uri={http://127.0.0.1:19000/photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,428] com.vmware.photon.controller.cloudstore.xenon.entity.VmService: [Req: d55f3bd6-c2d3-46bd-bd44-c3748bfeaf48] [/photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de] Patching service /photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de
INFO [2016-10-24 08:31:21,433] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={PATCH}, OperationId={12755}, Uri={http://127.0.0.1:19000/photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,433] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={POST}, OperationId={12759}, Uri={http://127.0.0.1:19000/core/node-selectors/default/forwarding?path=/core/local-query-tasks&target=ALL}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,445] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={POST}, OperationId={12759}, Uri={http://127.0.0.1:19000/core/node-selectors/default/forwarding?path=/core/local-query-tasks&target=ALL}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,446] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={GET}, OperationId={12762}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,446] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={GET}, OperationId={12762}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,448] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={PATCH}, OperationId={12764}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
INFO [2016-10-24 08:31:21,455] com.vmware.photon.controller.cloudstore.xenon.entity.DiskService: [Req: d55f3bd6-c2d3-46bd-bd44-c3748bfeaf48] [/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50] Patching DiskService /photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50
INFO [2016-10-24 08:31:21,463] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={PATCH}, OperationId={12764}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
ERROR [2016-10-24 08:31:21,463] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Command execution failed with exception
! com.vmware.photon.controller.common.clients.exceptions.SystemErrorException: FS type 99
! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.checkCreateVmResponse(HostClient.java:1640)
I used below command
esxcloud@esxcloud-management-vm [ ~ ]$ photon cluster create -n k8-cluster -k KUBERNETES --dns 100.99.8.39 --gateway 100.99.48.1 --netmask 255.255.248.0 --master-ip 100.99.49.165 --container-network 172.0.0.1/16 --etcd1 100.99.49.166 -s 1 -v cluster-tiny-vm
Using target 'http://100.99.49.162:28080'
etcd server 2 static IP address (leave blank for none):
Creating cluster: k8-cluster (KUBERNETES)
VM flavor: cluster-tiny-vm
Slave count: 1
Are you sure [y/n]? y
2016/10/26 09:29:22 photon: Task 'a1ebbce8-3c9f-4392-9d4d-703d783dd0fd' is in error state: {@step=={"sequence"=>"1","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed with error [Task "CREATE_VM": step "RESERVE_RESOURCE" failed with error code "InternalError", message "Please contact the system administrator about request #c05056b2-6878-4228-a2c3-2482e3df5c92"]. /photon/clustermanager/vm-provision-tasks/1b1cccb8-1ab0-4ed6-9db3-97a6880f5768]', data: 'map[]' }],"warnings"=>[],"operation"=>"CREATE_KUBERNETES_CLUSTER_SETUP_ETCD","startedTime"=>"1477474156774","queuedTime"=>"1477474156747","endTime"=>"1477474161777","options"=>map[]}}
API Errors: [photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed with error [Task "CREATE_VM": step "RESERVE_RESOURCE" failed with error code "InternalError", message "Please contact the system administrator about request #c05056b2-6878-4228-a2c3-2482e3df5c92"]. /photon/clustermanager/vm-provision-tasks/1b1cccb8-1ab0-4ed6-9db3-97a6880f5768]', data: 'map[]' }]
esxcloud@esxcloud-management-vm [ ~ ]$ docker version
Client:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 19:36:04 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.0
API version: 1.23
Go version: go1.5.4
Git commit: 4dc5990
Built: Wed Apr 13 19:36:04 2016
OS/Arch: linux/amd64
Links to:
photon-Controller-Tomcat-rc.yml
photon-Controller-Tomcat-service.yml
are dead 🙂