Automation – Page 182

Schedule automated backups of VCSA 6.0 vPostgres embedded database to Amazon S3

07.09.2015 by William Lam // 6 Comments

A couple of weeks back, I had received a question around backing up and restoring the Embedded vPostgres Database found within the new vCenter Server Appliance (VCSA) 6.0. At the time, the only thing I had seen was KB 2110294 and vSphere 6.0 Documentation here which recommends that a full VM backup be taken for either the vCenter Server for Windows as well as the VCSA to be able to properly protect your vCenter Server.

It was just recently that I came across VMware KB 2091961 which provides some details on just backing up the individual vPostgres DB. Having said that, just having a database backup is not sufficient to perform a proper restore in the case of completely losing your vCenter Server. There are other sources of data within the vCenter Server as well as the Platform Services Controller that are required and restoring a database would only work if you still had access to the original system. This is why a full VM backup is still the recommended approach.

For those who want to be able to just restore the database, the process listed in the KB is currently a manual step which uses a Python script that is provided in the KB. I thought it would be useful to demonstrate how you could schedule continuous backups during off peak hours using a simple cronjob and more interesting to me, is the how and where of the overall process? One option would be to mount a backup NFS share directly onto the VCSA and place all backups on that volume. Another option could have the backups directly uploaded to a Storage Cloud Provider like an Amazon S3 for example. I decided to take a look into the latter option.

In searching online, I found that Amazon offers a nice CLI called AWS CLI which provides S3 functionality like the 'cp' command and I was able to install it on the VCSA without any issues. You can find the instructions for installing the AWS CLI here and I would also recommend that you create a dedicate user assigned to the S3 bucket for storing the backups and then following the steps here to configure access to the AWS CLI. When asked about the Amazon Region as part of the configuration, I found this page to be helpful in listing the region names.

Disclaimer: Installing 3rd Party tools and products on the (VCSA) is not officially supported, you may be asked by GSS to remove them during troubleshooting.

If everything is installed correct, you should be able to run the following command to ensure you can reach the S3 bucket:

aws s3 ls s3:\\[NAME-OF-YOUR-S3-BUCKET]

To tie everything together, I created a simple shell script called backup_vcsa_vpostgres_db.sh which contains a couple of variables that you will need to edit:

VPOSTGRES_BACKUP_SCRIPT - The path to the Python vPostgres backup script
AWS_CLI - The full path to the AWS CLI binary
AWS_S3_BUCKET - The name of the S3 bucket using syntax s3:\\NAME-OF-YOUR-S3-BUCKET

Before creating the cronjob, I would recommend that you manually run the script to ensure everything works as expected and you are able to upload to your S3 bucket. Here is an example execution of the script which is backing up to my S3 bucket which I called "vcsa-backup".

You can quickly verify that the backup has been uploaded to the S3 bucket by running the "ls" command as shown earlier or you can login to the Amazon S3 console and you should be able to see the backup files as shown in the screenshot below.

To schedule the script to automatically run during a certain period, you can create a cronjob by running the following command:

crontab -e

For more information about setting up a cronjob, you can take a look here or Google your favorite resource. If you plan on storing backups with a Cloud Storage Provider and do not have direct internet access like most customers do, you can configure an HTTP(S) proxy by editing /etc/sysconfig/proxy If you prefer not to install AWS CLI, you can also use this simple bash script which uses an HTTP POST to upload to Amazon S3.

Quick Tip - Determining the vCenter Server OS platform (Windows or VCSA) using vSphere API

06.25.2015 by William Lam // Leave a Comment

The vSphere API is an extensively rich interface for being able to extract all sorts of useful information about your vSphere infrastructure. One useful trick that may come in handy for those requiring to perform operations directly against the vCenter Server guestOS itself is to figure out whether you are connecting to a Windows vCenter Server or the vCenter Server Appliance (VCSA)? Lets say you wish to automate the deployment of the recently released VSAN 6.0 Health Check Plugin and the process to install the plugin will differ between Windows vCenter Server and the VCSA, so it would be ideal if you can easily distinguish between the two

A simplistic solution would be to quickly test for something that would exist in either Windows or Linux, but what if you wanted to perform these operations using the vSphere API and the Guest Operations API to execute the commands within the guests? Well, luckily the vSphere API actually provides this information when connecting to a vCenter Server API endpoint and you can tell if you are connecting to a Windows vCenter Server or the VCSA.

To determine the guestOS type for the vCenter Server you are connecting to, there is a property called osType which you can query when you first connect. Below is a quick PowerCLI snippet for accessing this property, you can also use a variety of other vSphere SDKs to extract this property.

$server = Connect-VIServer -Server reflex.primp-industries.com

$server.ExtensionData.Content.About

Disconnect-VIServer -Server $server -Confirm:$false

The osType property for the VCSA is linux-x64

The osType property for vCenter Server for Windows is win32-x64

Quickly getting started with VMware AppCatalyst & AppCatalyst Vagrant Plugin

06.24.2015 by William Lam // 5 Comments

At Dockercon this week, the Cloud-Native Apps team at VMware introduced two new Tech Previews: VMware AppCatalyst and Project Bonneville. In addition, I also found out today that Fabio Rapposelli who works over in the CNA team has also released a Vagrant Plugin for VMware Catalyst as well. Having spent a couple of days playing with AppCatalyst before it was released, I thought this would be a good opportunity to show how to quickly get started with AppCatalyst but also to try out the new Vagrant Plugin that had just been released.

Before jumping in, What exactly is VMware AppCatalyst? It is a slimmed down desktop Hypervisor based on VMware Fusion that has been optimized for Developers who want to be able to quickly spin up Virtual Machines to run Docker Containers. Included with AppCatalyst is also VMware Photon, an open-source lightweight Linux container host that is used to quickly instantiate new VMs that are ready for running and building Docker Containers. Instead of a traditional GUI like VMware Fusion, AppCatalyst's is consumed completely via a REST API or CLI, which is also ideal for integrating with other 3rd Party Automation tools like Vagrant. Best of all, both VMware AppCatalyst & AppCatalyst Vagrant Plugin is completely Free!

Requirements:

Mac OS X 10.9.4 or 10.10
VMware Fusion must not be running

Quick INFO:

The AppCatalyst configuration file is located in ~/.appcatalyst.conf (if you wish to change system defaults)
The CLI to AppCatalyst is located at /opt/vmware/appcatalyst/bin/appcatalyst
The daemon for the AppCatalyst REST API is located at /opt/vmware/appcatalyst/bin/appcatalyst-daemon
SSH keys to the Photon VM is located in /opt/vmware/appcatalyst/etc/appcatalyst_insecure_ssh_key
Additional AppCatalyst Documentation can be found here

Exploring Appcatalyst CLI:

Step 1 - Download and install VMware AppCatalyst from here

Step 2 - Once AppCatalyst has been installed, you can explore the CLI and view the list of operations by running the following command:

/opt/vmware/appcatalyst/bin/appcatalyst

Step 3 - To create our first VM using AppCatalyst, run the following command:

/opt/vmware/appcatalyst/bin/appcatalyst vm create vGhetto1

Note: You will see that all VMs created by AppCatalyst will be stored in /Users/[USER]/Documents/AppCatalyst and you can change this by editing DEFAULT_VM_PATH in AppCatalyst configuration file

Step 4 - To power on our VM, run the following command:

/opt/vmware/appcatalyst/bin/appcatalyst vmpower on vGhetto1

Note: If you run into problems powering on your VM, there is a good chance that you may still have a VM that is running in VMware Fusion (also check docker-machine in case you are using that to power off any VMs provisioned using that tool)

Step 5 - Once the VM is powered on, we will probably want to retrieve the IP Address which could take a few seconds to be displayed by running the following command:

/opt/vmware/appcatalyst/bin/appcatalyst guest getip vGhetto1

Step 6 - Once we have the IP Address of our VM, we can then SSH to it using the SSH keys included with AppCatalyst by running the following command:

ssh -i /opt/vmware/appcatalyst/etc/appcatalyst_insecure_ssh_key [email protected]

At this point, you have successfully deployed a VM based on VMware Photon image using the AppCatalyst's CLI. You can now login and start running and building Docker Containers as Docker is already pre-installed on Photon. Next we will take look at using the AppCatalyst API.

Exploring Appcatalyst API:

UPDATE (01/22/16) - For the complete cheatsheet of using AppCatalyst API using cURL, check out this article here for more examples.

Step 1 - To use the AppCatalyst REST API, you will need to run the AppCatalyst Daemon by running the following command:

/opt/vmware/appcatalyst/bin/appcatalyst-daemon

Note: You can also run the AppCatalyst daemon in the background by adding '&' to the command

Step 2 - You can easily view the AppCatalyst API by opening a browser to http://localhost:8080 (assuming you did not change the port) which is built using Swagger. You can explore and even test the API using this interface if you have not worked with a Swagger interface before.

Step 3 - Lets quickly test the GET /vms operation which will list the VMs that are being managed by AppCatalyst. We will be using cURL by running the following command:

curl http://localhost:8080/api/vms

As long as the AppCatalyst daemon is running, you will be able to interact with the REST API using a variety of methods including the examples I have shown above.

Note: A known issue is that VMs powered on using the AppCatalyst API can not be managed by the CLI until they have been powered down using the API. Hopefully this issue will be resolved in a future update.

Exploring Appcatalyst Vagrant Plugin:

Step 1 - Ensure you have Vagrant already installed on your system, if not you can download it here.

Step 2 - Install the Vagrant Plugin by running the following command:

vagrant plugin install vagrant-vmware-appcatalyst

Step 3 - You can either run "vagrant init" or manually create a file name Vagrant file that contains the following:

# Set our default provider for this Vagrantfile to 'vmware_appcatalyst'
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_appcatalyst'

nodes = [
  { hostname: 'gantry-test-1', box: 'vmware/photon' },
  { hostname: 'gantry-test-2', box: 'vmware/photon' }
]

Vagrant.configure('2') do |config|

  # Configure our boxes with 1 CPU and 384MB of RAM
  config.vm.provider 'vmware_appcatalyst' do |v|
    v.vmx['numvcpus'] = '1'
    v.vmx['memsize'] = '384'
  end

  # Go through nodes and configure each of them.j
  nodes.each do |node|
    config.vm.define node[:hostname] do |node_config|
      node_config.vm.box = node[:box]
      node_config.vm.hostname = node[:hostname]
    end
  end
end

The above Vagrant file will create two VMs called gantry-test-{1,2} using the VMware Photon Box hosted on Atlas by HashiCorp.

Step 4 (Optional) - If you decide to use the example above with VMware Photon Box, you will need to install an additional Vagrant Plugin for managing Photon Guest by running the following command:

vagrant plugin install vagrant-guests-photon

Step 5 - Ensure that the AppCatalyst daemon is running before proceeding with the next step as the Vagrant Plugin uses the AppCatalyst REST API.

Step 6 - To start the deployment, run the following command:

vagrant up --provider=vmware_appcatalyst

Step 7 - To login to one of the VMs that have been created by Vagrant, simply run the following command and specify the name of the VM to SSH to:

vagrant ssh gantry-test-1

Step 8 - Finally, we can also confirm that the VMs created by the AppCatalyst Vagrant Plugin is also visible using just the AppCatalyst REST API and we can perform another "GET" operation to verify the two VMs that we had just created.

Hopefully this quick primer has been useful in getting you introduced and started with VMware AppCatalyst as well as the new AppCatalyst Vagrant Plugin. If you have any feedback or questions, feel free to leave a comment in the VMware AppCatalyst Forum and for feedback/questions on AppCatalyst Vagrant Plugin, you can file issues here.