Why am I seeing HTTP communication status 404 error when configuring vSphere with Tanzu & how to fix?

One thing I love about the VMware Community is the constant sharing of knowledge and information on a regular basis. I always enjoy discovering new tricks and tidbits from the community, especially as it helps me refine my own knowledge and understanding of a given technology or solution.

My good buddy Ariel Sanchez cc'ed me on Twitter yesterday referencing a blog post by Paul Wilk about an issue he was observing in his Nested ESXi environment when configuring vSphere with Tanzu.

This is interesting! Wonder if @lamw ir @eric_shanks have ever seen something like it

— Ariel Sanchez Mora @*protected email* (@arielsanchezmor) November 15, 2020

This was in regards to the dreaded 404 message displayed in the vSphere UI:

It took a bit more digging to figure out what port VCSA was actually serving the OVA file, because I would have assumed it was on 443. It turns out, it is being served on 5480 which is also the same port for hosting the Virtual Appliance Management Interface (VAMI), which I suspect is due to the fact that it has a lighthtpd running. The way I figured out 5480 was actually because I had been spending some time with vSphere with Tanzu configuration file which is stored under /etc/vmware/wcp/wcpsvc.yaml and there is a commented out configuration mentioning where the WCP Agent VM which is another word for Supervisor VM:

ovfurl: 'https://this_vc_pnid:5480/wcpagent/photon-ova-%%SIZE%%.ovf'

As you can see from the example, it defaults to 5480. Looking at the URL path, I was able to determine where these files actually lived on the VCSA filesystem and I found there was a symlink from /opt/vmware/share/htdocs/wcpagent pointing to /storage/lifecycle/vmware-wcp/wcpagent which is where the Supervisor VM (Photon) OVAs are stored on the VCSA.

To actually confirm my suspicion, we need to configure our ESXi host to allow for outbound connectivity to 5480. To do that, I had to use one of my older blog articles back in 2011 on how to create a custom ESXi firewall rule, since 5480 was not one of the default ports that is available for configuration.

Create /etc/vmware/firewall/wcp.xml on ESXi host with the following configuration:

<ConfigRoot>
  <service>
    <id>wcp</id>
    <rule id='0006'>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <porttype>src</porttype>
      <port>5480</port>
    </rule>
  </service>
</ConfigRoot>

Then run the following two ESXCLI commands to load our new firewall configuration and enable the new ruleset:

esxcli network firewall refresh
esxcli network firewall ruleset set -e true -r wcp

If we now enable Workload Management on our vSphere with Tanzu cluster, you will see that the "Download remote files" no longer throw a 404 but is progressing as expected!

So now that we know why this happening, the custom ESXi firewall rule is not really a good solution. Since we do not allow for any custom firewall policies in ESXi, customers must create this XML file and then package it up into a custom VIB for any type of automated and scalable solution. It is also not ideal because the optimized deployment workflow should just work out of the box and if we do require ports opened on ESXi, it should be done as part of the service and then disabled when not required.

Lastly, I did find it strange that we would host the OVA files behind something other than 443 which is pretty common when serving HTTP(s) files. The VCSA does have another web server which I thought would have made more sense, which is the main landing page and is served on 443. Since the OVA files is not actually stored in the current htdocs folder but rather symlinked. A quicker and more ideal permanent solution is to just symlink the OVA files to VCSA primary htdocs directory and then update the OVA URL in the wcpsvc.yaml configuration file. The other really nice benefit is that you do not have to make any changes to the ESXi firewall nor mess with custom firewall policies.

Disclaimer: This is not officially supported by VMware, especially as changes to the VCSA filesystem can be reverted the next time it is patched or upgraded.

Step 1 - SSH to VCSA and change into /etc/vmware-vpx/docRoot directory and then run the following command to create symlink:

ln -s /storage/lifecycle/vmware-wcp/wcpagent wcpagent

Step 2 - Edit /etc/vmware/wcp/wcpsvc.yaml and uncomment the kubevm and ovfurl section and then replace the address with either the Hostname or IP Address of the VCSA and remove port 5480

kubevm:
   ovfurl: 'https://192.168.30.200/wcpagent/photon-ova-%%SIZE%%.ovf'

Step 3 - Restart the wcp service for the changes to go into effect:

service-control --restart wcp

So there you have it, the reason and solution to why the HTTP 404 error is showing up when enabling vSphere with Tanzu. I definitely will be sharing this analysis with the Engineering team in case they were not aware and hopefully this will be resolved in a future update and this error will no longer show up and the system will automatically do the right thing.