Changing the default HTTP(s) Reverse Proxy Ports on ESXi 8.0

The process of changing the default ports for the ESXi Reverse Proxy service has always been pretty straight forward, which I had also shared back in 2015 HERE. While most customers stick with the default configuration (80 for HTTP and 443 for HTTPS), we do have some customers that need to change these ports to meet certain organization security and/or compliance requirements.

I recently came across a customer report where the previous method for changing the ESXi Reverse Proxy ports on an 8.0 host no longer worked and the only thing that was shared was that the user could no longer run ESXCLI directly within the ESXi Shell, which I thought was a strange observation.

I deployed the latest ESXi 8.0b as a Nested ESXi VM and I went through the instructions I had outlined in my blog post HERE and changed the HTTPS port from 443 to 4444, which was the setup the user was looking to do and I ran into the exact same issue. At first, I thought maybe we actually no longer support this capability and decided to quickly test by using the remote version of ESXCLI, which allows you to specify a port as part of the connection and it failed with the same error.

I first took a look at the ESXi VMkernel log (/var/log/vmkernel.log) and immediately I saw that there was an issue with the configuration change and that it was being blocked by the VMkernel with the following error message:

rhttpproxy: running in rhttpproxyDom(68): ipAddr = ::, port = 4444: Access denied by vmkernel access control policy

While I thought the ESXi Reverse Proxy service had successfully restarted, in fact, it had failed to bind to the new port and thus never started. This would explain why local ESXCLI command was unable to connect, because the reverse proxy service was actually down.

The error message jogged my memory a bit or at least gave me a hint as I vaguely recall seeing some security policy commands within ESXCLI. Long story short, since this was not a default configuration, the new ESXi Security Domain Enforcement policy had prevented the binding of the new port. The workaround was straight forward after a quick test.

Step 1 - If you already made the change, then you can use localcli (which bypasses hostd) to disable the enforcement policy for reverse proxy service or you can use ESXCLI to do that before making any configuration changes to reverse proxy configuration file

localcli system secpolicy domain set -n rhttpproxyDom -l disabled

Step 2 - Edit the reverse proxy configuration file (/etc/vmware/rhttpproxy/config.xml), if you have not already and then start service by running the following command:

/etc/init.d/rhttpproxy start

Confirm that the reverse proxy service is now running with the new port by checking its status:

/etc/init.d/rhttpproxy status

Step 3 - Finally, we will re-enable the default security policy for reverse proxy service with the enforcing value using the following command:

esxcli system secpolicy domain set -n rhttpproxyDom -l enforcing