How to Create Custom Firewall Rules in ESXi 5.0

In ESXi 5.0, the firewall system has been completely revamped to provide the same functionality as the classic ESX Service Console esxcfg-firewall command. To access the firewall configurations, you can use the following esxcli namespace: esxcli network firewall. By default, there are set of predefined services that a user can enable or disable upon startup.

To list the default firewall rules, you can run the following command:  

esxcli network firewall ruleset list

You can also create your own custom firewall rules/services, but unfortunately this has to go outside of the esxcli framework with a custom firewall rule configuration file. An example firewall rule can be viewed under /etc/vmware/firewall/ if you have FDM enabled and you should find a file called fdm.xml which looks a little something like this:
This XML configuration file describes the name of the firewall rule/service and also specifies the various ports, port type, protocol and direction of a given service.

In the following example, I will create a new firewall rule called "virtuallyGhetto" and it will have port 1337 using TCP for both inbound/outbound and port 20120 using UDP for both inbound/oubound. You will need to create a new XML file and specify a name which I have called /etc/vmware/firewall/virtuallyGhetto.xml

<ConfigRoot>
  <service>
    <id>virtuallyGhetto</id>
    <rule id='0000'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>1337</port>
    </rule>
    <rule id='0001'>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>1337</port>
    </rule>
    <rule id='0002'>
      <direction>inbound</direction>
      <protocol>udp</protocol>
      <porttype>dst</porttype>
      <port>20201</port>
    </rule>
    <rule id='0003'>
      <direction>outbound</direction>
      <protocol>udp</protocol>
      <porttype>dst</porttype>
      <port>20201</port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
  </service>
</ConfigRoot>

Next we will need to reload the firewall by performing a "refresh" operation and then list the rules again using the following command:  

esxcli network firewall refresh
esxcli network firewall ruleset list

We can also verify that the individual rulesets for our new firewall rule/service by running the following command and grepping for the rule in question:

esxcli network firewall ruleset rule list | grep virtuallyGhetto

The new ESXi firewall also allows you to specify specific IP Address or IP ranges to access a particular service. In the following example I disable the "allow all" and specify a particular range for the virtuallyGhetto service using the following commands:

esxcli network firewall ruleset set --allowed-all false --ruleset-id=virtuallyGhetto
esxcli network firewall ruleset allowedip add --ip-address=172.30.0.0/24 --ruleset-id=virtuallyGhetto

The new firewall rules/services are also viewable under the host configuration section "Security Profile" using the vSphere Client:

As you can see it is pretty straight forward to add your own firewall rules and this can easily be incorporated into your kickstart builds.

UPDATE1: How to persist custom firewall rules in ESXi 5, take a look at these two articles here and here.

UPDATE2: Duncan Epping just posted an article on creating your own vibs which will persist firewall rules, definitely take a look as another option.

UPDATE3: You can now easily create persistent firewall rules and other files using the new VIB Author Fling, please take a look at this article here for some examples.