In ESXi 5.0, the firewall system has been completely revamped to provide the same functionality as the classic ESX Service Console esxcfg-firewall command. To access the firewall configurations, you can use the following esxcli namespace: esxcli network firewall. By default, there are set of predefined services that a user can enable or disable upon startup.

To list the default firewall rules, you can run the following command:  

esxcli network firewall ruleset list

You can also create your own custom firewall rules/services, but unfortunately this has to go outside of the esxcli framework with a custom firewall rule configuration file. An example firewall rule can be viewed under /etc/vmware/firewall/ if you have FDM enabled and you should find a file called fdm.xml which looks a little something like this:
This XML configuration file describes the name of the firewall rule/service and also specifies the various ports, port type, protocol and direction of a given service.

In the following example, I will create a new firewall rule called "virtuallyGhetto" and it will have port 1337 using TCP for both inbound/outbound and port 20120 using UDP for both inbound/oubound. You will need to create a new XML file and specify a name which I have called /etc/vmware/firewall/virtuallyGhetto.xml

<ConfigRoot>
  <service>
    <id>virtuallyGhetto</id>
    <rule id='0000'>
      <direction>inbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>1337</port>
    </rule>
    <rule id='0001'>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>1337</port>
    </rule>
    <rule id='0002'>
      <direction>inbound</direction>
      <protocol>udp</protocol>
      <porttype>dst</porttype>
      <port>20201</port>
    </rule>
    <rule id='0003'>
      <direction>outbound</direction>
      <protocol>udp</protocol>
      <porttype>dst</porttype>
      <port>20201</port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
  </service>
</ConfigRoot>

Next we will need to reload the firewall by performing a "refresh" operation and then list the rules again using the following command:  

esxcli network firewall refresh
esxcli network firewall ruleset list

We can also verify that the individual rulesets for our new firewall rule/service by running the following command and grepping for the rule in question:

esxcli network firewall ruleset rule list | grep virtuallyGhetto

The new ESXi firewall also allows you to specify specific IP Address or IP ranges to access a particular service. In the following example I disable the "allow all" and specify a particular range for the virtuallyGhetto service using the following commands:

esxcli network firewall ruleset set --allowed-all false --ruleset-id=virtuallyGhetto
esxcli network firewall ruleset allowedip add --ip-address=172.30.0.0/24 --ruleset-id=virtuallyGhetto

The new firewall rules/services are also viewable under the host configuration section "Security Profile" using the vSphere Client:

As you can see it is pretty straight forward to add your own firewall rules and this can easily be incorporated into your kickstart builds.

UPDATE1: How to persist custom firewall rules in ESXi 5, take a look at these two articles here and here.

UPDATE2: Duncan Epping just posted an article on creating your own vibs which will persist firewall rules, definitely take a look as another option.

UPDATE3: You can now easily create persistent firewall rules and other files using the new VIB Author Fling, please take a look at this article here for some examples.