How to automate the creation multiple routable VLANs on single L2 network using VyOS

My personal homelab has a very simple network topology, everything is connected to a single flat network. This has served me well over the years, but sometimes it can prevent me from deploying more complex scenarios. Most recently while working with NSX-T and Project Pacific, I had a need for additional VLANs which my home router does not support. There are a number of software solutions that can be used including the popular pfSense, which I have used before.

Over the Winter break, a colleague introduced me to VyOS, which is another popular software firewall and router solution. I had not heard of VyOS before but later realized it was derived from Vyatta, which I had heard of, but development of that solution had stopped and VyOS is now the open source version of that software. Having never played with VyoS before, I thought this might be a good learning opopournity and started to dabble with VyOS over the holiday. At a high level, I have VyOS connected to two networks: Outside network as VyOS refers which is your local LAN and Inside network as VyOS refers which is an is an isolated vSphere Portgroup (VSS/VDS) that is not connected to anything and configured to pass all traffic (4095). From here, you can create multiple VLANs in VyOS which can then be untagged using Virtual Guest Tagging (VGT) by placing a Nested ESXi VM on the same isolated portgroup and then creating the respective portgroups within the Nested ESXi VM mapping to the VyOS VLANs you have created.

One of the nice benefits of this solution is that you can create multiple "Isolated" yet routable networks that can still reach your primary LAN network and still have  to access core infrastructure services running like Active Directory, DNS, etc. which was one of my requirements.  After figuring out how VyOS works and applying that to my specific use case, I thought why not build some basic automation to setup this solution as I probably will forget how I setup everything. Initially I was using the VyOS OVA but later found out it was an extremely out of date there was no public version of the latest VyOS release in OVA form. I decided to use their latest rolling release and apply some vSphere API Automation to not only install VyOS but also fully configure based on template containing VyOS commands. I know the latest version of VyOS now includes a REST API but its a bit of a chicken/egg to enable and not very friendly to use compared to the solution I have built.

Preparation Steps:

Step 1 - Create an "Isolated" Portgroup (VSS/VDS) and configure all VLANs (4095) to be passed for Virtual Guest Tagging.

Step 2 - Download the latest VyOS rolling release ISO from here and upload that to your ESXi datastore.

Step 3 - Create a new VM using Debian GNU/Linux 10 (64bit) guest with the following configurations:

1. Connect the first vNIC to your LAN network portgroup (e.g. VM Network)
2. Add a secondary vNIC and connect that to the Isolated network portgroup from Step 1
3. Add the following two VM Advanced Settings: cdrom.showIsoLockWarning = FALSE and msg.autoanswer = TRUE (required for Automation to work as CD-ROM is locked during VyOS installation)
4. Mount the VyOS ISO and power on the VM

Open the VM Console and wait for it to boot up to the login screen as show in the screenshot below. At this point, you do not have to do anything with the VM and proceed to the next section which will automatically handle the installation and configuration.

VyOS Installation Step:

Step 1 - Download both the VyOS PowerCLI Module and the VyOS configuration template (vyos.template)

Step 2 - Edit the vyos.template and only update the VLAN specific definitions (L13-23, L33-36, L39-42, & L46-60). You can define additional or remove some entries depending on your use. By default, the template will create 4 VLANs with the following configurations:

• VLAN 10 - 172.30.10.0/24
• VLAN 20 - 172.30.20.0/24
• VLAN 30 - 172.30.30.0/24
• VLAN 40 - 172.30.40.0/24

Save the file once you have finished making your changes

Step 3 - Import the VyOS PowerCLI Module

Import-Module ./VyOS.psm1

Step 4 - Next, we are going to run the New-VyOSInstallation function which will install VyOS to the VM that you had created earlier. You will need to provide the following arguments to the function which is described below:

• VMName - The name of the VyOS VM
• ManagementPassword - The password to configure for the vyos user

Here is an example of a running the function to install VyOS into a VM called VyOS-Router:

New-VyOSInstallation -VMName VyOS-Router -ManagementPassword VMware1!

If all goes well, you should see a series of commands being sent into the VM using my VM Keystroke automation function which I have blogged about in the past and the VyOS VM should reboot after its completed. Before proceeding to the next step, make sure the VM is at the login screen (no need to actually login, the configuration function will handle that for you).

VyOS Configuration Step:

Step 5 - Next, we are going to run the New-VyOSConfiguration function which will apply the configuration based the vyos.template from Step 2. In addition, you will also need to provide the following arguments to the function which is described below:

• VMName - The name of the VyOS VM
• ManagementPassword - The password to configure for the vyos user
• ConfigFile - The path to VyOS configuration template file
• ManagementAddress - The IP Address of the Outside Interface (eth0)
• ManagementGateway - The Gateway Addrress of the Outside Interface (eth0)
• ManagementDNSDomain - The DNS Domain on the Outside network
• ManagementDNSServer - The DNS Server on the Outside Network
• ManagementJumpHostIP - The IP Address of Windows Jumphost that can be used to RDP into various VLANs

Here is an example of applying the default vyos.template and setting up the VyOS VM to have (Outside) address configured to 192.168.30.155/24 (notice the network CIDR at the end of the address which his required), gateway set to 192.168.30.1 and DNS Server as 192.168.30.2 and DNS Domain as primp-industries.com and for Jumphost IP Address of 192.168.30.199 and setting the password of the vyos user to VMware1!

New-VyOSConfiguration -VMName VyOS-Router -ConfigFile vyos.template -ManagementAddress 192.168.30.155/24 -ManagementGateway 192.168.30.1 -ManagementDNSDomain primp-industries.com -ManagementDNSServer 192.168.30.2 -ManagementJumpHostIP 192.168.30.199 -ManagementPassword VMware1!

Step 5 - To verify that everything was configured correctly, you will want to test connectivity between all the VLANs that were created as well as ensuring it can reach back out to your Outside network. Deploy a single Nested ESXi VM (you can use my Nested ESXi Virtual Appliance) and attach the 2nd vNIC to the Isolated portgroup that you had created earlier.

Add a second vSwitch to the Nested ESXi host and add the respective portgroups and VLANs that you had defined. After that you can create a VM (in my example, I used PhotonOS) to verify each VLAN. In each VM, you should be able to ping the local gateway of that VLAN, ping across the different VLAN to the other VMs and also be able to reach out to the internet (assuming your Outside interface allows for that).

Although I suspect most folks will only build and install VyOS once, I think it would be nice to have a proper community edition of VyOS which natively supports OVF properties which can accelerate the initial configuration. In addition, I do like the fact that I can simply take a configuration file that contains the various VyOS and pass that into my script, perhaps another enhancement the VyOS folks could consider is allowing this same capability as part of an OVF property by simply asking users to "encode" their configuration using something like base64 so its compact and then pass that opaque string into the OVF property. From there, it can be decoded as part of the setup and automatically applied when the system has been initialized. The benefit here is that an end user only needs to provide the network configuration for the inside/outside networks and this encoded blob and the VyOS appliance would be ready to consume!