While a capable system like the Minisforum MS-A2 (see example BOM) is great for running VMware Cloud Foundation (VCF) 9 in a lab, equally capable networking hardware is just as essential.
Depending on your requirements, there are a TON of networking options in the market from low-end enterprise to prosumer hardware. One popular brand of router/switches that I have seen many folks from the VMware community deploy is from MikroTik.
RouterOS, MikroTik's operating system that runs across all their devices, provides Layer 2 and Layer 3 functionality, which is especially useful for a VCF lab environment.
My initial experience with some of MikroTik's consumer router/switches was not great as I did not realize Jumbo Frame support (MTU of 9000) was not a standard feature and while you can go larger than 1500 bytes on some devices, it was no where near the 9K that I was looking for.
Thanks to Daniel Krieger, who made me aware of the MikroTik MTU support page, I was able to finally identify which MikroTik devices can support MTU 9K+ and while setting up the VCF 9 setup for a recent VMUG event, I have been getting more experience with RouterOS, which does not have the most intuitive UX.
Here are a few MikroTik devices that can be used with VCF 9 setup:
- MikroTik HAP AX2 ( 5 x 1GbE)
- MikroTik 5009UG+S+ (8 x 1GbE, 1 x 2.5GbE, 1 x 10GbE SFP+)
- MikroTik CRS304-4XG-IN (5 x 10GbE)
Note: While playing with the CRS304, I have observed an issue during the VSS to VDS migration of the pNIC (2.5GbE is connected from MS-A2) where the interface on the MikroTik basically goes admin down for some reason, which of course breaks the migration and I have not been able to figure out why. I have not seen this problem with the AX2/5009UG and the only difference that I can see is the first two are marketed as Router where as the CRS304 is marketed as a Switch, but for all three setups, I am using the bridge mode. UPDATE (07/22/25) - Thanks to Daniel Krieger who saw my note about interface dropping, it turns out you need to force the speed on the interfaces connected to ESXi host, apparently when the pNIC is re-homed from VSS to VDS, it goes admin down on switch and never comes back up. The workaround is to disable auto-negotiate and force the desired speed of your physical NIC (e.g. 2.5GbE or 10GbE depending on your NIC speeds).
For those new to MikroTik's RouterOS, here is a quick step-by-step on the initial configurations that is required to deploy VCF 9.0, this is not meant to be an exhaustive list of configurations and if you have any tips/tricks, feel free to leave a comment as I am still learning about RouterOS and its functions.
Here are the minimum number of VLANs that would be required to deploy VCF 9.0
| VLAN | Purpose | Network |
|---|---|---|
| 30 | Management | 172.30.30.0/24 |
| 40 | vMotion | 172.30.40.0/24 |
| 50 | vSAN/Storage | 172.30.50.0/24 |
| 60 | ESXi & NSX Edge Tunnel Endpoint (TEP) | 172.30.60.0/24 |
| 70 | NSX Edge T0 Uplink | 172.30.70.0/24 |
Step 1 - Download and install the MikroTik WinBox client, which we will use to connect to our MikroTik device for configuration. While you can use the RouterOS web interface, if you somehow lock yourself out (ME!), the nice thing about the WinBox client is you can connect via the MAC Address in case you lost networking on the MikroTik device, which is quite nice for easy recovery.
Step 2 - Connect an ethernet cable from your laptop/desktop to the MikroTik device and then open the WinBox client. By default, the MikroTik should have a default address of 192.168.88.1 and you can refresh the neighbors list to automatically detect your device. Login using the admin user along with the credentials, which should be printed on the front of physical instructions.
Step 3 - Update the default admin password by going to Quick Set->System->Password along with updating the RouterOS to the latest version by going to Quick Set->System->Check For Updates
Step 4 - Enable VLAN Filtering on Bridge by going to Bridge->Bridge->(double click on bridge)->VLAN->VLAN Filtering
Step 5 - To enable Jumbo Frames, we need to update the MTU on ALL physical ports of your MikroTik, including SFP+ and ensure they are all uniform configuration or else the VLANs you define will default to the lower MTU value. Go to Interfaces->Ethernet->(double click on etherX)->General and update both MTU and L2MTU to the desired value. For my MikroTik device, it supports up 10218 bytes, so I am using value of 9200.
Repeat for all ports until they all have the same MTU/L2MTU configured.
Step 6 - To prevent the MikroTik from down'ing the interface during the VSS to VDS migration, which I was not the only one who had observed this quirk, the workaround is to manually specify the speed of the interface, rather than relying on auto negotiation. You will only need to do this for the interfaces that are connected to your ESXi host and to do so, go to Interfaces->Ethernet->(double click on etherX)->Ethernet and disable Auto Negotiation and manually specify the speed of the physical network interface.
Step 7 - Create our VLANs by going to Interfaces->VLAN->General and provide name, desired MTU, VLAN ID and then select bridge for the Interface.
After adding all desired VLANs, it should look like the following screenshot:
Step 8 - We can now assign the VLANs that we had just created to the specific ports as either tagged or untagged. To do so, go to Bridge->VLANs->New and provide comment, VLAN ID and interfaces to tag or untag. In our setup, we will tag interfaces ether1-ether4 with all of the VLANS we have created and if you want to place your local workstation on the Management VLAN, you can use an untagged VLAn so you do not have to setup VLAN tag on your system for simplicity purposes.
After assigning all of our VLANs, your screen should look like the following screenshot:
Step 9 - Next, we will create our IP addressing scheme and pools (DHCP) for each of our VLANs.
Go to IP->Addresses->New and provide comment, Address (e.g. 172.30.0.1/24) which will be your gateway along with the desired CIDR, Network and Interface which should map to the desired VLAN
After repeating this for all of your defined VLANs, your screen should look like the following screenshot:
Next, we need to create the IP Pools that will be associated with each VLAN. Go to IP->Pool->New and provide Name, Addresses range (e.g. 172.30.0.100-172.30.0.125) and then click OK to Save. You will then need to re-edit the entry and specify the Next Pool with the associated name (not sure why this is required but it does not allow you to specify it during the initial creation).
After completing the IP Pool creation for all VLANs, your screen should look like the following screenshot:
To ensure that everything has been configured correctly, you should be able to ping each of the gateway addresses that you had created from Step 8, this will ensure that all VLANs were properly configured on the MikroTik before deploying your VCF 9 environment.
Step 10 - To enable DHCP for our VLAN, go to IP->DHCP Server->DHCP and create an entry for each VLAN that you wish to enable by selecting the VLAN interface and the IP Pool you had created earlier.
To associate the network definition for each DHCP-enabled VLAN, go to IP->DHCP Server->Networks and create an entry and provide the address, gateway and optionally specify the DNS Server and DNS Domain.
just for reference, you are not within the rfc1918 private ip networks with your choice. you might want to switch to 10.* instead of 172.*
regards, raoul.
Good point!
Hi William, I want to be very kind, polite, but also provide you short feedback. I work mi MKs 25years+, highest possible knowledge from early ros 3.x versions till newest 7.20beta6 rc(as to day). Other skills are irelevant at this moment.
I literally had to read you article carefully 10x to be sure, that I am not halucinating. You work with vmware (I know exactly, the systems, thats the case of using simply vmware.
But please for your own good and to avoid career crash, MKs are super stable enterprise grade routers for a fraction of money of shit likeCisco. And you have so many red flags, I am still confused, what did you want to achieve. The biggest sign is using quickset... really?? Never ever!!!
You were not able to find yourself MTU specs based on chipset? ....
Do you know difference between CRS and router? CCR is not global name for switches like CRS is.
Your approach to vlans, tagging, mng vlan, vlan filtering, untagging is a mess. CRS has dedicated chip to deal with vlans and offers full HW acceleration. You missed entirelly stp/rstp, which should.be off. Also letting data tagged with vlan 1, or.even 0 (crs1xx switches) means that it remains tagged. Specifieng PVID on ports in bridge make the untagging. On switch, you have to create vlan.interface under bridge and tag bridge and this vlan with the same id. Also add mng ip
Also never enable vlan filtering without working mng vlan. Use the safe mode in case something goes.wrong.
And wimbox and ros is very intuitive. It works the same.way on newest ccrs as 15 years old mipsbe boards.
This is just as droplet in see of what is MK capable. Every single person blamingnMKs through the years lacks basic network skills.
Take this as friendly act from me to help you.
I can help you to get the best explanation of everything, which I practise for years on students from 14yo as they choose to start the carrier by my companies. Many of them are now senior netw admins, have their own companies. And everything because my unique methods and individual access to everybody.
There is unacceptable to say, something is not possible. It is a sign, that maybe a little thing was misunderstood
Thanks for the comment Peter, perhaps there's a language barrier in your response but polite and nonsense don't really go together ... If you would like to be helpful, perhaps you should share the exact commands to achieve the task mentioned above which is the 5 VLANs defined for deploying VCF, which honestly isn't very unique in terms of needing handful of VLANs and keeping things simple, having them routable is handy for debugging/etc. You mention Quick Start, but if you've read the article, you should be able to see that I did NOT recommend nor use Quick Start wizard, so again, not sure if you've used the latest RouterOS but all screenshots should demonstrate that I was going through and attempting to setup the minimal things needed for environment.
Again, if you would like to help, feel free to share the RouterOS commands as that might be quickest way to articulate what you think is the ideal configuration
I want to clarify from my point of view, that politness and nonsense were use exactly as both word were targeting 2 separate aspects. I rarely respond to problems solved in forums, because based on context it is usually wasting time.
The fact I took my time to respond to your article is because your minimalistic but also vm skills and knowledge (you know that feeling, that often the most valuable things are visible only for them, who deserve it). As we can assume both of us are profesionals and responsible for extreme complex systems, where even the smallest mistake or maybe design flaw, not double checking others work, could lead in excessive damage, losses, ...
I am from Europe. Maybe you will understand my usage of the nonsense term. Just assume you solve critical issue, every second off costs 100k eur and some of your coworkers(i assume you are in charge ) comes with totally messed work. In this case, the used word is very very soft. Because his ignorance made everything worse. This was as emergency explanation.
Another way is working with you students, I accept only they are capable to think outside the box, have full interest. Also they have to get their skills alone. I assign them a task and divide it to smallet parts, which makes them fully interested, because they achieve goals in small steps. And this motivation remains constant, because getting something done feels good. When I see someone struggle, I dont provide solution, but divide the big gap of missing knowledge to smaller steps and they are on their own. And finally the nonsense - some teenagers are kings of everything, the respect for their mentors is gone long away. I am not the person, that is scared, no matter situation. Many boys after my intensive, not agressive, but fearless conversation, just for 2 minutes are like new persons.
And finally to you, it was nothing insulting, but just friendly warning, that top skilled tech guy can lose respect when publishing this How to.
I apologize, if you felt it like insult.
And look at this all, would you react so fasf, when I didnt have used the nonsense word? (classical psychological ego play 🙂
To the mks and what you want to achieve. There are so many ways to do even the simplest things, that I would like too see block diagram of your concept with short description, of each segments purpose. I can.help.you with that and also explain all details, why I choose this solution, and also disadvantages of other.
I hope, you understood my clarification. Also the long response from me can be indicator of what I meant.
Peter 🙂
P.S. you can run fully working ros on vmware. It is marked as CHR - cloud hosted router.
Just one thing, I forgot afternreading your response. I know every single rc version of ROS. And it is worth to mention, that setting up MK the right way is not about commands, there are many other settings important to understand. Also differences betwes CRS and routers. Bridging, HW offloading, optimizing the system to get the best performance. This is not tplink or Ubiquity gui system, which in reality slows the experts very much.
I am responsible of 24/7 99.9999% uptime of couple of tens thousands router, either in datacenters or at customers. The network is fully redundant with auto reconfiguration and dynamic TE (traffic engineering) capacity allocation based on actual usage.
What is the point of using 172.16.x.x instead of 10.x.x.x or 192.168.. or even 100.64..., when they are used locally and are not routable even with bgp. Especially in this little example, which in fact is pure nonsense
the point is to do the right thing. even if the networks never leave your lab towards the interwebs, you should adhere to best practice and only use rfc1918. just my 2 cents. regards, raoul.
Yes he is within the Class B range of the RFC 1918 specification which allows a 20-bit block from 172.16.0.0 up to 172.31.255.255
my comment applied to an earlier version of the text, when the networks were partly out of rfc1918. you can still see them in the screenshots.
Yes, your point is valid. I was focused on the text and barely inspected screenshots. That caused my confusion, what were you talking about.
In this case, using wrong ip subnets is just a typing error when comparing the configuration provided. It lacks just everything.
In my lab, I have two Aruba 5406r zl2’s one is in my office and the other one is in the server room. The one in the server room has 5 J9538a 8 port SFP+ cards and 1 J9990a 20 copper and 4 SFP+ ports, two 700W PSU’s - J9830A. The one in the office has a single J9990a card and two 700W PSU’s - J9830A. The copper ports on the J9990a are all POE+. Very quiet. I use these in my lab because before I retired after 25 years of service, I managed an enterprise network that has a mix of 6 slot 5460rZL2 and 12 slot 5412rZL2.
Replying regarding @Peter Holkovič messages about MikroTik. I've worked with enterprise grade switches and routers for 30 years before I was introduced to and started to use MikroTik and RouterOS devices.
First, I have to say that I'm mostly a fan of the devices. For the price you get a huge amount of features. As William pointed out, there is an enormous amount of complexity surrounding what platforms support what features. The matrix on this is mind boggling.
However, I heartily DISAGREE with your statements that MikroTik devices are easy and anyone with network experience should understand them. They do NOT conform to any other switch manufacturers approaches to configuration. They are needlessly complex and and require way too many steps to configure basic capabilities.
Then they go and rip things out and totally change how the configurations work in such a way they break compatibility of previous configurations.
From a datacenter perspective, I gave up on MikoTik switches as they simply are NOT capable enough, are too confusing, and shifting configuration syntax in point releases can create havoc. I now use FS and FSOS based switches if I need to stick to a less expensive but still capable platform.
I still use MikroTik in a LOT of places, so please don't take my statements to mean they're all bad. I rely on my 60 GHz point-to-point cubes for campus area networks - they've been rock solid and stable for years. I still use a dozen CRS312s for basic 10gbit/sec connectivity. They have a dizzying array of radio/wireless and wired products - I love them for that!