Home Router Experiments

What does Gbit mean anyway?

Gigabit Network Interface Cards (NICs) are theoretically capable of pumping 1 Gbit/s in both directions simultaneously, reaching an actual total of 2 Gbit/s combined. That's a lot of things to perform for a GHz-class CPU. As a rule of thumb, you cannot usually reach such speeds with embedded sub-GHz CPUs without specific hardware acceleration.

For more information on this topic and a very worthwhile discussion, read on here:

https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and-need-a-router-read-this-first/90305

Software first

There aren't so many home router software bundles, so with a bit of patience you could try them all and see what they're worth:

• pfSense, OPNsense are based on FreeBSD
• IPFire is open-source and free, Linux-based
• Untangle, ClearOS, Sophos are commercial, Linux-based
• OpenWRT: free and Linux-based
• VyOS: free/commercial and Linux-based

Worth noting: a whole lot of experiments based on FreeBSD, OpenBSD, and Linux.

Check Wikipedia for more ideas: https://en.wikipedia.org/wiki/List_of_router_and_firewall_distributions

You can also roll your own with a straight OS install:

• FreeBSD with pf or ipfw
• OpenBSD with pf
• Debian or Arch with Shorewall or nftables

In no particular order and without any scientific or systematic method, here are my findings:

Ubiquiti EdgeRouter-X

https://store.ui.com/collections/operator-edgemax-routers/products/edgerouter-x

This cheap ($50) box comes pre-installed with Ubiquiti's EdgeOS, a Debian-based OS that still retains some compatibility as you can install some Debian packages through CLI. I wasn't much impressed with EdgeOS: the GUI is nice but hard to browse through. Adding a static DHCP lease forces you to click-click-click far too much, especially when you want to enter a hundred MAC and IP addresses. I ended up doing most things through ssh. Bonus points: you can open the box and directly access the JTAG for serial console fun, or re-flashing the firmware.

OpenWRT can be flashed on this tiny box. The results are excellent: all the comfort of OpenWRT's Luci GUI and the power of a true Linux CLI. Only trouble is the performance: I managed to get a window of about 900 Mbit/s in total, so about 450-450 up-down which is still far from maxing my connection. If your Internet bandwidth is below that you shouldn't hesitate a single moment and go for it, you won't find any better.

https://forum.openwrt.org/t/ubiquiti-edgerouter-x-loading-openwrt-and-performance-numbers/27470

I realize I could be running my home network on this tiny box and wouldn't notice any difference since I pretty much never use 1000+500 simultaneously, but the real fun is to manage to squeeze out every single bit/s from my line, because I can.

Mikrotik hEX S

https://mikrotik.com/product/hex_s

This box has the very same hardware as Ubiquiti's EdgeRouter-X, just sold with a different proprietary Linux-based firewall. Bit pricier too: $70 at the time of writing.

The firewall GUI looks outdated, like a web site from the 1990s that hasn't aged well. The left-side menu is just a long list of confusing acronyms with not even an attempt at explaining what they do. It takes ages reading the copious online documentation to just configure a basic NAT and a few rules. At that level I am wondering if this GUI is meant to set up a firewall or just torture humans.

I spent a lot of time on Reddit looking for clues and recommendations. Seems you can only love or hate that GUI, so count me with the haters on that one. I sent the hardware back to Amazon after a couple of weeks.

Good news: you can re-flash that box with OpenWRT! Bad news: the box is only spec'd for about 900 Mbit/s total with a few firewall rules and a NAT. Nice but won't saturate my line, and I cannot condone torture through GUI.

https://mikrotik.com/product/hex_s#fndtn-testresults

PCengines APU2C4

https://pcengines.ch/apu2c4.htm

That box is touted everywhere as the best one in its price range. For 100-150 euros you get 3 Intel NICs attached to an AMD quad-core processor. The part I like most is the open-source BIOS (coreboot) which keeps improving all the time. My poor box has been reflashed a million times to try all possible software options. In details:

• pfSense worked, but not great. Best speed I could reach was about 600 Mbit/s download. That was a while ago though. TekLager published several recommendations to squeeze better performance out of pfSense, which I haven't tried.

https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/

• OPNsense did just about the same as pfSense. The GUI is a lot more friendly and there are more functionalities offered, but the engine running underneath reached exactly the same disappointing throughput. Again: haven't tried in a while, possibly performs better today.
• FreeBSD+pf did a bit better, reaching 700 Mbit/s download. That was FreeBSD 11 and I had to tweak a lot of stuff to make it work. Most certainly better on the latest FreeBSD 13, but I haven't tried. I couldn't find any sexy GUI to display graphs or just a dashboard for a quick glance at my network status.
• OpenBSD+pf got the worst results, painfully reaching 300 Mbit/s download. Activating multi-processor support brought it up to 400 Mbit/s and the box got really warm during stress tests. There is of course no GUI but the pf configuration syntax is really clever and easy to write. The pf book is a fantastic source for learning how to build your own firewall. Even if the performance was really disappointing I learned so much about OpenBSD and firewalling that it was in itself a very worthwhile experiment. Would do it again just for fun.

If you are interested in building an OpenBSD firewall, here is a good tutorial: https://openbsdrouterguide.net/

• OpenWRT works fine on this box, though still not reaching symmetric Gbit. The latest OpenWRT version I tried (2021.02.0) seems to have a bug: I could only reach 300 Mbit/s download when upload had no trouble reaching my 500 Mbit/s limit. I remember getting far better results on OpenWRT 19.x versions.
• Debian+Shorewall wasn't so good. Can't remember what speeds I reached but it was far from saturating the line. Shorewall is also a beautiful piece of software for configuring a firewall, with adequate documentation and examples. The project seems to have been orphaned in 2018 and is still very useful and usable today, just don't expect new features.

I also tried Debian/nftables, with same disappointing performance. I didn't know anything about nftables when I started and read all available documentation. About 90% of the docs I found were meant to let me know why nftables is soooo much better than iptables and will end up replacing it one day, but only 10% are dedicated to explaining how to actually build a firewall, mostly by translating iptables rules to nftables. Since I didn't know about iptables either it didn't help me much. The syntax looks nice though, similar to BSD's pf with a few more warts. I wish they had written a manual.

• IPFire got me in trouble before I could use it. I managed to mess up the internal network config during install and couldn't recover from command-line afterwards as everything is meant to be controlled from the web interface. I was so annoyed I just gave up and didn't even try it out, though they claim to reach full symmetric Gbit on that hardware. Maybe another day.

I also ran a few experiments with Proxmox and installed virtual instances of OpenWRT and OPNsense with direct PCI passthrough to the NICs. The box reached the same performance overall as their direct install counterparts in both cases. The only real advantage of virtualizing your router is you can snapshot it at regular intervals. Proxmox is a lot of fun to play with.

Raspberry Pi 4

Spending a lot of time on OpenWRT forums, I stumbled upon many discussions about using an RPi4 as a home router for symmetric Gbit. It seemed hard to believe: those little critters are notoriously underpowered and have only one Ethernet port anyway, so what's the point?

Somebody tried one out with a USB3 Gbit dongle from various vendors and found out that they all yield very different performances. Most specifically: the TP-Link UE300 (a Realtek NIC for $11 on Amazon as I write this) seems to perform exceedingly well. This discovery was mostly met with disbelief at first but then some people started actually benchmarking the whole thing with great care and confirmed the outcome: an RPi4/UE300 combination running OpenWRT can sustain full symmetric Gbit without a sweat. Wow!

Of course I had a couple of RPi4 gathering dust somewhere at home. I just picked one up, swapped the SD card with a recent OpenWRT version, connected a TP-Link UE300 and started running tests. The results are just astonishing: full symmetric Gbit support on an internal network (941+ Mbit/s up and 941+ Mbit/s down). I have used one on my home network for a couple of weeks now and can confirm this is working astonishingly well.

I ended up bulletproofing my setup: got a box and a fan to keep it cool, a USB3 thumb drive to install the OS instead of running from a flaky SD card, and off you go. In terms of cost it ends up being twice as expensive as an EdgeRouter-X for double performance and it is so much fun to use I do not regret it a single moment.

More about that setup in another blog post.