I am hoping to get some advice before I purchase the hEX refresh. I am open to upgrading to a “better” option (RB760iGS, hAP ax³), as long as it is around $200 CA (Canadian Dollars).
Networking skill level: 2 out of 5
Use case:
Site to site VPN. Site A has pfsense Netgate SG-2100 while site B will have the Mikrotik router.
Use site B as a backup location for my data. Site A has a QNAP TS-453D, while site B has a QNAP TS-451.
Play with VPN setup/configuration
Connection/traffic should be encrypted in transit.
Any concrete advice/suggestion is highly appreciated.
Mikrotik routers basically all run the same software, so have the same feature set (with some minor differences).
I’d reject the hEX S (rb750igs) out of hand, because it has a MIPS CPU, and Mikrotik has pretty much committed itself to ARM devices for the future. Unless you need some hardware feature, like the SFP port of the micorSD slot, this is really not good value for money. Since you’re interested in VPNs, only ARM (and ARM64) devices support ZeroTier in Mikrotik-land, so this does not. All the others I list below support it. It’s also by far the weakest in terms of performance.
I think your choice basically comes down to two devices:
The hEX refresh: a nice/solid device; without enabling any sort of acceleration it can do about 500 Mbps of routing. It will get you around 200 Mbps with Wireguard VPN. If you have faster Internet, I’d go with something else. (USD 60)
The hAPax2; you get bonus WiFi (on Mikoritk devices this can be fully turned off) This will do around 1 Gbps of routing and 500-600 Mbps Wireguard. (USD 100)
Probably less suitable would be:
hAP ax3: basically the bigger brother of the ax2 with a 2.5G port. About +20-30% in routing, a bit more in VPN throughput. I don’t think it’s especially good value - but a solid device nonetheless. Also, it’s not very compact. (USD 139)
rb5009: if you want to go big. 3.5 Gbps of routing, and this is the first device that does 1+ Gbps of Wireguard. (USD 219)
The hAP ac2 is a really nice device at USD 80. I would not recommend it however, because it only has 16MB of flash memory, which is a really tight fit for Mikrotik’s OS, and Bad Things happen if it runs out of space, which can easily happen due to e.g. complicated or frequently changing configuration, frequent updates, log storage on flash, etc. The device is fully recoverable in these situations, but if you’re buying new, I don’t think it’s worth the (potential) hassle for the USD 20 difference compared to an ax2. All the other devices listed have at least 128 MB of storage, which is comfortable.
Altogether, I’d suggest the ax2 as a general purpose device - of course there may be circumstances I’m not aware of.
Hi lurker,
Thank you very much for the detailed reply. I also really appreciate the reasoning you put in the post. It allows me make an informed purchase decision.
Thank you again and hope you have an excellent rest of the week.
Thank you for a nice summary. May I ask you were did you get the WireGuard numbers? Is it from personal real-world experience? Unfortunately, MikroTik doesn’t publish WireGuard test results like they for IPSec.
I think it would be nice to publish wg performance data for the different models. It’s becoming sort of a standard for site-to-site tunnels involving Linux/BSD soho/smb routers. Or maybe someone with lots of devices lying around could provide us with a proper comparison???
If the numbers sound a bit hand wavy, it’s because they are.
I have actually measured these devices (excluding the hEX S) myself at some point, and my measurements lined up roughly with the numbers I found online.
My testing was done in lab conditions with a simple setup. (They were done at different points in time so there was no consistency in the software version used.) Default config with a single wireguard tunnel, UDP traffic with symmetric simultaneous send and receive, at least 10 different port numbers. The result was obtained by eyeballing packet loss, and the highest number where there was no consistent packet loss was considered the limit. (By “no consistent” I mean that I was willing to disregard the device deciding to do some housekeeping/management stuff intermittently.) This was repeated for 512 byte packets and the max that would fit through the tunnel without fragmentation.
Then comes the handwaving part: the 512/max throughput shows roughly 1:2 ratio and I used some point in the middle as a reference single number.
Also, I lock devices to the nominal CPU frequency. (Not necessarily the max RouterOS allows, so e.g. 716 MHz for the ac2.) I do this both for testing and production. I have found “auto” to mess with both synthetic and real world performance in a major way.
My overall conclusion is that the results roughly line up with the “512 byte / 25 filter rules” numbers published by Mikrotik (in their proportions of course), with 64bit platforms showing a clear advantage, which is to be expected for the algorithms used.
Several years ago I was looking specifically for devices capable of WireGuard for site-to-site small project. This is actually when I learned about MikroTik. People like you reporting their real-world experience and observed throughput were immensely helpful.
[offtopic] By the time I completed that project (very successfully), I got into RouterOS ecosystem and never turned back ever since. [/offtopic]
So, you mean take the official MT test results and divide by 2?
You had some pretty thorough testing there. When I tested a pair of ac2, I just used iperf between two wired PCs. I looked up my old notes and see that I could push a single stream 250-280 Mbps one way, mostly 250 sustained. The default iperf TCP packet size is 128 KB. The MT test shows 124 Mbps for 64-byte packets, mine is double that but one way. The 512 test shows 986 Mbps but keeping in mind those are for v6 (I wish MT re-published all tests for v7 like they did for hEX). I guess your estimate might be close enough with a lack of anything better available. Thanks again.
Sort of why I measured. I was involved in a couple of projects in the hundreds of devices for connecting remote equipment for monitoring/management, mostly for industrial automation applications. Of course encryption was mandatory. The required bandwidth for these is quite small, as you would imagine - but people like numbers, and when I was repeatedly asked about it, I measured it (These are places where even today they deploy 100M switches without any hesitation because they bought “a few” when these for still available to have some spares on hand - so asking about bandwidth is a bit redundant.)
This you misunderstood. I measured the performance through wg for both 512 and max unfragmented packets. The Mbps figure for these show a roughly 1:2 ratio. (This simply means that there is a per packet as well a per byte cost for both networking and encryption - which is not exactly surprising.)
One of the few measurements I actually documented was for the ac2:
with 512 byte packets I got 180 Mbps
with 1420 byte packets 420 Mbps
So I just said 250-350. (Which for most practical purposes is an okay estimate, I guess.) The 1:2 ratio doesn’t work all that well for this one. shrug
(These are places where even today they deploy 100M switches without any hesitation because they bought “a few” when these for still available to have some spares on hand - so asking about bandwidth is a bit redundant.)