I just bought a pair of RB1100AHx4 to replace RB3011 routers, especially to be able to do some HW acceleration for AES-GCM/AES-512 as the 3011 CPU’s were redlining most of the time.
Before assigning the ports on the new RB1100AHx4, I had a look at the block diagram here:
This raises a few question: to make it simple, I have 3 kind of ports: LAN, DMZ, and WAN, connected to the network over port-channels.
My initial idea was to split these among the 3 internal Switch chips:
physical ports 1-5 (Realtek GB Switch 1) => LAN
physical ports 6-10 (Realtek GB Switch 2) => DMZ
physical ports 11-13 (Realtek GB Switch 3) => WAN
99% of the traffic will be between LAN and WAN, there will be almost no traffic between different LAN vlans or DMZ vlans. This means that almost all traffic will be running over the CPU. So now I’m wondering if it wouldn’t be better to mix LAN, DMZ and WAN segments over all switches, for instance:
eth1 + eth6 + eth11 = LAN port-channel
eth2 + eth 7 + eth12 = DMZ port-channel
eth3 + eth8 + eth 13 = WAN port-channel
Maybe by doing this, the router will be able to offload more processing (fasttrack) to the 3 switches instead of having to run everything through the CPU?
There is no L3HW offload on the RB1100AHx4, so routed IP traffic will always use the CPU. So the round-robin ports likely isn’t a good idea.
Also keep in mind ether11 and ether12 support a offline “bypass”. If power is cycled (and switch is on), it could bridge unexpectedly… While I’m sure the hardware switch to control this works, but having a LAN and WAN on those ports leave the door open if someone didn’t check the bypass switch.
I would have at least considered the RB5009 in the mix with also a Marvel switch chip - 88E6393X.
You purchased an ARM32bit processor with 1gig RAM, and 128MB storage with a CPU AL21400
to replace
an an ARM32bit processor with 1gig RAM and 128MB storage with a CPU IPQ-8064
IPQ-8064 - quad core consists of dual-core Qualcomm® Krait™ CPU (1.4 GHz) for control plane and applications, with a dual-core 730 MHz Network Subsystem (NSS) to accelerate packet processing. VINTAGE - 2014
AL21400 - 4 core Cortex-A15 running at 1.4Ghz, VINTAGE 2016
However if looking for more IPSEC performance than an upscaled AL chip on the CCR2004 perhaps was more appropriate.
Ipsec performance in the order of 940.
If your budget is not the issue, then the CCR2116 is the no brainer choice with ispec at 2730 !!
They’re dated sure. RB1100AHx4-Dude has 2 x M.2 slots to use as a disks… Plus more ports than RB5009 & redundant power supplies. And at least ARM, so runs ZeroTier. Everything has a use.
When they make some RB5009 things with M.2 slots, we can talk about retirement of the RB1100’s . But a RB11000 is actually primary RouterOS test router ATM.
the RB1100AHx4 does not have HW offloading from the CPU, but still had HW acceleration to offload some tasks from the CPU (sorry, I wasn’t clear): https://wiki.mikrotik.com/wiki/Manual:IP/IPsec#Hardware_acceleration
This will be perfectly fine for my needs, as the RB3011 are already “almost” OK. The encrypted traffic is about 500Mbps max (for a couple hours), and 30-50Mbps average the rest of the time.
I took the RB1100Ahx4 because I got a very good deal on these: about 50 EUR/pc
good point for the ports 11/12 which work as a bypass. It was more meant as an example, I will not use those 2 ports in a mixed mode for sure. But thanks for the hint!
So you would 1 internal switch for each type of port? e.g:
physical ports 1-5 (Realtek GB Switch 1) => LAN
physical ports 6-10 (Realtek GB Switch 2) => DMZ
physical ports 11-13 (Realtek GB Switch 3) => WAN
or would you at least mix up ports 1-5 and 6-10 to make better use of the internal switches 1 and 2?
They’re dated sure. RB1100AHx4-Dude has 2 x M.2 slots to use as a disks… Plus more ports than RB5009 & redundant power supplies. And at least ARM, so runs ZeroTier. Everything has a use.
Indeed. Dated doesn’t mean outdated. I completely agree: if it works and fits the needs, why changing it? (I admit, that’s a personal opinion, but I’m against consumerism). My wife is 42 years old and has some milage too (and god knows, so do I) but that doesn’t mean I’m planning to replace her!
On a more serious topic, I’ve got some BEST POWER Ferrups UPSes running for 30+ years in production (yes, really). They buried several APC, Socomec and Eaton (new gen) UPSes that litteraly died in prod after a couple of years (transistor and/or caps that blew through the PCB…) and as an electronics enthusiast, I’m still using a Tektronix 462b scope and old Fluke multimeters. They are over 45 years old and work perfectly well!
It really depends on traffic pattern. Will LAN hosts communicate between each other? If yes, then placing them on same switch chip is the only sensible thing to do. If not, then it doesn’t matter much because traffic will pass the CPU regardless the distribution. And the 2.5Gbps interconnect between switch and CPU might become a bottleneck in both cases (but if a group of hosts consume large amount of bandwidth for communication of each other, then interconnect might not be a bottleneck because individual links will be saturated).
very few. it’s anecdotic. The Mikrotik routers are edge routers, and what I’ve defined as “LAN” segments are actually a transit to the inner firewall, which handles the inter-VLAN LAN traffic. There are a few exceptions but we’re talking about 5-10Mbps rates that the MT will have to handle between non-WAN segments.
99% of the traffic is from (several) DMZ to WAN and LAN (transit) to WAN and vice-versa
The WAN links on all sites are 1GB sym, so that0’s where the bottleneck is. The 2,5GB BW inside the MT bus is not a limiting factor for the years to come.
In this case distributing LAN ports between different switch chips and DMZ ports likewise would act in similar way as LACP bond does … sometimes multiplying total throughput and sometimes no gains at all. But in average I guess the “distributed” use might offer somehow higher total throughput.
My thought is keep the VLAN together on same switch chip (per VLAN). I get the idea to “load-balance” the switch-to-cpu ports…
But I just can’t see that having an practical/noticable effect. And, while likely LAN-to-LAN might be infrequent… but always possible you’d have HUGE backup/restore over LAN and that’s when HW offload ports be handy SO that doesn’t overwhelm the CPU if it did happen.
And, while likely LAN-to-LAN might be infrequent… but always possible you’d have HUGE backup/restore over LAN and that’s when HW offload ports be handy SO that doesn’t overwhelm the CPU if it did happen.
hum…I don’t catch your thought, could you re-explain to me what you mean?
Wen you transfer zilions of bytes from local server to local backup e.g.NAS then instead of bothering CPU with transmission it’s way better to use HW-offload to minimize CPU saturation
Well that will never happen. Backups will stay in the internal network, and the whole traffic is running over the internal Firewall. The mikrotik will never see such backup traffic, unless it is for inter-site backups (and it that case, it will use encrypted tunnels over the WAN port). But there will never be huge LAN to LAN traffic on the RB1100.
Yes but …
Looking at https://i.mt.lv/cdn/product_files/RB1100AHx4v5_180118.png you can clearly see that transfers between any port in switches 1..5 & 6..10 could never reach CPU as they could go at wirespeed from eg. Eth2 to Eth4 for local LAN transfers. When you transfer from e.g. Eth1 to Eth7 then the traffic goes via CPU and 2.5Gb links When you have 3 such streams then both 2.5Gb lings get saturated even if CPU is not involved. At the worst situation five streams of data for ports 1-6, 2-7, 3-8, 4-9 & 5-10 have to share both internal links. If CPU gets involved than situation goes even worse.
ok, so in this case you would setup LAGs with 1 port on the 1st switch and 1 port on the other switch?
Do we know how the RB1100 will select which port it uses to send out traffic (like built-in priority for ports able to communicate without running over the CPU)?
I didn’t notice they were going to switch or ISP alone, but you have TWO of them. Without out know more specifics, configuring the two router identically seem the best use. Is your your WAN or DMZ actually >1GB in speed – then bonding make more sense.
Personally I think the ether1 to ether1, ether5 to ether5, ether13 to ether13 be clean. I’m a VRRP “fan”, so this model allow that. And if you have the ports, keeping access-ports and using 3 cables avoid any complex briding things. With “one cable trunk” speed be limited to 1G on the RB1100AHx4 (without bonding).
Alternatively you could use one router as the WAN/DMZ router, and the other as the LAN router (or WAN on one and DMZ/LAN other split, etc.). That may be better/worse, but hard to know what your specific needs/usage is going to be…
It’s the traffic, not the port. Better to think that everything goes through CPU…UNLESS it’s the traffic on the same switch chip. e.g. on LAN if ether2 communicates to ether4, that’s HW offload (switched). But the second you cross a VLAN boundary, it goes through the CPU, full stop.
And similarly, if single VLAN crosses switch chips with the ports, it’s also goes through CPU.
But the second you cross a VLAN boundary, it goes through the CPU, full stop
Ok, so that means that even inter-VLAN routing is ALWAYS done on the CPU, no exceptions (e.g if vlan 10 and 20 both belong to the “LAN” Port-group, and both are exclusively tagged on switch1 (for instance port 1 and 2))?
I didn’t notice they were going to switch or ISP alone, but you have TWO of them.
Sorry I wasn’t clear enough. I bought 2 routers (+1 a 3rd one as spare), but they will be installed on 2 different sites. Each site will use only one RB1100AHx4 as a border router to the ISP (the router config is pretty much the same on both sites, with only a few minor differences). (I am planning to add VRRP on the main site, but with a 4G/5G router as backup. It’s gonna be a bit tricky because of CGN on the Mobile network…I might have to create a tunnel to announce my AS over BGP…I’m still working on the design)
Yup, between VLAN 10 and VLAN 20 would still route via the CPU. Only traffic that used multiple ports, both same VLAN, would remain in the switch chip.
The RB1100’s don’t support ANY Layer3/IP HW offload, so idea of “port group” really doesn’t exist – only same-VLAN can be offloaded across multiple ports.
. But a RB11000 is actually primary RouterOS test router ATM.