I have configured my CRS309 to be my lab inter-VLAN router with L3 HW offloading. So far, everything seems to work great and I wanted to check back on your opinions as I’m not sure how good of an idea this is. All the NAT lifting, site to site VPNs and so on is done by a separate hardware, as I guess the CRS309 could get overloaded with that, depending on how much traffic and firewall rules is going on there? Currently I do have around zero CPU load at the CRS309, even@little bit over 9 Gbps routing throughput between the two server networks.
I think that’s a huge bang for the buck, if this is really going to work flawlessly in the long-term. Compared to competitor’s hardware, the CRS309 costs next to nothing.
thanks! How can I check how many fast-track connections I’m currently using? I could monitor that value using Zabbix, I guess, and issue me an alert if I hit certain thresholds.
Currently no ACL rules at all and by 16K - 30K routes you mean connected (dynamic) and manual (static) routes? I don’t think I’ll ever hit that limit…
so i guess there’s not much going on with 30-60 connections. I’d have to create firewall rules for fast-tracking, I guess? it doesn’t seem those non-fast-tracked connections are bothering the CPU in any kind of way, haven’t seen it spiking above 4%.
ah yes, makes sense, no nat, no connection tracking
my nat firewall is a fortigate 60D, it can do at least wirespeed NAT with its ASICs (1Gbps that is). it is quite old but the ASIC is doing the magic.
Hi,
you almost have the same config as I have, except I have a trunk between my FW and my CRS309. It works extremely well in this setup, but I’m not using L3offloading on the switch. The routing is done by the FW. The biggest problem I have are the MT 10G Cu SFPs, which are getting extremely hot. I had to mount some heat sinks. Looks bad, but works even on hot summer days and stays silent.
I need the 10G Cu, because my servers and also the FW don’t have SFP+ slots, just on board 10G Cu. Otherwise I would suggest using DAC Cables instead. I may reconfigure this switch for routing with L3 offloading enabled, but I’ll wait for the ROS 7 to stabilize.
DAC cables don´t get hot, so they probably don´t need readings (they are just that: cables with a few electronic components, but the Cu modules have active signal processing).
I am aware of the recommendations and I keep between my S+RJ10 modules a free slot. Still without any cooling they have reached over 90°C at an ambient temperature of around 30°C.
I have tried some no name chinese Cu SFPs as well, if they worked, they didn´t get as hot as MT SFP+ modules.
All types of routes (type doesn’t matter for HW offload engine). And the number is not that large, there’s a gotcha: if there’s a connected network, then every active host in that connected network uses up one route slot. E.g. if one of interfaces connects to a /16 directly connected subnet, then theoretically this can mean up to 65k routes. In reality the number will be most of times lower, possibly not all IP addresses are in active use. Theoretically IPv6 will be even worse, but in reality IPv6 address space is used very sparsely even if a subnet uses /64 prefix. Still it will be slightly worse than IPv4 due to the fact many hosts use multiple IPv6 addresses concurrently.
ok so each host within all the routed LAN segments connected to the router takes up one route. in my case that is way under 100.
but what about the ip addresses on the internet? all the connections from the LAN side to the internet - they are all not NATed on the CRS309 but they are indeed routed. do they take up one route space each as well in the CRS309?
only local host consume switch resources,this because the switch has to be aware of ARP and local things like that, from the perspective of the routing in the switch internet destination ips only are related to one resource, the default gateway route
As @checchito already wrote: naive version of HW routing table would contain three rows:
destination IP address
destination IP mask
next hop MAC address
A default upstream route would be: 0.0.0.0 / 0.0.0.0 / aa:bb:cc:dd:ee:ff (single entry for whole internet with MAC address of upstream router’s interface facing towards CRS).
Likewise route for single directly connected device would be: 10.20.30.40 / 255.255.255.255 / 00:11:22:33:44:55 (entry only covering single end device) … and there would be many more entries with similar IP address, same IP mask and different MAC addresses.
Mind that mask is not subnet mask, it’s defining IP address range, handled by same device having specified MAC address … which makes opportunity for a very minor improvement if single device handles multiple consecutive IP addresses but not entire subnet (e.g. due to proxy ARP or similar) … but that’s hard to do because router would have to constantly scan HW routing table for any candidates which in reality are very few, only to save a few entries in the table.
ok that makes sense, thanks! but you’re saying it yourself in some way:
who says, the CRS309 doesn’t maintain one entry per WAN destination IP? something like this (where AA:BB:CC:DD:11:22 is the NAT firewall’s MAC):
While these entries could be combined from 6 entries to two (assuming that there would be enough to fill a valid mask), I kinda doubt that RouterOS will “defragment” aka “compact” those entries to as few as possible…?! So in turn that would mean that each and every target IP on the internet will take up one entry in the HW routing table… right?
One way to solve that programatically would be, to make automatically huge wildcard entries in RouterOS towards the default gateway’s MAC address and only exclude the ranges where we have locally connected interfaces or manually added, static routes.
In my home lab I have CRS309 and CRS305 connected to my 3 servers in ECMP configuration with L3HW. Works quite well and stable for me after I started to use ROS 7.2RC5.
One thing that puzzles me is that when I test the performance with iperf3, I get 9.7Gbit/s going in one direction, but only around 8-8.5Gbit/s going into the opposite direction (and that happens across both switches)…
