Community discussions

MikroTik App
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 6:00 pm

Hi

I have configured my CRS309 to be my lab inter-VLAN router with L3 HW offloading. So far, everything seems to work great and I wanted to check back on your opinions as I'm not sure how good of an idea this is. All the NAT lifting, site to site VPNs and so on is done by a separate hardware, as I guess the CRS309 could get overloaded with that, depending on how much traffic and firewall rules is going on there? Currently I do have around zero CPU load at the CRS309, even@little bit over 9 Gbps routing throughput between the two server networks.
CRS309.PNG
I think that's a huge bang for the buck, if this is really going to work flawlessly in the long-term. Compared to competitor's hardware, the CRS309 costs next to nothing.
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 10:22 pm

nice setup
As long as you consciously stay within the limits of this device, you'll be fine:

16K - 30K Routes
4.5K Fast-track Connections
1024 ACL Rules

and

i think you must stay below 100kpps /1gbps of traffic procesed by CPU
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:10 pm

nice setup
As long as you consciously stay within the limits of this device, you'll be fine:

16K - 30K Routes
4.5K Fast-track Connections
1024 ACL Rules

and

i think you must stay below 100kpps /1gbps of traffic procesed by CPU
thanks! How can I check how many fast-track connections I'm currently using? I could monitor that value using Zabbix, I guess, and issue me an alert if I hit certain thresholds.
Currently no ACL rules at all and by 16K - 30K routes you mean connected (dynamic) and manual (static) routes? I don't think I'll ever hit that limit...
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:29 pm

check if this terminal command works:
/ip firewall connection print count-only where fasttrack=yes
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:35 pm

check if this terminal command works:
/ip firewall connection print count-only where fasttrack=yes
that shows 0.
but /ip firewall connection print count-only shows 60...
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:38 pm

in winbox gui

ip firewall connections

there you can filter connections by different criteria
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:42 pm

yeah well that's the same as
/ip firewall connection print count-only
so i guess there's not much going on with 30-60 connections. I'd have to create firewall rules for fast-tracking, I guess? it doesn't seem those non-fast-tracked connections are bothering the CPU in any kind of way, haven't seen it spiking above 4%.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:49 pm

you are right, CRS is not doing NAT so no need for fast-track

your router firewall doing nat supports many gigabits of NAT? which device is ?
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Wed Apr 20, 2022 11:57 pm

you are right, CRS is not doing NAT so no need for fast-track

your router firewall doing nat supports many gigabits of NAT? which device is ?
ah yes, makes sense, no nat, no connection tracking :D
my nat firewall is a fortigate 60D, it can do at least wirespeed NAT with its ASICs (1Gbps that is). it is quite old but the ASIC is doing the magic.
 
User avatar
woland
Member Candidate
Member Candidate
Posts: 258
Joined: Mon Aug 16, 2021 4:49 pm

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 12:13 am

Hi,
you almost have the same config as I have, except I have a trunk between my FW and my CRS309. It works extremely well in this setup, but I'm not using L3offloading on the switch. The routing is done by the FW. The biggest problem I have are the MT 10G Cu SFPs, which are getting extremely hot. I had to mount some heat sinks. Looks bad, but works even on hot summer days and stays silent.
I need the 10G Cu, because my servers and also the FW don't have SFP+ slots, just on board 10G Cu. Otherwise I would suggest using DAC Cables instead. I may reconfigure this switch for routing with L3 offloading enabled, but I'll wait for the ROS 7 to stabilize.

BR W
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 2:19 am

important info about S+RJ10 module heat and temp

https://wiki.mikrotik.com/wiki/S%2BRJ10 ... l_guidance
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 10:32 am

Hi,
you almost have the same config as I have, except I have a trunk between my FW and my CRS309. It works extremely well in this setup, but I'm not using L3offloading on the switch. The routing is done by the FW. The biggest problem I have are the MT 10G Cu SFPs, which are getting extremely hot. I had to mount some heat sinks. Looks bad, but works even on hot summer days and stays silent.
I need the 10G Cu, because my servers and also the FW don't have SFP+ slots, just on board 10G Cu. Otherwise I would suggest using DAC Cables instead. I may reconfigure this switch for routing with L3 offloading enabled, but I'll wait for the ROS 7 to stabilize.

BR W
Wow, my SFP+ modules stay at around 38-40°C. The DAC cables don't show any temperature readings, unfortunately.
 
User avatar
woland
Member Candidate
Member Candidate
Posts: 258
Joined: Mon Aug 16, 2021 4:49 pm

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 10:47 am

Hi,

DAC cables don´t get hot, so they probably don´t need readings (they are just that: cables with a few electronic components, but the Cu modules have active signal processing).
I am aware of the recommendations and I keep between my S+RJ10 modules a free slot. Still without any cooling they have reached over 90°C at an ambient temperature of around 30°C.
I have tried some no name chinese Cu SFPs as well, if they worked, they didn´t get as hot as MT SFP+ modules.

W
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 11:05 am

... and by 16K - 30K routes you mean connected (dynamic) and manual (static) routes? I don't think I'll ever hit that limit...

All types of routes (type doesn't matter for HW offload engine). And the number is not that large, there's a gotcha: if there's a connected network, then every active host in that connected network uses up one route slot. E.g. if one of interfaces connects to a /16 directly connected subnet, then theoretically this can mean up to 65k routes. In reality the number will be most of times lower, possibly not all IP addresses are in active use. Theoretically IPv6 will be even worse, but in reality IPv6 address space is used very sparsely even if a subnet uses /64 prefix. Still it will be slightly worse than IPv4 due to the fact many hosts use multiple IPv6 addresses concurrently.
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 12:14 pm

... and by 16K - 30K routes you mean connected (dynamic) and manual (static) routes? I don't think I'll ever hit that limit...

All types of routes (type doesn't matter for HW offload engine). And the number is not that large, there's a gotcha: if there's a connected network, then every active host in that connected network uses up one route slot. E.g. if one of interfaces connects to a /16 directly connected subnet, then theoretically this can mean up to 65k routes. In reality the number will be most of times lower, possibly not all IP addresses are in active use. Theoretically IPv6 will be even worse, but in reality IPv6 address space is used very sparsely even if a subnet uses /64 prefix. Still it will be slightly worse than IPv4 due to the fact many hosts use multiple IPv6 addresses concurrently.
ok so each host within all the routed LAN segments connected to the router takes up one route. in my case that is way under 100.
but what about the ip addresses on the internet? all the connections from the LAN side to the internet - they are all not NATed on the CRS309 but they are indeed routed. do they take up one route space each as well in the CRS309?
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 5:17 pm

only local host consume switch resources,this because the switch has to be aware of ARP and local things like that, from the perspective of the routing in the switch internet destination ips only are related to one resource, the default gateway route
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 6:46 pm

As @checchito already wrote: naive version of HW routing table would contain three rows:
  1. destination IP address
  2. destination IP mask
  3. next hop MAC address

A default upstream route would be: 0.0.0.0 / 0.0.0.0 / aa:bb:cc:dd:ee:ff (single entry for whole internet with MAC address of upstream router's interface facing towards CRS).
Likewise route for single directly connected device would be: 10.20.30.40 / 255.255.255.255 / 00:11:22:33:44:55 (entry only covering single end device) ... and there would be many more entries with similar IP address, same IP mask and different MAC addresses.
Mind that mask is not subnet mask, it's defining IP address range, handled by same device having specified MAC address ... which makes opportunity for a very minor improvement if single device handles multiple consecutive IP addresses but not entire subnet (e.g. due to proxy ARP or similar) ... but that's hard to do because router would have to constantly scan HW routing table for any candidates which in reality are very few, only to save a few entries in the table.
 
azzurro
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 92
Joined: Mon Jan 17, 2022 2:55 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 10:44 pm

ok that makes sense, thanks! but you're saying it yourself in some way:
who says, the CRS309 doesn't maintain one entry per WAN destination IP? something like this (where AA:BB:CC:DD:11:22 is the NAT firewall's MAC):

37.2.1.1 / 255.255.255.255 / AA:BB:CC:DD:11:22
37.2.1.2 / 255.255.255.255 / AA:BB:CC:DD:11:22
37.2.1.3 / 255.255.255.255 / AA:BB:CC:DD:11:22
80.100.20.1 / 255.255.255.255 / AA:BB:CC:DD:11:22
80.100.20.2 / 255.255.255.255 / AA:BB:CC:DD:11:22
80.100.20.3 / 255.255.255.255 / AA:BB:CC:DD:11:22

While these entries could be combined from 6 entries to two (assuming that there would be enough to fill a valid mask), I kinda doubt that RouterOS will "defragment" aka "compact" those entries to as few as possible...?! So in turn that would mean that each and every target IP on the internet will take up one entry in the HW routing table... right?

One way to solve that programatically would be, to make automatically huge wildcard entries in RouterOS towards the default gateway's MAC address and only exclude the ranges where we have locally connected interfaces or manually added, static routes.

I wonder how RouterOS does it internally...
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Thu Apr 21, 2022 11:09 pm

I wonder how RouterOS does it internally

You can get a hint by Googling linux netfilter/nftables or checkout https://www.netfilter.org/ as a starting point.
 
mfedotov
just joined
Posts: 18
Joined: Mon Oct 25, 2021 3:32 am

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Sat Apr 23, 2022 1:13 am

In my home lab I have CRS309 and CRS305 connected to my 3 servers in ECMP configuration with L3HW. Works quite well and stable for me after I started to use ROS 7.2RC5.
One thing that puzzles me is that when I test the performance with iperf3, I get 9.7Gbit/s going in one direction, but only around 8-8.5Gbit/s going into the opposite direction (and that happens across both switches)...
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS309-1G-8S+IN as lab datacenter router - good idea?

Sat Apr 23, 2022 11:56 am

Are you sure that both test devices, running iperf3 (either client or server mode) are capable of saturating 10Gbps link with ease? It's not really peace a cake job and Tx is different than Rx ...

Who is online

Users browsing this forum: ameliask, Bing [Bot], kvitek79, mikronoob89, Vojta and 95 guests