Community discussions

MikroTik App
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 11, 2022 1:51 am

So I am in a situation where I am trying to figure out the best way to re-configure core network routing.
Currently using some old HPE kit to handle that and I have a CCR2004-16G and CRS354-48G both currently running v7. I need to perform basic L3 inter-VLAN routing of ~1Gb/s, averaging much less than that most of the time.
Little bit of insight into the design-
4 locations, each with isolated LAN segments & WiFi
This Router would be the "core" router with the other three sites being connected to this one. So performing all inter-vlan routing between sites & subnets, as well as internet facing traffic towards enterprise firewall would be on this unit.
Other VLAN are a mixture. WiFi, Guest WiFi (internet only), and isolated LAN segments. - So while most traffic is internal L2, there is some L3 expected to be wire-speed. An some of it is basic internet traffic. Besides peak times, such as when Off-site backups (Gig WAN limited) are occurring, the L3 requirements would be quite low, 200Mb/s or less likely.

I feel like I have no choice but to use CCR2004 if I expect to isolate subnets, but I worry it may struggle with near wire-speed L3.
My understanding tells me I need to use Firewall to ACL the subnets, Does this automatically eliminate the CRS354 as a choice? I know it can only do L3 wire-speed in Hardware offload mode. Does that mean its unable to isolate subnets and maintain wire-speed or can I use some kind of connection fast-track to check if its allowed an off-load it or not allowed block it? It would be purely SRC/DST by subnet isolation.

What would you suggest in my situation?

Follow up question:
If using CCR2004 for this: What configuration method would you use for the VLAN routing? Do I use Bridge-VLAN configuration on this hardware (Like the CRS3xx) or another method?

Thank you
 
DJSmiley
just joined
Posts: 18
Joined: Thu Feb 19, 2015 2:51 pm

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 11, 2022 2:11 pm

I wouldn't use a CRS for that. Also, what are the links between those 4 locations?

Do you have a fully L2 connection between those locations without any restrictions?
If using external providers for these links, a lot of them have mac limits in place.

I have similar setups running. Usually a CCR1009 or 4011 onsite, /32 subnets to connect over the primary WAN. (Thus preventing all mac adresses on the wan)
Secondary a DSL connection with a IPSec tunnel. Using OSPF this acts as a backup in case the primary link fails.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 11, 2022 8:24 pm

I wouldn't use a CRS for that. Also, what are the links between those 4 locations?
2x 100Mb private fiber
1x 1Gb private fiber
Do you have a fully L2 connection between those locations without any restrictions?
If using external providers for these links, a lot of them have mac limits in place.
Yes, full Layer2 available. But we opt to isolate sites with L3 transport to prevent any kind of flooding on the WAN links.
I have similar setups running. Usually a CCR1009 or 4011 onsite, /32 subnets to connect over the primary WAN. (Thus preventing all mac adresses on the wan)
Secondary a DSL connection with a IPSec tunnel. Using OSPF this acts as a backup in case the primary link fails.
Remote sites do not have backup connections/internet available, they are connected with a single fiber WAN here. where they get Internet/Network resources from the main site.
One site does have its own servers on-site, but still rely on internet and other internal resources from the main location.
This has only been a issue once in 5+ years, where someone bored into the fiber for one of the locations, cutting them off for roughly 18 hours until the fiber could be dug up and fully repaired.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 11, 2022 10:03 pm

The answer to initial question depends on when it's needed. If it's needed now, then the answer is CCR2004. If it'll be needed in a few months, then the answer is CRS354.

When running ROS v6, CRS devices are switches with low-capacity routing ... CRS354 say around 200Mbps on all ports combined.

When running ROS v7, which is not ready for prime time yet, but it's being improved as we speak, CRS3xx get wire-speed routing capacity. With some gotchas, but if use case fits, they're beasts. You can read more about L3 hardware offloading. The way offloading works is that connections are offloaded after they are allowed. Checks are still done by (slow) CPU, but this mesns checking a few packets initially and vast majority is then handled by switch chip.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 11, 2022 10:52 pm

I would prefer sooner than later.
I guess I felt that basic L2/L3 would be okay to run on v7 currently and the distributer who sold me the hardware agreed.
It wasn't until after I got it and did some more digging into the configuration options that I started to wonder if I even need the CCR2004.
At the time of purchase it was planned to do all routing on the CCR2004 an leave the switches just doing L2 & MLAG for redundancy (you stumbled upon my comment in another post)

I have read the L3 Hardware Offloading page previously and just reread it again now, which is where my concern of firewall (CPU) limitation would come into play on the CRS354.
If I can still achieve wire-speed L3 with minimal CPU impact using basic ACL/Fastpath rules on the firewall that makes the most sense to me. I will need the port capacity anyway.
For example:
/ip firewall filter
add action=drop chain=forward connection-state=new dst-address-list=LAN src-address-list=Guest
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
If its as simple as that to deny access between some VLAN an Off-load the rest, that shouldn't be an issue for the CRS354 should it?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Wed Jan 12, 2022 12:58 pm

If its as simple as that to deny access between some VLAN an Off-load the rest, that shouldn't be an issue for the CRS354 should it?

You're right, it shouldn't be a problem ... because packets, hitting rule #1 will be rare (unless somebody from Guest list decides to DOS your router). At that moment already offloaded connections should still be routed wire speed, but adding new connections to offloaded list would be slow.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Wed Jan 12, 2022 11:40 pm

I don't see a inter-VLAN DOS attempt taking place, but I guess its always possible.

For new connections to be first checked against firewall with L3 Offload enabled, and achieve VLAN isolation do I need to enable IP Firewall VLAN on the bridge?
/interface/bridge/settings
use-ip-firewall=yes
use-ip-firewall-for-vlan=yes
Or do I need to disabled L3 hardware off-load on all Tagged trunk interfaces of the switch? This sounds like it would impact performance more an CPU process connections that don't need to be. But I guess I am just not sure of the correct approach.

if I should stick with my original intention of using the CCR2004 as the router, and error on the side of caution you suggested (CCR now/CRS later) should I be using the same Bridged-VLAN configuration type that I use on the CRS354? Keeping in mind I do intend to run v7, as this L3 routing is really the extent of my requirements.

Thanks again
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Thu Jan 13, 2022 9:45 pm

Inter-VLAN traffic passes normal firewall rules. The settings "use-ip-firewall=yes" and "use-ip-firewall-for-vlan=yes" force using firewall filters for traffic passing bridge inside L2 network (either all-untagged LAN for the former setting or intra-VLAN for the kater setting). Both settings are meant for special cases, but your case (inter-VLAN routing and firewalling) is not one.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Fri Jan 14, 2022 12:53 am

Inter-VLAN traffic passes normal firewall rules. The settings "use-ip-firewall=yes" and "use-ip-firewall-for-vlan=yes" force using firewall filters for traffic passing bridge inside L2 network (either all-untagged LAN for the former setting or intra-VLAN for the kater setting). Both settings are meant for special cases, but your case (inter-VLAN routing and firewalling) is not one.
So no need to disable L3 offload per interface either then I assume, since you didn't comment on that?

Can I pry again for thoughts on the CCR2004 configuration?
Or anyone else for that matter.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Fri Jan 14, 2022 7:55 am

No need to disable HW offload on any interfaces. If client feels it needs help from router (e.g. because destination of packet is not within same subnet), then client sends packet with MAC address of router. If CRS is acting as a router, then frame will be targeting CRS' own MAC address (of the bridge) and switch chip of CRS will know to forward it to CPU. Unless L3HW configured switch chip to alter frame (needs to set new dst-mac and new VID) and send it out autonomously.

I don't see any reason not to configure CCR in same manner as CRS ... e.g. single bridge with vlan-filtering=yes and all ports members (either trunk or access or hybrid). If miracle happens and CCR2004 gets L3HW offload some time, then your config will be ready for it. If miracle doesn't happen, it will (most likely) perform equally well as if using any other method (if not better). The added value is that configs of CCR and CRS will be similar so you'll master both configs with ease :wink:
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 18, 2022 12:01 am

No need to disable HW offload on any interfaces. If client feels it needs help from router (e.g. because destination of packet is not within same subnet), then client sends packet with MAC address of router. If CRS is acting as a router, then frame will be targeting CRS' own MAC address (of the bridge) and switch chip of CRS will know to forward it to CPU. Unless L3HW configured switch chip to alter frame (needs to set new dst-mac and new VID) and send it out autonomously.
Thanks, the comments about Off-loading preventing rules from being Firewall processed worried me. I am glad its a simple couple of rules to both ensure ACL work and Off-loading still happens on existing connections.
I don't see any reason not to configure CCR in same manner as CRS ... e.g. single bridge with vlan-filtering=yes and all ports members (either trunk or access or hybrid). If miracle happens and CCR2004 gets L3HW offload some time, then your config will be ready for it. If miracle doesn't happen, it will (most likely) perform equally well as if using any other method (if not better). The added value is that configs of CCR and CRS will be similar so you'll master both configs with ease :wink:
Appreciate your feedback on this.
I too hope that L3 off-load comes to move devices, such as the CCR2004. Anyway that Mikrotik configurations can be more uniform between device types is a step in the right direction. I think a lot of confusion comes from the hardware configurations from hardware of old still being out there, an while they work on newer hardware, may not be the best practice any longer.

When looking at just newer Mikrotik Hardware, are the days of worrying about which VLAN method to use a thing of the past? Am I safe to assume that all new hardware should be able to perform fine with single-bridge, vlan-filtering enabled configurations? Obviously within reason, right hardware specifications for the data an roles its going to fulfill of course.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Tue Jan 18, 2022 8:19 am

When looking at just newer Mikrotik Hardware, are the days of worrying about which VLAN method to use a thing of the past? Am I safe to assume that all new hardware should be able to perform fine with single-bridge, vlan-filtering enabled configurations? Obviously within reason, right hardware specifications for the data an roles its going to fulfill of course.

All mikrotik devices support the unified configuration using single vlan-filtering enabled bridge. The only gotcha is that most devices (i.e. all older, except CRS3xx and a few other in ROS v7) don't offload L2 operations to hardware even if hardware capable of doing it is there. Meaning that performance (throughput) is less than great.
As we've seen with RB4011 this can change, so I'm looking forward to see more devices supporting L2 HW offload in future ROS v7 releases. The problem with this is only that it's not sure if and when this will happen for a particular device. If one is picking hardware for network, one should pick hardware which supports features and capacity needed with current ROS releases. And consider any potential future performance improvements as a free gift. Or one can gamble and pick cheaper hardware with expectation of decent performance in the future.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Wed Feb 09, 2022 8:06 pm

I wanted to provide an update.
I did opt to go with the CCR2004-16P-2S+ as the core routing device.
It is connected via 2x2 LACP to a MLAG configuration on the pair of CRS354-48Gs. An to put it kindly, things are stable as long as I don't touch it.

As its recommended L3-HWO is disabled when you make Bridge/VLAN changes on the CRS354, the moment I do that on either switch, the CCR2004 starts complaining it has a loop an slowly locks up. (Neither of the CRS354s throw any errors/warnings about loops)
It cannot be remedied until I completely shutdown the LACP interface and bring it back up after I am done with changed on the 354s an turned L3-HWO back on.
I can disable 3 of 4 interfaces of the LACP on the CCR2004 it stops but unless I take down all four interfaces after the L3HWO is back on, it does not stop.

I feel I may also have some Root bridge/Bridge configurations that I need to fine tune. Related to Path cost that is causing network changes to take a really long time to sort themselves out.
This has gotten better with the complete removal of a old procurve switch that was connected to the CRS354s but compared to when the network was mostly Procurve, things are not as quick to change.
Related to this: Should the path cost be lowest on the CRS354 "Core switches" or the CCR2004 "core router". Most but not everything traverses the router, so I am unsure what should have the lowest value an be considered my root bridge. Also, does it matter which Bridge interface it chooses as its root port when you are manually setting Admin MAC? The CRS354s currently have selected a random Fiber SFP as its root an not the LACP to the CCR2004 as its root port. Which seems odd to me as logically that LACP is the core connection of the network.

An then there are things like this where packet loss to the CCR2004 is higher than it should be compared to a almost identically connected RB1100.
Relative to my workstation its PC > CRS328 > CRS354s an the RB1100 an CCR2004 are connected to the 354s. Only difference is the RB is one interface an the CRS2004 is the LACP to MLAG.
ping_compare.png
Over-all if some of the weirdness was gone, such as LACP loops occurring when I go to make vlan changes or random port flaps or what seems like long network changes I would be completely satisfied with the hardware. This really is not a complex setup. Router, Redundant switches an a handful of routed VLANs. It should not be a problem.
You do not have the required permissions to view the files attached to this post.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Thu Feb 10, 2022 12:23 am

Oh this one is interesting. There isn't even a cable plugged into ether1 but at 4:20a today the port came up some how...
ghost_connecting.png
So that's fun.
You do not have the required permissions to view the files attached to this post.
 
rooin
newbie
Topic Author
Posts: 47
Joined: Tue Feb 22, 2011 10:44 am

Re: L3 Routing - CCR2004-16G-2S+ or CRS354-48G-4S+2Q+ v7

Thu Feb 10, 2022 9:32 pm

Anyone else experienced issues when using LACP bonding on CCR2004-16G-2S+ hardware?
Today we started experiencing intermittent connectivity/routing issues on the CCR2004. As the usual culprit of its issues has been the 2x2 LACP to MLAG connection it has with the pair of CRS354s. I disabled that and just connected it via ether1 to one of the CRS354s an now the problem has seemingly gone away.
No more random 2-3 packet drops on ping or reported disconnects of user sessions on various applications. (Both internal an out to the web) Which is strange because internal LAN would not even traverse the CCR2004 in my configuration.
Edit Clarification- Local LAN, non-wireless, non-remote site subnets, would not be routed and/or need to be handled by the CCR2004.
All internet connections would.

Who is online

Users browsing this forum: inteq, ricardobar and 9 guests