Community discussions

MikroTik App
 
elmomac
just joined
Topic Author
Posts: 3
Joined: Sun Apr 30, 2023 3:17 am

CCR2216 + HW Offload + BGP = crash

Sun Apr 30, 2023 3:32 am

Hi Team,

I have put in SUP-114337 for issues with my recently installed CCR2216.

We have the following setup:
- 80G (2x40G) LAG to 2 x CRS354's. These are mLAG's on the CRS side.
- 20G (2x10G) LAG to ISP A
- 20G (2x10G) LAG to IXP A
- 10G to IXP B

We have approx 17 x IPv4 & IPv6 BGP peers/sessions, and around 580K routes. As soon as we enable L3HW offloading under "Switch" settings, the device stops routing all traffic and reports L3HW routes are full and crashes. We have tested 7.8, and all 7.9rc versions - the only way to fix is to turn off L3HW and reboot the device. We have even tried to disable all bar 3 of the BGP peering sessions where we end up with less than 10K routes overall, and the device still randomnly crashes with L3HW turned on.

With L3HW offloading turned off, we can still route approximately 20Gbps at 20-30% CPU, however this would be substantially reduced with being able to turn on L3HW.

Thanks!
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CCR2216 + HW Offload + BGP = crash

Tue May 09, 2023 3:47 pm

Hi, and sorry for the late reply!

We have made significant improvements to L3HW in RouterOS v7.10. Give it a try once it gets released.

Regarding the configuration of your CCR2216, VLAN interfaces must be set on the bridge; otherwise, L3HW cannot perform inter-VLAN hardware routing. Bypassing the bridge and setting VLAN interfaces directly on switch ports or bonds (as documented here) breaks hardware routing.

The correct configuration would be setting "interface=bridge" in /interface/vlan, then adding bonds under /interface/bridge/port.
 
dragoalato1988
just joined
Posts: 7
Joined: Sun Aug 29, 2010 2:06 pm

Re: CCR2216 + HW Offload + BGP = crash

Tue May 09, 2023 5:56 pm

Hi, and sorry for the late reply!

We have made significant improvements to L3HW in RouterOS v7.10. Give it a try once it gets released.

Regarding the configuration of your CCR2216, VLAN interfaces must be set on the bridge; otherwise, L3HW cannot perform inter-VLAN hardware routing. Bypassing the bridge and setting VLAN interfaces directly on switch ports or bonds (as documented here) breaks hardware routing.

The correct configuration would be setting "interface=bridge" in /interface/vlan, then adding bonds under /interface/bridge/port.
Hello support, I kindly ask for information. I'm trying to make a ccr2216 configured as a mpls P router work in hardware offload. Unfortunately, even with 7.9 I see that the traffic is only managed by the cpu. Do you have a roadmap for solving this problem? if yes for when?
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CCR2216 + HW Offload + BGP = crash

Wed May 10, 2023 11:16 am

Hello support, I kindly ask for information. I'm trying to make a ccr2216 configured as a mpls P router work in hardware offload. Unfortunately, even with 7.9 I see that the traffic is only managed by the cpu. Do you have a roadmap for solving this problem? if yes for when?

Hi,
MPLS HW Offloading is on the roadmap, but it is too early to speak about any release dates or RouterOS versions. It is not a problem but a new feature that requires a full development cycle.
 
dragoalato1988
just joined
Posts: 7
Joined: Sun Aug 29, 2010 2:06 pm

Re: CCR2216 + HW Offload + BGP = crash

Wed May 10, 2023 9:41 pm

Hello support, I kindly ask for information. I'm trying to make a ccr2216 configured as a mpls P router work in hardware offload. Unfortunately, even with 7.9 I see that the traffic is only managed by the cpu. Do you have a roadmap for solving this problem? if yes for when?

Hi,
MPLS HW Offloading is on the roadmap, but it is too early to speak about any release dates or RouterOS versions. It is not a problem but a new feature that requires a full development cycle.

Thank you for your quick answer. we are now testing 7.10 and it seems to work quite well. To make up for the current lack of the mpls l3hw function we were thinking of using P crs317 routers with routeros 6.x as router. while for PE, using routeros 7.X
There could be problems? can we trust that mpls P l3hw will work this way?
thanks and good job
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: CCR2216 + HW Offload + BGP = crash

Thu May 11, 2023 9:55 am

Why do you need to use RouterOS v6? Can you use RouterOS v7 with l3hw disabled (l3-hw-offloading=no) instead?
 
dragoalato1988
just joined
Posts: 7
Joined: Sun Aug 29, 2010 2:06 pm

Re: CCR2216 + HW Offload + BGP = crash

Fri May 12, 2023 3:52 am

Why do you need to use RouterOS v6? Can you use RouterOS v7 with l3hw disabled (l3-hw-offloading=no) instead?
we are trying to implement the right mpls distribution, and we need low latency, especially for the connection of router P network nodes. we are testing PE-P-PE, but we detect unexpected results.

the configuration connected with 10gb dac is
CCR2216 - PE -> Routeros 7.10beta5
crs317 - P -> Routeros 6.49
CCR2116 - PE. -> Routeros 7.10beta5

in this setup to get hardware offload where should "explicit null" be turned on?
 
elmomac
just joined
Topic Author
Posts: 3
Joined: Sun Apr 30, 2023 3:17 am

Re: CCR2216 + HW Offload + BGP = crash

Wed May 17, 2023 2:28 pm

Hi, and sorry for the late reply!

We have made significant improvements to L3HW in RouterOS v7.10. Give it a try once it gets released.

Regarding the configuration of your CCR2216, VLAN interfaces must be set on the bridge; otherwise, L3HW cannot perform inter-VLAN hardware routing. Bypassing the bridge and setting VLAN interfaces directly on switch ports or bonds (as documented here) breaks hardware routing.

The correct configuration would be setting "interface=bridge" in /interface/vlan, then adding bonds under /interface/bridge/port.
I have updated the support request with an updated supout.

Unfortunately i have tried 7.10 beta and after 5 hours we have experienced crashing and instability again. L3HW back off for now.
 
nocnetlatin
just joined
Posts: 7
Joined: Tue Aug 22, 2023 2:27 am
Location: Argentina
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 22, 2023 1:53 pm

Hi, I have the same problem with 4 ccr2216, I have 12 active bgp sessions per ccr, less than 12k routes, and I have to disable the l3hw from the WAN interface, otherwise it crashes, does not let traffic pass, does not let you enter by console, does not show you anything in neighbor, I have to restart it manually. Now all this is in random mode, it can work fine for 1 month or 5 hours or less.

The CCR2216 is configured like this.
qsfp1 WAN (2 bgp +2 vlan+ in bridge+no l3hw)
qsfp2 (9bgp + 15vlan+in bridge+l3hw)
sfp28-5 (1bgp+out bridge+l3hw)
sfp28-12(bypass vlan of qsfp1+bridge+no l3hw)
ether1 (out bridge+1bgp+no l3hw)

I have had this for months, and I have 4 providers, 1 for each ccr2216.
 
nocnetlatin
just joined
Posts: 7
Joined: Tue Aug 22, 2023 2:27 am
Location: Argentina
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 29, 2023 8:11 pm

Hi, well, I have re-installed ROS with netinstall in version 7.11 (stable) in one of my CCR2216 (This equipment crashed with L3HW or without L3HW, no matter day, time, amount of traffic) and in that rush I enabled a replacement CRS504 with BGP+Switch and the truth is that I did not find myself in need to put the CCR2216 back in production.

The CRS504 handles about 110gbps of troughput with 12 BGP peers + 12K routes. CPU at 20% peak time. I deduce that the error comes with something from the 2216 and not from a mikrotik qualified engineer config as raised later in this thread.

I have a CRS326 24S+2Q as CDN with FNA, GGC, OCA with 48gbps troughput at 1% CPU on average doing only BGP+3Bondings. All this with L3HW. I have a CRS305 passing 6gbps troughput only with IPv6+6 peer BGP with 1% CPU.

So L3HW works perfect, I don't understand where is the click with the CCR2216.

The configuration was copied as it was on the CCR2216 to the CRS504.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 29, 2023 9:25 pm


I have a CRS326 24S+2Q as CDN with FNA, GGC, OCA with 48gbps troughput at 1% CPU on average doing only BGP+3Bondings. All this with L3HW. I have a CRS305 passing 6gbps troughput only with IPv6+6 peer BGP with 1% CPU.
Which RouterOS Version on that CRS326 ?
 
nocnetlatin
just joined
Posts: 7
Joined: Tue Aug 22, 2023 2:27 am
Location: Argentina
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 29, 2023 10:11 pm


I have a CRS326 24S+2Q as CDN with FNA, GGC, OCA with 48gbps troughput at 1% CPU on average doing only BGP+3Bondings. All this with L3HW. I have a CRS305 passing 6gbps troughput only with IPv6+6 peer BGP with 1% CPU.
Which RouterOS Version on that CRS326 ?
Image
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 29, 2023 10:12 pm

nice, thank you for sharing.

your situation with 2216 is very striking
 
nocnetlatin
just joined
Posts: 7
Joined: Tue Aug 22, 2023 2:27 am
Location: Argentina
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 29, 2023 10:14 pm

nice, thank you for sharing.

your situation with 2216 is very striking
Si amigo, veo que sos de Colombia, estoy sufriendo.-

Yes my friend, I see U are from Colombia, I'm suffering.
Last edited by nocnetlatin on Fri Sep 01, 2023 7:26 pm, edited 1 time in total.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Tue Aug 29, 2023 10:19 pm

nice, thank you for sharing.

your situation with 2216 is very striking
Si amigo, veo que sos de Colombia, estoy sufriendo.-
Si amigo, tu caso me parece preocupante
es muy notable el hecho de que el CRS 504 pueda soportar ese escenario mientras que el CCR2216 no
desafortunadamente no conozco a nadie que tenga un escenario similar al tuyo para poder realizar una comparación

your case seems worrying to me
it is very remarkable the fact that the CRS 504 can support that scenario load and the CCR2216 cannot
unfortunately I don't know of anyone who has a scenario similar to yours to be able to make a comparison
 
User avatar
PapsGrey
just joined
Posts: 22
Joined: Wed Dec 01, 2021 10:58 am
Location: Accra-Ghana
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Sun Oct 22, 2023 2:25 pm

Please any update on the L3HW crashing with BGP?
I have about 38 pices of the CCR2216 on my network with MPBGP on ROS7.11.2, and yet have routers crashing randomly until rebooted.
 
User avatar
PapsGrey
just joined
Posts: 22
Joined: Wed Dec 01, 2021 10:58 am
Location: Accra-Ghana
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Sun Oct 22, 2023 10:01 pm

Please any update on the L3HW crashing with BGP?
I have about 38 pices of the CCR2216 on my network with MPBGP on ROS7.11.2, and yet have routers crashing randomly until rebooted.

Anybody fixed this issue, please?
 
elmomac
just joined
Topic Author
Posts: 3
Joined: Sun Apr 30, 2023 3:17 am

Re: CCR2216 + HW Offload + BGP = crash

Tue Oct 24, 2023 3:50 am

To confirm, Mikrotik has fixed all of the issues I was experiencing in this thread in 7.12

I am running the latest 7.12 RC without issue.
 
User avatar
PapsGrey
just joined
Posts: 22
Joined: Wed Dec 01, 2021 10:58 am
Location: Accra-Ghana
Contact:

Re: CCR2216 + HW Offload + BGP = crash

Wed Oct 25, 2023 11:02 am

To confirm, Mikrotik has fixed all of the issues I was experiencing in this thread in 7.12

I am running the latest 7.12 RC without issue.

Thanks a lot for the feedback. I will upgrade to 7.12 and test.

Who is online

Users browsing this forum: No registered users and 25 guests