Community discussions

MikroTik App
 
LightnetBarry
just joined
Topic Author
Posts: 16
Joined: Tue Jun 05, 2012 2:56 pm

BGP, disappearing routes and raw filter rules

Thu Sep 02, 2021 2:43 pm

We experienced a strange situation earlier in the week.

I'm sharing it in the hope that :
a) it will be helpful to someone
b)the cause can be addressed (or I can find which version addresses it)

We use a CCR1072 as a BGP router peering with an IX and a transit provider. The router has been running v6.46.8 since Nov 2020 and very stable.
On 21st August a processor became stuck at 100% activity (running routing, management, etc processes)
On 29th August the router experienced a kernel panic and rebooted. The stuck processor cleared, all eBGP and iBGP sessions came back up and full routing was restored within a few minutes.
All OK so far.

8 hours later traffic levels started to drop across the router, then all traffic ceased and the router became unreachable. Then it recovered, then traffic levels dropped and stopped, and again, and again...
Recovery was always proceeded by BGP session timers expiring but didn't last for long. As a last resort the router was rebooted and everything was good again overnight.
10 hours later the process occurred again, exactly the same pattern: diminishing traffic, BGP session expiry, traffic recovery...
Another reboot and an upgrade to 6.47.10, another 10 hours at which point I drove across the country to the DC and replaced the router. All good again.

Next morning, traffic diminished, sessions dropped, yada, yada. I dropped all the IX sessions and it stayed up for an hour. I dropped the transit and brought back the IX sessions, it lasted a little longer. I dropped the iBGP, no difference.

Eventually we discovered that connection tracking was turned on. My theory is that this was somehow impacting available memory and eventually the routing table began to grow smaller until it was non-existent, then BGP keepalives began to not get responded to and sessions dropped. This allowed ConnTrack entries to expire and some memory to be freed up, routing table to be rebuilt and everything to begin again. That's my theory even though SNMP shows no extra memory use (circa 14GB free constantly).

Connection tracking was set to 'NO' rather than 'AUTO' on the router.

However... Connection tracking is disabled by an entry in the raw filter table and I also use the raw table to process packets to mitigate against DDoS. I accept a lot of pps/dest and beyond that add the dest address to a drop rule for 10 minutes. This works really well btw.
When I export my config the ConnTrack=NO declaration is way before the raw table rules however on reboot the raw rules are placed before the ConnTrack=NO rules and my accept rule means that no packets reach the tracking rule so all connections are tracked. Since moving the rule to disable connection tracking back to position 0, the problem has not recurred.

This occurs on both 6.46.8 and 6.47.10

The obvious solution is to ensure that a 'place-before=0' parameter is added to the auto generated connection tracking rules, or that another method is used to disable connection tracking since the current placement into the raw filter table is not working (and causing very strange problems which are hard to track down!). For now I have simply made manual copies of these rules and placed them at the top of the list in case of future reboots.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 953
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: BGP, disappearing routes and raw filter rules

Thu Sep 02, 2021 6:23 pm

Do You user Winbox to manage the router? I have seen reports here that it can change the rules order, if the screen is refresh intensive. Look for it in the forum - even version 3.29 has it, looks like.

This may be the cause of your "roaming rule".

Who is online

Users browsing this forum: Bing [Bot], karlisi, kivimart, Omerik, peterda and 114 guests