Traffic drops with BGP routing

RouterOS Forwarding Protocols
1

I have a strange problem that I wanted to get input on.
I have two edge routers. One is peered to one upstream ISP1 and the other to another ISP2. Both using default route (not full routes)
I want to advertise all prefixes to both ISPs but I get packet drops that cause certain sites to not load right (facebook, youtube, vpn connections).
Both routers are CCR1072 V6.49.7
Here is a diagram to show how things are connected.

This is a simplified diagram as the real network has many more downstream sites/prefixes/etc. But it captures the basic idea and problem.

Anyone have an idea why I get packet drops?
It is site dependent, if traffic happens to come back in the same router that it went out, not problem.
Screenshot 2022-10-27 091942.jpg

2

In case of asymmetric routing make sure that you are not using any of the connection tracking related features.

3

Do have a bunch of fw filter rules that rely on conn tracking. Traffic is not natted.
thank that could still be the problem?

4

make sure you dont have rp-filter enabled

check it on

ip → settings

also try disabling any firewall rule in chain forward dropping invalid packets

5

rp-filter should be set to loose mode for assymetric routing.

But overall, assymetric routing is bad traffic engineering. What you should do is announce the largest possible aggregates of all your prefixes to both upstreams, equally without prepending or more specifics.

More specifics should be used for DDoS blackholing/scrubbing instead during DDoS mitigation/detection.

Then use something like PathVector for traffic engineering in an automated fashion.

6

Absolutely.
NAT is not relevant to the problem. Doing connection tracking (and thus accepting established/related connections and somewhere down the road dropping the remaining traffic) is.

When the traffic returns on another router, that router knows nothing about that connection, so it is not established/related by its standpoint, thus it gets dropped.

7

RP was set to loose and I have default route installed from upstream BGP peer so really does nothing. Turned off for now.

8

yeah, I do accept est/rel traffic but I dont drop other traffic, just check it against rules like the rate of inbound connections being too high, DNS request coming.
I may try turning any drop rule off and see if it helps.

9

thanks, not dropping invalid.

10

rp-filter is on loose.
Yeah, that is what I’m trying to work towards. announcing all prefixes to both upstreams without prepending. and I announce DDoS blackholing out both sides also.
Not sure how PathVector or similar would fit in.

11

Step 1: Fix the issue you’re facing here
Step 2: Get full tables from upstream transits
Step 3: Enable loose mode rp-filter
Step 4: Do not use stateful-ness on edge routers
Step 5: Announce prefixes aggregated, equally, no prepends to all transits, IXPs and also PNIs/Bilaterals.
Step 6: Install PathVector on-prem server and configure it to auto traffic engineer your routing based on latency and other parameters of your choice.

For other best practices, follow the guide for ISPs:
http://forum.mikrotik.com/t/how-to-edge-router-and-bng-optimization-for-isps/150007/1

12

I found what I think is the primary issue. As I’m using a collapsed config at the edge, my core network and edge and NAT are the same router.
I have some firewall rules to help protect the network from bogons, bad ports, etc.
one of the rules, and I thought this one was gone long ago, was a PSD (ports scan) rule that was killing some traffic. Removed that rule and things are running much better.

I’m working towards separating border from network and NAT. QoE is at data center and BNG is at tower site. NAT is upstream from QoE box and is tough to implement at BNG due to multiple paths that are available to many BNG units.