Finally figured out an issue we've been having.

CCR 1072 has well over 1 million active connections, set the timeout to 10 minutes, still over 1 million active connnections.

I noticed that two IP addresses make up the majority of those addresses, they're hammering on all of our subnets. The port scanner portion of the firewall is blocking them but they're still being listed under the connections tab and take up massive resources, completely bog down the 72 core router and use a ton of RAM.

Black hole the addresses seems to help but aside from manually watching the router what can be done?

The firewall filter forward rules are preventing them from doing any damage but they are still using nearly a million connections and causing issues with the router.

You can move the filter from the forward/input filter to the prerouting filter in raw.
That will solve your connection tracking issue, but BE CAREFUL: when somehow a legitimate server address ends up on the blacklist, you/your clients will no longer be able to communicate with that server.
Hackers are sending "portscans" with spoofed source addresses like 1.1.1.1 and 8.8.8.8 so those servers will get on your blacklist and customers will not be happy about that.
You need to have some form of whitelist in the automatic blacklist rules to avoid that, and even then it is not certain that they will not spoof addresses you did not think about.

Prerouting doesn't show up in filter on mine, only mangle.

Tried typing it manually, no change. As soon as I turn off the blackhole the connections jumps from 150K to 400K+ and keeps climbing.

You need to look in the "raw" instead of "filter" firewall.

Thanks.

That's the part I didn't catch.

Been using Mikrotik since 2007 and have never used that part of the firewall.

Always something left to learn.

Okay, now I can catch the traffic but when I enable the drop rule on Raw it drops way too much legit traffic.

If I don't use the drop rule then they still overwhelm the router even with the Filter drop rule in place.

The only thing that works so far is to blackhole the IPs.

Why do you have connection tracking enabled for those connections to begin with? Sounds like you aren't doing NAT.

If your raw drop rule is set for a specific list, that list shouldn't contain "legit" IPs -> shouldn't drop legit traffic.
If it does, you're doing something wrong.
For example I'm piling up IPs that are hammering my DNS with abusive queries (todays favourite query is for "lavrov.in") and in raw I'm only dropping that specific type of traffic, coming from any of my wan interfaces with any of my local IPs as destination, so I don't get locked out for some reason, spoofed IP or whatever.
firewall/raw: Does the job for now.

The list rules between filter and raw were identical, grabbing port scanners. They created a list called PSD and dropped the traffic.

When I enabled the raw drop rule traffic dropped from 1Gbps to 200Mbps.

I changed the tag to PSD2 and it seems to be working but if I disable the black hole routes for the offending IP addresses then tracking picks back up into the millions again.

Even with tracking disabled it still overwhelms the router and we've still got someone port scanning our entire network creating roughly 1 million active connections.

Um, you didn't move the port scanning detection rules in raw too, did you? that won't work.
You only keep the drop rule in raw, that's it.

highly suggest you use this service and see if it cleans up your issue.
Try it for one month, nothing ventured nothing gained, designed specifically for MT.

https://itexpertoncall.com/promotional/moab.html#prime

If only the drop rule is in raw the port scanner still fills up the connection tracking within about a minute, 1 million + connections.

Disabling connection tracking breaks BGP.

DIsabling connection tracking as such cannot break BGP. What breaks it is the fact that with connection tracking disabled, the action=accept connection-state=established,related stops working, hence the incoming BGP traffic is dropped by further firewall rules.

And yes, a drop rule in raw cannot completely prevent the connection list from growing, because the first few initial packets from each source address still make it beyond the raw stage, i.e. to connection tracking, until the port scanner detector wakes up and adds the source to the address-list used as src-address-list in the action=drop rule in raw.

You run BGP and don't understand how stateful / stateless firewalls work? I second the suggestion to get a consultant (though not the one above that is also a useless blacklist). You're clearly in over your head here. Using PSD just opens you to further attack when someone decides to spoof the IP of your BGP peers or DNS servers etc.

DIsabling connection tracking as such cannot break BGP. What breaks it is the fact that with connection tracking disabled, the action=accept connection-state=established,related stops working, hence the incoming BGP traffic is dropped by further firewall rules.

When connection tracking was disabled the BGP handshake never took place. Disabling connection tracking broke BGP in 6.45.9. It would not connect. I agree this is not normal behavior, I've read several articles this week that recommend disabling connection tracking while using BGP.

You run BGP and don't understand how stateful / stateless firewalls work? I second the suggestion to get a consultant (though not the one above that is also a useless blacklist). You're clearly in over your head here. Using PSD just opens you to further attack when someone decides to spoof the IP of your BGP peers or DNS servers etc.

I've been doing this since 2007. It's never come up.

I agree I need help, that's why I came here and asked for help.

and yeah, for the record we do have NAT in play.

You run BGP and don't understand how stateful / stateless firewalls work? I second the suggestion to get a consultant (though not the one above that is also a useless blacklist). You're clearly in over your head here. Using PSD just opens you to further attack when someone decides to spoof the IP of your BGP peers or DNS servers etc.

Well, if my upstream provider would not take appropriate actions to prevent this its time to find another one. I cannot image that an ISP would not filter this on its borders. It should never be possible that a spoofed packet, carrying the source-IP resembling ANY IP of that ISP (who you are peering with) to enter its network through his peers.
If this spoofing activity would take place inside the ISP's own network its also time to find a new ISP because then they clearly don't have their act together.

When connection tracking was disabled the BGP handshake never took place. Disabling connection tracking broke BGP in 6.45.9. It would not connect. I agree this is not normal behavior, I've read several articles this week that recommend disabling connection tracking while using BGP.

Well, over time, I've read many posts recommending to disable firewall completely in order to make xyz work, but that doesn't mean that it is the right thing to do. It's internet, anyone can publish anything and there is no ultimate authority to tell which thoughts are gems and which are crap.

I don't think the BGP handshake didn't take place; I think it wasn't successful.

If your firewall rules form up a stateful firewall, some other things may stop working as well if you disable connection tracking, because it's the connection tracking what provides the statefulness.

You cannot get a useful analysis without providing enough data. In this particular case, without seeing your actual firewall rules, it is not possible to suggest how to modify them so that you could disable connection tracking and still have BGP working. On the other hand, as you use NAT, you cannot disable connection tracking anyway, as NAT depends on connection tracking.

You run BGP and don't understand how stateful / stateless firewalls work? I second the suggestion to get a consultant (though not the one above that is also a useless blacklist). You're clearly in over your head here. Using PSD just opens you to further attack when someone decides to spoof the IP of your BGP peers or DNS servers etc.

I've been doing this since 2007. It's never come up.

I agree I need help, that's why I came here and asked for help.

and yeah, for the record we do have NAT in play.

See link below, pick one closest to you. These consultants pay a lot of money to get certified and reason they are there is to assist in situations like this.

https://mikrotik.com/consultants

Like I said, I provided you to a link with a chap that will help you out.
You can choose to be smart, or be a donkey, which is no relation to a llama ;-)
(from one oldman to another)

I've got it situated.

Thanks.

No worries, been there thats all..........