Today I got a TCP SYN Flood attack on one of my clients.
The attack wasn't that big (~150-200kpps and a few hundred mbps) but it managed to bring almost everything down. The uplink is 10gbit so it was nowhere near physical medium congestion and I confirmed that my upstream did not had any congestion issues either (ie: the attack was not volumetric)
Until FastNetMon kicked in to blackhole the victim's IP, the router was essentially unreachable and everything behind it either had high latency or packet loss.
Winbox and SSH could not connect at all.
I use raw filters to no-track all the forwarded traffic. With UDP attacks even with 1mpps and 7-8gps the router wouldn't even sweat! no-track works like a charm for these attacks!
But with TCP SYN Flood the router just freaked out.
I don't use TCP SYN Cookies nor connection tracking.
For a brief moment I was able to connect with winbox and I saw that most of the cpu usage was on 'networking'.
The router is a CCR1036-8G-2S+ running ROS v6.38.5 with FW v3.33.
Does anyone else has had similar experience with this?
Was this normal? Shouldn't the router just forward the traffic? Why did SYN Flood cause so much cpu usage?
Any tips on how to avoid high CPU usage on future SYN Flood attacks?
Re: TCP SYN Flood attack causing high cpu
I forgot to mention that by dropping the traffic (using raw rules) the cpu usage would go down to 0% again.
But I don't find this to be a robust solution. Since I don't do any connection tracking or have congestion at the uplink level the router should just forward the packets. I don't see why it would have increased CPU usage only for TCP SYN Flood (as I mentioned UDP attacks pass through fine without increased CPU usage).
But I don't find this to be a robust solution. Since I don't do any connection tracking or have congestion at the uplink level the router should just forward the packets. I don't see why it would have increased CPU usage only for TCP SYN Flood (as I mentioned UDP attacks pass through fine without increased CPU usage).
Re: TCP SYN Flood attack causing high cpu
Any ideas, anyone? 
Re: TCP SYN Flood attack causing high cpu
I've seen this same thing on x86 RouterOS (although it did handle more traffic than your CCR). I believe this is Linux kernel related. No reason the router should blow up but it does. I didn't have specific RAW rules in place so maybe in my case with more CPU available I'll get it to a reasonable point.
I'm planning to do more thorough testing, just need to find the time. Unless someone out there can save me the effort and break it down for us?
I'm planning to do more thorough testing, just need to find the time. Unless someone out there can save me the effort and break it down for us?
-
-
shadowskippie
Member Candidate
- Posts: 213
- Joined:
Re: TCP SYN Flood attack causing high cpu
I use this, It's not perfect.
Code: Select all
add action=jump chain=Input comment="Syn Attack Protection" connection-state=new jump-target=Syn_Protection protocol=tcp src-address-list=\
High_Connections tcp-flags=syn
add action=accept chain=Syn_Protection limit=1k,10 protocol=tcp
add action=drop chain=Syn_Protection protocol=tcp
add action=add-src-to-address-list address-list=High_Connections address-list-timeout=1d chain="Intrazone Untrust Input" connection-limit=100,32Re: TCP SYN Flood attack causing high cpu
This is not a solution IMHO.
I am not looking to filter the attack. I want to forward it without bringing the whole router down.
200kpps and 70-100mbit traffic for a CCR is too little to cause 100% CPU on all 36 cores especially when using RAW filters to not track any of the forwarded traffic (which the whole point of it is to deal with DDoS attacks).
I am not looking to filter the attack. I want to forward it without bringing the whole router down.
200kpps and 70-100mbit traffic for a CCR is too little to cause 100% CPU on all 36 cores especially when using RAW filters to not track any of the forwarded traffic (which the whole point of it is to deal with DDoS attacks).
Re: TCP SYN Flood attack causing high cpu
have you tried removing any rule to activate fast path and let fastnemon blackhole traffic?
Why not considering contact a DDoS protected company to use for incoming traffic forwarded trought gre tunnel? You will save lot of headache
Why not considering contact a DDoS protected company to use for incoming traffic forwarded trought gre tunnel? You will save lot of headache
Re: TCP SYN Flood attack causing high cpu
The problem is most likely with the linux kernel and how it handles SYN packets. No matter what the rules in firewall it made no difference on my tests (I mean for better, because firewall can make it way worse depending on the rules obviously).
The main load is on the 'networking' process in Tools / Profile. Firewall doesn't seem to play any role whatsoever when properly (un)configured.
Also with rp_filter enabled it causes extra cpu load on the routing process.
The only thing that drops the cpu load down immediately is dropping the SYN Flood with a raw rule.
Trying to forward the traffic simply brings down the router (management-wise. It still forwards packets but with increased latency and packet loss).
Fastnetmon doesn't block traffic. It's not in the data path. It talks to the edge routers either via BGP or API and tells them what to block.
Fastnetmon is of no concern here. The problem is the high cpu usage (and ultimately total lockout) when trying to forward even a moderate TCP SYN Flood not how to mitigate it.
The main load is on the 'networking' process in Tools / Profile. Firewall doesn't seem to play any role whatsoever when properly (un)configured.
Also with rp_filter enabled it causes extra cpu load on the routing process.
The only thing that drops the cpu load down immediately is dropping the SYN Flood with a raw rule.
Trying to forward the traffic simply brings down the router (management-wise. It still forwards packets but with increased latency and packet loss).
Fastnetmon doesn't block traffic. It's not in the data path. It talks to the edge routers either via BGP or API and tells them what to block.
Fastnetmon is of no concern here. The problem is the high cpu usage (and ultimately total lockout) when trying to forward even a moderate TCP SYN Flood not how to mitigate it.
Re: TCP SYN Flood attack causing high cpu
Yes i know the conseguencies on MT. DDoS Mitigation is my job
If you want help your router to support 2x DDoS you're receiving now, disable route cache. You will see your cpu usage immediately goes down.
Put rp_filter in loose mode and enable tcp syncookie.
Set (only if you use router as border one and you not do nat or similar services)
/ip firewall connection tracking set enabled=no
Use only raw rules and setup something like this:
You will block most know UDP Amplification script.
this is the best configuration we found to allow MT absorb attacks, you can't get better performance.
Now to do real tcp mitigation you should apply an external device (in line or out of line is your choice) to filter some more specific packets (strings, ttl, flags...). If you not feel safe to use in line, consider to use fastnemon that detect a ddos and inject a route to forward /32 to that device.
Or if you have a budget, choose a company that does ddos mitigation and you will sleep better
If you want help your router to support 2x DDoS you're receiving now, disable route cache. You will see your cpu usage immediately goes down.
Put rp_filter in loose mode and enable tcp syncookie.
Set (only if you use router as border one and you not do nat or similar services)
/ip firewall connection tracking set enabled=no
Use only raw rules and setup something like this:
Code: Select all
/ip firewall raw
add chain=prerouting action=jump jump-target=udp-filters in-interface=NETIX log=no log-prefix="" protocol=udp
add chain=prerouting action=jump jump-target=tcp-filters in-interface=NETIX log=no log-prefix="" protocol=tcp
add chain=udp-filters action=accept in-interface=NETIX src-port=53 limit=2500,100:packet log=no log-prefix="" protocol=udp
add chain=udp-filters action=drop in-interface=NETIX src-port=53 log=no log-prefix="" protocol=udp
add chain=udp-filters action=drop in-interface=NETIX src-port=389 log=no log-prefix="" protocol=udp comment=LDAP
add chain=udp-filters action=drop in-interface=NETIX src-port=80 log=no log-prefix="" protocol=udp comment="UDP SRC 80"
add chain=udp-filters action=drop in-interface=NETIX src-port=443 log=no log-prefix="" protocol=udp comment="UDP SRC 443"
add chain=udp-filters action=drop in-interface=NETIX dst-port=80 log=no log-prefix="" protocol=udp comment="UDP DST 80"
add chain=udp-filters action=drop in-interface=NETIX dst-port=443 log=no log-prefix="" protocol=udp comment="UDP DST 443"
add chain=udp-filters action=notrack log=no log-prefix=""
add chain=tcp-filters action=notrack log=no log-prefix=""
add chain=prerouting action=notrack log=no log-prefix=""
/ip firewall filter
add chain=forward protocol=tcp tcp-flags=syn,rst action=drop
this is the best configuration we found to allow MT absorb attacks, you can't get better performance.
Now to do real tcp mitigation you should apply an external device (in line or out of line is your choice) to filter some more specific packets (strings, ttl, flags...). If you not feel safe to use in line, consider to use fastnemon that detect a ddos and inject a route to forward /32 to that device.
Or if you have a budget, choose a company that does ddos mitigation and you will sleep better
Re: TCP SYN Flood attack causing high cpu
Why didn't you use tcp syn cookie?
/ip settings set tcp-syncookies=yes
/ip settings set tcp-syncookies=yes
Re: TCP SYN Flood attack causing high cpu
With syncookie you ask to routeros to be a proxy and it help a bit on syn floods.
Another trick we added is to put in mangle some tcp checks to mark and put ip in blacklist.
You can put any mark rule to put ip in address list. With that you can drop packets from src ip or blackhole it.
Another trick we added is to put in mangle some tcp checks to mark and put ip in blacklist.
Code: Select all
/ip firewall> mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=add-src-to-address-list tcp-flags=syn,rst protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""
1 X chain=prerouting action=add-src-to-address-list tcp-flags=!fin,!syn,!rst,!ack protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""
-
-
texmeshtexas
Member Candidate
- Posts: 151
- Joined:
Re: TCP SYN Flood attack causing high cpu
that first mangle rule
/ip firewall> mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=add-src-to-address-list tcp-flags=syn,rst protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""
sure seems to put alot of addresses on the list instantly. Does this require both syn AND rst flags to be set or is it syn OR rst?
/ip firewall> mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=add-src-to-address-list tcp-flags=syn,rst protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""
sure seems to put alot of addresses on the list instantly. Does this require both syn AND rst flags to be set or is it syn OR rst?
Re: TCP SYN Flood attack causing high cpu
It is true that the logical operation between the items on the list is not obvious, but the answer to "what logical operation between the values on the list is used for this particular field?" is "the more useful one". So while for dst-port=a,b (and most other match conditions), the only operation to make sense is "or" (so the condition expands to dst-port=a or dst-port=b), in case of tcp-flags=a,b, the operation useful in most cases is "and" (so the condition expands to tcp-flags.a = true and tcp-flags.b = true). So if you would (theoretically) want to do take the same action for tcp-flags.syn=true or tcp.flags.rst=true, you would have to use two rules, one with tcp-flags=syn and another one with tcp-flags=rst. Also bear in mind that a mere tcp-flags=syn does not care about the other flags, so if you want to match packets which have only the SYN flag set, you have to use tcp-flags=syn,!ack,!cwr,!ece,!fin,!psh,!rst,!urg.Does this require both syn AND rst flags to be set or is it syn OR rst?
Re: TCP SYN Flood attack causing high cpu
Almost two years have passed, and absolutely nothing has changed.
CCRs still cannot route (not drop) a moderate flood of SYN packets.
CCRs still cannot route (not drop) a moderate flood of SYN packets.
-
-
edwinlai33
newbie
- Posts: 28
- Joined:
Re: TCP SYN Flood attack causing high cpu
Strongly agree, suffered many timesAlmost two years have passed, and absolutely nothing has changed.
CCRs still cannot route (not drop) a moderate flood of SYN packets.
-
-
edwinlai33
newbie
- Posts: 28
- Joined:
Re: TCP SYN Flood attack causing high cpu
did you enable FastPath? if enable , it should not consume cpuAlmost two years have passed, and absolutely nothing has changed.
CCRs still cannot route (not drop) a moderate flood of SYN packets.
Re: TCP SYN Flood attack causing high cpu
FastPath needs Route Cache to be enabled.
With route cache enabled, the router becomes almost completely unresponsive during any moderate SYN flood attack.
With route cache enabled, the router becomes almost completely unresponsive during any moderate SYN flood attack.
-
-
edwinlai33
newbie
- Posts: 28
- Joined:
Re: TCP SYN Flood attack causing high cpu
if no fastpath in CCR, the performance is worse than X86 when suffer similar attack.
Re: TCP SYN Flood attack causing high cpu
Thank you for your opinion.
Re: TCP SYN Flood attack causing high cpu
we experiance the same high cpu load on ccr and moderate syn-flood.
We were able to catch the attackers in some scenarios and filter them in raw. But in one case we are not able to catch them:
- when the attacker uses all the ip of some /22 to all our IPs so they have a low number of syn packet rate for each src ip.
Any idea on how to catch them?
regards
Ros
We were able to catch the attackers in some scenarios and filter them in raw. But in one case we are not able to catch them:
- when the attacker uses all the ip of some /22 to all our IPs so they have a low number of syn packet rate for each src ip.
Any idea on how to catch them?
regards
Ros
Re: TCP SYN Flood attack causing high cpu
Yes, viewtopic.php?f=2&t=152953
Any idea on how to catch them?
regards
Ros
Set your own treshold and it will create blocks of /24 and you can combine those later in list-2 manually to larger ranges if needed.
You need the schedule, 2 minutes or a bit more, to call the script. Then two lines, one to add IP addresses to list-1 and the blocker line (list-2) in RAW.
-
-
lowprofiler
just joined
- Posts: 4
- Joined:
Re: TCP SYN Flood attack causing high cpu
Could anyone share some experience about this issue? I had issues for several years ago with my CCR1036 which coudn't sustain any SYN attack even though we are talking about 3-10mbit. The router becomes the bottleneck and everyone is offile.
I really can't understand how people are using these routers in enterprise environment when it "lacks" of this simple protection - I have tried each and every setting available - same result almost, atleast not something i can accept. Volume attack like UDP is not a problem, only SYN attacks with special flags.
Please share your experiences. The same problem is also actual to PFsense :/ - Time to switch to Cisco or other brand?
I really can't understand how people are using these routers in enterprise environment when it "lacks" of this simple protection - I have tried each and every setting available - same result almost, atleast not something i can accept. Volume attack like UDP is not a problem, only SYN attacks with special flags.
Please share your experiences. The same problem is also actual to PFsense :/ - Time to switch to Cisco or other brand?
Re: TCP SYN Flood attack causing high cpu
Hi guys
i know this is an old thread, but somehow fits my problem description we are getting the same flavour of attack.. both 1016 and 1036 max out cpu.. BGP and PPPOE servers..
the attacks only last average 20 seconds.. then stop.. our main backhaul internet provider told us that the attack monitored on their traffic monitoring tools reached 42.1gbps around 7,2mpps
the latest around 30gbps attacks.. somehow they are attacking all ASNs swapping traffic with this backhaul ISP link supplier, basically all smaller ISPs clients with ASN are getting attacked.. attacks often happend 3 to 4x per day on specific peak times of internet usage.. lunch, afternoon and evening times.. on same specific timmings.. short periods of 15 to 20 seconds..
on our BGP and CGNAT boxes.. we have seen CPU load reach 100% even with low traffic around 800 to 1gbps traffic.. the highest peak we have managed to log on our traffic flow tool was peak of 1.7gbps udp.
all the mikrotik firewall filter rules and raw rules we have seen on mikrotik site, and other sites.. none worked not even a single packet trapped into syn flood neither tcp or udp packets.. it simply just crashes everything until winbox disconnects and after couple of seconds connection on WAN starts working again..
Any ideas? we use TeamCYMRU UTRS2.0 but not catching anything at all no time to let the mikrotik Think... bare in mind.. that the ports are SFP+... on the backhaul main ISP link provider they have other hardware gear with 100Gbps ports and monitoring tools, they are trying to mitigate on their side also.. but so far unsucesfull as the attacker seems to be using some sort of IP random spoof with random ports.. tcp and udp.. no specific port at all.. and as i have mentioned initially attacks are ramping up over 40gbps traffic amplified on the attack..
i know this is an old thread, but somehow fits my problem description we are getting the same flavour of attack.. both 1016 and 1036 max out cpu.. BGP and PPPOE servers..
the attacks only last average 20 seconds.. then stop.. our main backhaul internet provider told us that the attack monitored on their traffic monitoring tools reached 42.1gbps around 7,2mpps
the latest around 30gbps attacks.. somehow they are attacking all ASNs swapping traffic with this backhaul ISP link supplier, basically all smaller ISPs clients with ASN are getting attacked.. attacks often happend 3 to 4x per day on specific peak times of internet usage.. lunch, afternoon and evening times.. on same specific timmings.. short periods of 15 to 20 seconds..
on our BGP and CGNAT boxes.. we have seen CPU load reach 100% even with low traffic around 800 to 1gbps traffic.. the highest peak we have managed to log on our traffic flow tool was peak of 1.7gbps udp.
all the mikrotik firewall filter rules and raw rules we have seen on mikrotik site, and other sites.. none worked not even a single packet trapped into syn flood neither tcp or udp packets.. it simply just crashes everything until winbox disconnects and after couple of seconds connection on WAN starts working again..
Any ideas? we use TeamCYMRU UTRS2.0 but not catching anything at all no time to let the mikrotik Think... bare in mind.. that the ports are SFP+... on the backhaul main ISP link provider they have other hardware gear with 100Gbps ports and monitoring tools, they are trying to mitigate on their side also.. but so far unsucesfull as the attacker seems to be using some sort of IP random spoof with random ports.. tcp and udp.. no specific port at all.. and as i have mentioned initially attacks are ramping up over 40gbps traffic amplified on the attack..
Re: TCP SYN Flood attack causing high cpu
It would seem this issue has HIGH IMPACT, but LOW OCCURRENCE rate. In other words only a few users seem to suffer this attack but it has significant consequences.
The question I have is what are MT Coders putting into the OS, that will help those with this issue? Clearly the available routers settings have not done much in this regard and should have something more effective available to users. OR is this users not knowing how to setup the router (highly unlikely).
My understanding is that in general the MT OS is unable to cope effectively with any sort of attacks at all and to ask a water bottle to put out a forest fire is ridonkulous.
The question I have is what are MT Coders putting into the OS, that will help those with this issue? Clearly the available routers settings have not done much in this regard and should have something more effective available to users. OR is this users not knowing how to setup the router (highly unlikely).
My understanding is that in general the MT OS is unable to cope effectively with any sort of attacks at all and to ask a water bottle to put out a forest fire is ridonkulous.
Who is online
Users browsing this forum: urbok and 83 guests