Community discussions

 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Topic Author
Posts: 817
Joined: Tue Oct 11, 2005 4:53 pm

TCP SYN Flood attack causing high cpu

Wed Oct 04, 2017 6:35 pm

Today I got a TCP SYN Flood attack on one of my clients.

The attack wasn't that big (~150-200kpps and a few hundred mbps) but it managed to bring almost everything down. The uplink is 10gbit so it was nowhere near physical medium congestion and I confirmed that my upstream did not had any congestion issues either (ie: the attack was not volumetric)

Until FastNetMon kicked in to blackhole the victim's IP, the router was essentially unreachable and everything behind it either had high latency or packet loss.

Winbox and SSH could not connect at all.

I use raw filters to no-track all the forwarded traffic. With UDP attacks even with 1mpps and 7-8gps the router wouldn't even sweat! no-track works like a charm for these attacks!
But with TCP SYN Flood the router just freaked out.

I don't use TCP SYN Cookies nor connection tracking.

For a brief moment I was able to connect with winbox and I saw that most of the cpu usage was on 'networking'.

The router is a CCR1036-8G-2S+ running ROS v6.38.5 with FW v3.33.

Does anyone else has had similar experience with this?
Was this normal? Shouldn't the router just forward the traffic? Why did SYN Flood cause so much cpu usage?

Any tips on how to avoid high CPU usage on future SYN Flood attacks?
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Topic Author
Posts: 817
Joined: Tue Oct 11, 2005 4:53 pm

Re: TCP SYN Flood attack causing high cpu

Wed Oct 04, 2017 6:56 pm

I forgot to mention that by dropping the traffic (using raw rules) the cpu usage would go down to 0% again.

But I don't find this to be a robust solution. Since I don't do any connection tracking or have congestion at the uplink level the router should just forward the packets. I don't see why it would have increased CPU usage only for TCP SYN Flood (as I mentioned UDP attacks pass through fine without increased CPU usage).
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Topic Author
Posts: 817
Joined: Tue Oct 11, 2005 4:53 pm

Re: TCP SYN Flood attack causing high cpu

Fri Oct 06, 2017 3:41 pm

Any ideas, anyone? :(
 
jd603
just joined
Posts: 7
Joined: Tue Dec 23, 2014 4:41 am

Re: TCP SYN Flood attack causing high cpu

Mon Oct 09, 2017 11:50 pm

I've seen this same thing on x86 RouterOS (although it did handle more traffic than your CCR). I believe this is Linux kernel related. No reason the router should blow up but it does. I didn't have specific RAW rules in place so maybe in my case with more CPU available I'll get it to a reasonable point.

I'm planning to do more thorough testing, just need to find the time. Unless someone out there can save me the effort and break it down for us? :-)
 
User avatar
shadowskippie
Member Candidate
Member Candidate
Posts: 211
Joined: Tue Dec 21, 2010 6:20 pm

Re: TCP SYN Flood attack causing high cpu

Tue Oct 10, 2017 10:07 am

I use this, It's not perfect.

add action=jump chain=Input comment="Syn Attack Protection" connection-state=new jump-target=Syn_Protection protocol=tcp src-address-list=\
    High_Connections tcp-flags=syn
add action=accept chain=Syn_Protection limit=1k,10 protocol=tcp
add action=drop chain=Syn_Protection protocol=tcp
add action=add-src-to-address-list address-list=High_Connections address-list-timeout=1d chain="Intrazone Untrust Input" connection-limit=100,32
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Topic Author
Posts: 817
Joined: Tue Oct 11, 2005 4:53 pm

Re: TCP SYN Flood attack causing high cpu

Tue Oct 10, 2017 1:03 pm

This is not a solution IMHO.

I am not looking to filter the attack. I want to forward it without bringing the whole router down.

200kpps and 70-100mbit traffic for a CCR is too little to cause 100% CPU on all 36 cores especially when using RAW filters to not track any of the forwarded traffic (which the whole point of it is to deal with DDoS attacks).
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: TCP SYN Flood attack causing high cpu

Tue Oct 17, 2017 12:31 am

have you tried removing any rule to activate fast path and let fastnemon blackhole traffic?

Why not considering contact a DDoS protected company to use for incoming traffic forwarded trought gre tunnel? You will save lot of headache
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Topic Author
Posts: 817
Joined: Tue Oct 11, 2005 4:53 pm

Re: TCP SYN Flood attack causing high cpu

Tue Oct 17, 2017 12:49 am

The problem is most likely with the linux kernel and how it handles SYN packets. No matter what the rules in firewall it made no difference on my tests (I mean for better, because firewall can make it way worse depending on the rules obviously).
The main load is on the 'networking' process in Tools / Profile. Firewall doesn't seem to play any role whatsoever when properly (un)configured.
Also with rp_filter enabled it causes extra cpu load on the routing process.

The only thing that drops the cpu load down immediately is dropping the SYN Flood with a raw rule.
Trying to forward the traffic simply brings down the router (management-wise. It still forwards packets but with increased latency and packet loss).

Fastnetmon doesn't block traffic. It's not in the data path. It talks to the edge routers either via BGP or API and tells them what to block.
Fastnetmon is of no concern here. The problem is the high cpu usage (and ultimately total lockout) when trying to forward even a moderate TCP SYN Flood not how to mitigate it.
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: TCP SYN Flood attack causing high cpu

Tue Oct 17, 2017 1:08 am

Yes i know the conseguencies on MT. DDoS Mitigation is my job :-)

If you want help your router to support 2x DDoS you're receiving now, disable route cache. You will see your cpu usage immediately goes down.

Put rp_filter in loose mode and enable tcp syncookie.

Set (only if you use router as border one and you not do nat or similar services)
/ip firewall connection tracking set enabled=no

Use only raw rules and setup something like this:
/ip firewall raw
add    chain=prerouting action=jump jump-target=udp-filters in-interface=NETIX log=no log-prefix="" protocol=udp

add    chain=prerouting action=jump jump-target=tcp-filters in-interface=NETIX log=no log-prefix="" protocol=tcp

add   chain=udp-filters action=accept in-interface=NETIX src-port=53 limit=2500,100:packet log=no log-prefix="" protocol=udp

add    chain=udp-filters action=drop in-interface=NETIX src-port=53 log=no log-prefix="" protocol=udp

add  chain=udp-filters action=drop in-interface=NETIX src-port=389 log=no log-prefix="" protocol=udp comment=LDAP

add  chain=udp-filters action=drop in-interface=NETIX src-port=80 log=no log-prefix="" protocol=udp comment="UDP SRC 80"

add  chain=udp-filters action=drop in-interface=NETIX src-port=443 log=no log-prefix="" protocol=udp comment="UDP SRC 443"

add  chain=udp-filters action=drop in-interface=NETIX dst-port=80 log=no log-prefix="" protocol=udp comment="UDP DST 80"

add  chain=udp-filters action=drop in-interface=NETIX dst-port=443 log=no log-prefix="" protocol=udp comment="UDP DST 443"

add    chain=udp-filters action=notrack log=no log-prefix=""

add    chain=tcp-filters action=notrack log=no log-prefix=""

add    chain=prerouting action=notrack log=no log-prefix=""

/ip firewall filter

add chain=forward protocol=tcp tcp-flags=syn,rst action=drop
You will block most know UDP Amplification script.

this is the best configuration we found to allow MT absorb attacks, you can't get better performance.

Now to do real tcp mitigation you should apply an external device (in line or out of line is your choice) to filter some more specific packets (strings, ttl, flags...). If you not feel safe to use in line, consider to use fastnemon that detect a ddos and inject a route to forward /32 to that device.

Or if you have a budget, choose a company that does ddos mitigation and you will sleep better
 
User avatar
Anumrak
Forum Veteran
Forum Veteran
Posts: 855
Joined: Fri Jul 28, 2017 2:53 pm

Re: TCP SYN Flood attack causing high cpu

Thu Oct 26, 2017 11:58 am

Why didn't you use tcp syn cookie?
/ip settings set tcp-syncookies=yes
 
berlo
newbie
Posts: 43
Joined: Sat May 13, 2017 5:11 pm

Re: TCP SYN Flood attack causing high cpu

Thu Oct 26, 2017 1:09 pm

With syncookie you ask to routeros to be a proxy and it help a bit on syn floods.

Another trick we added is to put in mangle some tcp checks to mark and put ip in blacklist.
/ip firewall> mangle print
Flags: X - disabled, I - invalid, D - dynamic
 0    chain=prerouting action=add-src-to-address-list tcp-flags=syn,rst protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""

 1 X  chain=prerouting action=add-src-to-address-list tcp-flags=!fin,!syn,!rst,!ack protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""
You can put any mark rule to put ip in address list. With that you can drop packets from src ip or blackhole it.
 
texmeshtexas
newbie
Posts: 33
Joined: Sat Oct 11, 2008 11:17 pm

Re: TCP SYN Flood attack causing high cpu

Sun Aug 12, 2018 8:25 am

that first mangle rule
/ip firewall> mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=add-src-to-address-list tcp-flags=syn,rst protocol=tcp address-list=ddos-source address-list-timeout=6h in-interface=NETIX log=no log-prefix=""
sure seems to put alot of addresses on the list instantly. Does this require both syn AND rst flags to be set or is it syn OR rst?
 
sindy
Forum Guru
Forum Guru
Posts: 2441
Joined: Mon Dec 04, 2017 9:19 pm

Re: TCP SYN Flood attack causing high cpu

Sun Aug 12, 2018 12:02 pm

Does this require both syn AND rst flags to be set or is it syn OR rst?
It is true that the logical operation between the items on the list is not obvious, but the answer to "what logical operation between the values on the list is used for this particular field?" is "the more useful one". So while for dst-port=a,b (and most other match conditions), the only operation to make sense is "or" (so the condition expands to dst-port=a or dst-port=b), in case of tcp-flags=a,b, the operation useful in most cases is "and" (so the condition expands to tcp-flags.a = true and tcp-flags.b = true). So if you would (theoretically) want to do take the same action for tcp-flags.syn=true or tcp.flags.rst=true, you would have to use two rules, one with tcp-flags=syn and another one with tcp-flags=rst. Also bear in mind that a mere tcp-flags=syn does not care about the other flags, so if you want to match packets which have only the SYN flag set, you have to use tcp-flags=syn,!ack,!cwr,!ece,!fin,!psh,!rst,!urg.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.

Who is online

Users browsing this forum: avn, sindy and 51 guests