Hello,

I'm hoping theres an easy answer for this. We are seeing some low end DDOS attacks around 1-2 gbit/sec. The attack signature is SYN packets with random source IPs with a destination within the network that the CCRs do edge routing for. The CPU level is basically 100% the whole time. We did some internal testing, with a 200 megabit sustained syn flood against a host behind a CCR, but with normal traffic we can easily route that much without seeing 1 or 2% cup usage. Is there anything we can set on these so they don't fall over past 1gb/sec of syn traffic with random source addresses? Details below:

Output from test running at about 400k PPS / 200 megabits per second:

CPU usage: 20%

output of profile:

[admin@MikroTik] > /tool profile
NAME CPU USAGE
snmp all 0%
console all 0%
ssh all 0%
networking all 17.1%
winbox all 0%
mpls all 0.1%
management all 0.1%
routing all 5.5%
idle all 76.6%
profiling all 0%
bridging all 0.1%
unclassified all 0.1%

Output of /ip firewall export:

[admin@MikroTik] > /ip firewall export
# jan/04/2015 18:32:36 by RouterOS 6.24
# software id =
#
/ip firewall connection tracking
set enabled=no

/queue export is empty, IP settings print output:

[admin@MikroTik] /ip settings> print
ip-forward: yes
send-redirects: no
accept-source-route: yes
accept-redirects: no
secure-redirects: no
rp-filter: no
tcp-syncookies: no
max-arp-entries: 8192
arp-timeout: 30s
icmp-rate-limit: 10
icmp-rate-mask: 0x1818
allow-fast-path: yes

Hi,

Couple of things to consider, if you attempt to limit TCP SYN packets then there is no guarantee valid packets belonging to real clients won't be dropped anyway causing denial of service. In my opinion limiting SYN traffic is counterproductive.

Building firewall filter rules and possibly traffic queues to deal with 1-2 gbit/s of TCP SYN will also consume CPU.

What TCP port is under attack? If it's port 80 then I would suggest signing up for a service like Cloudflare.

What TCP port is under attack? If it's port 80 then I would suggest signing up for a service like Cloudflare.

The services that are targeted aren't http based unfortunately, so couldflare isn't an option. Our total connectivity is 20 gigabits, so ideally we would like to pass any traffic we recieve to our core layer, which can apply traffic ACLs at line rate. Really, I just want the CCRs to be able to forward all the packets it receives, including these bogus syn packets, so they don't fall over and lock up the rest of the legitimate traffic coming in.

any thoughts?

here is my test running with 4 xeon server with iperf, packet size 64b, proto udp and 10 parallel connection. It reached 2.7Mpps, cpu usage 25%. Though it showed erroneous cpu usage under winbox title bar after doing the tools profile the cpu usage got corrected too.
Config :
allow fast path off
conntrack on
3 firewall rules (on rule all traffic accepted on forward chain)
3 simple queues with dst 0.0.0.0/0
Screenshots attached.

Looks like i'd hit another soft spot of CCR. On the above test I found that no traffic was passing thru the queues. I disable all queues and enabled just one queue with target 0.0.0.0/0 dst ether6 max limit 900k and there it goes. The cpu usage rose to 100% winbox disconnected no telnet no ssh. I was unable to make support file.

Seeing basically the same issue - CCR cannot handle any tcp+syn type DDoS at all and falls over. 6.18 (our tested most stable version, no plans to upgrade for now since otherwise stable.)

But is a big disappointment. Like CCR otherwise, but may be forced back to ASR or MX gear due to these issues.

Plan to spend some time in the lab testing as well to see if there are any setting or other tweaks that can help prevent this, it sounds like you have already started - please keep us posted with findings since it's unlikely Mikrotik themselves will offer any assistance.

On 6.24 we can handle about 1.1 or 1.2M PPS of syn flood. After that the packet loss causes the router to lock up. I'm hoping the CCR1072 will double that when it finally arrives.

Still, we're probably going to have to move in a different direction for our edge routing since we use 10G ethernet. Otherwise its still a solid product, and we will probably continue to employ these elsewhere in our network where this type of traffic can be mitigated upstream. Its worth mentioning that Vyatta has similar issues, so I'm thinking this type of thing is probably related to linux networking as a whole. We are going to try a few alternative options, and I'll report back within a few weeks on what we find.

1.2 Mpps (syn) is not a bad amount of load on ccr 1036 but too bad if it hits any of the simple queue and see ccr 1036 bouncing on the table

Until Mikrotik improve the routing core and optimise the packet flow on the CCR to balance across all CPU cores I won't be using any of the CCR range for deployments above 1 Gigabit per port.

The 72 core CCR which I believe is 10 GbE only looks great on paper but will be pointless unless released with ROS 7 and able to route port to port at 20 Gb full wirespeed and 20+ mpps with 64 byte packets.

In our environment we simply either drop SYN DDOS attacks above 1 mpps at the edge of the network or use BGP blackhole with our transit providers if the attack is causing issues.

@dottxt I'd be interested to know what service you are trying to protect and your strategy for mitigating an attack of 1-2 mpps?

I completely agree that routerOS has trouble routing past 30Gb/s. I tested this using a bridge and some weird software only setups within the CCR itself by attempting to test how much bandwidth it can really handle using 65535 MTU bridge and same sized packets with everything done internally so the physical ports arent involved. It also helps prevent disconnection when using traffic generator to a totally seperate and unused interface.

As for preventing attacks i found an interesting solution that works if you are not expecting random incoming traffic.
First create a whitelist/some way to access ports and services you need.
Detect and whitelist your clients (either by authorisation or L2 + src and port)
Temporarely whitelist their outgoing connections
tarpit and drop the rest.

You still will need firewall rules for the whitelisted stuff but you can just blacklist external ips that attempt to access protected services randomly and just tarpit and drop all traffic from them. If you are worried that the attacker IP is a shared service you can just use temporarely blacklist. Better to have a website temporarely unavailable than an entire network slow down. I never use rate detection because it makes things more complicated especially if you expect a lot of traffic such as from a compute cluster.

I use this because my network setup is very complicated where i cannot say drop input from WAN because in my config there is no WAN port and internal and non internal traffic need to communicate. Its just my ISP that is weird. I not only have to protect against internet traffic but against LAN traffic from their other customers because they perform NAT and their port isolation doesnt really work. I cannot blindly place an entire ip segment (LAN) in whitelist either because my WAN port is connected to a big LAN where my ISP does the NAT and DHCP.

I think there are examples in mikrotik wiki on stopping syn attacks.

Is this still a problem?