Page 1 of 1

suggestion for a 5 gig router after testing ccr 1036

Posted: Fri Sep 19, 2014 8:04 pm
by hashbang
I installed ccr 1036-8g-2s+, a brief explanation of config :
1) 2 bgp peers + 6 filters
2) 2 isp (ether 7 & 8)
3) 2 lan outs with one lan having 10 vlans (ether 4 & 5)
4) 41 queues
5) 3 rules for ssh bruteforcing protection 1 rule for packet marking.
all other ports are free
Bandwidth running 800mbps download and 150mbps upload. Router cpu consumption had reached 25% approx. I have few questions as ccr 1072 is not going to be launched soon. They are :
Why such a drastic difference in benchmarks of MT and practical test
What h/w is suggested if I need 5gig of routing (if x86 then which hw)
already ordered 5 ccr-1036
No major stability issue found apart from vlan packet drop (which got solved).
BGP filter behaving strangely (posted somewhere already in bgp section).
This router is already running from 80days. 35days idle, 35days 300mbps and from 14 days full load.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Sat Sep 20, 2014 4:09 am
by Zorro
some part of ROS, on which you config relying on so much - obbviously had scalability issues, as you noticed thats 25% CPU load was peak.
as for simple answer - CCR1072 ? :)

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Sat Sep 20, 2014 8:28 am
by hashbang
which part ? Does MT needs more optimization ? Practical thru is noway near what tilera advertise.
MT is silent about release date of ccr 1072

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Sun Sep 21, 2014 2:15 pm
by joegoldman
you should profile and see what is most consuming. At the moment I run about 10 queues, 2 port bonding with 5 BGP peers (1 full table + IX tables, for IPv4 and IPv6) with about 500-600mbit aggregate peak (100k-130k pps), sits at about 15%. There are 2 packet marking rules per queue set up and also about 20 filter rules.

A big difference may be your packet rate causing higher CPU.

If Queuing is the big performance hit, try ROS6.19 if you haven't, meant to be big improvements. If queueing starts to get too much might be an idea to split out the tasks, assign some CCR's to the edge for BGP and have the queuing routers not having to do big routing lookups etc, so you might be able to get fastpath routing or the like in.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Sun Sep 21, 2014 8:02 pm
by hashbang
Mine is having 200k pps and 950mbps, cpu usage reaching peak 24%. Screenshots below

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Mon Sep 22, 2014 2:19 am
by joegoldman
You say you only have 3 or 4 rules for your firewall, maybe do a /ip firewall export (so filtering, NAT and Mangle) so we can have a look - it might be doing more work than it needs to as the firewall shouldn't be taking so much. My Firewall (with 19 filter rules currently) is usually 3rd or 4th in process list. routing, networking and queuing usually the top.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Mon Sep 22, 2014 9:47 pm
by hashbang
5 rules to be precise. 4 rules for bruteforcing of telnet and ssh as per wiki
1 rule for blocking dst address list which has 142 ip addresses (zeus network block)
1 rule for making packets based on dscp 14
no nat rule

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Mon Sep 22, 2014 10:45 pm
by lambert
Joe asked for :
/ip firewall export
The reason he asked for that is that it is entirely possible for you to write the rules you described in such a way as to spin your CPU for every packet or only when necessary.

You can change IP addresses in the rules for privacy, but showing us exactly what the rules are is the only way to allow us to help you figure out what is wrong.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Tue Sep 23, 2014 10:42 am
by hashbang
/ip firewall filter
add action=drop chain=forward comment="zeus drop" connection-state=new \
dst-address-list=zeus
add action=add-src-to-address-list address-list=level1 address-list-timeout=\
1m chain=input connection-state=new dst-port=22,23 protocol=tcp
add action=add-src-to-address-list address-list=level2 address-list-timeout=\
1m chain=input connection-state=new dst-port=22,23 protocol=tcp \
src-address-list=level1
add action=add-src-to-address-list address-list=level3 address-list-timeout=\
6h chain=input connection-state=new dst-port=22,23 protocol=tcp \
src-address-list=level2
add action=drop chain=input connection-state=new src-address-list=level3
/ip firewall mangle
add action=mark-packet chain=prerouting comment="dscp" dscp=\
14 new-packet-mark=cache_mark passthrough=no

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Tue Sep 23, 2014 9:57 pm
by lambert
/ip firewall filter
add action=drop chain=forward comment="zeus drop" connection-state=new dst-address-list=zeus

add action=add-src-to-address-list address-list=level1 address-list-timeout=1m chain=input connection-state=new dst-port=22,23 protocol=tcp
add action=add-src-to-address-list address-list=level2 address-list-timeout=1m chain=input connection-state=new dst-port=22,23 protocol=tcp src-address-list=level1
add action=add-src-to-address-list address-list=level3 address-list-timeout=6h chain=input connection-state=new dst-port=22,23 protocol=tcp  src-address-list=level2
add action=drop chain=input connection-state=new src-address-list=level3

/ip firewall mangle
add action=mark-packet chain=prerouting comment="dscp" dscp=14 new-packet-mark=cache_mark passthrough=no
I don't see much room for improvement there. You might bring the block rule for level3 above the add rules.

It might be less cpu intensive to jump to a "new input connections" chain on connection-state=new. It would add one step for new connections but would eliminate 3 evaluations and all address-list lookups for all established and related connections. The other option would be to explicitly allow established and related connections before evaluating the connection-state=new rules but that would be two rules instead of one jump rule.

In fact, eliminating the address-list lookups for every established and related connection state packet may be enough of a win to be worth having allow rules for established and related connections before the drop rule on your forward chain. I do not know if the firewall does the address-list lookup concurrent with the connection-state comparison or only if the connection-state matches when they are written into one rule.

On the mangle rule, it might be faster to make a connection-mark with passthrough and then mark packets based on connection mark. Or just add an accept rule for !dscp=14 before the packet mark. Not sure. I haven't spent enough time thinking about it. It may be fine the way it is.

I would be very interested to know if you try any of these suggestions and what, if any, effect they have for you.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Wed Sep 24, 2014 8:50 am
by TrollMan
If you mark connection, and then only mark packet for that marked connection you will also get a performance increase.
I would try a more clean setup for a performance baseline than the one you have, like lambert says, your FW rules have room for performance. Allow rules first since they are hit by traffic you want, lower down put your drops when possible.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Wed Sep 24, 2014 12:23 pm
by hashbang
ty, Rule orders changed
/ip firewall filter
add action=add-src-to-address-list address-list=level3 address-list-timeout=\
6h chain=input comment=l3 connection-state=new dst-port=22,23 protocol=\
tcp src-address-list=level2
add action=add-src-to-address-list address-list=level2 address-list-timeout=\
30s chain=input comment=l2 connection-state=new dst-port=22,23 protocol=\
tcp src-address-list=level1
add action=add-src-to-address-list address-list=level1 address-list-timeout=\
1m chain=input comment=l1 connection-state=new dst-port=22,23 protocol=\
tcp
add action=drop chain=forward comment="zeus drop" connection-state=new \
dst-address-list=zeus
add action=drop chain=input comment="ssh brute drop" connection-state=new \
src-address-list=level3
/ip firewall mangle
add action=mark-connection chain=prerouting comment="CACHE Mark" \
dscp=14 new-connection-mark=cache_conn
add action=mark-packet chain=prerouting comment="CACHE Mark" \
connection-mark=cache_conn dscp=14 new-packet-mark=cache_mark \
passthrough=no

Lets see wht changes it does to cpu usage. I'd started graphing the cpu usage.

Re: suggestion for a 5 gig router after testing ccr 1036

Posted: Wed Sep 24, 2014 9:33 pm
by hashbang
No noticeable change
peak cpu usage 24%