R1CH
September 5, 2013, 3:11pm
1
I recently upgraded my internet to 150mbps, however I’m only getting 100mbps throughput before the CPU on the RB951G is maxing out.
I have a queue tree on two interfaces (in / out) with 12 queues on each interface, and a firewall setup with packet tagging on the mangle chain (34 items). The majority of the traffic should only need to traverse two firewall rules before being tagged, eg HTTP downloads (passthrough disabled). The brochure for the RB951G shows 25 firewall rules still achieving 698 mbps throughput, so I’m rather confused why I’m having such large firewall performance problems.
The rest of the setup is pretty much default - wireless bridged with the port 2-5 LAN, DHCP server, routing and conntrack.
My profile looks like this during a large HTTP download:
I already tried overclocking to 650MHz but that only gave me an improvement to 106mbps, I’d rather not go any further as it already runs quite warm. Any suggestions for what would be causing such a high firewall load when most of the traffic is hit by the 2nd rule with no passthrough?
pcunite
September 5, 2013, 4:07pm
2
May we see your Firewall, Mangle, and Queue trees exports? Edit out any personal information. Also what version of RouterOS?
R1CH
September 5, 2013, 6:19pm
3
6.3, here’s the output:
Firewall:
# sep/05/2013 20:13:26 by RouterOS 6.3
/ip firewall connection tracking
set enabled=yes tcp-close-timeout=30s tcp-close-wait-timeout=30s \
tcp-fin-wait-timeout=30s tcp-last-ack-timeout=30s \
tcp-syn-received-timeout=15s tcp-syn-sent-timeout=15s \
tcp-time-wait-timeout=30s udp-timeout=30s
/ip firewall filter
add chain=input protocol=icmp
add chain=input connection-state=established
add chain=input connection-state=related
add chain=input dst-port=2222,8291 in-interface=ether1-gateway protocol=tcp
add action=drop chain=input in-interface=\
ether1-gateway
add action=drop chain=forward connection-state=invalid
/ip firewall mangle
add action=mark-packet chain=prerouting \
connection-mark=streaming-out new-packet-mark=streaming-out passthrough=\
no
add action=mark-packet chain=prerouting new-packet-mark=\
tcp-ack packet-size=40 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting \
connection-mark=http-slow new-packet-mark=http-slow passthrough=no
add action=mark-connection chain=prerouting connection-bytes=2048576-0 \
in-interface=ether1-gateway new-connection-mark=http-slow passthrough=no \
protocol=tcp src-port=80,443
add action=mark-packet chain=prerouting \
connection-mark=http new-packet-mark=http passthrough=no
add action=mark-connection chain=prerouting in-interface=ether1-gateway \
new-connection-mark=http passthrough=no protocol=tcp src-port=80,443
add action=mark-packet chain=prerouting connection-mark=\
dota2 new-packet-mark=dota2 passthrough=no
add action=mark-connection chain=prerouting \
connection-state=new in-interface=bridge-local new-connection-mark=dota2 \
passthrough=no protocol=udp src-port=27005
add action=mark-packet chain=prerouting \
connection-mark=starcraft2 new-packet-mark=starcraft2 passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-port=\
1119 new-connection-mark=starcraft2 out-interface=ether1-gateway \
passthrough=no protocol=udp
add action=mark-connection chain=postrouting connection-state=new \
new-connection-mark=starcraft2 out-interface=ether1-gateway passthrough=\
no protocol=udp src-port=6112-6119
add action=mark-connection chain=postrouting connection-state=new dst-port=\
1119 new-connection-mark=starcraft2 out-interface=ether1-gateway \
passthrough=no protocol=tcp
add action=mark-connection chain=prerouting dst-port=\
1935 in-interface=bridge-local new-connection-mark=streaming-out \
passthrough=no protocol=tcp
add action=mark-packet chain=prerouting connection-mark=\
skype new-packet-mark=skype passthrough=no
add action=mark-connection chain=prerouting \
dst-port=9009 in-interface=bridge-local new-connection-mark=skype \
passthrough=no protocol=tcp
add action=mark-connection chain=prerouting in-interface=ether1-gateway \
new-connection-mark=skype passthrough=no protocol=tcp src-port=9009
add action=mark-packet chain=prerouting \
connection-mark=misc new-packet-mark=misc passthrough=no
add action=mark-connection chain=prerouting in-interface=ether1-gateway \
new-connection-mark=misc passthrough=no src-address=63.251.64.0/24
add action=mark-connection chain=prerouting \
connection-rate=0-100k in-interface=ether1-gateway new-connection-mark=\
ssh protocol=tcp src-port=22
add action=mark-connection chain=prerouting connection-rate=100k-1G \
in-interface=ether1-gateway new-connection-mark=ssh-download passthrough=\
no protocol=tcp src-port=22
add action=mark-packet chain=prerouting connection-mark=ssh-download \
new-packet-mark=ssh-download passthrough=no
add action=mark-packet chain=prerouting connection-mark=ssh \
new-packet-mark=ssh passthrough=no
add action=mark-connection chain=postrouting connection-state=new dst-port=22 \
new-connection-mark=ssh out-interface=ether1-gateway passthrough=no \
protocol=tcp
add action=mark-packet chain=prerouting \
connection-mark=udp-fast new-packet-mark=udp-fast passthrough=no
add action=mark-connection chain=prerouting dst-port=65180-65281 \
in-interface=ether1-gateway new-connection-mark=udp-fast passthrough=no \
protocol=udp
add action=mark-connection chain=postrouting dst-port=\
53,5060,5061,5080,27910,1119 new-connection-mark=udp-fast out-interface=\
ether1-gateway passthrough=no protocol=udp
add action=mark-packet chain=prerouting new-packet-mark=misc \
passthrough=no protocol=icmp
add action=mark-packet chain=postrouting new-packet-mark=\
octoshape-out out-interface=ether1-gateway passthrough=no protocol=udp \
src-port=8247
add action=log chain=prerouting disabled=yes packet-mark=\
no-mark
add action=mark-packet chain=prerouting new-packet-mark=\
untagged packet-mark=no-mark passthrough=no
add action=log chain=postrouting disabled=yes out-interface=ether1-gateway \
packet-mark=untagged
/ip firewall nat
add action=masquerade chain=srcnat \
out-interface=ether1-gateway to-addresses=0.0.0.0
Tree:
/queue tree
add limit-at=148M max-limit=148M name=queue1 parent=bridge-local queue=\
default
add limit-at=500k max-limit=144M name=prio8-untagged-in packet-mark=untagged \
parent=queue1 queue=stream-red
add limit-at=30M max-limit=145M name=prio6-http-in packet-mark=http parent=\
queue1 priority=6 queue=stream-red
add limit-at=1G max-limit=1G name=prio2-udp-fast-in packet-mark=udp-fast,misc \
parent=queue1 priority=2 queue=default
add limit-at=1G max-limit=1G name=prio2-ssh-in packet-mark=ssh parent=queue1 \
priority=2 queue=default
add limit-at=30M max-limit=145M name=prio7-http-slow-in packet-mark=http-slow \
parent=queue1 priority=7 queue=stream-red
add limit-at=30M max-limit=145M name=prio7-ssh-slow-in packet-mark=\
ssh-download parent=queue1 priority=7 queue=default
add limit-at=14M max-limit=14M name=queue2 parent=ether1-gateway queue=\
default
add limit-at=1G max-limit=1G name=prio2-udp-fast-out packet-mark=udp-fast \
parent=queue2 priority=2 queue=ack-red
add limit-at=1G max-limit=1G name=prio2-ssh-out packet-mark=ssh parent=queue2 \
priority=2 queue=default
add limit-at=1G max-limit=1G name=prio2-tcp-acks-out packet-mark=tcp-ack \
parent=queue2 priority=2 queue=ack-red
add limit-at=500k max-limit=13M name=prio8-untagged-out packet-mark=untagged \
parent=queue2 queue=default
add limit-at=2M max-limit=14M name=prio6-http-out packet-mark=http parent=\
queue2 priority=6 queue=default
add limit-at=1M max-limit=14M name=prio7-http-ssh-out packet-mark=\
ssh-download,http-slow parent=queue2 priority=7 queue=default
add limit-at=8M max-limit=14M name=prio4-streaming-out packet-mark=\
streaming-out parent=queue2 priority=4 queue=pfifo-500
add limit-at=1G max-limit=1G name=prio3-games-in packet-mark=starcraft2,dota2 \
parent=queue1 priority=3 queue=sfq
add limit-at=1G max-limit=1G name=prio3-games-out packet-mark=\
starcraft2,dota2 parent=queue2 priority=3 queue=default
add limit-at=50k max-limit=50k name=prio8-octoshape-out packet-mark=\
octoshape-out parent=queue2 queue=default
add limit-at=500k max-limit=144M name=prio8-notag-in packet-mark=no-mark \
parent=queue1 queue=default
add limit-at=2M max-limit=14M name=prio8-notag-out packet-mark=no-mark \
parent=queue2 queue=default
add limit-at=1G max-limit=1G name=prio2-tcp-acks-in packet-mark=tcp-ack \
parent=queue1 priority=2 queue=default
add limit-at=2M max-limit=14M name=prio7-misc-out packet-mark=misc parent=\
queue2 priority=7 queue=default
add limit-at=30M max-limit=145M name=prio5-streaming packet-mark=\
streaming-in,streaming-out parent=queue1 priority=5 queue=default
add limit-at=4M max-limit=13M name=prio5-skype packet-mark=skype parent=\
queue2 priority=5 queue=default
add limit-at=5M max-limit=30M name=prio5-skype-in packet-mark=skype parent=\
queue1 priority=5 queue=default
No way you’ll be able to push more than you are now with the 951 and that ruleset… frankly I’m surprised you’re getting past 30Mbps… Nothing wrong with your rules that I can see, just I’ve found the 951 (and RB2011UAS) to be much slower than advertised in “real world” situations.
(and that 698Mbps benchmark is with ALL 1518 byte packets, I’ve found for ‘real world’ examples it ends up somewhere between the 64 and 512 byte benchmarks under the best of circumstances)
R1CH
September 5, 2013, 9:47pm
5
A full speed HTTP download should be almost all 1500 byte packets, and HTTP is matched by the 3rd mangle rule so it shouldn’t have to traverse the entire firewall chain. It’s rather disappointing that the highest speed consumer device can’t keep up with modern connections. Hopefully MT will release something with higher speed CPUs soon.
biomesh
September 5, 2013, 10:22pm
6
I would take a backup of your config and run with the default rules that come with the device when using quickset. See if/how the performance differs.
pcunite
September 5, 2013, 11:46pm
7
Excellent advice. To the OP, remove all rules except the most basic needed to operate the unit. Then do another test and let us know. This is a very interesting topic.
janisk
September 6, 2013, 11:22am
8
in some places you mark packets directly without marking connection first - that can take some more resources than marking connection first and then marking packets of the connection only based on connection mark. Jump is very cheap, and afterwards you do not need to check for parameter you checked when jumping
Also, you can jump to other chain for packets that have no connection mark to mark the connection, and then just mark the packets based on connections afterwards.
also, untagged seem to be useless as you can check for no-mark so adding the mark just wastes resources.
One more thing, if you can - move rules that accept the most packets as much up as you can, but without braking the logic.
R1CH
September 6, 2013, 6:39pm
9
Thanks for the suggestions. I removed the redundant rules and split the no-marked packets into a separate chain, unfortunately it barely made any difference to the throughput, the CPU still maxes out at 100% during large downloads
pcunite
September 6, 2013, 8:05pm
10
Rich,not take bufferbloat and ACK packets into account. They just run packets through it over a LAN. Because you’re using the product in a real world setting you need to make some adjustments for this.
If you’re still up for some more testing, please remove all Filter, Nat, and Mangle rules you currently have. Then copy and paste this into a new terminal window. Change the 10M to be 90% of your actual upload. I already set 135M to be 90% of your download. Also try them at 80% too.
# Minimum firewall protection
/ip firewall filter
add chain=input action=accept connection-state=new in-interface=bridge-local comment="Allow LAN access to the router itself"
add chain=input action=accept connection-state=established
add chain=input action=accept connection-state=related
add chain=input action=accept protocol=icmp comment="Allow ping ICMP from anywhere"
add chain=input action=drop comment="Drop"
add chain=forward action=accept connection-state=new in-interface=bridge-local comment="Allow LAN traffic to move through the router"
add chain=forward action=accept connection-state=established
add chain=forward action=accept connection-state=related
add chain=forward action=drop comment="Drop"
# NAT masquerade
/ip firewall nat add chain=srcnat action=masquerade out-interface=ether1-gateway comment="Turn on masquerading"
# Minimum Mangle for fast HTTP speeds
/ip firewall mangle
add chain=prerouting action=mark-connection protocol=udp port=53 connection-state=new new-connection-mark="DNS" comment="DNS"
add chain=prerouting action=mark-packet passthrough=no connection-mark="DNS" new-packet-mark="DNS"
add chain=postrouting action=mark-connection protocol=udp port=53 connection-state=new new-connection-mark="DNS"
add chain=postrouting action=mark-packet passthrough=no connection-mark="DNS" new-packet-mark="DNS"
#
add chain=postrouting action=mark-packet passthrough=no protocol=tcp tcp-flags=ack packet-size=0-123 new-packet-mark="ACK" comment="ACK"
add chain=prerouting action=mark-packet passthrough=no protocol=tcp tcp-flags=ack packet-size=0-123 new-packet-mark="ACK"
#
add chain=prerouting action=mark-connection protocol=tcp port=80,443 connection-state=new new-connection-mark="HTTP" comment="HTTP"
add chain=prerouting action=mark-packet passthrough=no connection-mark="HTTP" new-packet-mark="HTTP"
#
add chain=prerouting action=mark-connection connection-mark=no-mark new-connection-mark="OTHER" comment="OTHER"
add chain=prerouting action=mark-packet passthrough=no connection-mark="OTHER" new-packet-mark="OTHER"
# Minimum HTB Queue tree
/queue tree
add name="LEVEL_A_UP" parent=ether1-gateway queue=default max-limit=10M
add name="LEVEL_A_DOWN" parent=bridge-local queue=default max-limit=135M
add name="LEVEL_B_UP" parent=ether1-gateway queue=default max-limit=10M
add name="LEVEL_B_DOWN" parent=bridge-local queue=default max-limit=135M
#
add name="ACK_U" parent="LEVEL_A_UP" packet-mark="ACK" queue=default priority=1
add name="ACK_D" parent="LEVEL_A_DOWN" packet-mark="ACK" queue=default priority=1
add name="DNS_U" parent="LEVEL_A_UP" packet-mark="DNS" queue=default priority=2
add name="DNS_D" parent="LEVEL_A_DOWN" packet-mark="DNS" queue=default priority=2
#
add name="HTTP_U" parent="LEVEL_B_UP" packet-mark="HTTP" queue=default priority=1
add name="HTTP-D" parent="LEVEL_B_DOWN" packet-mark="HTTP" queue=default priority=1
add name="OTHER_U" parent="LEVEL_B_UP" packet-mark="HTTP" queue=default priority=8
add name="OTHER-D" parent="LEVEL_B_DOWN" packet-mark="OTHER" queue=default priority=8
pcunite:
…10M to be 90% of your actual upload. I already set 135M to be 90% of your download. Also try them at 80% too.
Thats a good idea, some connections do act funny with the ROS queueing when they get near saturation, unless you have ACK at a higher priority.
. I guess I just need more CPU power and will have to wait for a higher spec product.