Speed drop after update to 7.1stable

Hi Guys,

First of all I’m using a RB3011. My Internet Connection is normally around 550 - 650MBit which I do get with 6.49.2.
The CPU reaches about (max) ~50% load during speedtest (firewall enabled). After update to 7.1 stable it maxes out at around 250-300MBit and the cpu reaches 100% with firewall enabled and around 80% with firewall disabled (no change in max dl-speed with disabled firewall).
Fasttrack rules are in place of course.

I don’t know where to start to be honest.

Do you guys have any idea?

Cheers

PS: I can do a screenshot with 7.1 tomorrow. I’ve reverted back to 6.49.2 because of family constraints :slight_smile:
6.49.2.jpg

Probably best to report this directly to support, including supout files from both v6 and v7.

Disable route cache in ROS v6 and then compare the speeds.

I think the disable route cache setting is broken in v6, I did some tests a few versions ago, it didn’t change anything.

See if the fasttrack rules are working and the fasttrack counters are going up. The lack of route caching in RouterOS v7 means that you will get lower speed test results without fasttrack, but I would expect Fasttrack to help more than it is if it was working.

You can see my response here to post about a similar seemingly slower IPv6 performance on RouterOS v7: v7.1 is released! - Announcements - MikroTik community forum

You should get similar results on IPv4 as well if fasttrack is not used, just like IPv6.

Yes, I suspect the disable route cache setting was made into a dummy switch in RouterOS v7 that doesn’t actually do anything (because there is no route cache there), and this change (making it a dummy switch) was accidentally backported to RouterOS v6, breaking the ability to turn off route caching from recent v6 versions.

Good Morning,

Disabling route-cache caused 6.49.2 also to max out at around 350MBit (but with the CPU at ~50%).

Yes they are. Its also shown as active in IP-Settings (IPv4 Fasttrack Active - counting up)

Thx for your help indeed!

The route cache is gone from v7 and is not coming back, so if it is the only reason for the speedtest results you are seeing, then unfortunately you will not be able to replicate those speedtest results with v7. Route caching would give an artificial boost to things like speedtests making it look like your router could handle more traffic than it could in normal real world situations, so on RouterOS v6 you get the 620Mbps speedtest result when your router most likely can’t handle more than 300-350Mbps of real world traffic with route caching on in v6.

Are you using the default firewall ruleset, or did you modify things? If you modified things it could be that you are only fasttracking a portion of your traffic instead of all traffic.

Moving a huge video file to a remote NAS or to a NAS from another subnet doesn’t count as “real world” ?
Or database backups, or other big files.

Ehm - don’t get me wrong but I can download from the internet with ~55MB from CDN-networks (for example via ddownload or rapidgator for example) so I assume that the speedtest tells me the truth and does not tells me “fictional facts”.
Also the test-results on https://mikrotik.com/product/RB3011UiAS-RM#fndtn-testresults tells another language.

Sorry missed your question regarding the firewall. I use a self made firewall →

[admin@router-main] /ip firewall filter> export 
# dec/09/2021 15:16:04 by RouterOS 6.49.2
/ip firewall filter
add action=accept chain=input comment="WireGuard Docker Container" dst-port=51820 log=yes protocol=udp
add action=drop chain=input comment="drop ssh brute forcers" dst-port=22 protocol=tcp src-address=!192.168.100.0/24 src-address-list=ssh_blacklist
add action=add-src-to-address-list address-list=ssh_blacklist address-list-timeout=1w3d chain=input connection-state=new dst-port=22 protocol=tcp src-address-list=ssh_stage3
add action=add-src-to-address-list address-list=ssh_stage3 address-list-timeout=1m chain=input connection-state=new dst-port=22 protocol=tcp src-address-list=ssh_stage2
add action=add-src-to-address-list address-list=ssh_stage2 address-list-timeout=1m chain=input connection-state=new dst-port=22 protocol=tcp src-address-list=ssh_stage1
add action=add-src-to-address-list address-list=ssh_stage1 address-list-timeout=1m chain=input connection-state=new dst-port=22 protocol=tcp
add action=accept chain=input comment="default configuration" connection-state=established,related
add action=accept chain=input src-address-list=allowed_to_router
add action=accept chain=input protocol=icmp
add action=drop chain=input
add action=jump chain=forward comment="jump to kid-control rules" jump-target=kid-control
add action=drop chain=forward comment="Block YouTube on FireTV Sandro" layer7-protocol=YouTube src-address=192.168.100.67
add action=fasttrack-connection chain=forward comment=FastTrack connection-state=established,related
add action=accept chain=forward connection-nat-state=dstnat
add action=accept chain=forward comment="Established, Related" connection-state=established,related
add action=drop chain=forward comment="Drop invalid" connection-state=invalid log=yes log-prefix=invalid
add action=drop chain=forward comment="Drop tries to reach not public addresses from LAN" dst-address-list=not_in_internet in-interface-list=LAN_list log=yes log-prefix=!public_from_LAN out-interface-list=!LAN_list
add action=drop chain=forward comment="Drop incoming packets that are not NAT`ted" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN_list log=yes log-prefix=!NAT
add action=jump chain=forward comment="jump to ICMP filters" jump-target=icmp protocol=icmp
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface-list=WAN_list log=yes log-prefix=!public src-address-list=not_in_internet
add action=drop chain=forward comment="Drop packets from LAN that do not have LAN IP" in-interface-list=LAN_list log=yes log-prefix=LAN_!LAN src-address=!192.168.100.0/24
add action=accept chain=icmp comment="echo reply" icmp-options=0:0 protocol=icmp
add action=accept chain=icmp comment="net unreachable" icmp-options=3:0 protocol=icmp
add action=accept chain=icmp comment="host unreachable" icmp-options=3:1 protocol=icmp
add action=accept chain=icmp comment="host unreachable fragmentation required" icmp-options=3:4 protocol=icmp
add action=accept chain=icmp comment="allow echo request" icmp-options=8:0 protocol=icmp
add action=accept chain=icmp comment="allow time exceed" icmp-options=11:0 protocol=icmp
add action=accept chain=icmp comment="allow parameter bad" icmp-options=12:0 protocol=icmp
add action=log chain=icmp comment="deny all other types"
add action=drop chain=icmp comment="deny all other types"

/ip firewall layer7-protocol
add name=YouTube regexp="^.+(youtube.com).*\$"

/ip firewall nat
add action=dst-nat chain=dstnat dst-port=3074,27014-27050 in-interface-list=WAN_list protocol=tcp to-addresses=192.168.100.59
add action=dst-nat chain=dstnat dst-port=3074,3075,3076,3077,3078,3079 in-interface-list=WAN_list protocol=udp to-addresses=192.168.100.59
add action=dst-nat chain=dstnat comment="SSH to zeus 22" dst-port=8080 in-interface-list=WAN_list protocol=tcp to-addresses=192.168.100.211 to-ports=22
add action=dst-nat chain=dstnat comment=wireguard dst-port=51820 in-interface-list=WAN_list protocol=udp to-addresses=192.168.100.210 to-ports=51820
add action=dst-nat chain=dstnat dst-port=443 in-interface-list=WAN_list protocol=tcp to-addresses=192.168.100.210 to-ports=443
add action=dst-nat chain=dstnat comment=temp_rule_for_letsencrypt_do_not_change dst-port=80 in-interface-list=WAN_list protocol=tcp to-addresses=192.168.100.210 to-ports=80
add action=masquerade chain=srcnat comment="lan --> internet" out-interface-list=WAN_list src-address=192.168.100.0/24
add action=masquerade chain=srcnat disabled=yes

/ip firewall raw
add action=drop chain=prerouting in-interface-list=WAN_list src-address-list=blacklist

cheers

Route caching gives a boost to speed tests and big file downloads in the same way because of how it works. So yes, big file downloads to one system will also decrease in speed in v7, but you are not doing big file downloads all the time like that. Where route caching starts to really harm the performance of the device is when you have a bunch of users behind it going to a bunch of different websites and other online things. In RouterOS v6 most of this regular browsing traffic will miss the cache, and the cache harms performance more than helping. That is why it was removed from the Linux kernel nearly a decade ago.

Those test results are on RouterOS v6. As MikroTik has been revising test results for RouterOS v7 they are often 20-50% lower than what they were on RouterOS v6 for the same device.

Please see this for more details: CCR2004 High CPU Usage ROS7 - #9 by raimondsp

The block Youtube on fireTV layer 7 rule needs to be moved down and adjusted. It will be extremely heavy on the router and is probably the reason you are getting slower fasttrack speeds than I would expect. Layer7 matchers are so heavy that they often completely kill the performance. If you disable it temporarily you should find higher speeds with fasttrack, and then you can figure out how to adjust the rule in the list and in terms of the conditions so that it doesn’t have to scan so much traffic.

You could also remove some things like the ssh brute force - you probably shouldn’t have ssh open to the world anyway so there is no need for address lists for brute force, and you can cut down on the number of rules for ICMP. And some other changes, like moving the accept dstnat rule below the accept established,related. And you have the raw blacklist rule as well, which is probably not necessary for a home router.

You may even want to try using the MikroTik default firewall for comparison temporarily:

                    /interface list member add list=LAN interface=bridge comment="defconf"
                     /interface list member add list=WAN interface=ether1 comment="defconf"
                     /ip firewall nat add chain=srcnat out-interface-list=WAN ipsec-policy=out,none action=masquerade comment="defconf: masquerade"
                     /ip firewall {
                       filter add chain=input action=accept connection-state=established,related,untracked comment="defconf: accept established,related,untracked"
                       filter add chain=input action=drop connection-state=invalid comment="defconf: drop invalid"
                       filter add chain=input action=accept protocol=icmp comment="defconf: accept ICMP"
                       filter add chain=input action=accept dst-address=127.0.0.1 comment="defconf: accept to local loopback (for CAPsMAN)"
                       filter add chain=input action=drop in-interface-list=!LAN comment="defconf: drop all not coming from LAN"
                       filter add chain=forward action=accept ipsec-policy=in,ipsec comment="defconf: accept in ipsec policy"
                       filter add chain=forward action=accept ipsec-policy=out,ipsec comment="defconf: accept out ipsec policy"
                       filter add chain=forward action=fasttrack-connection connection-state=established,related comment="defconf: fasttrack"
                       filter add chain=forward action=accept connection-state=established,related,untracked comment="defconf: accept established,related, untracked"
                       filter add chain=forward action=drop connection-state=invalid comment="defconf: drop invalid"
                       filter add chain=forward action=drop connection-state=new connection-nat-state=!dstnat in-interface-list=WAN comment="defconf: drop all from WAN not DSTNATed"
                     }

If all that does not help you may just need a faster device.

It is not a typical traffic pattern for an internet router. You aren’t doing file transfers like that all the time, only occasionally. Still, even if you make the argument that it should be considered typical, route caching has been gone from the Linux kernel for almost 10 years, and MikroTik cannot put it back.

I wasn’t picking on route caching beeing gone, I was picking on your definition of “real world”.
There are users out there that bought routers for home use, not to be a small ISP in their home, nobody keeps at home tens of machines doing random stuff on the internet.
And yes there are users that do video editing and 3D rendering from a home office. And yes they have 500Mbps+ connections.
And yes they bought a hEX or hAP ac2/ac3/Audience/ for that connection, which currently work fine with ROS v6, not so much with IPv6, but that will probably change if IPv6 FastTrack gets implemented.
It’s not all about ISPs.
LE: removed Chateau, that’s v7 only(?).

These days there are a lot of systems in homes - families with kids, you can have a bunch of laptops, phones, tablets, and gaming systems. In most cases the total traffic used is fairly low because things are not being used all the time, so whether such heavy users have 300Mbps connections or 1Gbps connections they really only use 10-20Mbps on average. So the 1Gbps connection isn’t really any faster for them than the 300Mbps connection if only 10-20Mbps is used.

And for the people who do a lot of file transfers and bulk traffic that route caching really helps with, in cases where there is a CPU bottleneck, then unfortunately they will get slower speeds on v7, and the choice will be to remain on RouterOS v6 forever or upgrade to a more powerful device. I suspect some users will instead try to wait for this to be “fixed” in v7, thinking it is just a bug that can be fixed, when this will never happen.

I am hoping that at some point MikroTik goes back and revises the product pages for existing devices to show the v7 performance instead of v6, so that people know what to expect.

@mducharme
In my market many clients have multiple [4 to5] real-time HD iptv streams running all day long plus many other ream time activities by other family members plus plus plus … they all have I Gbps service and yes all working well ….

You don’t need that much bandwidth for a few video streams. 4K streams are usualy under 50Mbps, HD ones are insignificant.

I deleted my whole firewall and took this one as successor Log into Atlassian - MikroTik Documentation wich should be MT approved I guess. There is no reasonable difference in CPU load.

But to be honest I do not get the point why I do need another device - I did speedtests now with and without Route-Cache on v6 (not tested v7 until know)

enabled-route-cache – Download ~ 550MBit

lcd                                 0.5%
spi                                 2.5%
ethernet                           10.2%
console                             0.2%
firewall                           10.5%
networking                         12.7%
winbox                              0.2%
management                            1%
profiling                             0%
traffic-accou...                    0.5%
bridging                              3%
unclassified                          5%
total                              46.3%

disabled-route-cache – Download ~ 340Mbit

lcd                                 0.7%
spi                                   3%
ethernet                            7.5%
console                             0.2%
firewall                           17.2%
networking                         18.5%
winbox                              0.5%
management                          1.7%
routing                             3.2%
dhcp                                  0%
profiling                           0.5%
traffic-accou...                    1.2%
bridging                            5.5%
unclassified                          8%
total                              67.7%

So from a CPU-Load perspective I do not understand why a more powerful device is needed (btw - I thought that the 3011 with dual 1,4GHz ARM CPU “is” powerful - which one would you assume to fullfill the needs for my internet-connection then?)

Cheers

Please read this.