I have been updating my RB760iGS routers, about 200 of them, to the latest version (7.xx) from 6.48. Since I have done that the speed test results and the bandwidth test results have dropped significantly. For example all the units I have that are still at 6.48 can do about 750 down and up simultaneously (bandwidth testing) and after the update can only do 360 (bandwidth testing), and the speed test results from Okala come in at about 180. We have done significant testing to make sure that the only difference causing this is the updated firmware (tested on about 15 different units).
We also use E60iUGS, which is supposed to be more powerful but seems to have even slower speed results after the firmware has been updated. Today I did a speed test to Okala and could only get 75 down and 400 up, without the update I can get about 960 up and down. I have a certified Mikrotik network guy that knows everything about Mikrotiks and has also determined the same results.
One of major changes between ROS v6 and ROS v7 is version of linux kernel used. ROSv7 uses linux kernel which drops functionality of "route caching" (or something like that). Which in practice means that in certain scenarios, routing speed gets degraded quite drastically.
And no, there's nothing to be done ... other than keep using ROS v6 (go for latest long-term which is 6.49.18) unless there's some functionality introduced in v7 that you absolutely require (and with RB760iGS architecture - MMIPS - I doubt there are some such functionalities, most new functionalities are limited to ARM/ARM64 architectures).
As to E60iUGS: there were performance issues with ether1 and SFP+ ports (when looking at block diagram of device, it can be seen that these two ports are connected directly to CPU as opposed to other ports which are connected to switch chip). Performance issues are supposedly solved (or at least much improved) in latest v7 (could be it's actually 7.21rc that includes that fix ... personally I don't have hEX refresh or hEX S 2025, so I'm following specifics only vaguely).
Never would i have thought that a firmware update would make these routers basically useless by reducing the speed so slow that Mikrotiks specs and test results are now essentially false advertising. Apparently they didn't do the testing on this as well. I mean who would have thought to even test this, isn’t upgrading supposed to mean better? I bought 400 of these units with the intention of using them to support as much as 500 up and down, which they initially did, but after updating them, which everyone says we should be doing, they now can only confidently do 100 up and down. That’s a far cry from what I was sold. Thank you though for commenting.
mkx: thank you for this info as I don’t think anyone else in this forum has mentioned this issue, now I know.
What @mkx said is absolutely true. Between the two kernels there is a downgrade in routing performance. This is usually around 30%.
This doesn't explain your results, however. Something else must have also changed. There were many changes in almost every are, so you'll have to do some work to narrow it down.
I believe that with fasttrack the original hex s is still considered Gb-capable. Even if I recall wrong, it should still be near.
I don't know if you're doing pppoe, but there were quite a few changes...
The most I have seen in speed on a bandwidth test has been 750 up and down simultaneously but now I see 550 up and 360 down. In a test from okala I struggle to get 300 down and that’s download, by itself. The programming is very basic using just 1 nat rule so it shouldn’t be causing any issues. We have spent around 50 hours or so just testing for the reason of the speed loss on around 15 different hex’s, all of them come in at the exact same speed results and only after the firmware update, which would make the firmware update virtually the only cause. We tested the units at 6.48 and got the expected results, then we upgraded and got the slower speeds, then we dropped the firmware back down to 6.48, which is where the hex started from, and the results did NOT go back to what they were doing before the update. So if you upgrade then try to downgrade the firmware you basically ruined the hex. We are fully convinced that the firmware is causing the issue after all the testing we did, but no one else has mentioned it in this forum and Mikrotik hasn't responded to my trouble ticket. Still 30% is quite a drop in speed, as you mentioned in your post, because I think we all purchase certain units based on the specs you need for a specific job and upgrading them severely changes the test results in which you purchased them is not very cool.
At the end of the day I have 300 brand new in the box units that I can’t use because they now fall short of what they were bought for and I don’t know what to do with them.
A fairly popular (on this forum at least) rule of thumb about real-life routing performance of certain device is to look at official test results (published on official device page) ... then take number under "routing - 25 ip filter rules - 512 bytes packet size". Depending on particular configuration actual performance can be higher or lower. For RB760iGS that number is 260Mbps. Yes, there are users which report (near) gigabit routing of this device ... and there are others complaining of not getting even that much.
And another detail: official test results are only performed once (right before official model launch), so elder devices (such as RB760iGS) will have ROSv6 results published even though v7 is the ROS already for a while now. But this can't be blatantly called a scam, devices were produced with ROS v6 installed and users are still able to run v6 on these devices, nobody forces anybody to run v7 on these devices (I highly doubt that any of original hEX S'es came with v7 from factory).
So, considering the said rule of thumb, routing results you're seeing (from 360Mbps to 550Mbps) are pretty normal for this device. Even "mere" 300Mbps is not outragely low.
For comparison: at home I've got a RBD52G runing 7.19.6. "Rule of thumb" test result says it should be able to route at around 980Mbps (again a "v6" number) ... even though I have to use PPPoE (which does hit router's CPU) I still regularly get around 950Mbps (my subscription is 1000/300 Mbps) without loading router's CPU beyond 30% (none of cores are maxed out), but I have to use "best" speedtest client, in my case that's a CLI client running on linux.
By gigabit I of course didn't mean simultaneous. (All routing numbers for every manufacturer add all flows together.) 750 Mbps x 2 sounds about right. In fact more than I would have guessed; I would have thought more or less 1Gbps to be the maximum even with v6.
I think it isn't worth arguing about the removal of the route cache. It's a done deal, and it affects everyone. The linux kernel team made this decision. It's a small(ish) respite that the performance drop is really only there when there are a small number of endpoints, so it predominantly affects speed tests and not so much real-world traffic.
Also, software (to our sorrow) usually doesn't become faster. Instead more features are supported. This is the case for the Linux networking stack as well.
Again, something else is probably going on. E.g. it has happened before that a specific version had a severe regression in a given setup.
The fact that downgrading doesn't seem to fix the issue is even more unexpected. When upgrading from v6 to v7, actually the whole configuration is frozen and retained, so nothing should be different after a downgrade.
There was time to update 200 devices and time to investigate on 15 devices afterwards what could be wrong, but no time to do some throughput testing upfront.
What purpose do these RBs serve? All the same config? How config looks like? Until now this is more like a hypothetical question.
I appreciate your response on my issue but I don’t seem to be making progress towards finding out what can be done to fix this. All I can say is these hex’s were doing the speed we needed before the upgrades and now they aren’t and nothing has changed. Not upgrading them isn’t really an option as a lot of the updates are for security. I could definitely run on the 6.48 but I also take the chance of a security breach. I should be able to update without fearing the reduction of performance to this degree anyway. I guess I was just hoping for someone that would have come back and said “yea i had that problem and this is what I did to fix it”.
1.5Gbps combined up and down is probably not possible with ROS7 on the devices. But reaching near wirespeed in one direction is still possible.
You have to make sure that fasttrack is enabled and has all conditions it needs to be effective, then with a "normal" firewall setup such as the configuration you get when resetting to defconf the old hEX S will be able to route 980Mbps (measured on the interface, using ookla would show 920ish results) which is 98% of wirespeed, in one direction, on recent RouterOS 7.
Try to reset a device back to defconf and measure. But dont use btest running on the router because the tool itself consumes CPU.
Here are screenshots (IPv4 and IPv6) I made last February on the old hEX using defconf that even has PPPoE overhead:
Wait, OP. By "just having a nat rule", do you mean that you don't have the fasttrack-connection rule in forward? Because that would totally explain the dramatic speed decrease to, or even under, 300 Mbps.
ROS v6 more or less ceased to evolve functionality wise around version 6.47 ... so vast majority of changes in latest v6 versions were bug fixes and security fixes. AFAIK there's no notable known security problem which is fixed in v7 and is not fixed in latest v6 ... which is 6.49 and security wise there's big difference between 6.48 you were running and 6.49 I was suggesting.
There's a discussion about wanted LTS vrreion of ROS ... and currently 6.49.18 seems to be an LTS in its truest meaning: older version with security fixes (while all the development of new features is done in v7). Unfortunately (for some users) v6 is not an option for newer device models ... but it's IMO a very feasible option in your case.
Not using fasttrack really wouldn’t explain the slow speeds since I have never used it on any tik and right now I can go into one of my 6.48 version units and get 750 full duplex and go into any 7.xx units and only get 360 down. I could see where fasttrack may help if I had more rules but using only 1, fasttrack won’t have any affect.
mkx,
When i said nothing has changed I meant nothing in my programming or network operations has changed. I’ve been programming these units the same for about 5 years now, the only thing that has changed is the firmware updates. I just think Mikrotik should have stated in the “fixes” statement telling us about it, of course if they even tested for it. If I would have known that my routers was going to take a hit of this size I most likely would have updated out of 6.xx.
No it not only helps when you have many rules. Besides, if you think about it, in a typical FW setup with lots of filter and NAT rules, if done correctly then most of the packets only need to be processed by a single firewall rule anyway, the "accept established, related" filter rule that should be conveniently placed at the top of the chain. All the NAT rules and other filter rules placed below in the chain only have to process the first packet of a connection.
So the fact that your firewall "has only one rule" makes almost no difference, what the performance cost of the firewall concern, compared to a correctly configured firewall with full rule set.
FastTrack not only skips the FW rules but also uses shortcuts through many parts of the stack. Packets will even travel a different part on the bridge. For the old hEX / hEX S with RouterOS 7, having FastTrack available and usable will speed up routing over 3Ă— for a configuration with FW rules and enabled connection tracking.
The Ookla tests from my linked screenshots above would only reach 280Mbps instead of 920Mbps if the FastTrack rule is disabled, while keeping all other settings the same.
When ROS v7 came out, people noticed decrease in routing speed immediately. There were discussions about it going left and right, also a few MT staffers contributed ... and the outcome was what we mentioned here: that the performance drop is due to change in linux kernel behaviour and MT can't do anything about it. But you're right, this was never published in any of official documents (but so there isn't an easily-accessible official transition manual available - could be it even doesn't exist - where such information should really go ... because change logs in MT world are relative to previous version ... so you'd have to read change log of v7.0 when going from 6.48 and everything before and after that ... did you?)
There were suggestions tgat MT should redo official performance tests for all (pre-v7) devices and publish both results with clear distinction between v6 and v7 results. MT didn't want to do it (saying that testing costs money and that tests were valid at time of product launch running certain ROS version which is still true and that v6 devices are supposed to keep running v6). Most of forum regulars of course don't agree with such argumentation. The problem with published test results is also that actual ROS version running on test device while performing tests is not mentioned in test report.
Fast forward a few years ... you're quite late with v6 to v7 transition, v7 is around for around 5 years (first 7.0 betas came out in middle of 2019, 7.0 was launched in 2020) so I don't expect you to remember those discussions (if you even followed them). But OTOH I wonder what made transition on those old devices so critical now (not to mention your lack of preparation ... ups, I mentioned it).
