RB750Gr3 difference between workstation speedtest vs bandwitch test

RouterOS General
1

Hi,

I use RB750Gr3 as a wired router for couple devices with 300/40 Mb/s WAN connection.
My config is almost default. I did not change anything in firewall filter rules.

When i checking connection speed with using speedtest I have results:
Download: 33.50 Mbit/s
Upload: 42.13 Mbit/s
Always result are similar. Download about 30-35 Mb/s and upload always about maximum. I checked on 3 different pc/laptop connecting directly to router on different Ethernet ports.

but when I checking speed using Bandwidth test in MT to server: 38.104.52.187 i have results closer to my connection max speeds: 272/38 Mb/s

MT WAN is connected: 1Gb/s
PC is connected to MT: 1GB/s

Mikrotik: 6.47.8
Routerboard: 6.47.8

I tried reset configuration, no changes.

What i changed in default settings:

/ip pool
add name=dhcp_50 ranges=192.168.77.50-192.168.77.55

/ip dhcp-server
add address-pool=dhcp_50 disabled=no interface=bridge lease-time=1d name=dhcp_50

/ip address
add address=192.168.77.77/24 comment=defconf interface=bridge network=192.168.77.0

/ip dhcp-server lease
add address=192.168.77.99 comment=ADMIN_LAN mac-address=MAC server=dhcp_50

/ip dhcp-server network
add address=192.168.77.0/24 comment=defconf gateway=192.168.77.77 netmask=24

Rest is default.

CPU usage is close to 0. I replaced cables. No changes. I checked what I could but still don’t know where is a problem.

I will be appreciate for any clues.

BR,
Lukasz.

2

Any clues?

3

I’ve read a similar complaint about hEXr3 multiple times here on the forum but none of these has ever been resolved.

So just one question - is there anything else connected to the hEX during the speedtest, that has negotiated a slower link speed than 1 Gbit/s? Other people have noticed that the switch chip does not accept further frames from the CPU until it manages to send out ones received from the CPU previously, which it already has in its internal queues. So other traffic, even local, running from the CPU via such a slow link may slow down the speedtest traffic.

4

Is this connection a “gateway” connection. (LAN-WAN with NAT) or just a LAN-LAN passthrough ?
There a 2 switch settings in the hEXr3, so that setting and the used ethernet port may play a role here.
https://i.mt.lv/cdn/product_files/RB750Gr3-dsw_161117.png
https://i.mt.lv/cdn/product_files/RB750Gr3-esw3_190642.png

5

Hi,
Thank You for response.

I use 3 MT ports:
eth1 → WAN
eth2 → Switch 1Gb/s - in MT auto negotiated 1GBs
eth5 → RPI - in MT auto negotiated 1GBs

I did a test couple minutes ago. I Disabled eth2 and did a test on RPI using speedtest cli. No changes. Normally some of devices connected to switch are fast Ethernet only. Could it cause mentioned problem?

I use LAN-WAN with NAT.

6

OK. WAN-LAN. Flow to ether2 or ether5, or both? Have no clue for the performance hit, just trying to think of some reason.

You mention a switch on ether2. Could that switch be applying “backpressure”? Is there any indication in the interface counters of the hEX for this or for any other non-optimised traffic flow?

7

This is what I had in mind, but as you say you’ve now got the same result with only the WAN link and the RPi link, both negotiated at 1 Gbps, and nothing else connected to the hEX, it is not the reason. Plus if the 100 Mbit/s devices are not connected directly to the hEX but to an external switch, but the switch itself is connected to the hEX by a 1 Gbit/s link, it is unlikely to be related at all, as the external switch would have to use Ethernet flow control, which is rarely the case.

Also the switching mode as suggested by @bpwl cannot be related in this case (single WAN link and single LAN link), as the internal channel between the CPU and the switch chip is a 2 Gbit/s one in the switch mode so traffic in both directions can use the full bandwidth of the ports (if the CPU managed to handle it). In the 1+3+5/2+4 mode, you would have to connect both the WAN and the LAN link to the same group to get a throughput limit to 500 Mbit/s per direction if both direction were maxed out symmetrically. So nothing close to the 30 Mbit/s you can see.

8

Flow from WAN eth1 to bridge (eth-2-3-4-5).

Could You tell what I should check?

eth2 (to switch):
Rx Drop = 38 Total Rx 128GB transfered
no Tx Drop

I did a test right now (from PI):
Testing download speed…
Download: 28.76 Mbit/s
Testing upload speed…
Upload: 38.91 Mbit/s

Bandwidth Test to: 38.104.52.187
Rx: 262.5 Mbps
Tx: 38.91 Mbit/s

Upload is always max or close to max (max = 40 Mb/s).

I will be appreciate for any clues what I should check, what test I can make because I have this issue longer time. I can add that on previous MT was the same situation (RB750GR2) i was thought that was related with high cpu usage (20-30% 1d average), on RB750GR3 from last month after I replaced devices is: Max: 3%; Average: 0%; Current: 1% [Daily]. That is why I replaced GR2 on GR3 but it wasn’t solution of my issue.

Maybe ISP blocking me on some way? But why Bandwidth test in MT always shows normal speed but on devices inside LAN always is the same about 30Mb/s download and 38Mb/s upload.

I using MT RB750 since first version. RB750 → RB750GR2 → RB750GR3.

Problem is strange.

9

Check the two block diagrams found on the product page: https://mikrotik.com/product/RB750Gr3#fndtn-downloads
It has always been unclear to me what the method is to choose between the two modes.
“with disabled switching” and “with enabled switching”, what does it mean?

However, it seems that when you want max performance out of these devices it would be prudent to put WAN on ether1, a switch on ether2, and take ether3-ether5 out of the bridge.
That would seem to guarantee that each port has a dedicated path to the CPU and no tricks required to access them.

10

I guess that switching case is when two or more ports are members of a bridge and thus switch traffic between ports. Thus the “is switched” parallelogram, some ports might be used in non-switching configuration (e.g. WAN port).
Non-switching case is when none of ports are members of a bridge, i.e. routing mode.

11

My guess was the setting of the hEX was done with : https://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features#Switch_All_Ports_Feature
But the device is NOT listed in that wiki setting, it’s for other chip sets. So I had it wrong. Can’t find the setting in the wiki.

The other reference I find around MT7621A is that it will not do hardware offloading if the STP protocol is enabled. Default is protocol set to RSTP, must be “none” for switch HW acceleration.
(https://wiki.mikrotik.com/wiki/Manual:Interface/Bridge#Bridge_Hardware_Offloading)

Here is no LAN switching used, but who knows what the impact is on the packet flow. (Probably guessing wrong again. It’s only a hunch.)

12

Ok so this might be a stupid question but have you tested other speed testing sites? I mean just because you get crap speed on one speed test site/connection does not mean you connection is bad.
If you download a ISO file or something else that is big can you get the proper speed then?

I use to have a RB750Gr3 as my WAN router and it had no issues pushing 800 Mbit and this with NAT similar to you setup.

If you always have bad speed can you please output full config with hide-sensitive just in case something strange have been added to config?
Also you can test to netinstall the router or change build version.

If you go down the netinstall route or reset config again can you please test this again but without changing the default config at all just to make sure that any added config does not add the issues.

13

It’s not a hardware problem, that’s for sure.
He stated above that he had the same issue with 750Gr2 which is a completly different SoC (QCA9556 vs MT7621A) but even 750Gr2 can handle more than 100Mbps.
It’s somewhere between the chair and the speedtest server, but closer to the chair since I had a 750Gr2 and now a 750Gr3 in use with no issues like that.

14

Let me build on @znevna’s direction of thinking: what does Wireshark analysis of output file of a tcpdump running on the RPi during the speedtest show? Which delay is bigger, between the data packet from the server and the ACK of it sent by the RPi or between the ACK sent by the RPi and the subsequent packet from the server? And what is the initial TCP window size indicated by the RPi during the download test, compared to the initial TCP windows size indicated by the remote server during the upload test? It could be that the TCP settings on the RPi use a short window so the round trip delay limits the throughput (the window gets full before the first ACK reaches the server, so the server pauses sending).