Strange slow internet connections

ChrisVanMeer · August 4, 2016, 4:25pm

Hi

See my setup below:

                                     ┌────────────────┐                                          
                                     │ Local network  │                                          
                                     │ 192.168.0.0/24 │                                          
                                     └────────────────┘                                          
━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━ ━  
                                                                                                 
                     .254             .253            .12                                        
┌──────────┐     ┌──────────┐     ┌──────────┐       ┌───┐                                       
│ Internet │ ◀───│  RB2011  │ ◀───│  CRS125  │ ◀─────│Pi1│  D:90.64 Mbits/s - U:15.09 Mbits/s    
└──────────┘150Mb└──────────┘ 1Gb └──────────┘ 100Mb └───┘                                       
                                        ▲                                                        
                 Λ                 1Gb  │                                                        
 ┌─────────────▶▕ ▏      .251           │ .252         .9                                        
 │ 5Ghz          V──────────┐     ┌──────────┐       ┌───┐                                       
 │               │  hAP ac  │────▶│   hEX1   │ ◀─────│Pi2│  D:33.48 Mbit/s - U:15.07 Mbit/s      
 │               └──────────┘ 1Gb └──────────┘ 100Mb └───┘                                       
 │                                      ▲                                                        
 │                                 1Gb  │                                                        
 │                                      │ .250        .11                                        
 Λ                                ┌──────────┐       ┌───┐                                       
▕ ▏      .10                      │   hEX2   │ ◀─────│Pi3│  D:35.25 Mbits/s - U:15.10 Mbits/s    
 V──────────┐                     └──────────┘ 100Mb └───┘                                       
 │  Laptop  │                                                                                    
 └──────────┘

CRS125, hEX1 and hEX2 are setup in the same vlan, same /24 subnet, all ports switched to ether1 of the device at hand.
All uplinks 1Gb. No firewalling.
3x Raspberry Pi 3, all the same model, 100Mbit port.

When I do a speedtest-cli from Pi2 and Pi3, the speed is drastically lower than the speedtest-cli from Pi1.

iPerf from Pi3 → Pi2:

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   112 MBytes  94.2 Mbits/sec

iPerf from Pi2 → Pi1:

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   113 MBytes  94.3 Mbits/sec

When I do a speedtest from my laptop - connected through WIFI (on a hAP AC that is connected to hEX1) I get the full 150Mbit

Could someone help me troubleshoot this? I don’t know where to look anymore.

Thanks!
Chris

ChrisVanMeer · August 7, 2016, 2:48pm

bump

IntrusDave · August 7, 2016, 2:50pm

Can you post an “/interface export” for each of the Mikrotik devices?

ChrisVanMeer · August 7, 2016, 3:49pm

RB2011

# aug/07/2016 16:51:40 by RouterOS 6.35.4
# software id = RJBB-BDDK
#
/interface bridge
add admin-mac=4C:5E:0C:C7:A7:8E auto-mac=no mtu=1500 name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=ether1
set [ find default-name=ether2 ] name=ether2-master
set [ find default-name=ether3 ] master-port=ether2-master
set [ find default-name=ether4 ] master-port=ether2-master
set [ find default-name=ether5 ] master-port=ether2-master
set [ find default-name=ether6 ] name=ether6-master
set [ find default-name=ether7 ] master-port=ether6-master
set [ find default-name=ether8 ] master-port=ether6-master
set [ find default-name=ether9 ] master-port=ether6-master
set [ find default-name=ether10 ] master-port=ether6-master
/interface bridge port
add bridge=bridge interface=ether2-master
add bridge=bridge interface=ether6-master
add bridge=bridge interface=sfp1
add bridge=bridge interface=wlan1

CRS125

# aug/07/2016 16:52:30 by RouterOS 6.35.4
# software id = 66SH-5RHJ
#
/interface bridge
add admin-mac=E4:8D:8C:C4:06:7A auto-mac=no name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=ether1-master
set [ find default-name=ether2 ] master-port=ether1-master
set [ find default-name=ether3 ] master-port=ether1-master
set [ find default-name=ether4 ] master-port=ether1-master
set [ find default-name=ether5 ] master-port=ether1-master
set [ find default-name=ether6 ] master-port=ether1-master
set [ find default-name=ether7 ] master-port=ether1-master
set [ find default-name=ether8 ] master-port=ether1-master
set [ find default-name=ether9 ] master-port=ether1-master
set [ find default-name=ether10 ] master-port=ether1-master
set [ find default-name=ether11 ] master-port=ether1-master
set [ find default-name=ether12 ] master-port=ether1-master
set [ find default-name=ether13 ] master-port=ether1-master
set [ find default-name=ether14 ] master-port=ether1-master
set [ find default-name=ether15 ] master-port=ether1-master
set [ find default-name=ether16 ] master-port=ether1-master
set [ find default-name=ether17 ] master-port=ether1-master
set [ find default-name=ether18 ] master-port=ether1-master
set [ find default-name=ether19 ] master-port=ether1-master
set [ find default-name=ether20 ] master-port=ether1-master
set [ find default-name=ether21 ] master-port=ether1-master
set [ find default-name=ether22 ] master-port=ether1-master
set [ find default-name=ether23 ] master-port=ether1-master
set [ find default-name=ether24 ] master-port=ether1-master
/interface bridge port
add bridge=bridge interface=ether1-master
add bridge=bridge interface=sfp1
/interface ethernet switch port
set 0 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 1 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 2 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 3 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 4 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 5 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 6 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 7 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 8 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 9 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 10 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 11 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 12 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 13 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 14 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 15 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 16 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 17 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 18 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 19 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 20 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 21 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 22 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 23 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 24 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"
set 25 per-queue-scheduling="wrr-group0:1,wrr-group0:2,wrr-group0:4,wrr-group0:8,wrr-group0:16,wrr-group0:32,wrr-group0:64,wrr-group0:128"

hEX1

# aug/07/2016 16:51:45 by RouterOS 6.35.2
# software id = 605X-W8XD
#
/interface bridge
add name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=ether1-master
set [ find default-name=ether2 ] master-port=ether1-master
set [ find default-name=ether3 ] master-port=ether1-master
set [ find default-name=ether4 ] master-port=ether1-master
set [ find default-name=ether5 ] master-port=ether1-master
/interface bridge port
add bridge=bridge interface=ether1-master

hAP ac

# aug/07/2016 16:51:54 by RouterOS 6.36
# software id = CZHA-1KZ2
#
/interface bridge
add admin-mac=6C:3B:6B:18:86:09 auto-mac=no name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=ether1-master
set [ find default-name=ether2 ] master-port=ether1-master
set [ find default-name=ether3 ] master-port=ether1-master
set [ find default-name=ether4 ] master-port=ether1-master
set [ find default-name=ether5 ] master-port=ether1-master
/interface bridge port
add bridge=bridge interface=sfp1
add bridge=bridge interface=wlan1
add bridge=bridge interface=wlan2
add bridge=bridge interface=ether1-master
/interface bridge settings
set allow-fast-path=no

hEX2

# aug/07/2016 16:52:37 by RouterOS 6.35.4
# software id = U4JT-960V
#
/interface bridge
add name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=ether1-master
set [ find default-name=ether2 ] master-port=ether1-master
set [ find default-name=ether3 ] master-port=ether1-master
set [ find default-name=ether4 ] master-port=ether1-master
set [ find default-name=ether5 ] master-port=ether1-master
/interface bridge port
add bridge=bridge interface=ether1-master

IntrusDave · August 7, 2016, 3:58pm

hex1 and hex2 don’t need to have bridges. They are already in a switch group, the bridge just makes for more CPU load.
If you are not using the sfp on the switch, you can remove the bridge from that one too.

Have you verified that the pi’s get full speed when connected directly to the switch?

BlackVS · August 7, 2016, 4:04pm

What was the reason to create bridge with one interface on Hex1/Hex2 ? Except specially to make higher load of CPU of course %)
Disable both them.
Why to not plug Pi3 directly in Hex1 or CRS125? Try both variants and check speedtest results. If they are the same then problem not in Hex1/Hex2.

Also during speedtests from Pi2/Pi3 will check CPU load on Hex1, Hex2.

IntrusDave · August 7, 2016, 4:05pm

i think I just said that..

ChrisVanMeer · August 7, 2016, 4:07pm

Ok, moved the IP addresses of hEX1 and hEX2 from the bridge interface to the ether1 port and then removed the bridges.
The reason why I created those bridges? I don’t know really, now to come and think of it.

Removed SFP1 from the CRS125.
No change in speedtest-cli outputs.
CPU speeds stays low.

Yes, when connected to the CRS125, all Pi’s get the full speed.
I have interchanged them all in a way that every Pi was at given point in either CRS125, hEX1 or hEX2 and the results remained the same: when connected to hEX1 or hEX2, the speeds are low. I also have an Apple TV connected to hEX2, that also just get 30Mbit when doing a speedtest.

IntrusDave · August 7, 2016, 4:11pm

Give us this from hex1:

/export compact hide-sensitive

ChrisVanMeer · August 7, 2016, 4:18pm

# aug/07/2016 18:14:13 by RouterOS 6.35.2
# software id = 605X-W8XD
#
/interface ethernet
set [ find default-name=ether1 ] name=ether1-master
set [ find default-name=ether2 ] master-port=ether1-master
set [ find default-name=ether3 ] master-port=ether1-master
set [ find default-name=ether4 ] master-port=ether1-master
set [ find default-name=ether5 ] master-port=ether1-master
/ip address
add address=192.168.0.252/24 interface=ether1-master network=192.168.0.0
/ip dns
set allow-remote-requests=yes servers=192.168.0.254
/ip firewall filter
add chain=input comment="defconf: accept ICMP" protocol=icmp
add chain=input comment="defconf: accept established,related" \
    connection-state=established,related
add chain=forward comment="defconf: accept established,related" \
    connection-state=established,related
/ip route
add distance=1 gateway=192.168.0.254
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www-ssl disabled=no
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/ipv6 address
add address=2001:470:7879:2::248
/ipv6 route
add distance=1 gateway=2001:470:7879:2::254
/system clock
set time-zone-name=Europe/Amsterdam
/system identity
set name=hEX1
/system routerboard settings
set cpu-frequency=720MHz protected-routerboot=disabled silent-boot=yes
/tool romon
set enabled=yes

IntrusDave · August 7, 2016, 4:24pm

Very odd. the only thing I can think of would be a defective cable or maybe power supply.
have you tried swapping hex1 and hex2? Different cables and ports?

ChrisVanMeer · August 7, 2016, 4:37pm

Yes swapped hEX1 with 2. Swapped the power supply.

Didn’t swap the cable, it runs through the walls…but my theory is that the cable is good, since my laptop gets high speeds through wifi that is connected to hEX1

IntrusDave · August 7, 2016, 4:46pm

Ahhh, missed that detail.

So - if the laptop is fine, then the hex and switch are fine. that leaves the Pi’s.
try manually setting the speed and duplex to 100M/Full on the HEX and the Pi.
Also, check the stats on the ports and see if they are getting errors.

ChrisVanMeer · August 7, 2016, 5:08pm

I can’t get why a pi connected to the crs125 is doing fine and when I connect that same pi to hEX1, speed drops. Both ports came up as 100mb no errors.

And the iperf from pi2 and pi3 to pi1 get the full bandwith.

ChrisVanMeer · August 8, 2016, 5:10am

Hmm, when I do a quick compare between my MikroTiks, it appears that both hEX’s have an input voltage of 11.5V.
The rest of them has 24V.

Does the input voltage has anything to do with the throughput speed?

Zorro · August 8, 2016, 8:56am

if you read SoC datasheed(used in Pi)you will be surprised how its achieve even That speed, because its Extrmely weak, not only compared to used in RB SoC, but at all.
my point is: numbers are good. nothing “wrong” happen. its how ~it should be.

ChrisVanMeer · August 8, 2016, 9:03am

The hEX2 also has an Apple TV (gen4) attached to it, it gets the same speed.
But connected directly to the switch, it gets up to 100 Mbit.
Same goes for the Pi, so the hardware can handle it.

Something on the hEX is slowing it down.

Another thought: iPerf’s from Pi2 and Pi3 to Pi1 shows 94Mbit, but that is on the same subnet, so no routing involved.
Speedtest goes to the internet, so routing is involved…could that be something?

IntrusDave · August 8, 2016, 1:55pm

you may have some rules in the router that are slowing it down. have you tried disabling all of the queue, filter, and mangle rules?

ChrisVanMeer · August 8, 2016, 5:01pm

Yep, tried that as well.
And if it was the solution, then it wouldn’t explain why the wired devices on / behind hEX1 have lower speed than wireless devices behind hEX1.

IntrusDave · August 9, 2016, 1:56pm

The only thing left is a compatibility issue with the Pi and the Hex