Switch with low performance!

RouterOS General
1

Hi!

I bought a CSS326-24G-2S + RM and it was just a bad joke! i couldnt barely push over 100Mbps on the switch! and some freezes from time to time

Then i bought a CRS326-24G-2S+RM
this one wont freezes at least!
but the performance is bad!

Heres what happens…

On both cases the switch was connected to a RB1100Dx4 (which is running just fine) at 1Gbps link speed
several devices at 100Mbps (at least 15) connected to the switch. if i run a speed test from internet i cant push over 50Mbps!

If i use any other CHEAP switch on the same setup and port it works just perfect ! and i can push to almost 100mbps (limited by the eth max speed)

But with the mikrotik switches i cant get a stable throughput !

Then i tought can it be the config ?

So i just made a bridge with hardware off-load (aka master port) without vlans! just like a regular switch

same results
tried with vlans.. same results

now im running bonding 2 ethernet + vlans
and i got slightly better troughput
(YES Bonding does support hw-offload with xor or 802.11ad on crs326)

Heres my Question
I got tons of RX-Overruns on the switch trunk port

RX-Overruns happens when the device is slower than the other device right ?

on this case…

RB1100Dx4====>CRS326===>MkDevice

I got 0 Overruns on the 1100
i got tons Overruns on the CRS (on both interfaces (eth1 and eth2 which is the bonding slaves)
i got 0 Overruns on The other MK-Devices..

so the crs is the slow one right ?

Just to be clear… if i connect the the same mk-device direct to the rb1100 works flawless

How can i solve this ?
I already tried with fixed link speed and auto-negotiation
Flow control on and off

PS: eth2 has less overruns because it was added later with bonding
switch bonding bridge crs326.png

2

This what happens when i run some btest from the devices to the rb1100 through the switch MT Devices ===>CRS326====>RB1100Dx4 (upload)
Works nicely!! 8 devices could reach almost 800Mbps, but the other way around just sucks RB1100Dx4====>CRS326====>MT Devices (Download)
I cant reach the same amount of traffic on the devices.. starts to drop packets some devices reach 98Mbps for a short period some just reach 7Mbps. its just driving me crazy.
dude btest crs 1100.png

3

I use the same switch, you can’t be configuring it correctly. I get wire speed switching. Post a export of your configuration.


Sent from my SM-A520W using Tapatalk

4

Thanks for you reply

Heres the config.

/interface bridge
add name=bridge-vlans priority=0x20 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] name=ether1-trunk rx-flow-control=on tx-flow-control=on
set [ find default-name=ether2 ] name=ether2-trunk rx-flow-control=on tx-flow-control=on
set [ find default-name=ether3 ] name=ether3-mrb-hades rx-flow-control=on tx-flow-control=on
set [ find default-name=ether4 ] loop-protect=off name=ether4-jayme rx-flow-control=on tx-flow-control=on
set [ find default-name=ether5 ] name=ether5-mrb-aegon rx-flow-control=on tx-flow-control=on
set [ find default-name=ether6 ] name=ether6-mrb-skadi rx-flow-control=on tx-flow-control=on
set [ find default-name=ether7 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether8 ] name=ether8-mrb-arya rx-flow-control=on tx-flow-control=on
set [ find default-name=ether9 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether10 ] advertise=10M-full,100M-full name=ether10-mrb-tyrion rx-flow-control=on tx-flow-control=on
set [ find default-name=ether11 ] advertise=10M-full,100M-full name=ether11-ptp-loja rx-flow-control=on tx-flow-control=on
set [ find default-name=ether12 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether13 ] name=ether13-saveiros rx-flow-control=on tx-flow-control=on
set [ find default-name=ether14 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether15 ] name=ether15-mrb-pegasus rx-flow-control=on tx-flow-control=on
set [ find default-name=ether16 ] name=ether16-mrb-daenerys rx-flow-control=on tx-flow-control=on
set [ find default-name=ether17 ] name=ether17-mrb-heimdall rx-flow-control=on tx-flow-control=on
set [ find default-name=ether18 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether19 ] name=ether19-mrb-zeus rx-flow-control=on tx-flow-control=on
set [ find default-name=ether20 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether21 ] name=ether21-mrb-poseidon rx-flow-control=on tx-flow-control=on
set [ find default-name=ether22 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=ether23 ] name=ether23-mrb-diana rx-flow-control=on tx-flow-control=on
set [ find default-name=ether24 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=sfp-sfpplus1 ] rx-flow-control=on tx-flow-control=on
set [ find default-name=sfp-sfpplus2 ] rx-flow-control=on tx-flow-control=on
/interface vlan
add interface=bridge-vlans name=vlan2403-mrb-hades vlan-id=2403
add interface=bridge-vlans name=vlan2404 vlan-id=2404
add interface=bridge-vlans name=vlan2405-mrb-aegon vlan-id=2405
add interface=bridge-vlans name=vlan2406 vlan-id=2406
add interface=bridge-vlans name=vlan2407 vlan-id=2407
add interface=bridge-vlans name=vlan2408 vlan-id=2408
add interface=bridge-vlans name=vlan2409 vlan-id=2409
add interface=bridge-vlans name=vlan2411-ptp-loja vlan-id=2411
add interface=bridge-vlans name=vlan2412 vlan-id=2412
add interface=bridge-vlans name=vlan2413-saveiros vlan-id=2413
add interface=bridge-vlans name=vlan2415-mrb-pegasus vlan-id=2415
add interface=bridge-vlans name=vlan2416 vlan-id=2416
add interface=bridge-vlans name=vlan2417-mrb-heimdall vlan-id=2417
add interface=bridge-vlans name=vlan2419-mrb-zeus vlan-id=2419
add interface=bridge-vlans name=vlan2421-mrb-poseidon vlan-id=2421
add interface=bridge-vlans name=vlan2423-mrb-diana vlan-id=2423
/interface bonding
add mode=balance-xor name=bonding1 slaves=ether2-trunk,ether1-trunk transmit-hash-policy=layer-2-and-3
/interface vlan
add interface=bonding1 name=vlan500-mngt vlan-id=500

/interface bridge port
add bridge=bridge-vlans interface=ether23-mrb-diana pvid=2423
add bridge=bridge-vlans interface=ether19-mrb-zeus pvid=2419
add bridge=bridge-vlans interface=ether17-mrb-heimdall pvid=2417
add bridge=bridge-vlans interface=ether15-mrb-pegasus pvid=2415
add bridge=bridge-vlans interface=ether13-saveiros pvid=2413
add bridge=bridge-vlans interface=ether21-mrb-poseidon pvid=2421
add bridge=bridge-vlans interface=ether3-mrb-hades pvid=2403
add bridge=bridge-vlans interface=ether5-mrb-aegon pvid=2405
add bridge=bridge-vlans interface=ether11-ptp-loja pvid=2411
add bridge=bridge-vlans interface=ether4-jayme pvid=2404
add bridge=bridge-vlans interface=ether6-mrb-skadi pvid=2406
add bridge=bridge-vlans interface=ether7 pvid=2407
add bridge=bridge-vlans interface=ether8-mrb-arya pvid=2408
add bridge=bridge-vlans interface=ether9 pvid=2409
add bridge=bridge-vlans interface=ether10-mrb-tyrion pvid=2410
add bridge=bridge-vlans interface=ether12 pvid=2412
add bridge=bridge-vlans interface=ether14
add bridge=bridge-vlans interface=ether16-mrb-daenerys pvid=2416
add bridge=bridge-vlans interface=bonding1

/interface bridge vlan
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether3-mrb-hades vlan-ids=2403
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether5-mrb-aegon vlan-ids=2405
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether11-ptp-loja vlan-ids=2411
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether13-saveiros vlan-ids=2413
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether15-mrb-pegasus vlan-ids=2415
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether17-mrb-heimdall vlan-ids=2417
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether19-mrb-zeus vlan-ids=2419
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether21-mrb-poseidon vlan-ids=2421
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether23-mrb-diana vlan-ids=2423
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether12 vlan-ids=2412
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether10-mrb-tyrion vlan-ids=2410
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether9 vlan-ids=2409
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether8-mrb-arya vlan-ids=2408
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether7 vlan-ids=2407
add bridge=bridge-vlans tagged=bridge-vlans,bonding1 untagged=ether6-mrb-skadi vlan-ids=2406
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether4-jayme vlan-ids=2404
add bridge=bridge-vlans tagged=bonding1,bridge-vlans untagged=ether16-mrb-daenerys vlan-ids=2416

5

The first thing I would suspect is the flow control. As it is used rarely, it may have got silently broken. And when it works properly, it has the bad habit of pushing peripheral problems to the core - a slow destination cannot handle incoming packets so it says “stop sending”, and thus the frames accumulate at the previous device until its buffers overflow and it has to start dropping them anyway. But instead of dropping packets of the slow peripheral device on itself, the core equipment drops any packets on the interface facing that peripheral device, affecting unrelated flows.

Second point, do you really have an IP interface in each of your million and six VLANs? Not that the load of the CPU coming from processing all the ARP requests in all the VLANs should affect switch chip performance, but I’d keep only those /interface vlan which are really necessary.

6

Thanks for your reply!

The flow control was enabled later for test purpose! still i got the same results with on/off

each vlan has a pppoe-server for my customers, and ip address only to manage the ap devices
but like i said before, the first thing i tried was without vlans, just like a regular switch.

I’m considering to replace the RB1100Dx4 for a CCR1009 to see what happens…

7

I’m having trouble with HW-Offload ! when i set it to on i got a lot a overruns on the trunk interface and low throughput!
when i set to off i got no overruns and the throughput is way better! but the cpu load is way higher!

can u guys tell me why ?
Its supposed to be the other way around right ?
Wire speed with hw-offload

8

https://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features

9

Well, if linking to manuals, this link is more appropriate.

I don’t like the /interface vlan name=vlan500-mngt to be attached directly to interface=bond1 directly while all other /interface vlan are attached to the /interface bridge name=bridge-vlan which has bond1 as one of its slave interfaces, but I’m not sure how it is eventually related to the low throughput in hw-offload mode which really should be higher than a software bridged mode.

10

I Apreciate your reply guys.

i’m not even running bonding interface anymore… just a single interface… eth1 as trunk interface.. and the mngt-vlan is now on the bridge as well
i’ve already read all those manuals and articles about the CRS! thanks tho

hw-offload gave me worst performance! i know … doesnt make sense!

11

I would start from the default configuration in steps. Create a single bridge with hw-offload, vlan-filtering=no, insert the machine “into” an already existing trunk connection. If it works fine, configure vlan filtering rules permitting all vlans actually used on that trunk and activate vlan-filtering. If it still works fine, add another port as an access one to one of the used VLANs, and see whether the flow between that access port and one of the trunk ports tested previously is still OK, etc.

12

Hi there!

thats was the first thing i did, step by step
My laptop on the routerboard was flawless, then the switch on the same port… i coudnt get the same throughput (the first setup the switch was just a bridge with hw-offloading)
But like i said before, with hw-offload i could get almost 800Mbps in one direction, but not the other way around.. and most important for me (download speed).

the next step im gonna try with swos then default routeros version.

for now its working… not like i expected… but its working.

13

Well, if you say that it throttles “one direction”, what happened when you swapped the roles of the ports? Was it always throttling port A → port B or always the other RouterBoard → the PC regardless whether port A was connected to the PC or to the other RouterBoard?

14

if you using bonding interface - bonding is the same as a physical ethernet. So a vlan to bonding1 is the same as a vlan to ether1.. if not bonding

Ther is a big different in having vlan to the interface - and to the bridge. If having the vlan to a interface - the interface would tag tjis out to only this port. If tagged to a bridge - all ports in the bridge will have the vlan tagged. And last -but not least- if put in bridge -it goes untagged into this bridge.

Nowdays its no more software bridge like earlier. traffic do not goes trough connection tracking like in old times. but depends on setup,.

15

Hey sindy,
Doesnt matter the port i use as trunk, the same thing happen.

16

I confirm this issue

happened to me on
CRS 326
CRS 328 24P

tested on routeros 6.43.4

very frustrating, disappointing and problematic, you buy a mikrotik switch to improve performance and you get to opposite result

17

I solved my CRS 326 RX Overflow problem switching the uplink port that was on an ethernet port to an SFP port. It seems the SFP port can handle all the traffic better then one ethernet port receiveing and transmiting data from and to other ethernet ports.

18

thank you for sharing your experience i will get a SFP cable to try

19

If this helps, it sounds to me as if the connection on the copper-only port negotiated a half duplex operation mode, as otherwise there is no sane reason why an SFP should perform different from a copper-only port. Have you used an optical SFP or a copper one?

20

on CRS 326 SFP ports are 10g maybe that’s the reason why using SFP helps

i have replicated the same issue in multiple customers, issue present on CRS 326 and CRS 328 when i can get SFP i will try

i think this can be the case of “Head-of-line blocking” issue, of course i dont have proof of that, i am a customer, i dont have a LAB to try and prove that