Solved: Cloud Router HUB(!) 125

RouterOS General
1

I’ve managed to replicate and isolate an issue two of my customers of CRS125 has run into. I’m not sure if this is a bug or a config fault. I’ve replicated this with ROS 6.15 6.38.1 and 6.39rc55.

What happens is that the CRS125 starts sending out each packet (TX) to ALL active ports. It basically starts acting as a hub rather than a switch. And as most hubs - if there is a 10mbit (or 100mbit) device connected to one of its ports, all ports are limited to 10mbit (or 100mbit) TX. RX is not limited.

How to replicate:
You need:

  • a CRS125 running default config (my testbox is a CRS125-24G-1S-2HnD-in )
  • a 10mbit (or 100mbit) device (for my testing I used a RB750gl with one port configured to 10mbit full duplex, and I changed its ip to 192.168.88.2 for easy access while testing, and turned off dhcp server. Other than that I left it at default config.)
  • two 1gbit devices (for my testing I used my laptop and a RB493g which act as my home router.)

I connected:

  • port 1 of the CRS125 to my RB493g.
  • port 3 of the CRS125 to my laptop
  • port 5 of the CRS125 to the forced 10mbit port of the RB750gl.

Starting out with the default config running on the CRS125 I ran the following commands (from a laptop connected with wlan, as cable connected device will be disconnected during the config):

/interface vlan
add interface=ether2-master-local name=vlan-gjest vlan-id=4
add interface=ether2-master-local name=vlan-lan vlan-id=5

/interface bridge
add name=bridge-gjest

/interface bridge port
add bridge=bridge-local interface=vlan-lan
add bridge=bridge-gjest interface=vlan-gjest
remove [ find interface=ether2-master-local ]

/interface ethernet switch egress-vlan-tag
add tagged-ports="ether21-slave-local,ether22-slave-local,ether23-slave-local,\
    ether24-slave-local,switch1-cpu" vlan-id=5
add tagged-ports="ether21-slave-local,ether22-slave-local,ether23-slave-local,\
    ether24-slave-local,switch1-cpu" vlan-id=4

/interface ethernet switch ingress-vlan-translation
add new-customer-vid=5 ports="ether2-master-local,ether3-slave-local,ether4-sl\
    ave-local,ether5-slave-local,ether6-slave-local,ether7-slave-local,ether8-\
    slave-local,ether9-slave-local,ether10-slave-local,ether11-slave-local,eth\
    er12-slave-local,ether13-slave-local,ether14-slave-local,ether15-slave-loc\
    al,ether16-slave-local" sa-learning=no
add new-customer-vid=4 ports="ether17-slave-local,ether18-slave-local,ether19-\
    slave-local,ether20-slave-local" sa-learning=no

So far bandwidth from my laptop to/from the RB493g is good and looking at the interface statistics shows that the CRS125 still act as a switch.
Now I run this final command:

/interface ethernet switch vlan
add ports="ether2-master-local,ether3-slave-local,ether4-slave-local,ether5-sl\
    ave-local,ether6-slave-local,ether7-slave-local,ether8-slave-local,ether9-\
    slave-local,ether10-slave-local,ether11-slave-local,ether12-slave-local,et\
    her13-slave-local,ether14-slave-local,ether15-slave-local,ether16-slave-lo\
    cal,ether21-slave-local,ether22-slave-local,ether23-slave-local,ether24-sl\
    ave-local,switch1-cpu" vlan-id=5
add ports="ether17-slave-local,ether18-slave-local,ether19-slave-local,ether20\
    -slave-local,ether21-slave-local,ether22-slave-local,ether23-slave-local,e\
    ther24-slave-local,switch1-cpu" vlan-id=4

Now when I test bandwidth, TX towards my laptop is limited to 10mbit. Looking at the interface statistics I see that every packet that is sent towards my laptop is also sent towards the RB750g.
Once I disconnect the RB750g, my laptop get a full 1gbit TX. And once I reconnect the RB750g, the TX drops to 10mbit.
TX on ether1 is not affected, probably because it does not have a master-port set. RX is not affected at any ports.

If I put a 1gbit switch between the CRS125 and the 10mbit port on RB750g, the issue does not occur.

So, can anyone tell me if this is a missconfig or a bug?

One of the customers who ran into this did not use vlan. He just had several master ports, which automatically creates some vlans for internal use.

Below is the full config export compact of the CRS125

# mar/21/2017 17:32:55 by RouterOS 6.39rc55
# software id = 6I0S-QQRB
#
/interface bridge
add fast-forward=no name=bridge-gjest
add admin-mac=D4:CA:6D:1E:96:57 auto-mac=no fast-forward=no mtu=1500 name=\
    bridge-local
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce \
    disabled=no distance=indoors frequency=auto mode=ap-bridge ssid=CRS125 \
    wireless-protocol=802.11 wps-mode=disabled
/interface ethernet
set [ find default-name=ether1 ] name=ether1-gateway
set [ find default-name=ether2 ] name=ether2-master-local
set [ find default-name=ether3 ] master-port=ether2-master-local name=\
    ether3-slave-local
set [ find default-name=ether4 ] master-port=ether2-master-local name=\
    ether4-slave-local
set [ find default-name=ether5 ] master-port=ether2-master-local name=\
    ether5-slave-local
set [ find default-name=ether6 ] master-port=ether2-master-local name=\
    ether6-slave-local
set [ find default-name=ether7 ] master-port=ether2-master-local name=\
    ether7-slave-local
set [ find default-name=ether8 ] master-port=ether2-master-local name=\
    ether8-slave-local
set [ find default-name=ether9 ] master-port=ether2-master-local name=\
    ether9-slave-local
set [ find default-name=ether10 ] master-port=ether2-master-local name=\
    ether10-slave-local
set [ find default-name=ether11 ] master-port=ether2-master-local name=\
    ether11-slave-local
set [ find default-name=ether12 ] master-port=ether2-master-local name=\
    ether12-slave-local
set [ find default-name=ether13 ] master-port=ether2-master-local name=\
    ether13-slave-local
set [ find default-name=ether14 ] master-port=ether2-master-local name=\
    ether14-slave-local
set [ find default-name=ether15 ] master-port=ether2-master-local name=\
    ether15-slave-local
set [ find default-name=ether16 ] master-port=ether2-master-local name=\
    ether16-slave-local
set [ find default-name=ether17 ] master-port=ether2-master-local name=\
    ether17-slave-local
set [ find default-name=ether18 ] master-port=ether2-master-local name=\
    ether18-slave-local
set [ find default-name=ether19 ] master-port=ether2-master-local name=\
    ether19-slave-local
set [ find default-name=ether20 ] master-port=ether2-master-local name=\
    ether20-slave-local
set [ find default-name=ether21 ] master-port=ether2-master-local name=\
    ether21-slave-local
set [ find default-name=ether22 ] master-port=ether2-master-local name=\
    ether22-slave-local
set [ find default-name=ether23 ] master-port=ether2-master-local name=\
    ether23-slave-local
set [ find default-name=ether24 ] master-port=ether2-master-local name=\
    ether24-slave-local
set [ find default-name=sfp1 ] name=sfp1-gateway
/ip neighbor discovery
set ether1-gateway discover=no
set sfp1-gateway discover=no
/interface vlan
add interface=ether2-master-local name=vlan-gjest vlan-id=4
add interface=ether2-master-local name=vlan-lan vlan-id=5
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=bridge-local name=default
/interface bridge port
add bridge=bridge-local interface=wlan1
add bridge=bridge-local interface=vlan-lan
add bridge=bridge-gjest interface=vlan-gjest
/interface ethernet switch egress-vlan-tag
add tagged-ports="ether21-slave-local,ether22-slave-local,ether23-slave-local,\
    ether24-slave-local,switch1-cpu" vlan-id=5
add tagged-ports="ether21-slave-local,ether22-slave-local,ether23-slave-local,\
    ether24-slave-local,switch1-cpu" vlan-id=4
/interface ethernet switch ingress-vlan-translation
add new-customer-vid=5 ports="ether2-master-local,ether3-slave-local,ether4-sl\
    ave-local,ether5-slave-local,ether6-slave-local,ether7-slave-local,ether8-\
    slave-local,ether9-slave-local,ether10-slave-local,ether11-slave-local,eth\
    er12-slave-local,ether13-slave-local,ether14-slave-local,ether15-slave-loc\
    al,ether16-slave-local" sa-learning=no
add new-customer-vid=4 ports="ether17-slave-local,ether18-slave-local,ether19-\
    slave-local,ether20-slave-local" sa-learning=no
/interface ethernet switch port
set 0 dscp-based-qos-dscp-to-dscp-mapping=no
set 1 dscp-based-qos-dscp-to-dscp-mapping=no
set 2 dscp-based-qos-dscp-to-dscp-mapping=no
set 3 dscp-based-qos-dscp-to-dscp-mapping=no
set 4 dscp-based-qos-dscp-to-dscp-mapping=no
set 5 dscp-based-qos-dscp-to-dscp-mapping=no
set 6 dscp-based-qos-dscp-to-dscp-mapping=no
set 7 dscp-based-qos-dscp-to-dscp-mapping=no
set 8 dscp-based-qos-dscp-to-dscp-mapping=no
set 9 dscp-based-qos-dscp-to-dscp-mapping=no
set 10 dscp-based-qos-dscp-to-dscp-mapping=no
set 11 dscp-based-qos-dscp-to-dscp-mapping=no
set 12 dscp-based-qos-dscp-to-dscp-mapping=no
set 13 dscp-based-qos-dscp-to-dscp-mapping=no
set 14 dscp-based-qos-dscp-to-dscp-mapping=no
set 15 dscp-based-qos-dscp-to-dscp-mapping=no
set 16 dscp-based-qos-dscp-to-dscp-mapping=no
set 17 dscp-based-qos-dscp-to-dscp-mapping=no
set 18 dscp-based-qos-dscp-to-dscp-mapping=no
set 19 dscp-based-qos-dscp-to-dscp-mapping=no
set 20 dscp-based-qos-dscp-to-dscp-mapping=no
set 21 dscp-based-qos-dscp-to-dscp-mapping=no
set 22 dscp-based-qos-dscp-to-dscp-mapping=no
set 23 dscp-based-qos-dscp-to-dscp-mapping=no
set 24 dscp-based-qos-dscp-to-dscp-mapping=no
set 25 dscp-based-qos-dscp-to-dscp-mapping=no
/interface ethernet switch vlan
add ports="ether2-master-local,ether3-slave-local,ether4-slave-local,ether5-sl\
    ave-local,ether6-slave-local,ether7-slave-local,ether8-slave-local,ether9-\
    slave-local,ether10-slave-local,ether11-slave-local,ether12-slave-local,et\
    her13-slave-local,ether14-slave-local,ether15-slave-local,ether16-slave-lo\
    cal,ether21-slave-local,ether22-slave-local,ether23-slave-local,ether24-sl\
    ave-local,switch1-cpu" vlan-id=5
add ports="ether17-slave-local,ether18-slave-local,ether19-slave-local,ether20\
    -slave-local,ether21-slave-local,ether22-slave-local,ether23-slave-local,e\
    ther24-slave-local,switch1-cpu" vlan-id=4
/ip address
add address=192.168.88.1/24 comment="default configuration" interface=\
    bridge-local network=192.168.88.0
/ip dhcp-client
add comment="default configuration" dhcp-options=hostname,clientid disabled=\
    no interface=ether1-gateway
add comment="default configuration" dhcp-options=hostname,clientid disabled=\
    no interface=sfp1-gateway
/ip dhcp-server network
add address=192.168.88.0/24 comment="default configuration" dns-server=\
    192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 name=router
/ip firewall filter
add action=accept chain=input comment="default configuration" protocol=icmp
add action=accept chain=input comment="default configuration" \
    connection-state=established
add action=accept chain=input comment="default configuration" \
    connection-state=related
add action=drop chain=input comment="default configuration" in-interface=\
    ether1-gateway
add action=drop chain=input comment="default configuration" in-interface=\
    sfp1-gateway
add action=accept chain=forward comment="default configuration" \
    connection-state=established
add action=accept chain=forward comment="default configuration" \
    connection-state=related
add action=drop chain=forward comment="default configuration" \
    connection-state=invalid
/ip firewall nat
add action=masquerade chain=srcnat comment="default configuration" \
    out-interface=ether1-gateway
add action=masquerade chain=srcnat comment="default configuration" \
    out-interface=sfp1-gateway
/system clock
set time-zone-name=Europe/Oslo
/system leds
set 0 interface=wlan1
/tool mac-server
set [ find default=yes ] disabled=yes
add interface=ether2-master-local
add interface=ether3-slave-local
add interface=ether4-slave-local
add interface=ether5-slave-local
add interface=ether6-slave-local
add interface=ether7-slave-local
add interface=ether8-slave-local
add interface=ether9-slave-local
add interface=ether10-slave-local
add interface=ether11-slave-local
add interface=ether12-slave-local
add interface=ether13-slave-local
add interface=ether14-slave-local
add interface=ether15-slave-local
add interface=ether16-slave-local
add interface=ether17-slave-local
add interface=ether18-slave-local
add interface=ether19-slave-local
add interface=ether20-slave-local
add interface=ether21-slave-local
add interface=ether22-slave-local
add interface=ether23-slave-local
add interface=ether24-slave-local
add interface=wlan1
add interface=bridge-local
/tool mac-server mac-winbox
set [ find default=yes ] disabled=yes
add interface=ether2-master-local
add interface=ether3-slave-local
add interface=ether4-slave-local
add interface=ether5-slave-local
add interface=ether6-slave-local
add interface=ether7-slave-local
add interface=ether8-slave-local
add interface=ether9-slave-local
add interface=ether10-slave-local
add interface=ether11-slave-local
add interface=ether12-slave-local
add interface=ether13-slave-local
add interface=ether14-slave-local
add interface=ether15-slave-local
add interface=ether16-slave-local
add interface=ether17-slave-local
add interface=ether18-slave-local
add interface=ether19-slave-local
add interface=ether20-slave-local
add interface=ether21-slave-local
add interface=ether22-slave-local
add interface=ether23-slave-local
add interface=ether24-slave-local
add interface=wlan1
add interface=bridge-local
2

I finally figured this one out!
The problem is explained here: https://support.radware.com/app/answers/answer_view/a_id/15364/~/when-should-source-mac-learning-be-disabled-on-vlans%3F

Somehow I managed to set sa-learning=no in both entries at switch->ingress-vlan-translation
Changing that to yes (which is default) solves the problem.

Now an interesting observation: Some of those ingress vlan translations that are dynamically created when the CRS is set up with multiple master ports get the sa-learning set to ‘no’. I haven’t figured the pattern when this happens and when it does not happen. But I’m guessing this was the source of trouble for the first customer who reported this issue, as he had no vlans, but several master ports in his config.