Switching loop - why?

lapsio · Sun Jul 15, 2018 3:55 am

I'm quite new to Layer 2 (unfortunately i started from top of OSI and stepped down successively) so I decided to get some grip here. In order to test various more advanced configs I decided to create something like this:

susecap607.png

ports with dots represent tagged ports, colors represent untagged ports. coloured lines represent vlans carrying networks to corresponding ports. When I created config with single trunk (so that all VLANs go through CRS317), everything worked. However when I added LACP group and moved 2 VLANs to this second trunk it didn't quite work. It sometimes worked, sometimes didn't. Switch really didn't like this config and expressed it by really non deterministic behavior. Sometimes everything worked fine but after reboot nothing worked again. It's not problem with LACP itself because when I disabled bottom trunk it was deterministic again.

I think there's some nasty loop happening here but I don't know why. I enabled VLAN filtering on bridges on both switches and VLANs supported by trunks did not overlap. I also set frames-type to "allow-only-vlan-tagged" on trunks to avoid accidental default vlan loop. Both switches have protocol-mode set to none because mstp was cutting out CRS317 from router. I'm not sure how to enable mstp for router.

Where's my mistake?

this is config with single trunk (one that works):

/interface bridge
add admin-mac=CC:2D:E0:51:8E:E0 auto-mac=no name=br-hardware protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=9112 mtu=9000
set [ find default-name=ether2 ] l2mtu=9112 mtu=9000
set [ find default-name=ether3 ] l2mtu=9112 mtu=9000
set [ find default-name=ether4 ] l2mtu=9112 mtu=9000
set [ find default-name=ether5 ] l2mtu=9112 mtu=9000
set [ find default-name=ether6 ] l2mtu=9112 mtu=9000
set [ find default-name=ether7 ] l2mtu=9112 mtu=9000
set [ find default-name=ether8 ] l2mtu=9112 mtu=9000
set [ find default-name=ether9 ] l2mtu=9112 mtu=9000
set [ find default-name=ether10 ] l2mtu=9112 mtu=9000
set [ find default-name=ether11 ] l2mtu=9112 mtu=9000
set [ find default-name=ether12 ] l2mtu=9112 mtu=9000
set [ find default-name=ether13 ] l2mtu=9112 mtu=9000
set [ find default-name=ether14 ] l2mtu=9112 mtu=9000
set [ find default-name=ether15 ] l2mtu=9112 mtu=9000
set [ find default-name=ether16 ] l2mtu=9112 mac-address=CC:2D:E0:51:8E:EE mtu=9000
set [ find default-name=ether17 ] l2mtu=9112 mtu=9000
set [ find default-name=ether18 ] l2mtu=9112 mtu=9000
set [ find default-name=ether19 ] l2mtu=9112 mtu=9000
set [ find default-name=ether20 ] l2mtu=9112 mtu=9000
set [ find default-name=ether21 ] l2mtu=9112 mtu=9000
set [ find default-name=ether22 ] l2mtu=9112 mtu=9000
set [ find default-name=ether23 ] l2mtu=9112 mtu=9000
set [ find default-name=ether24 ] l2mtu=9112 mtu=9000
set [ find default-name=sfp-sfpplus1 ] l2mtu=9112 mtu=9000
set [ find default-name=sfp-sfpplus2 ] l2mtu=9112 mtu=9000
/interface vlan
add interface=br-hardware name=vlan4-ccr vlan-id=400
add interface=br-hardware name=vlan10-ccr vlan-id=1000
/interface bonding
add mode=802.3ad mtu=9000 name=bond-ccr slaves=ether15,ether16 transmit-hash-policy=layer-2-and-3
/interface list
add exclude=dynamic name=discover
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/user group
add name=public policy=ssh,read,web,!local,!telnet,!ftp,!reboot,!write,!policy,!test,!winbox,!password,!sniff,!sensitive,!api,!romon,!dude,!tikapp
/interface bridge port
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether17
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether18
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether19
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether20
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether21
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether22
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether23
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether24
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus1
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus2 pvid=2
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether1 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether2 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether3 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether4 pvid=99
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether5 pvid=301
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether6 pvid=302
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether7 pvid=303
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether8 pvid=304
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether9 pvid=401
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether10 pvid=402
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether11 pvid=403
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether12 pvid=404
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether13 pvid=405
add bridge=br-hardware frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether14 pvid=406
add bridge=br-hardware frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=bond-ccr pvid=3
/ip neighbor discovery-settings
set discover-interface-list=discover
/interface bridge vlan
add bridge=br-hardware tagged=sfp-sfpplus2,br-hardware vlan-ids=1000
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether9 vlan-ids=401
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether10 vlan-ids=402
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether11 vlan-ids=403
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether12 vlan-ids=404
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether13 vlan-ids=405
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether14 vlan-ids=406
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether5 vlan-ids=301
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether6 vlan-ids=302
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether7 vlan-ids=303
add bridge=br-hardware tagged=sfp-sfpplus2 untagged=ether8 vlan-ids=304
add bridge=br-hardware untagged=ether1,ether2,ether3,ether4 vlan-ids=99
add bridge=br-hardware tagged=br-hardware,sfp-sfpplus2 vlan-ids=400
/interface list member
add interface=vlan10-ccr list=discover
/ip address
add address=192.168.10.6/24 interface=vlan10-ccr network=192.168.10.0
add address=192.168.4.22/24 interface=vlan4-ccr network=192.168.4.0
/ip cloud
set update-time=no
/ip dns
set servers=192.168.10.9
/ip route
add distance=1 gateway=192.168.10.2
/system clock
set time-zone-autodetect=no time-zone-name=Europe/Warsaw
/system identity
set name=CRS326SWAG
/system leds
set 0 interface=ether1
add interface=vlan10-ccr leds=user-led type=interface-activity
/system ntp client
set enabled=yes primary-ntp=158.75.5.245
/system routerboard settings
set boot-os=router-os silent-boot=no
/user aaa
set default-group=public

Config with 2 trunks is exactly the same except this part:


/interface bridge vlan
add bridge=br-hardware tagged=sfp-sfpplus2,br-hardware vlan-ids=1000
add bridge=br-hardware tagged=bond-ccr untagged=ether9 vlan-ids=401
add bridge=br-hardware tagged=bond-ccr untagged=ether10 vlan-ids=402
add bridge=br-hardware tagged=bond-ccr untagged=ether11 vlan-ids=403
add bridge=br-hardware tagged=bond-ccr untagged=ether12 vlan-ids=404
add bridge=br-hardware tagged=bond-ccr untagged=ether13 vlan-ids=405
add bridge=br-hardware tagged=bond-ccr untagged=ether14 vlan-ids=406
add bridge=br-hardware tagged=bond-ccr untagged=ether5 vlan-ids=301
add bridge=br-hardware tagged=bond-ccr untagged=ether6 vlan-ids=302
add bridge=br-hardware tagged=bond-ccr untagged=ether7 vlan-ids=303
add bridge=br-hardware tagged=bond-ccr untagged=ether8 vlan-ids=304
add bridge=br-hardware untagged=ether1,ether2,ether3,ether4 vlan-ids=99
add bridge=br-hardware tagged=br-hardware,bond-ccr vlan-ids=400

And this one doesn't work. Inb4 I'm using one vlan per port because I want to have stateful firewall between each device inside single network so I'm using CCR1009 to bridge VLANs with bridge ip firewall enabled (I'm essentially using CCR1009 as firewall in this case). So all 40x vlans are in one bridge, all 30x in another etc. Those are separate bridges on CCR. VLAN interfaces are assigned directly to physical interfaces and then those VLAN interfaces are put in appropriate bridges. Like this:


# jul/15/2018 03:11:01 by RouterOS 6.42.5
# software id = ...
#
# model = CCR1009-7G-1C-1S+
# serial number = ...
/interface vlan
add interface=combo1-vlan-rb mtu=2000 name=vlan3-rb vlan-id=300
add interface=sfp-sfpplus1 mtu=2000 name=vlan3.1-crs vlan-id=301
add interface=sfp-sfpplus1 mtu=2000 name=vlan3.2-crs vlan-id=302
add interface=sfp-sfpplus1 mtu=2000 name=vlan3.3-crs vlan-id=303
add interface=sfp-sfpplus1 mtu=2000 name=vlan3.4-crs vlan-id=304
add interface=sfp-sfpplus1 mtu=2000 name=vlan4-crs vlan-id=400
add interface=combo1-vlan-rb mtu=2000 name=vlan4-rb vlan-id=400
add interface=sfp-sfpplus1 mtu=2000 name=vlan4.1-crs vlan-id=401
add interface=sfp-sfpplus1 mtu=2000 name=vlan4.2-crs vlan-id=402
add interface=sfp-sfpplus1 mtu=2000 name=vlan4.3-crs vlan-id=403
add interface=sfp-sfpplus1 mtu=2000 name=vlan4.4-crs vlan-id=404
add interface=sfp-sfpplus1 mtu=2000 name=vlan4.5-crs vlan-id=405
add interface=sfp-sfpplus1 mtu=2000 name=vlan4.6-crs vlan-id=406
add interface=sfp-sfpplus1 name=vlan10-crs vlan-id=1000
add interface=combo1-vlan-rb name=vlan10-rb vlan-id=1000

/interface bridge port
add bridge=br-public interface=vlan3-rb
add bridge=br-primary interface=vlan4-rb
add bridge=br-primary interface=vlan4.1-crs
add bridge=br-primary interface=vlan4.2-crs
add bridge=br-primary interface=vlan4.3-crs
add bridge=br-primary interface=vlan4.4-crs
add bridge=br-primary interface=vlan4.5-crs
add bridge=br-primary interface=vlan4.6-crs
add bridge=br-public interface=vlan3.1-crs
add bridge=br-public interface=vlan3.2-crs
add bridge=br-public interface=vlan3.3-crs
add bridge=br-public interface=vlan3.4-crs
add bridge=br-primary interface=vlan4-crs
add bridge=br-service interface=vlan10-crs
add bridge=br-service interface=vlan10-rb

EDIT:

Actually it turns out bonding setup doesn't work even if there's no connection between switches and everything works if instead of bonding, single cable is used as second trunk. I'm confused...

lapsio · Sun Jul 15, 2018 5:36 am

It turns out sometimes VLAN interfaces on CCR1009 randomly don't get up and require disabling and reenabling... ._.

Switching loop - why? [SOLVED]

Switching loop - why?

Re: Switching loop - why? [SOLVED]

Who is online