VLAN Issues

kieranmcdonnell · September 22, 2020, 4:05pm

Hi,
I’m a Mikrotik newbie, having some strange issues trying to use a Mikrotik in an existing network.
We bought a CRS328-4C-20S-4S+ to move our existing point-to-point links off an old 3750X and increase our fibre density.

The remote sites use VLANs that are trunked through the 3750X/CRS328 to the core (Cisco 6807).
We’ve had to put out some new links recently due to Covid, so we have 3 sites on each device currently.

The Mikrotik is in RouterOS mode and has a single brdige. There is a VLAN for each remote site, with the uplink interface and the remote site interface configured as tagged.
When I try to do a test ping through a new LACP bond on sfp-plus3 and sfp-plus4 I am getting intermittent packet loss when traversing the Mikrotik.
If I put a VLAN interface on the Uplink bond I get no loss. If I move it to the bridge I start to get packet loss.

We want to use 2 links out of the CRS328 so we have a link to each core to provide additional fault tolerance, having them at 10G so that it reduces the contention on the sites connected into it.
However I’m worried about moving over the main uplink to the Mikrotik due to the packet loss. Pinging from the 3750X to the CSR328 gets 100% response. Changing the uplink to a single non-LACP 10G link doesn’t appear to have any impact on the response rate.

I’m not sure what I’m missing with the Mikrotik as this would be simple on a Cisco switch, which I’m familiar with.

I work for a UK NHS trust, so the remote sites are all providing patient care to the public so I want to be confident before I move everything over.

I’ll put together a diagram of the connection now, but if anyone has any advice on where I should start looking or what information I can provide to get better assistance I’d appreciate it.

kieranmcdonnell · September 23, 2020, 8:41am

The black line is a VLAN for device management. The coloured lines indicate tagged VLANs.

In the current setup with the VL12 and 13 going through the CRS328, through a trunk to the 3750X and to the core.
My plan was to bring the bonded link to the core and enable the VLAN path through the CRS328 to the core, then remove it from the 3750X’s link to the core. This would allow me to move site by site in a change window.
However when I tried to move the management VLAN the packet lost meant I couldn’t configure the 3750X through the CRS328.

tdw · September 23, 2020, 9:56am

Post the output of /export hide-sensitive, preferably in a code block (the square brackets icon) to make it more readable.

kieranmcdonnell · September 23, 2020, 2:11pm

Thanks for responding.
VL910 is currently used for management. VL671 is being used as the test VLAN.

Here’s the export from the CRS328

# sep/23/2020 14:57:32 by RouterOS 6.46.6
# software id = CB73-39XZ
#
# model = CRS328-4C-20S-4S+
/interface ethernet
set [ find default-name=combo4 ] comment="Interswitch Link"
set [ find default-name=sfp-sfpplus3 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,10000M-full
set [ find default-name=sfp-sfpplus4 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full,10000M-full
set [ find default-name=sfp1 ] comment="Equinox House [Ramsay]"
set [ find default-name=sfp2 ] comment=Archways
set [ find default-name=sfp3 ] comment="Tang Hall HC"
/interface bridge
add admin-mac=C4:AD:34:CE:20:0F auto-mac=no comment=defconf name=bridge
/interface vlan
add interface=bridge name=DRS_Test vlan-id=671
add interface=bridge name=MGMT vlan-id=910
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface bridge port
add bridge=bridge comment=defconf interface=combo1
add bridge=bridge comment=defconf interface=combo2
add bridge=bridge comment=defconf interface=combo3
add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=combo4
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp1 pvid=553
add bridge=bridge comment=defconf interface=sfp2 pvid=777
add bridge=bridge comment=defconf interface=sfp3
add bridge=bridge comment=defconf interface=sfp4
add bridge=bridge comment=defconf interface=sfp5
add bridge=bridge comment=defconf interface=sfp6
add bridge=bridge comment=defconf interface=sfp7
add bridge=bridge comment=defconf interface=sfp8
add bridge=bridge comment=defconf interface=sfp9
add bridge=bridge comment=defconf interface=sfp10
add bridge=bridge comment=defconf interface=sfp11
add bridge=bridge comment=defconf interface=sfp12
add bridge=bridge comment=defconf interface=sfp13
add bridge=bridge comment=defconf interface=sfp14
add bridge=bridge comment=defconf interface=sfp15
add bridge=bridge comment=defconf interface=sfp16
add bridge=bridge comment=defconf interface=sfp17
add bridge=bridge comment=defconf interface=sfp18
add bridge=bridge comment=defconf interface=sfp19
add bridge=bridge comment=defconf interface=sfp20
add bridge=bridge comment="LACPUplink to tex/1/31" frame-types=admit-only-vlan-tagged interface=*23 pvid=777
add bridge=bridge interface=sfp-sfpplus3
/interface bridge vlan
add bridge=bridge comment="sfp1 - Equinox House [Ramsay]" tagged=*21,sfp1,combo4 vlan-ids=553
add bridge=bridge comment="Clifton Park [Ramsay] - 3750X" tagged=*21,sfp2,combo4 vlan-ids=555
add bridge=bridge comment="sfp2 - Archaways" tagged=sfp2,*21,combo4 vlan-ids=560
add bridge=bridge comment="sfp3 - Tang Hall HC" tagged=sfp3,*21,combo4 vlan-ids=554
add bridge=bridge comment="Occupational Health - 3750X" tagged=sfp4,*21,combo4 vlan-ids=559
add bridge=bridge comment="Management VLAN" tagged=*21,combo4 untagged=bridge vlan-ids=910
add bridge=bridge comment="DRS - to be moved" tagged=combo4,*23 vlan-ids=671
add bridge=bridge comment="Medical Records - to be moved - 3750X" tagged=*21,sfp5,combo4 vlan-ids=670
add bridge=bridge comment="DRS - New VLAN" tagged=*21,sfp6,combo4 vlan-ids=551
add bridge=bridge comment="Medical Records - New VLAN - 3750X" tagged=*21,sfp5,combo4 vlan-ids=552
/ip address
add address=192.168.254.81/29 interface=MGMT network=192.168.254.80
add address=192.168.195.53/29 interface=DRS_Test network=192.168.195.48
/ip route
add distance=1 gateway=192.168.254.86
/system clock
set time-zone-name=Europe/London
/system routerboard settings
set boot-os=router-os
/tool sniffer
set filter-interface=*23 filter-ip-address=192.168.195.48/29
/tool user-manager database
set db-path=flash/user-manager

3750X Uplink to CRS328

interface GigabitEthernet1/0/48
 description Mikrotik-P2P
 switchport trunk allowed vlan 551-555,559,560,670,671,910
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 777
 switchport mode trunk

6807 To CSR328 [Currently single 10G]

interface TenGigabitEthernet2/1/31
 description Physical ports to Mikrotik-P2P
 switchport
 switchport mode trunk
 switchport trunk allowed vlan 671

6807 to CSR328 [Bonded config]

int range Te1/1/31,Te2/1/31
 description Physical ports to Mikrotik-P2P
 switchport
 switchport mode trunk
 switchport trunk allowed vlan 671
 channel-group 131 mode active
!
interface Port-channel131
 description LACP to Mikrotik-P2P
 switchport
 switchport mode trunk
 switchport trunk allowed vlan 671

tdw · September 23, 2020, 2:51pm

Firstly the Mikrotik bridge is currently operating as an unmanaged switch, the pvid= settings in the /interface bridge port section and all of the /interface bridge vlan section are ignored until the bridge has the vlan-filtering=yes setting.

It appears you have also deleted some interfaces which has caused the references to *21 and *23 in various places. Although interfaces appear to have names they are internally referenced by an index so if you delete an interface, e.g. a bond, and recreate it there will be a different index even if you use the same name.

If you have PVST+ configured on the Cisco interfaces it doesn’t play nicely with the default RSTP on the Mikrotik.

From the interface comments I’m guessing you are in York.

tdw · September 23, 2020, 3:20pm

For info Cisco trunks with a native VLAN, e.g.
interface SOMEPORT
switchport trunk encapsulation dot1q
switchport trunk native vlan R
switchport trunk allowed vlan R,S,T
switchport mode trunk
translates to
/interface bridge port
add bridge=bridge ingress-filtering=yes interface=PORTNAME pvid=R
/interface bridge vlan
add bridge=bridge tagged=…,PORTNAME,… vlan-ids=S
add bridge=bridge tagged=…,PORTNAME,… vlan-ids=T

and purely tagged trunks, e.g.
interface SOMEPORT
switchport trunk encapsulation dot1q
switchport trunk allowed vlan S,T
switchport mode trunk
translates to
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=PORTNAME
/interface bridge vlan
add bridge=bridge tagged=…,PORTNAME,… vlan-ids=S
add bridge=bridge tagged=…,PORTNAME,… vlan-ids=T

kieranmcdonnell · September 23, 2020, 11:09pm

Yes, I’ve created and removed the bond a few times trying to work out where my issues were.
I think the Cisco core is configured on RPVST, so I’ll have a look at moving that to RSTP.

I’ll read over your config pairs and try and get better at the Mikrotik CLI. Thank you very much for taking time to help!

tdw · September 24, 2020, 10:57am

Switching to RSTP would depend on your core network - it isn’t suitable for network topologies which pass groups of VLANs along differing redundant paths, you would have to use MSTP which the CRS3xx also support without loosing hardware offloading.

As you don’t particularly need spanning tree in this scenario you could certainly block/filter them to evaluate the Mikrotik.