CRS354 - vlans work only with specific networks

joshuapl · Sun Mar 17, 2024 8:16 pm

Hello,

I've been struggling for some time with setting up a small constellation of CRS354 and CRS326 switches in a large building, with each one connected by fiber link (I'm using 10Gbps LC/UPC links). These mikrotik's, together with RB1100AHx4 (this later) are going to replace current setup, with old Stormshield router and very old Netgear switches (usually GSM7248 and M4100).

I have been preparing the complete setup for some time, gathered all switches (4x CRS354 and 1x CRS326) on my desk, connected with fiber patchcords, with simulated environment (using old Netasq router to set up networks and dhcp, it was replaced by current Stormshield so configuration is very similar) connected to the main CRS354 which is going to be a main switch. I've learned a lot of things inbetween, like how is management network created etc.

And finally, when I moved to the site and, recreated the connections between switches and connected (of course in parallel to working infrastructure, I'm not that crazy!) all inputs to the main switch... some things worked, some didn't. And in quite a strange way...

So, first: interfaces ether1..ether7 are used for input networks. These ports are set up on bridge with their own PVIDs, so are (I assume) treated as untagged. Some of them are provided by Stormshield (it has 2 network providers and 4 local subnets), some by additional router RB3011 which has additional 2 network providers and 2 local subnets. This makes 6 subnets, but there is another thing - one WAN needs to be forwarded flawlessly to another server room, where it is connected to dedicated server with direct internet link.

When I tested the subnets, spread between switches (thank God I had an ethernet tester with DHCP), they mostly worked flawlessly. I was getting proper networks on proper ports. However, with three exceptions:
- two LANs provided by Mikrotik don't seem to be working
- WAN passthrough doesn't work, neither.

I tried various diagnosis methods, and finally in despair I have set up another untagged ports on the same CRS354 where original networks are connected. To be precise:
- PVID 109 (from Stormshield) on ether2 & ether45 (network 10.0.9.0/24)
- PVID 118 (from Mikrotik) on ether6 & ether46 (network 10.0.18.0/24)
- PVID 119 (from Mikrotik) on ether5 & ether47 (network 10.0.19.0/24)
- PVID 111 (unfiltered WAN) on ether7 & ether48

I have connected ports 45..48 to another small mikrotik (let's call it CLIENT) to diagnose

And the results are crazy. Network with PVID 109 is working fine, router 10.0.9.1 is easily pingable from CLIENT, but nothing else is. This would mean that I've done something wrong or other networks are somehow weird, but here comes the strangest thing: If I connect network 10.0.18 to ether2 then it is instantly reachable by client (of course I need to switch port on client, too). Same with 10.0.19. Doesn't help with unfiltered WAN, though - so that's even stranger. It looks like PVID 109 network for some reason is treated in different way... but why?

I am attaching my config, it's not that complex so I hope someone will be able to spot what I'm doing wrong.

# jan/26/1970 00:14:52 by RouterOS 7.8
# software id = DU6V-AYZV
#
# model = CRS354-48G-4S+2Q+
# serial number = HF209CSJ9S2
/interface bridge
add comment=defconf name=bridge vlan-filtering=yes
/interface vlan
add interface=bridge name=management vlan-id=251
add interface=bridge name=park vlan-id=102
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge comment="PARK in" interface=ether1 pvid=102
add bridge=bridge comment="ZSZ in" interface=ether2 pvid=109
add bridge=bridge comment="ZSZ-VIP in" interface=ether3 pvid=108
add bridge=bridge comment="Kosiba in" interface=ether4 pvid=106
add bridge=bridge comment="Bistro in" interface=ether5 pvid=119
add bridge=bridge comment="External in" interface=ether6 pvid=118
add bridge=bridge comment=defconf interface=ether7 pvid=111
add bridge=bridge comment=defconf interface=ether8 pvid=109
add bridge=bridge comment=defconf interface=ether9 pvid=109
add bridge=bridge comment=defconf interface=ether10 pvid=109
add bridge=bridge comment=defconf interface=ether11 pvid=109
add bridge=bridge comment=defconf interface=ether12 pvid=109
add bridge=bridge comment=defconf interface=ether13 pvid=109
add bridge=bridge comment=defconf interface=ether14 pvid=109
add bridge=bridge comment=defconf interface=ether15 pvid=109
add bridge=bridge comment=defconf interface=ether16 pvid=109
add bridge=bridge comment=defconf interface=ether17 pvid=109
add bridge=bridge comment=defconf interface=ether18 pvid=109
add bridge=bridge comment=defconf interface=ether19 pvid=109
add bridge=bridge comment=defconf interface=ether20 pvid=109
add bridge=bridge comment=defconf interface=ether21 pvid=109
add bridge=bridge comment=defconf interface=ether22 pvid=109
add bridge=bridge comment=defconf interface=ether23 pvid=109
add bridge=bridge comment=defconf interface=ether24 pvid=109
add bridge=bridge comment=defconf interface=ether25 pvid=109
add bridge=bridge comment=defconf interface=ether26 pvid=109
add bridge=bridge comment=defconf interface=ether27 pvid=109
add bridge=bridge comment=defconf interface=ether28 pvid=109
add bridge=bridge comment=defconf interface=ether29 pvid=109
add bridge=bridge comment=defconf interface=ether30 pvid=109
add bridge=bridge comment=defconf interface=ether31 pvid=109
add bridge=bridge comment=defconf interface=ether32 pvid=109
add bridge=bridge comment=defconf interface=ether33 pvid=109
add bridge=bridge comment=defconf interface=ether34 pvid=109
add bridge=bridge comment=defconf interface=ether35 pvid=109
add bridge=bridge comment=defconf interface=ether36 pvid=109
add bridge=bridge comment=defconf interface=ether37 pvid=109
add bridge=bridge comment=defconf interface=ether38 pvid=109
add bridge=bridge comment=defconf interface=ether39 pvid=109
add bridge=bridge comment=defconf interface=ether40 pvid=109
add bridge=bridge comment=defconf interface=ether41 pvid=109
add bridge=bridge comment=defconf interface=ether42 pvid=109
add bridge=bridge comment=defconf interface=ether43 pvid=109
add bridge=bridge comment=defconf interface=ether44 pvid=109
add bridge=bridge comment=defconf interface=ether45 pvid=109
add bridge=bridge comment=defconf interface=ether46 pvid=118
add bridge=bridge comment=defconf interface=ether47 pvid=119
add bridge=bridge comment=defconf interface=ether48 pvid=111
add bridge=bridge comment=defconf interface=ether49 pvid=251
add bridge=bridge comment=defconf interface=qsfpplus1-1
add bridge=bridge comment=defconf interface=qsfpplus1-2
add bridge=bridge comment=defconf interface=qsfpplus1-3
add bridge=bridge comment=defconf interface=qsfpplus1-4
add bridge=bridge comment=defconf interface=qsfpplus2-1
add bridge=bridge comment=defconf interface=qsfpplus2-2
add bridge=bridge comment=defconf interface=qsfpplus2-3
add bridge=bridge comment=defconf interface=qsfpplus2-4
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
/interface bridge vlan
add bridge=bridge comment=Management tagged=bridge,qsfpplus1-1,sfp-sfpplus1,sfp-sfpplus3,sfp-sfpplus4 untagged=ether49 vlan-ids=251
add bridge=bridge comment="ZSZ INF" tagged=qsfpplus1-1,sfp-sfpplus1,sfp-sfpplus3,sfp-sfpplus4 untagged=ether2 vlan-ids=109
add bridge=bridge comment=Park tagged=qsfpplus1-1,sfp-sfpplus1,sfp-sfpplus3,sfp-sfpplus4,bridge untagged=ether1 vlan-ids=102
add bridge=bridge comment="ZSZ VIP" tagged=qsfpplus1-1,sfp-sfpplus1 untagged=ether3 vlan-ids=108
add bridge=bridge comment=Kosiba tagged=qsfpplus1-1,sfp-sfpplus4,sfp-sfpplus3 untagged=ether4 vlan-ids=106
add bridge=bridge comment=Bistro tagged=sfp-sfpplus3 untagged=ether5 vlan-ids=119
add bridge=bridge comment=External tagged=qsfpplus1-1 vlan-ids=118
add bridge=bridge comment="WAN ZSZ" tagged=sfp-sfpplus1 vlan-ids=111
add bridge=bridge comment=ext tagged=sfp-sfpplus2 vlan-ids=18
/interface list member
add interface=ether49 list=WAN
add interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add interface=ether11 list=LAN
add interface=ether12 list=LAN
add interface=ether13 list=LAN
add interface=ether14 list=LAN
add interface=ether15 list=LAN
add interface=ether16 list=LAN
add interface=ether17 list=LAN
add interface=ether18 list=LAN
add interface=ether19 list=LAN
add interface=ether20 list=LAN
add interface=ether21 list=LAN
add interface=ether22 list=LAN
add interface=ether23 list=LAN
add interface=ether24 list=LAN
add interface=ether25 list=LAN
add interface=ether26 list=LAN
add interface=ether27 list=LAN
add interface=ether28 list=LAN
add interface=ether29 list=LAN
add interface=ether30 list=LAN
add interface=ether31 list=LAN
add interface=ether32 list=LAN
add interface=ether33 list=LAN
add interface=ether34 list=LAN
add interface=ether35 list=LAN
add interface=ether36 list=LAN
add interface=ether37 list=LAN
add interface=ether38 list=LAN
add interface=ether39 list=LAN
add interface=ether40 list=LAN
add interface=ether41 list=LAN
add interface=ether42 list=LAN
add interface=ether43 list=LAN
add interface=ether44 list=LAN
add interface=ether45 list=LAN
add interface=ether46 list=LAN
add interface=ether47 list=LAN
add interface=ether48 list=LAN
add interface=qsfpplus1-1 list=LAN
add interface=qsfpplus1-2 list=LAN
add interface=qsfpplus1-3 list=LAN
add interface=qsfpplus1-4 list=LAN
add interface=qsfpplus2-1 list=LAN
add interface=qsfpplus2-2 list=LAN
add interface=qsfpplus2-3 list=LAN
add interface=qsfpplus2-4 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
/ip address
add address=192.168.251.1/24 comment=defconf interface=management network=192.168.251.0
add address=10.0.2.11/24 interface=park network=10.0.2.0
/ip firewall filter
add action=drop chain=input disabled=yes in-interface=*3F port=67 protocol=udp
/system identity
set name=Router354-1-PPD3-S3-1
/system routerboard settings
set boot-os=router-os enter-setup-on=delete-key

Thanks in advance!
--
Jacek

joshuapl · Sun Mar 17, 2024 10:24 pm

OK, few more thoughts since I've finally written

First of all, I was a bit confused while configuring VLANs on bridge: what is the purpose of defining untagged interfaces in /interface/bridge/vlan? If I set a PVID for an interface, it automatically appears as untagged interface for this VLAN. I see, that if I set PVID specific to one vlan and set it as untagged on another vlan, it appears as "CURRENT-UNTAGGED" for both vlans when executing /interface/bridge/vlan/print - but probably wouldn't work well for the network with other PVID, I assume. Nevertheless it's really confusing and I wonder what was the idea.

Another thing: this unfiltered network is connected to ether7. When I try sniffing on this port, I get something like this:

ether7     133.809   72  <-   CC:3E:5F:DF:87:80  01:80:C2:00:00:00  802.2       64    0
ether7     135.809   73  <-   CC:3E:5F:DF:87:80  01:80:C2:00:00:00  802.2       64    0
ether7     137.809   74  <-   CC:3E:5F:DF:87:80  01:80:C2:00:00:00  802.2       64    0
ether7     139.81    75  <-   CC:3E:5F:DF:87:80  01:80:C2:00:00:00  802.2       64    0
ether7     141.809   76  <-   CC:3E:5F:DF:87:80  01:80:C2:00:00:00  802.2       64    0
ether7     143.809   77  <-   CC:3E:5F:DF:87:80  01:80:C2:00:00:00  802.2       64    0

However, once I set ether7 as disabled on bridge, I immediately see the proper traffic on that port.

Why does it behave this way? ether7 is an untagged port

Greetings,
--
Jacek

joshuapl · Sun Mar 17, 2024 10:34 pm

Oh, and one more thing: I know, that management interface's vlan has to be set as tagged on bridge interface. Does it have any effect on other vlans?

anav · Sun Mar 17, 2024 11:29 pm

For all your switches, only the manag3ment vlan need be identified..... (assuming its 192.168.251.0/24)
I would take one port off bridge and use it as an emerg access like give it an IP address of 192.168.55.1/24 and then any pc with IPV4 settings set to 192.168.55.5 for example and your in!

/interface list
add name=MGMT
/interface list members
add interface=management list=MGMT
add interface=emergaccessport list=MGMT ( if you elect to have one )

Complete the bridge ports....... example of two lines only' ( first for access ports, second for trunk ports )
add bridge=bridge comment="PARK in" interface=ether1 pvid=102 ingress-filtering=yes frame-types=admit-only-priority-and-untagged
add bridge=bridge comment=defconf interface=qsfpplus1-1 ingress-filtering=yes frame-types=admit-only-vlan-tagged

/ip neighbor discovery-settings
set discover-interface-list=MGMT
/ip dns
set allow-remote-requests=yes servers=192.168.251.1
/ip route
add dst-address=0.0.0.0/0 gateway=192.168.251.1 routing-table=main
/tool mac-server
set allowed-interface-list=MGMT
/tool mac-server mac-winbox
set allowed-interface-list=MGMT

joshuapl · Mon Mar 18, 2024 1:36 pm

Thanks for clarification. I was experimenting with setting other vlans as I had some strange behaviour with one of the switches being unreachable through management net as long as it had another vlan set as untagged on one of the SFP ports. Actually, 192.168.251.0/24 is exactly meant to be set manually on computer's interface to access all the switches - by connecting to separate Management (ether49) port on CRS354 and ether24 on CRS326. I didn't remove it from bridge since it makes me able to reach all switches from one of them without having to walk kilometers and numerous stairs

Thanks for other suggestions on bridge ports. Actually ingress-filtering=yes is set, I should have done "verbose" export. But "frame-types=admit-only-priority-and-untagged" on access ports was not set and it is a good idea! Do I assume correctly, that without this a hacker could connect to any access port and by setting VLAN 251 tagged would be able to access management network?

Greetings,
--
Jacek

CRS354 - vlans work only with specific networks

CRS354 - vlans work only with specific networks

Re: CRS354 - vlans work only with specific networks

Re: CRS354 - vlans work only with specific networks

Re: CRS354 - vlans work only with specific networks

Re: CRS354 - vlans work only with specific networks

Who is online