SOLVED: bridged VLANs are not working anymore

Spartacus · March 20, 2019, 6:54pm

Hi all,
I have configured bridged VLANs with VLAN-Filtering some time ago. Everything worked fine for months and my subnets did not see each other. This was done under Router-OS 6.4.
In order to allow some Subnets to communicate, I installed a FW rule:

add action=accept chain=forward comment=\
    "Allow inter VLAN communication with VLAN friends" dst-address-list=\
    VlanFriends in-interface-list=LAN src-address-list=VlanFriends

List “VlanFriends” contains the list of the Subnets which can communicate each other.
As mentioned everything works fine since months. But now I noticed that the FW-rule will not apply anymore. I can disable this rule and communication between all subnets is possible. I am confused, because nothing changed since initial configuration except OS Update. Current OS is now 6.44.1. I noticed changes in bridge setup and I found some new options.

I do not understand the new options. Ca this be the reason why communication between subnets is possible without the rule above?The yellow marked fileds are new and do I have to change something?

Christian

anav · March 20, 2019, 7:35pm

Spartacus my dear friend. Still in the VLAN battle I see.
Well rest your shield and sword for a bit and post your latest config.
/export hide-sensitive file=yourconfigmar20.

I will take up the cause and defend the ramparts while you take a break!

Spartacus · March 20, 2019, 9:18pm

Hi anav,
great! You are still there, and yes, I am fighting with my router again!

I was wondering about this topic today and I am confused. I gues that changes in the OS may couse the issue. Please find below the config. I have shorten it because it shows more than 1000lines and i guess CAPSMAN config, DHCP and some other things are not relevant for my issue. But FW-Rules and interface settings should be complete. I also removed the address Lists. They show only a couple of Clients and subnets. If anything is missing, please let me know: Thanks for support

Regards,
Christian

/interface list member
add comment=Elementmedia interface=ether1 list=WAN
add comment=Admin interface=vlan1 list=LAN
add comment=Office interface=vlan10 list=LAN
add comment=VoiP interface=vlan20 list=LAN
add comment=FritzBox interface=vlan99 list=LAN
add interface=ether8 list=MAC-WinBox
add comment=Sonos interface=vlan30 list=LAN
add comment=IPTV interface=vlan40 list=LAN
add comment=SmartHome interface=vlan50 list=LAN
add comment=WLAN interface=vlan60 list=LAN
add comment=Gast interface=vlan70 list=LAN
add comment=Gaming interface=vlan80 list=LAN
add comment=SONOS disabled=yes interface=vlan99 list="Sonos Control"
add comment=SONOS interface=vlan60 list="Sonos Control"
add comment=SONOS interface=vlan10 list="Sonos Control"
add interface=vlan10 list=IPTV
add interface=vlan40 list=IPTV
add interface=vlan60 list=IPTV
add comment=SONOS interface=vlan50 list="Sonos Control"

/interface bridge
add fast-forward=no igmp-snooping=yes name=br_vlan protocol-mode=stp \
    vlan-filtering=yes
	
/interface vlan
add comment=Admin interface=br_vlan name=vlan1 vlan-id=1
add comment=Office interface=br_vlan name=vlan10 vlan-id=10
add comment=VoiP interface=br_vlan name=vlan20 vlan-id=20
add comment=Sonos interface=br_vlan name=vlan30 vlan-id=30
add comment=IPTV interface=br_vlan name=vlan40 vlan-id=40
add comment=SmartHome interface=br_vlan name=vlan50 vlan-id=50
add comment=WLAN interface=br_vlan name=vlan60 vlan-id=60
add comment=Gast interface=br_vlan name=vlan70 vlan-id=70
add comment=Gaming interface=br_vlan name=vlan80 vlan-id=80
add comment=FritzBox interface=br_vlan name=vlan99 vlan-id=99

/interface bridge port
add bridge=br_vlan hw=no interface=sfp1
add bridge=br_vlan frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2 pvid=10
add bridge=br_vlan frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether3 pvid=30
add bridge=br_vlan frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether4 pvid=20
add bridge=br_vlan frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether10 pvid=99
add bridge=br_vlan frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether5 pvid=50
add bridge=br_vlan interface=ether6

/interface bridge vlan
add bridge=br_vlan comment=Admin tagged=br_vlan,vlan1 untagged=sfp1,ether6 \
    vlan-ids=1
add bridge=br_vlan comment=Office tagged=sfp1,br_vlan,vlan10 untagged=ether2 \
    vlan-ids=10
add bridge=br_vlan comment=VoIP tagged=sfp1,br_vlan,vlan20 untagged=ether4 \
    vlan-ids=20
add bridge=br_vlan comment=FritzBox tagged=sfp1,br_vlan,vlan99 untagged=\
    ether10 vlan-ids=99
add bridge=br_vlan comment=SmartHome tagged=sfp1,br_vlan,vlan50 untagged=\
    ether5 vlan-ids=50
add bridge=br_vlan comment=Sonos tagged=sfp1,br_vlan,vlan30 untagged=ether3 \
    vlan-ids=30
add bridge=br_vlan comment=IPTV tagged=sfp1,br_vlan,vlan40 vlan-ids=40
add bridge=br_vlan comment=WLAN tagged=sfp1,br_vlan,vlan60 vlan-ids=60
add bridge=br_vlan comment=Gast tagged=sfp1,br_vlan,vlan70 vlan-ids=70
add bridge=br_vlan comment=Gaming tagged=sfp1,br_vlan,vlan80 vlan-ids=80

/ip firewall filter
add action=accept chain=input comment="accept established,related" \
    connection-state=established,related
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="allow AdminSubnet" in-interface-list=\
    LAN src-address-list=AdminSubnet
add action=accept chain=input comment="Allow LAN DNS queries-TCP" dst-port=53 \
    in-interface-list=LAN protocol=tcp
add action=accept chain=input comment="Allow LAN DNS queries-UDP" dst-port=53 \
    in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN NTP queries" dst-port=123 \
    in-interface-list=LAN protocol=udp
add action=drop chain=input comment=" drop everything"
add action=fasttrack-connection chain=forward comment=\
    " fasttrack established,related" connection-state=established,related
add action=accept chain=forward comment=" accept established,related" \
    connection-state=established,related
add action=drop chain=forward comment="prevent SPAM" dst-port=25 \
    in-interface-list=LAN out-interface-list=WAN protocol=tcp
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=accept chain=forward comment=\
    "SONOS. Forward UPnP Device Discovery events from Players" in-interface=\
    vlan30 out-interface-list="Sonos Control" port=1900,1901,6969 protocol=\
    udp
add action=accept chain=forward comment="SONOS: forward Multicast traffic" \
    dst-address=239.255.255.250 log-prefix=MultiCast
add action=accept chain=forward comment=\
    "SONOS: Forward Contoller events  from Players" in-interface-list=\
    "Sonos Control" log=yes log-prefix=FromPlayer out-interface=vlan30 port=\
    3400,3401,3500,4444,4070,5353 protocol=tcp
add action=accept chain=forward comment=\
    "SONOS: forward  Controller events to Players" dst-port="" in-interface=\
    vlan30 log-prefix=ToPlayer out-interface-list="Sonos Control" port=\
    3400,3401,3500,4444,4070,5353 protocol=tcp
add action=accept chain=forward comment=\
    "accept Internet Access from \"Allow WAN\"" in-interface-list=LAN \
    out-interface-list=WAN src-address-list=AlllowWAN
add action=accept chain=forward comment="Accept AdminSubnet-> PrivateSubnet" \
    dst-address-list=PrivateSubnets in-interface-list=LAN src-address-list=\
    AdminSubnet
add action=accept chain=forward comment=\
    "Allow inter VLAN communication with VLAN friends" disabled=yes \
    dst-address-list=VlanFriends in-interface-list=LAN src-address-list=\
    VlanFriends
add action=drop chain=forward comment="drop everything" log=yes log-prefix=\
    drop
/ip firewall nat
add action=redirect chain=dstnat comment=\
    "Force Users to Router for DNS - TCP" dst-port=53 protocol=tcp
add action=redirect chain=dstnat comment=\
    "Force Users to Router for DNS - UDP" dst-port=53 protocol=udp
add action=masquerade chain=srcnat comment="masquerade LAN->WAN" \
    out-interface=ether1 src-address-type=""

anav · March 21, 2019, 2:25am

****My first comment would be to NOT use vlan=1 if possible.
Create vlan11 for admin vlan.
Bridges would still retain their default pvid setting of 1, but the key here is not to assign a dhcp subnet to the bridge!! (instead set to vlan11)!
By the way that is a quick change (ip address to vlan11 interface and dhcp-server to vlan 11 interface - no need to change pool or dhcp server network easy peasy)
I refuse to ensure your interface list members is accurate LOL
What happened to eth 6,7,8,9,10 LOL

This rule looks to be wrong from my perspective. I dont think you tag VLANS, just ethports,wlans!! That could be one significant error.
add bridge=br_vlan comment=Admin tagged=br_vlan,vlan1 untagged=sfp1,ether6
vlan-ids=1

in any case if we move to vlan11 for the admin network it would be…
/interface bridge vlan
add bridge=br_vlan comment=Admin tagged=br_vlan, untagged=sfp1ether6 \ ******** see below for another modification and why.**
vlan-ids=11

FOR THE REST OF YOUR interface bridge vlan rules, REMOVE THE VLAN from the tagged or untagged portions (vlanIDs are the only way to identify vlans in these rules)

Including these four
add bridge=br_vlan comment=IPTV tagged=sfp1,br_vlan,vlan40 vlan-ids=40
add bridge=br_vlan comment=WLAN tagged=sfp1,br_vlan,vlan60 vlan-ids=60
add bridge=br_vlan comment=Gast tagged=sfp1,br_vlan,vlan70 vlan-ids=70
add bridge=br_vlan comment=Gaming tagged=sfp1,br_vlan,vlan80 vlan-ids=80

Personally since there is no difference in the interfaces listed here (all sfp1, br-vlan) one could write it as
add bridge=br_vlan comment=IPTV-WLAN-Gast-Gaming tagged=sfp1,br_vlan vlan-ids=40,60,70,80

******* HOWEVER it is clear that sfp1 is a trunk PORT.
/interface bridge port
add bridge=br_vlan hw=no interface=sfp1

and yet (in this rule I want you to get rid of…
/interface bridge vlan
add bridge=br_vlan comment=Admin tagged=br_vlan,vlan1 untagged=sfp1,ether6
vlan-ids=1

sfp1 is untagged as if it was an access port???
Therefore the new correct rule should be
/interface bridge vlan
add bridge=br_vlan comment=Admin tagged=br_vlan, sfp1 untagged=ether6
vlan-ids=11

I will stop there and let you adjust as that may fix all as you see fit and the Firewall rules need not be reviewed.

Spartacus · March 21, 2019, 8:54am

Hi anaw,
thanks for quick response. I do not understand all of your comments, but I have to be very careful with what I do, becauseit is in production and small changes can caouse an big issue.
I will start with the admin vlan ID. Not sure how long this takes, because it is long time ago that i configured this.

And yes, sfp1 is the trunk port and a cisco SG200 is connected. I also have to change vlan1 on this device. Let me start with this and I come back if have have changed.

And maybe you can explai this here a little more detailled. I do not understand, what you mean with:

This rule looks to be wrong from my perspective. I dont think you tag VLANS, just ethports,wlans!! That could be one significant error.
add bridge=br_vlan comment=Admin tagged=br_vlan,vlan1 untagged=sfp1,ether6
vlan-ids=1

and btw:
eth 6-10 are not connected

Thanks,
Christian

Spartacus · March 21, 2019, 9:40am

Hi,
ok. Seems to be not so easy to change Management VLAN. The Cisco devices and the CAPs are configured on VLAN1. If I change it on the Mikrotik from ID1 to ID11, I cannot access my Cisco switches anymore and I am not sure how to change the management vlan of the cisco from VLAN1 to Vlan11

From SFT1 of the Microtik router, all VLANS were trunked to the Cisco SG200 n(GE25) and from the SG200 (GE26) to a SG250. CAP 1 is connected to Port 11 and CAP2 to Port 12. All Ports are tTrunk Ports

(e.g. GE25 Trunk 1UP, 10T, 11T, 20T, 30T, 40T, 50T, 60T, 70T, 80T…).

VLAN1 is the default VLAN and untagged. If I add VLAN11 I can only trunk it as an tagged VLAN and I am not sure, if I can connect to the switches. I found settings on the Cisco where I can configure the VLAN ID. But I am afraid to change it to VLAN11

Isn´t it much more easy to change PVID from br_vlan into 11 on Mikrotik?

Christian

Spartacus · March 21, 2019, 11:07am

Hi anav,
let me summarize what I did until now: I hope I understood correctly:

changed the vlan settings under “Bridge” and I removed the VLANS itselves from “tagged” and “untagged” section.
I also changed SFP1 from “untagged” to “tagged” in Admin LAN

Seems to be working. I cannot notice any changes from prior configuration:
I configured a VLAN 11 on both Cisco Switches

Next step should be to change Admin LAN from ID 1 to ID 11, but I am a little bit afraid to loosing connection to the Cisco Switches (see post above)
Maybe you can tell me, how to avoid it.
Christian

anav · March 21, 2019, 12:03pm

Hi Spartacus, You dont have to change your admin vlan if its going to cause other issues with other equipment. In your case it may be just smarter to let it work as is.

I went through the pain of this with
dlink managed switch 24 port
GS110 netgear managed switch
TWO 260GS MT low frill managed switches
TWO capAC access points.

Finally got it all working but it caused me all kinds of grief.
In all cases, the bridge on my router has default pvid=1 (as per the standard setup that pcunite developed)
all switches still have pvid=1 as the default on the incoming trunk port from the router (so dont change this on the trunk port or in general settings).
all switches have vlan11 tagged on the trunk port.
my capacs still have pvid=1 on their bridge with vlan11 tagged on bridge

The changes you have made for fixing the obvious errors, seems to have done the trick!!!
If you are going to try changes ensure you have a large block of hours that will not interfere with anything LOL

Spartacus · March 21, 2019, 12:25pm

Hi anav,
the changes I made did not have any effects. This means, that Access accross all Subnets is possible. And all changes I did during the last 5,6 months is an OS Update. Seems to be that something happend here.

I need to separate the VLANs from each other but not sure, where I can start. I am also not sure, if this is depending on the PVID of the Bridge or the Switches. So what can I do next?
To change the Default VLAN is tricky and if I switch VLAN1 to VLAN11 access disapears. I think I have to setup a new VLAN in parallel and switch device by device to the new Admin LAN. I can use VLAN99 for this, because it is no longer used and I will configure this as 172.16.99.x. with PVID99. But the question is, if this is the solution? II don´t know.

What do you think should be the next step?
Christian

mkx · March 21, 2019, 1:13pm

For starters, vlanxx interfaces have no place in this config section:

/interface bridge vlan

Spartacus · March 21, 2019, 1:44pm

Hi
@mkx:
What does this mean? I do not understand this.

@ all
I cannot access the router anymore with winbox. not sure what happend. And I also cannot reset the RB3011 via Reset Button. It is ignored. Does anyone has an Idea how to get access to the RB3011 again?

mkx · March 21, 2019, 2:14pm

For example:

/interface bridge vlan
add bridge=br_vlan comment=Office tagged=sfp1,br_vlan> ,vlan10 > untagged=ether2 vlan-ids=10

The part, painted in red should not be there. And that’s for all configuration lines in that section.

Can you connect using webfig (normal http connection)? Can you connect using MAC winbox?

anav · March 21, 2019, 2:40pm

Mkx, I specifically made heavy usage of bold and colour just for you to see what changes I had recommended. Glad that you agreed with me, or should I humbly say, the master confirmed the novices advice LOL.

Spartacus… you must have failed the most important commandment.
THOU SHALT NOT CONFIGURE MIKROTIK ROUTER WITHOUT SAFE MODE ENABLED>
You have sinned and will be punished.

Sorry dude, I thought after one of your jpegs you stated
“Seems to be working. I cannot notice any changes from prior configuration:”

and I took that as whatever issues you were having were solved. Perhaps I will have a look at your FW rules then.

Spartacus · March 21, 2019, 2:56pm

Hi anav,
I am back on the Router. And I know what happend. I disabled this rule:

add action=accept chain=input comment="allow AdminSubnet" in-interface-list=\
    LAN src-address-list=AdminSubnet

and this was why i cannot connet to the Router. Reset worked and now I recovered the Config.

But:“Seems to be working. I cannot notice any changes from prior configuration:” means, that the changes I made today did not have any effect on the issue from yesterday. And this means, that the issue with VLANS is still present. The Rule

add action=accept chain=forward comment=\
    "Allow inter VLAN communication with VLAN friends" dst-address-list=\
    VlanFriends in-interface-list=LAN src-address-list=VlanFriends

is disabled and each VLAN can communicate with each other. This is not correct. I disabled all rules in the FW and it is the same. VLANS can communicate each other. And this was not the case when I entered the initial Config 5,6 Months ago. Due to no configuration chages, something happend to the OS.

Christian

anav · March 21, 2019, 3:17pm

Hi Good Sir,
I will have a look at your rules FW rules as the vlan config seems pretty solid.

Nothing major found yet…
(1) Dont see why you need this rule???
add action=accept chain=input comment=“Allow LAN NTP queries” dst-port=123
in-interface-list=LAN protocol=udp
I have NTP setup up as per below and don’t have any such rule in my input chain???
/system ntp client
set enabled=yes server-dns-names=time.nrc.ca,time,nrc.chu.ca

(2) add action=drop chain=forward comment=“prevent SPAM” dst-port=25
in-interface-list=LAN out-interface-list=WAN protocol=tcp.
I would do this slightly differently
BUT ONLY IF YOU DO NOT USE PORT 25 for anything else!!!
Same rule but in raw firewall rules…
add action=drop chain=prerouting comment=“prevent SPAM” dst-port=25
protocol=tcp. This would stop cold any traffic heading for port 25 in any direction to/from and across the router.

(3) This rule is confusing to me… its wide open…
add action=accept chain=forward comment=“SONOS: forward Multicast traffic”
dst-address=239.255.255.250 log-prefix=MultiCast

This allows all traffic coming from the internet and from your lan that is headed to that particular dst-address???
I dont even know where to begin to understand if this is good bad ugly or normal…. ???
This may be the source of leakage??

(4) Okay this one allows sonos control traffic to vlan30. Makes sense. Of course in the case of vlan30 to sonos control and sonos control to vlan30.
Return traffic is permitted on the same ports. (thus if let vlan30 hit sonos control on port 1900 the router would allow return traffic to vlan 30 on port 1900 (its established) but it seems that its just a control port and that the sonos control responds on different ports and thats what you are trying to setup??? Iin fact like FTP, established AND related traffic is allowed and thus if the sonos opens up ports to talk back to vlan 30 you may not need this rule. Just not sure how the whole SONOS thing works out. In any case, not the issue right now.

add action=accept chain=forward comment=
“SONOS: Forward Contoller events from Players” in-interface-list=
“Sonos Control” log=yes log-prefix=FromPlayer out-interface=vlan30 port=
3400,3401,3500,4444,4070,5353 protocol=tcp

(5) Lets try to narrow our forward rules a touch to be more accurate. In other words, lets refrain from over use of in-interface-list=LAN
Can you define an interface list for allowed wan? and then just change in-interface-list=LAN with the new list but keep everything else.

add action=accept chain=forward comment=
“accept Internet Access from "Allow WAN"” in-interface-list=LAN \ (modify this line)
out-interface-list=WAN src-address-list=AlllowWAN (keep this line)

(6) Its not clear to me what this rule is trying to accomplish?? Please explain.

add action=accept chain=forward comment=“Accept AdminSubnet-> PrivateSubnet”
dst-address-list=PrivateSubnets in-interface-list=LAN src-address-list=
AdminSubnet

Spartacus · March 21, 2019, 5:26pm

Hi anav,
I do not think that the issue is in the FW rules. It must be within the VLAn Config because:

when I initally setup the Router, I startet with VLAN config and I have had no implemented rules. FW was blank. The effect was, that VLANs could not see each other. This was the purpose.
I configured the VLANFrends-Rule in order to route between specific Subnets (all within the list)
today I disconnected router from WAN and removed all the rules and routing between VLANS was possible.

It must be within the VLAN-Config, or are my thoughts wrong?

The Rule #6 is bullshit. No idea why this is there. I have disabled it.

Christian

Spartacus · March 24, 2019, 1:59pm

Hi, it´s me again.
I have learned that VLANs are separated on Layer2 but Layer 3 must be restricted by FW-rules. But I do not really understand, why it is not working.
Why is inter-VLAN Traffic not dropped by the “Drop everything” -Rule at the end of the Forward Chain. If someone can have a look at the Rules again in order find the issue.
Not sure why nothing is dropped here…

(Please find interface-lists in my posts before; address-lists e.g Admin-Subnet, VLANFriends, SonosControl, contain Subnets in the format 172.16.x.0/24; AllowWAN contains a lost of all Subnets in the same format)
Thanks a lot,
Christian

anav · March 25, 2019, 12:22am

Try deleting the UPNP rule and see what affect that has??
Other than that, go rule by rule disabling to find out where the issue is.

Spartacus · March 25, 2019, 8:20am

Hi anav,
n idea what is going on here. I´ve disabled nearly all Forward rules but it is still not working as it should.
I have no idea what rule is wrong! And I am wondering why no packages were dropped! This seems to be very stange!

Christian

mkx · March 25, 2019, 8:41am

Post complete configuration (not just firewall) in ASCII … screenshots don’t show everything.