VLAN between CRS328 and RB1100AHx4 not working

kobuki · Sat Oct 05, 2019 3:07 pm

I'd like to achieve a simple network (to be later expanded), where the RB1100AHx4 is the main gateway and the CRS328-24P-4S+ the distribution/access switch. For now, I have 2 VLANs, ID 10 and 20. I want to connect the VLANs between the devices and provide DHCP on VLAN 10, while VLAN 20 is an administrative VLAN. I've a created the config (see below), but for the life of me I can't make it work. What am I missing? If possible, I want to use the CRS328 with HW-offloaded VLAN filtering. Please, give some advice. This task should have been straightforward but I'm fighting with it for about a day. As I understand, RB1100AHx4 doesn't support HW offloading of VLANs and neither VLAN tables, even if the chip would be capable of that.
.

CRS328

/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no name=br-trunk protocol-mode=none pvid=20 vlan-filtering=yes
/interface bridge port
add bridge=br-trunk interface=ether1 pvid=10
/interface bridge vlan
add bridge=br-trunk tagged=ether1 untagged=ether9,ether10,ether11,ether12 vlan-ids=10
add bridge=br-trunk tagged=ether1 untagged=ether17,ether18,ether19,ether20 vlan-ids=20

/interface vlan
add interface=br-trunk name=vlan10-lan vlan-id=10
add interface=br-trunk name=vlan20-adm vlan-id=20

/ip address
add address=10.72.17.253/24 interface=vlan10-lan network=10.72.17.0
add address=10.72.18.253/24 interface=vlan20-adm network=10.72.18.0


RB1100AHx4

/interface bridge
add name=br-trunk protocol-mode=none pvid=10 vlan-filtering=yes
/interface bridge port
add bridge=br-trunk interface=ether2 pvid=10
/interface bridge vlan
add bridge=br-trunk tagged=ether2 vlan-ids=10,20

/interface vlan
add interface=br-trunk name=vlan10-lan vlan-id=10
add interface=br-trunk name=vlan20-adm vlan-id=20

/interface ethernet switch port
set 1 default-vlan-id=10 # ether2
/interface ethernet switch vlan
add independent-learning=no ports=ether2,switch1-cpu switch=switch1 vlan-id=10
add independent-learning=no ports=ether2,switch1-cpu switch=switch1 vlan-id=20

/ip address
add address=10.72.17.254/24 interface=vlan10-lan network=10.72.17.0
add address=10.72.18.254/24 interface=vlan20-adm network=10.72.18.0

mkx · Sat Oct 05, 2019 4:16 pm

1st rule: don't use pvid on bridge, rather explicitly configure vlan interface with appropriate vid (as you have it later in the config)
2nd rule: don't ever use pvid on trunk interfaces, run them all tagged (right now you have configuration mismatch... ether1 n CRS and ether2 on RB have pvid=10 set meaning it should be untagged on the wire while bridge vlan says it should be tagged on the wire)
3rd rule: never use bridge vlan-filtering=yes and anything VLAN-related under /interface ethernet switch at the same time, it's mixing apples and bananas ...
4th rule: (for completeness) never use VID=1 for tagged ... many settings have implicit default vid=1 set and when using VLAN ID 1 explicitly this can cause some unexpected behaviour

kobuki · Sat Oct 05, 2019 4:23 pm

1st rule: don't use pvid on bridge, rather explicitly configure vlan interface with appropriate vid (as you have it later in the config)
2nd rule: don't ever use pvid on trunk interfaces, run them all tagged (right now you have configuration mismatch... ether1 n CRS and ether2 on RB have pvid=10 set meaning it should be untagged on the wire while bridge vlan says it should be tagged on the wire)
3rd rule: never use bridge vlan-filtering=yes and anything VLAN-related under /interface ethernet switch at the same time, it's mixing apples and bananas ...
4th rule: (for completeness) never use VID=1 for tagged ... many settings have implicit default vid=1 set and when using VLAN ID 1 explicitly this can cause some unexpected behaviour

Thanks for the advices. I already removed the /ethernet switch part, disregard that please. I never use VLAN ID 1, I tend to use 10, 20, etc. However, I don't know how it's possible to not set the PVID, as it's 1 on default or need to set explicitly. Do you mean I should leave it as 1?

mkx · Sat Oct 05, 2019 4:30 pm

Yup, leave pvid set to 1 or whichever vid you're not going to use.

How things work: if a port has pvid set, it will add VLAN tag to any untagged packets on ingress. And natural configuration would be to have same port set as untagged member of same VLAN ... so that VLAN tags get stripped on egress.
Similar principle holds for bridge interface itself (that's what pvid for bridge is all about).

Another thing: on CRS you'll have to add access ether ports to bridge with appropriate pvid set in bridge port section ... it's not enough to list them as untagged ports in bridge vlan section.

kobuki · Sat Oct 05, 2019 4:53 pm

Yup, leave pvid set to 1 or whichever vid you're not going to use.

How things work: if a port has pvid set, it will add VLAN tag to any untagged packets on ingress. And natural configuration would be to have same port set as untagged member of same VLAN ... so that VLAN tags get stripped on egress.
Similar principle holds for bridge interface itself (that's what pvid for bridge is all about).

Another thing: on CRS you'll have to add access ether ports to bridge with appropriate pvid set in bridge port section ... it's not enough to list them as untagged ports in bridge vlan section.

Thanks again. I'm aware of the basic concepts but with other, simpler managed/web-managed switches this is as simple as 1x1. In ROS there is too much choice and it's not very hard to get lost sometimes. I didn't include my bridge ports section, but here it is. The important ports are set up with the proper PVID.

/interface bridge port
add bridge=br-trunk interface=ether1 pvid=10
add bridge=br-trunk interface=ether9 pvid=10
add bridge=br-trunk interface=ether10 pvid=10
add bridge=br-trunk interface=ether11 pvid=10
add bridge=br-trunk interface=ether12 pvid=10
add bridge=br-trunk interface=ether17 pvid=20
add bridge=br-trunk interface=ether18 pvid=20
add bridge=br-trunk interface=ether19 pvid=20
add bridge=br-trunk interface=ether20 pvid=20

Now it's about time I actually tried fixing the config...

kobuki · Sat Oct 05, 2019 5:14 pm

OK, I changed the bridge/port config per suggestions, but it still doesn't work. By that I mean not a single ping is working between the 2 devices with this setup on neither VLAN. With torch or packet capture it's obvious that the packets are not tagged properly so they don't flow in the right VLAN. I'm a bit stuck again. Current config below.
.

CRS328

/interface bridge
add admin-mac=x auto-mac=no name=br-trunk protocol-mode=none vlan-filtering=yes
/interface bridge port
add bridge=br-trunk interface=ether1
add bridge=br-trunk interface=ether9 pvid=10
add bridge=br-trunk interface=ether10 pvid=10
add bridge=br-trunk interface=ether11 pvid=10
add bridge=br-trunk interface=ether12 pvid=10
add bridge=br-trunk interface=ether17 pvid=20
add bridge=br-trunk interface=ether18 pvid=20
add bridge=br-trunk interface=ether19 pvid=20
add bridge=br-trunk interface=ether20 pvid=20
/interface bridge vlan
add bridge=br-trunk tagged=ether1 untagged=ether9,ether10,ether11,ether12 vlan-ids=10
add bridge=br-trunk tagged=ether1 untagged=ether17,ether18,ether19,ether20 vlan-ids=20

/interface vlan
add interface=br-trunk name=vlan10-lan vlan-id=10
add interface=br-trunk name=vlan20-adm vlan-id=20

/ip address
add address=10.72.17.253/24 interface=vlan10-lan network=10.72.17.0
add address=10.72.18.253/24 interface=vlan20-adm network=10.72.18.0


RB1100AHx4

/interface bridge
add name=br-trunk protocol-mode=none vlan-filtering=yes
/interface bridge port
add bridge=br-trunk interface=ether2
/interface bridge vlan
add bridge=br-trunk tagged=ether2 vlan-ids=10,20

/interface vlan
add interface=br-trunk name=vlan10-lan vlan-id=10
add interface=br-trunk name=vlan20-adm vlan-id=20

/ip address
add address=10.72.17.254/24 interface=vlan10-lan network=10.72.17.0
add address=10.72.18.254/24 interface=vlan20-adm network=10.72.18.0

mkx · Sat Oct 05, 2019 7:41 pm

First of all, try to restart (cold boot by first powering down) both devices to make sure that running config is actually what's configured.

BTW, if you're trying to ping CRS' address, you can't because br-trunk has to be tagged member of itself. Same goes with RB ...

If things still don't behave after that, try to figure out which device is misbehaving. If you configure two computers with static IP addresses (from the same IP subnet) and connect them to ports belonging to same VLAN, can they ping each other? (make sure device firewall allows ICMP echo requests/replies)
When you determine that both CRS and RB individually work correct, it's time to check the trunk connection between the two.

BTW2: device doesn't need L3 config (vlan interface, IP address, ...) if it is not meant for it to interact with that VLAN. Switching traffic of that VLAN can be done without all those settings. In your case CRS probably doesn't need VLAN10 interface nor IP address (you mentioned that VLAN 20 is going to be admin VLAN). Hence br-trunk doesn't have to be tagged member of itself. ether ports still have to be setup properly (pvid for access ports, tagged/untagged members of correct VLANs) for device to do switching.

mkx · Sat Oct 05, 2019 7:51 pm

BTW3: when things start to behave, make sure you configure also VLAN security parameters on /interface bridge port ... in particular frame-types and ingress-filtering properties of individual ports.

kobuki · Sat Oct 05, 2019 8:05 pm

BTW, if you're trying to ping CRS' address, you can't because br-trunk has to be tagged member of itself. Same goes with RB ...

.
Thanks. This seems to have been the key... Weird but logical. I'll do some tests and set this thread solved for others if all is fine. Not many live devices on the access ports yet, they need to be installed but without network it's a bit hard so I had to finish my configs with VLANs first. Now I can ping both sides and the 2 available devices from the router on VLAN 20 with static IPs. There's some uncertainty in ROS as how settings are provisioned - I noticed that for some changes to stick, I had to disable/enable VLAN filtering on the bridge. Though it was on a bit older release (now the devices are on latest LTS).

WRT your BTWs: Yes, I plan to remove the IP from VLAN 10 on the CRS328. It was for testing (I tried a lot of things). I'll set up a VPN on the router to access all the internal LANS for management/setup/etc, if I remove all L3 access from the switch, how would you suggest accessing it for management?

mkx · Sat Oct 05, 2019 9:20 pm

... if I remove all L3 access from the switch, how would you suggest accessing it for management?

Not all of it, you should keep vlan20 and related L3 setup ... And make sure you firewall VLAN20 from the rest of LANs (and WAN) on your main router.

The trouble with (over-configured) L3 devices is that they can become routers between subnets (VLANs) in which they have L3 setup if admin doesn't prevent it (by setting appropriate drop drules) ... so if a device should not route, it shouldn't have more than one L3 interface (most of the time).

kobuki · Sat Oct 05, 2019 9:26 pm

Not all of it, you should keep vlan20 and related L3 setup ... And make sure you firewall VLAN20 from the rest of LANs (and WAN) on your main router.

The trouble with (over-configured) L3 devices is that they can become routers between subnets (VLANs) in which they have L3 setup if admin doesn't prevent it (by setting appropriate drop drules) ... so if a device should not route, it shouldn't have more than one L3 interface (most of the time).

Yeah, I agree. I always set up only what's needed. I intend to do all L3 routing and filter subnet forwarding on the router, and all L2 related traffic management/isolation on the switch.

VLAN between CRS328 and RB1100AHx4 not working [SOLVED]

VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working [SOLVED]

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Re: VLAN between CRS328 and RB1100AHx4 not working

Who is online