After many hours of reading and trying I still can’t seem to get communications to happen properly between the Microtik RB750Gr3’s vlan trunk and the upstream devices (Adtran switches).
The overall topology is straight-forward; Adtran trunk port <==> MikroTik LAN1 port, MikroTik WAN port <==> Comcast mdm/rtr, MikroTik LAN2-4 are access ports for vlans 1,2,4 respectively. MikroTik router provides Internet, DHCP and DNS services for the 9 vlans along with any inter-vlan routes. Vlan-1 is native default on Adtran for untagged traffic.
Here’s what I’ve done in a nutshell…
kept stock firewall rules, but added an allow (input and forward) for new connections from !WAN;
removed the switch ‘master’ on ether2 and it’s slaves on ether3-5;
defined ether1 as WAN, defined ether2 as LAN1 (a.k.a. ‘trunk’), defined ether3-5 as LAN2-4;
defined 9 vlan interfaces [the ‘trunk’], binding each to LAN1;
defined bridge interfaces for each of the 9 vlans;
defined a bridge-port on each bridge interface for it’s related vlan interface (i.e. each bridge has a vlan interface port);
defined bridge ports for LAN2,3,4 on three of the vlans-bridges (1,2,4) for use as untagged access ports;
assigned address/net to each vlan interface;
defined 9 DHCP servers, one for each vlan, associated with 1 of the 9 DHCP Pools accordingly;
bound the 9 DHCP servers to their respective vlan-bridge interfaces;
And here’s what I find in testing…
PC connected to LAN2-4 gets appropriate DHCP assignments for the designated vlans;
PC can ping each of the L3 vlan addresses;
PC has access to Internet;
PC can NOT ping any of the nodes on any of the 9 vlans;
MikroTik can NOT ping any of the nodes on any of the 9 vlans;
Winbox shows activity on the ethernets, bridges and firewall;
Winbox does enumerate TCP/UDP connections for each of the 9 vlans… list grows over time;
Winbox shows DHCP activity on many of the 9 vlans;
Adtran switch console can NOT ping any of the MikroTik-resident L3 addresses;
Adtran can NOT access Internet;
…and that’s where it sits with the vlan trunk obviously ‘alive’, but no L3 communications between the MikroTik and the Adtran-carried networks.
I’m guess’n I’ve missed something small but critical because while the bridges show activity, there seems to be no cross-communications happening… all help welcomed!
You’ve chosen an interesting time to attempt this because the way it’s done is going to change with the next version of ROS. (v6.41)
At that point, the bridge will be vlan-aware and you’ll be able to do what you want much more easily.
In today’s version, the best way to go about it, given your requirements to have tagged + untagged ports on the same vlan…
Step1: don’t use untagged vlan on the trunk interface at all - i.e. don’t bridge vlan1 because if you connect a trunk interface to another port using a bridge, then the tagged traffic may be forwarded as well (I haven’t experimented with this in a while so things may be different now)
Then step 2 is to make a vlan interface on ether1 for each vlan you wish to support:
names = e1.vlan2 , e1.vlan3 , e1.vlan4 , etc… set the vlan-ids appropriately.
Step 3:
Make bridges for each vlan:
bridge2 , bridge3 , bridge4, etc - but only for the vlans where you wish to pass the traffic to an untagged port on the Mikrotik for that vlan.
For vlans where you don’t want the Mikrotik to forward anything - just participate in the vlans themselves - use the e1.vlanXX interface as the IP interface for that vlan.
For all vlans where you have a bridge, use the bridgeXX interfaces as the IP interfaces.
Step 4:
Then make sure your IP forward filters aren’t causing your inter-vlan communication issues.
The switch-chip in the 750Gr3 doesn’t support VLANs. It let’s you configure them but they won’t work as you’re finding out.
That kind of behavior is solved in 6.41rc+ the device will toggle hardware features on automatically if the device and configuration allow it. It will fail back to software if you require the feature but don’t have hardware that natively supports it with ASICs or other hardware acceleration techniques.
I have been using the new VLAN aware bridges on several 750Gr3 devices since the initial release during a few of the 6.40rc before the got pulled until 6.41rc.
Regardless of version any VLANs on the 750Gr3 will be switched by the CPU. I get roughly 300 mbps through my 750Gr3 consistently with iPerf so it has worked out fine for what I use them for.
I’m not clear on this point… if you’re saying that the ‘access’ port may also link to the other vlans in the trunk, in this case that’d be okay as it is for monitoring only. The issue I’m having is that none of the ‘access’ ports can communicate with the other nodes on their respective vlans - save for the IP assigned to the vlan in the MikroTik. I.E. When connected to LAN3 (ether4), the PC gets an appropriate DHCP suite for that vlan (#2), and can ping the vlan’s local IP addr, but can not communicate with any other node on that subnet via the trunk… and visa versa.
HHhhhmmmmm, I believe this is exactly what was implemented (as described in the OP). The only difference being that we wish to be able to ‘tap’ any of the vlans for monitoring, hence the 9 bridges… with only 3 having ports on the ethernet (LANx) interfaces at any one time. It seems to act like the bridges don’t switch the tagged vlan trunk traffic but somehow manage to reveal TCP/UDP connection status to RouterOS and get DHCP info from the servers on the MikroTik.
Yeah, noticed that… but did see that some things are properly (for this chipset) blocked - such as auto-tagging untagged vlan packets arriving on an ethernet interface.
Okay, so help me understand the packet flow in these hybrid switch/routers… do vlan’d packets (tagged or ‘native’) arriving on the trunk interface automatically land in the switch-cpu? OR, should one also bind the switch-cpu interface to each of the various vlan-bridges as an additional port?
In their basic modes, it’s best (or at least it works for me) to think of the HW switch groups and the software bridges as being simple dumb switches - i.e. they agnostically pass tagged and untagged frames “as-is”
So if you bridge ether1 and ether2 with a soft bridge, for instance, then both will be “trunks” if tagged traffic is present, or just a simple two-port vlan.
The HW switch also adds some vlan tagging/filtering features but honestly, the configuration for that is convoluted to the point that I never bothered to learn it, but instead made other configurations that achieve the same result w/o the headache.
The other atomic component in the mix is the vlan interface. I like to think of them as having a “front” side and a “back” side. The front transmits whatever came in the back, but with the vlan-id added as an 802.1q header. The front side listens for 802.1q-tagged frames that match, it strips the vlan header and transmits the result on the back side. So a single-tagged frame would come out the back as untagged. A double-tagged frame would have the outer tag stripped but the inner tag intact. Etc.
The back is usually connected to the CPU, but this can be redirected to a bridge - by adding the vlan interface to a bridge as a port of that bridge.
The front is connected to whatever interface you build the vlan as a sub-interface on… i.e. if you create vlan interface myVlan with interface=ether2 then the “front” is connected to ether2. This front side may also be a bridge. In this scenario, the vlan interface will send/receive tagged frames on the bridge - i.e. all ports of the bridge will be able to send/receive the tagged frames.
If you make a bridge called vlan21 and add two vlan interfaces to it as ports (connecting the back sides of the two) then the bridge will contain the untagged traffic of vlan21 but it will leave the bridge via the subinterfaces, thus it will be tagged anywhere else.
The HW switch master/slave ports cause them to act as a dumb switch, and the master port is where you build your configurations. It’s pretty much the same as if there were only one physical interface (the master) which was then connected to a simple dumb switch.
Step 1, upgrade to 6.41rc.
Step 2, forget everything you’ve read about words like “switch-chip” and “cpu.”
Step 3, configure your device to use the VLAN aware bridges present in 6.41rc. You can find example configurations in the /interface bridge wiki page.
Longer summary, you’ll use the new /interface bridge vlan table to establish what VLAN-IDs are present on the bridge and which ports those VLANs are tagged or untagged at. This works in conjunction with /interface bridge port and the PVID value (untagged VLAN). Additionally you can set security parameters like allowing only tagged frames or allowing only untagged frames. You’ll need to add the bridge interface as a tagged interface for each VLAN not the PVID (1 by deafult) of the bridge. The layer 3 interfaces where the IP for the VLAN is should have it’s interface set to the bridge.
… assuming OP wants to run beta code on their device. If it’s just a lab, then that’s cool, but I wouldn’t ever want to run RC code on anything production.
Thanks to both responders! … I love having options, esp. if they’re about two ways to skin the cat [where the heck did that phrase come from - nasty visual that]. I will play with both concepts in the prototype setup, but using an RC in production is, IMHO, a bit risky - esp. with some history of ‘features’ gone away for a while Either way, it’s good to learn a bit more about how these devices ‘think’
Okay, time to get lost in the maze of packets and ports.
Ya, I guess I’m more daring than most. I take the approach of run local tests in a lab and if it works then deploy until proven wrong. This of course means I’d need to add a new test to my process. I’d never condone blindly upgrading to the latest image each release. It is important though to not just blindly stick to a bugfix release. Especially when a feature may have improved your life or the customer experience significantly. The conversion of legacy VLAN and switch chip implementations to the new bridges is one I’d be braver than most on (as long as hardware forwarding is implemented in your device if you need it). Unless the conversion scripts become staggeringly good I’m betting we’ll be doing touch-ups on every device after the upgrade. So when deploying new hardware it might make sense to get this work out of the way up front. Also, especially for new users it can be incredibly frustrating to learn either the legacy software bridge approach that you (Zerobyte) and I prefer over the convoluted switch chip implementation of yesteryear. These new folks might as well learn the new VLAN aware bridge out of the gate and grow with it.
That said, I understand everyone won’t have access to platforms or automation capable of performing their own in house release testing and no matter how good that testing is it is a thing that needs constant updating to both stay and become more relevant. The time that can take is often something businesses do not budget for. So, different strokes for different folks. Mine is a test and upgrade often approach vs dealing with feature limitations from extremely old code. With that I probably hit more bugs than the average fella but hey, it keeps me fresh.
I’ve taken the plunge into 6.41rc## and the bridge stuff is certainly cleaner… thanks for the heads-up.
With reference to Cisco or Adtran style vLAN config in the switches, seems like the bridged-vlan’s in MikroTik under this RC are much like the vLAN Trunks in those switches, with the bridge-ports being where one declares an interface port as either ‘access’ (untagged) or ‘trunk’ (tagged)… is that a fair was to view things? IF so… seems like it might now be possible to endow the Winbox/Webview interface with config language that’s more familiar to those who know/use the ‘Cisco way’… just my $.02
On a related note: looking at the example setup in the Wiki manual:interface /bridge, “8.3 VLAN Example #3 (InterVLAN Routing by Bridge)”… our particular setup is very similar with the exception that we do NOT wish to do inter-vLAN routing, just basic NAT out the WAN port. SO, could that be accomplished by simply moving the vLAN interfaces (not the bridge vlans) off of the bridge and directly onto the ethernet port (with the IP addresses still tied to vLAN interfaces - needed for DNS, DHCP and Gateway services to the various vLANs)? We believe our only real need for the bridge is to allow access to designated ‘trunk’ vlans from untagged traffic on specific ports… e.g. ether5 is setup as a bridge port with PVID 4 for untagged traffic… but we’re unclear on that.
Regardless of approach if you don’t want the internal “VLANs” to talk to one another you’ll have to use the firewall. I’d use the bridges and VLANs personally if you place the VLANs right on the Ethernet interfaces it will tag traffic. Additionally, using the hw-offload you’ll gain access to hardware acceleration if it is ever enabled for the models you use.
ZeroByte:
Your description of how to conceptualize the vLAN interface was one of the keys to getting a better understanding of how vLan tags are handled… in the current stable release, and also in the pending RC… where they may fade in significance if I get the way the new vLan-aware bridges function. For anyone else reading this, here’s a way to think about the vLAN [logical] interface…
The two keys things needed to declare a vLAN interface (think: shim or port-filter) are: the physical interface that the vLAN interface is to be connected to (the ‘raw’ or ‘interface’ side of the vLAN interface), and the vLan ID (VID) that the interface will operate on. Any packets arriving at that physical interface with a vLan tag matching the VID, will pass-thru the vLAN interface and exit as an untagged (one layer deep) packet… which will be passed to whatever port the vLAN interface is subsequently attached to (the ‘filtered’ or ‘port’ side of the vLAN interface) – like a bridge port. And, likewise, packets arriving at the ‘port’ side of the vLAN interface will be endowed with a vLan tag for the declared VID and exit the ‘interface’ side of the vLAN interface. A bi-directional un/tagger… any number of which can be built on a given physical interface.
Idlemind:
Having now set these up for an eleven-vLan configuration, it isn’t hard to see how the ‘new order’ vLan-aware bridges will make life, and configuration, way easier with increased flexibility. The RC version was fun to tinker with, but in the end I opted for a stable build in the interim [avoids finger pointing when Murphy visits]. One question did arise; vLan-aware bridges can be assigned to have both “tagged” and “untagged” ports… since I doubt that both forms of a packet exist ‘in’ the bridge, I’m curious to know if the bridges carry the tagged traffic and have automagically instantiated virtual vLAN interfaces attached to the “untagged” ports, or is the situation more the flip of that?
Thanks again to both for your thoughts and assistance.
Not having access to the code I can’t say for sure how MikroTik does it. That said, process wise it’s fairly easy. A packet arrives on a port, the bridge processes it by viewing it’s different tables like /interface bridge port and /interface bridge vlan. It uses those tables to determine first if it should accept the packet. Then it is able to make decisions like, if the packet arrived without a tag, what is the PVID of the bridge port? It then keeps track of the packet as belonging to that VLAN. It is then capable of determining if it needs to send that packet out to another bridge port based on it’s independent mac address table for that VLAN. When that decision is made the port and vlan tables can be referenced again to determine if it needs to apply a tag to the packet.
How this really happens in code may vary widely but there is no reason interfaces have to magically exist or be created outside of the new VLAN aware bridge, internal to it’s operation it knows how to perform those tasks.
Now their are some “dynamic” entries in the different tables and that may get the mind going. In reality those are just easy points of cross reference. These are commonly seen when you apply a PVID to a bridge port. The bridge will create a dynamic entry in the VLAN table. Nothing too crazy it’s just doing it’s thing to keep everything straight.
Either way, it’s good to learn a bit more about how these devices ‘think’ 