VLAN Setup using hardware features - RB2011UiAS and Cisco SG 200-08

[anteater]

The concept is simple, I’d like to create a VLAN for trusted devices, and a second one for untrusted devices that still need internet access at home. Things such as the tv, washing machine and guest wireless. I have the RB2011 in the basement where most of the trusted devices also are, a WiFi hotspot and a Cisco SG 200-08 managed switch in the media room. Since I wanted to keep devices in the media room from “seeing” each other I figure the solution is therefore a VLAN. So I set about configuring the RB2011 with access ports, a Bridge for each VLAN (so that the two internal switches to the RB2011 can be handled together) and unique DHCP server instances for each VLAN (to make it readily apparent which VLAN the device is on). I also wanted to have the hardware handle the VLANs because I wanted to avoid clogging up the RB2011 CPU.

I found this page, but it’s not really applicable because some of the setting locations change between the RB2011 and CRS series:
http://wiki.mikrotik.com/wiki/Manual:CRS_examples

I found this page:
http://wiki.mikrotik.com/wiki/Manual:Switch_Chip_Features

and then of course there are a number of examples using a bridge to handle the VLAN but then that presumably forces traffic on the CPU which isn’t desired.

---

All seems to work as expected with the VLANs and access ports on the RB2011, however under no conditions am I able to have the trunk to the Cisco switch work. It defaults to a management VLAN of 1, and I believe it was mentioned that “untagged” is equivalent to “vlan 1” and it also may neglect to add a tag. I tried using a windows pc that has vlan capability on its NIC and all works as expected, no VLAN gets it’s IP from the default dhcp server, 1 from the management, 2 from the trusted, 3 from the untrusted; and that computer seems to work fine; for that reason I believe that everything is in order, regardless of what comes into the RB2011, it should have a home right? Anyway as soon as I enable what I think is an access port on the Cisco, and add the tagged vlan2 to the trunk port, it stops being able to get an IP address and its clients aren’t getting through. I’m running the latest Cisco firmware, and RouterOS 6.37.1

So the question is, this seems straightforward enough, I want a trunk port configured between the mikrotik and the cisco, but I cannot get it to work! Access ports on the RB2011 seem to work fine so I believe the issue is in the VLAN settings between the two ends on the trunk port, and since the computer seems to work as expected connected where the switch is supposed to be I believe the issue is probably the Cisco switch, both are supposed to support 802.1Q.

Attached is a diagram of the physical network, what I think I have configured inside the RB2011, and screenshots of the settings available in the Cisco switch, which dies after three steps after factory reset (change password, create VLAN id, add VLAN to trunk port and access port) and what I believe the internal processing is doing within the RB2011 along with my confusion indicated regarding the Mikrotik configuration.

This is going on an entire week, I’m out of ideas to troubleshoot further! I’ll try and pull out the relevant details of the Mikrotik config in a later post. There appears to be a limit of 3 attachments, and port 1 on both devices is the trunk.


Home Network Diagram Nov 23.pdf (504 KB)

[anteater]

Here is the remaining relevant parts within Winbox, I was twiddling with vlan1, since with it removed I could still get the cisco to obtain an IP address, but it was not passing VLAN2 from the client regardless. The master for port 2-5 is port1, and for 7-9 is port 6. Port 10 is the WAN port and there is no SFP module in use.

[huntah]

I think “native” VLAN in mikrotik is VLAN0. .
For trunk port to Cisco you should have setting “secure” and “add if missing” (ie Tagged interface) on Mikrotik interface (Cisco is connected to ether1 I presume) and default VLAN ID 0 (for native VLAN).
Also re-enabe VLAN 0 in Switch - VLAN menu.

[anteater]

Thank you huntah, however many evenings later I still appear to be plagued with PEBKAC or some sort of bug. It’s been very frustrating that this seems to be like a house of cards, where it appears to be working then changing one thing tumbles all the way back to nothing with no idea why.

I’d like to scale this problem down to access ports only, forget the trunk, forget the Cisco switch. Also, since I’ve been fighting this for weeks in frustration I purchased an RB3011 so there was no question about interoperability, some of the port numbers are now changing since the desired trunk port will be on port 10, and the WAN on port 1. The problem with the current configuration I have is that untagged (VLAN0) and VLAN3 appear to work correctly. However VLAN2 is an abomination, with intermittent communication to the router (not even talking about the WAN yet) even though it is configured identically. I’ve tried rebooting everything to no avail, which is why this takes so long to finally give up and concede the problem is above me. To recover enough from whatever is wrong I simply switched port 2 back to the default VLAN0 instead of VLAN2 to put it back into the untagged VLAN and things are back up for my client connected there. I notice that the MAC address of the bridges is the same, which perhaps packets are going into the wrong bridge, or also a possibility is the correct tag is not getting added to packets entering the switch from the CPU. The bridges and vlans on interfaces still seems like voodoo in this scope. Is the vlan tagging occurring symmetrically?.. I have no idea how to tell until it works and thus far it’s been inexplicable with what appears to be cache somewhere making for it impossible to correlate a change with an outcome. I was even as far as getting the correct IP on a client on the remote managed switch but internet access was non-functional (remember my statement about the house of cards and how I’m no longer even attempting the trunk!).

I’ll attach the non functional configuration, which clearly after weeks of spinning my wheels getting nowhere I’m not aware of something fundamental. I believe the problem is within either the /Switch configuration but why, VLAN3 works? or in the bridges (packet getting routed out the wrong vlan?). The fact I get the correct IP I assume confirms the /switch menu and dhcp server and bridge all sort of work???

If I don’t have it by now I’m quite certain I need a working example to start from.

RB3011, RouterOS 6.37.2:
broken vlan.rsc (38.1 KB)

[anteater]

and here are more images of the current state

[anteater]

and here’s the last of the settings

[huntah]

I see you havent assigned Admin MAC to bridge.
Please do so and try again. I had bunch of problems without setting Admin MAC on bridges.
Of course set each bridge to unique MAC.

Try it and post results..
Also if it is not working try firstly without bridges, just interfaces (I know there are two switch groups but for test just to see if it is a Bridge problem or switch problem).

[anteater]

Thanks again huntah, this is progress!

-Manually assigning an admin MAC address resolved the glitchy connection to the router issue. I used the factory MAC for the first few ports but changed the first octet to “02” for each of the three bridges, if that makes any difference to routing behavior I thought I’d mention it
-assigning an address to the eth6vlan2 (or eth6vlan3) interface and putting the dhcp server on it seems to work as expected with snappy internet access

-I then left the vlan3 stuff alone since I need that port to work in order to call it a night (works, don’t touch!), and switched to the bridge for vlan2. I’m a little stumped what’s wrong but admittedly haven’t fought much with it tonight. an uneducated guess is that masquerade isn’t working or I’m inadvertently sending tagged traffic out the wan port and the ISP is unhappy with that. I am behind a second firewall/NAT until I get things running so what is truly bizarre is:
-ping 8.8.8.8 from a computer on the vlan2 bridge works
-web browsing seems to be very slow but it does eventually display text only sometimes
-access to the primary firewall protecting my “testbed” works flawlessly, at 172.16.1.254, same with pinging it relentlessly

If I can get to the other private subnet of the primary firewall somehow I crossed the 192.168.2.x->172.16.1.x NAT translation, but then I’m stuck with no fully working access beyond (and 192.168.1.x works fine). The untagged bridge (vlan0) still has internet access that works. I currently am using the “switch2” ports only (eth6-10), leaving “switch1” with untagged and no vlan settings other than defaults.

I checked the VLAN2 computer for weird arp and route entries and looked fine. nslookup for google also seems to work and I see the activity in the Mikrotik’s DNS cache. I found the Bridge->Settings->Use IP Firewall and turned that on already, so the existing single masquerade rule I would think would look after then NAT translation for both raw interfaces and the bridge. Interestingly, tracert does appear to make the journey to 8.8.8.8 even though the browser doesn’t work. I tried “fallback” and “always strip” on the WAN port thinking that should probably strip any vlan tags off since disabled seems to turn off all vlan processing and I didn’t want the headache of vlans on switch 1 yet (no change).

Since I made a Mikrotik support request and linked to this post I figured I’d provide update in case they get to it that I’m no longer stuck the same place.

Not currently stumped anymore but still not fully working… updates to come if I really do get stuck or get it to work (I plan to now investigate outgoing packets for VLAN tags and decomposing traffic contents for NAT translation). That’ll only prove what’s wrong, not how to fix it at which point I’ll censor my MAC addresses again and post what I’ve got.

[huntah]

Glad to help.

I will point out some more things I spotted. Firstly switch1 and switch2 should have secure (not fallback) VLAN Mode set
Here are some mistakes that I see comparing diagram and settings from pictures:

1. Ether1 (wan .. i guess it should be unttaged native vlan) set VLAN mode to disabled (not fallback or secure) and VLAN header “Leave as is” and Defaut VLAN unspecified (no number).
2. port10 (ether10) on your diagram should be trunk so set mode to secure and VLAN Header “Add if Missing” you can than set PVID (Default VLAN) as you wish
3. Ether9 should be Access VLAN3 > so set it secure, always strip and Default VLAN ID 3
4. Also your masquerade rule is quite long (I did not test it what it does). I would try that:

/ip firewall nat add chain=srcnat action=masquerade out-interface=ether1

This should solve your strange behavior on opening pages slowly and unpredictably..

Also If you have bridges then you use DHCP Server and IP addresses on bridges (dont forget admin-mac). If you test it without bridges put addresses and DHCP directly on interfaces and dont forget to remove interfaces from Bridge port!

I think this is all .. Correct this and it should route as planned on diagram..

[anteater]

Thanks as always huntah, sadly I found that following your last set of suggestions left the vlan2’s internet access (or was it vlan3, I don’t remember which I was testing at the time) glitchy as it had always been. The complicated masquerade was actually not part of my active config, or part of the problem since I had wiped the unit over the weekend again (leftover from a complicated firewall).

By fluke I tore out the vlan “interfaces” from the ether2 and ether6, and put the raw ether2 and ether6 into the bridge. It seems to work in preliminary testing (trunk included) so before starting to gather everything for a writeup for the next person (or at the very least my own future reference!) a discussion what happened is in order.

-the glitchy access to the router on the config in accordance with my last internal diagram was the MAC address not being manually assigned
-it doesn’t seem like the bridge of vlan pseudo interfaces works 100% of the time, its glitchy but to some extent partially works or is sluggish. I thought bridging vlan interfaces would unwrap the VLAN from the packet and provide that untagged packet to the bridge (and going the other way it would know which vlan to tag it to because it’s only got vlanX to exit the bridge on) but that doesn’t 100% work.

Where I’m at now by fluke and with no idea why it works, (most of my issue I believe was caused by the bridge):
-put full eth2 and eth6 in a bridge
-put vlan interfaces on that bridge
-put dhcp servers and addresses on each of the bridge’s vlan interfaces
-and then because one of my test devices was a simplistic IoT device I needed to enable UPnP since I think it was getting lost even though it had an IP and gateway by DHCP

So:
-a vlan interface on a physical interface and then put the dhcp and IP address on that vlan interface works fine
-put physical interfaces in a bridge, then put a vlan interface on the bridge then put the dhcp and ip address on that vlan interface works
-a vlan interface on a physical interface and then bridge those vlanX interfaces (because there’s two internal switches) and then put the dhcp and ip address on that bridge doesn’t work, it’s flakey and slow despite certain services working (dns, ping, tracert)

If this isn’t an epiphany moment and by fluke is appearing to work despite being wrong, let me know since the time I’ve spent on this has been excessive; I plan to compose material aimed to prevent it for the next person.