Community discussions

MikroTik App
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

VLAN Tagging over SFP

Fri Sep 30, 2022 3:57 am

Trying to get VLAN tags to work over multi-level fiber switches and it's just not working at all.

Edge CCR1072

SFP8 (out) -> CRS326Q-24S+2Q+ (SFP23 in) -> SFP24 (out) -> CRS354-48G-4S+2Q+ (SFP2 in)

For my SFP ports I am using all S+85DLC03D.

DHCP is working fine. I have another 24 port Gigabit mikrotik if I plug it into ETH1 on the CCR1072 and configure tagging the same way, it works. But tagging configured the same on the other two switches running over SFP won't tag.

The machines hooked to the 326 and the 354 get IPs, but from the default DHCP pool, I can get to them, just aren't tagging. Crazy thing is, if I go into hosts on the 326 or 354 both show the macs of the servers connected to each port with the correct tagging!

Hosts of 326 - https://cdn.microtronix-tech.com/imgs/f ... yiZQij.png
Hosts of 354 - https://cdn.microtronix-tech.com/imgs/f ... H0ntwa.png

Here's my setup on the switches:
CRS326Q-24S+2Q+ sfp23 to ccr1072
sfp24 to CRS354-48G-4S+2Q+

https://cdn.microtronix-tech.com/imgs/f ... TsORjI.png
https://cdn.microtronix-tech.com/imgs/f ... H1gKzE.png

Firmware

https://cdn.microtronix-tech.com/imgs/f ... lqxsQ4.png

CRS354-48G-4S+2Q+ port 14 and 16 tagged 200
port 29 tagged 3

https://cdn.microtronix-tech.com/imgs/f ... m4ix74.png
https://cdn.microtronix-tech.com/imgs/f ... 2LDg8y.png
https://cdn.microtronix-tech.com/imgs/f ... oaZ9nI.png
https://cdn.microtronix-tech.com/imgs/f ... dY7lpJ.png

Firmware
https://cdn.microtronix-tech.com/imgs/f ... AHLKSE.png

Like I said, I'm 110% positive VLANS are configured on the CCR1072 correctly, as if I bring a 24 port into ETH1 it works and tags.

Did I configure something wrong in my levels of tagging from switch 1 to switch 2? Or something else entirely? Could it be those SFP plugs don't support tagging??

One thing I have yet to try, I will tomorrow if I don't get any posts is plugging the 354 directly into the CCR1072 on SFP8 and see if it works. If that works, then it's my multi-level tagging and I'm missing something on SFP23 and SFP24.

Thanks for any help you can give! I'm at a loss here.
Last edited by jfreak53 on Fri Sep 30, 2022 2:19 pm, edited 1 time in total.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 883
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN Tagging over SFP

Fri Sep 30, 2022 7:37 am

Can you explain why you are using Force VLAN ID? See this post for the reason I ask.
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP

Fri Sep 30, 2022 2:13 pm

Yes, I don't want the person using that server hooked to that port to tag different traffic to get in our internal network. I want to force them on that lan regardless. If I don't check that they can change their lan.

But I've tried with it off, doesn't work either.
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP

Fri Sep 30, 2022 11:27 pm

Interesting, I plugged the 48 port directly into the SFP on the CCR1072 and the tagging worked! No changes to tagging at all, it just worked plugged in. To me that means that my multi-level tagging is what's causing the issue, so the issue is on port 24 and 23 on the CRS326Q.

Any thoughts? Since the issue is obviously not on the CCR1072 since it works direct. Thanks all!
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 883
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN Tagging over SFP

Sat Oct 01, 2022 6:26 am

Yes, I don't want the person using that server hooked to that port to tag different traffic to get in our internal network. I want to force them on that lan regardless. If I don't check that they can change their lan.

But I've tried with it off, doesn't work either.
In my opinion, you are using the incorrect method to try to enforce the use of a specific vlan from a specific port. You should read the documentation. I doubt that this is the cause of the problem you are reporting, but you really have not provided enough information for a remote diagnosis of the problem you reported.

In addition, we have no idea what your level of experience is with vlans in general or with swos configuration.

Also, in your OP can you explain exactly what you mean by
Trying to get VLAN tags to work over multi-level fiber switches and it's just not working at all.
Specifically, what do you mean by "multi-level" in this context. Also, what exactly do you mean by "not working at all". Because some things are working, e.g. you said dhcp is working, but getting an address from the wrong pool.

In your latest post #4
To me that means that my multi-level tagging is what's causing the issue, so the issue is on port 24 and 23 on the CRS326Q.
What exactly do you mean by multi-level tagging? Are you talking about Q-in-Q?

If you haven't found the SwOS documentation for the CRS-3xx here is where it lives. CRS3xx and CSS326-24G-2S+ series Manual

Pay particular attention when you read this section. VLAN and VLANs
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP

Sat Oct 01, 2022 2:55 pm

By multi level I mean two level of switches, router, switch one for passthrough tagging only, switch 2 for actual tagging ports. Tagging ports works if I plug it directly into router. Doesn't work as a second level after switch 1.
 
JustinM
just joined
Posts: 1
Joined: Sat Oct 01, 2022 9:50 pm

Re: VLAN Tagging over SFP

Sat Oct 01, 2022 9:58 pm

Hi, we have exactly the same issue, connect the switches CRS328-24P-4S+RM over copper ports VLANs are working fine, connect them with S+85DLC03D, untagged traffic works fine but VLAN traffic fails.

Will test some other SFPs to see if it's related to the S+85DLC03D or the CRS328-24P-4S+RM.

Justin
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 883
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN Tagging over SFP

Sun Oct 02, 2022 2:06 am

By multi level I mean two level of switches, router, switch one for passthrough tagging only, switch 2 for actual tagging ports. Tagging ports works if I plug it directly into router. Doesn't work as a second level after switch 1.
Unfortunately, the word "tagging" gets used to mean different things by different people, and what makes it especially confusing in this case is that the two uses are describing the exact opposites. When I use the word tagging, I try to limit that to the operation related to the IEEE 802.1Q tags that get "inserted" into the ethernet frames while they are "on the wire", and that's what I mean by a "tagged" vlan on a port. But even MikroTik uses the term tagging to describe what happens when an untagged ethernet frame (standard ethernet as is used by an out of the box PC) gets received on an access port, and the port vlan id (PVID; what SwOS uses the term "Default VLAN", what Cisco calls the "Native VLAN") is used to specify into which vlan the switch will "classify" the frame as belonging to. The point is I am not sure what you mean by your statement "switch 2 for actual tagging ports". It isn't clear to me if you mean switch 2 has access ports, or instead that switch 2 is applying tags on egress, as those are two very different things.

Now to your problem. Adding a switch in series should work, assuming it is configured correctly and is working hardware, and that the SFP modules are compatible with the switch firmware.

So my suggestion would be to make a backup of your CRS326Q (so you can restore later, when you know what needs to be fixed), then look at the examples in the documentation, and configure two ports as identical trunk ports (or hybrid if one of the vlans on the link is untagged) and be sure the tagged and untagged vlans on those trunk ports agree with the settings on the devices they connect to. And I would recommend using strict mode like they do in the examples.

If you still can't get it to work with SwOS, would you consider using ROS instead of SwOS on the CRS326-24S+2Q+RM? At least that has a text mode config that you can export to upload to the forum.
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP

Sun Oct 02, 2022 3:37 am

If you still can't get it to work with SwOS, would you consider using ROS instead of SwOS on the CRS326-24S+2Q+RM? At least that has a text mode config that you can export to upload to the forum.
That is an awesome idea!! I should never be using the 326 for tagging since it's purpose is bandwidth transport to TOR switches. Even if I do at some point have to tag a port I can use the switch screen of ROS.

This is a great idea! Thanks!
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP

Sun Oct 02, 2022 3:39 am

Any ramifications of using ROS instead of SwOS on this that you can think of? Might it heat up more or choke on bandwidth because of the ROS overhead?
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 883
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN Tagging over SFP

Sun Oct 02, 2022 9:10 am

Any ramifications of using ROS instead of SwOS on this that you can think of? Might it heat up more or choke on bandwidth because of the ROS overhead?
Disclaimer, I have never used any MikroTik switch other than the lowest end RB250 and then later the RB260 aka CSS106-5G-1S, so I am by no means an expert on the use of CRS326 or CRS354 switches.

You will want to use the switch hardware (the 98DX8332 28 port switch ASIC) in the CRS326-24S+2Q+RM. You don't want anything but management running on the QCA9531 CPU. Look at the CRS326-24S+2Q+RM test results The way I read the results, the bottom are the results for software bridging and routing using the QCA951 CPU, the top switching results are using the 98DX8332 switch ASIC.

As long as you have a single bridge and bridge hardware offloading, then my guess is you will get very similar results to what you would get with SwOS, since the heavy lifting is done by the ASIC and not limited by the CPU (on a stick, so also limited by the 1Gbps link between the CPU and the switch ASIC). So I doubt there is much ROS overhead, because the primary thing ROS is doing in this case is just "setting up the switch ASIC, and then letting it do the work.

You probably only want the CRS3xx to have a single vlan interface, and that i just for management. All switching will stay local on the 98DX8332 ASIC. In the "Basic VLAN" example, this is vlan 99.

Relevant documentation:

CRS326-24S+2Q+RM block diagram

RouterOS Bridging and Switching

CRS3xx, CRS5xx, CCR2116, CCR2216 switch chip features

Basic VLAN switching This is a "cookbook recipe" with no explanations. But if you tried something similar, it would proably do what you want. You would need to configure two SPF interfaces as trunks (similar to ether1 in the example config). vlan 99 is the management access to the CRS326-24S+2Q+RM

Bridge VLAN Filtering - this has the explanations.
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP  [SOLVED]

Mon Oct 03, 2022 2:27 am

Out of a whim before I switched to ROS I upgraded from 2.10 to 2.13.....it worked!! No other changes, it just worked.

So now that it's working, I'm trying to figure out without success, and I've googled a few times but not finding what I need.

I need to assign multiple ips to one server, one nic. I've got DHCP server disabled on this VLAN, but when I assign one Mac multiple ips in DHCP leases it tells me the client already exists.

Only way I've been able to do it is in arp assigning multiple ips to a mac, but then I can't ping any ips. Do I then need to static those ips inside the server? How do I prevent a server on the VLAN grabbing an ip it has not been assigned?
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 883
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN Tagging over SFP

Mon Oct 03, 2022 6:26 am

This topic isn't the correct place to ask that question in my opinion. I don't see how it is related to SwOS in any way.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: VLAN Tagging over SFP

Mon Oct 03, 2022 10:12 am

No DHCP server can assign multiple IP addresses to same client (i.e. single client ID, most of times it tightly resembles client's MAC address).

There are two ways to deal with the situation:
  1. configure client device (your server) with multiple MAC addresses and make it run multiple DHCP clients.
    IMO this is mission impossible (I don't believe one can run multiple DHCP clients over single network interface ... and probably it's not possible to assign mutiple MAC addresses as well. There are some was of getting around it, but they are absolutely not trivial.
    And, IMO, servers should not be configured as DHCP clients. It's pretty possible that DHCP server is not available for some reason when server boots and in that case, it'll assume some APIPA address (169.x.y.z) which is not what you'd want to see.
  2. configure client device (your server) statically with multiple IP addresses on same network interface. It's not exactly trivial in some OSes to make routing right if routing depends on local IP address used.

In case it's not obvious: I suggest you to go with way #2.
 
jfreak53
newbie
Topic Author
Posts: 45
Joined: Fri Oct 04, 2019 3:18 am

Re: VLAN Tagging over SFP

Mon Oct 03, 2022 1:40 pm

Thank you MKX. Last one, how do I stop a server from grabbing an ip it's not been assigned in arp?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: VLAN Tagging over SFP

Mon Oct 03, 2022 6:30 pm

You can play with manual entries in ARP table. But I guess that you really should solve the problems in another way, blocking server from using unauthorized IP address(es) by monitoring and blocking ARP traffic is most of time inefficient (it's only too easy to change MAC address of network interface). If you don't trust server's admin, you can monitor server's behaviour and have a word with said admin if (s)he breaks the rules.
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 883
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: VLAN Tagging over SFP

Tue Oct 04, 2022 12:38 am

Last one, how do I stop a server from grabbing an ip it's not been assigned in arp?
Can you explain what that means?

ARP returns the mac address for an IP on the attached LAN. RARP (mostly obsolete) returns ip address from gateway's arp cache. DHCP is the usual thing that "assigns" ip addresses. Summary: ARP, Reverse ARP(RARP), Inverse ARP (InARP), Proxy ARP and Gratuitous ARP

What problem are you trying to solve?

Who is online

Users browsing this forum: No registered users and 12 guests