Page 1 of 1

Bridge is resetting CoS to 0 (was: Setting CoS from DSCP on PPPoE server)

Posted: Tue Jun 25, 2019 12:28 am
by NathanA
(3 July 2019 update and TL;DR: the PPPoE server wasn't the problem; see my most recent post where I discuss MT bridges secretly resetting CoS on forwarded VLAN frames)

'lo all,

Let's hypothesize that I am running a PPPoE server on a 802.1q (ethertype 0x8100) VLAN. So, IP packets to be forwarded over a logical PPPoE interface come in as raw IP packets on one ethernet interface, and when they egress the router, they leave as VLAN-encapsulated PPPoE (0x8864 inside of 0x8100).

As the router forwards IP packets down a PPPoE tunnel, I want it to look at the DSCP field of the packet it is forwarding, and copy the top 3 bits to the VLAN CoS (802.1p) field. If no PPPoE server is involved, this works fine using IP Mangle rule with "action=set-priority new-priority=from-dscp-high-3-bits". But with a PPPoE server in the mix, it doesn't work. The IP Mangle rule's packet and byte counters DO increase, but I'm guessing that in this arrangement, the VLAN tag is not applied until after postrouting chain (because the PPPoE interface is the "out" interface for the purposes of forwarding, not the VLAN).

I can't simply drop the VLAN in a bridge, run the PPPoE server on the bridge interface, and set "use-ip-firewall-for-pppoe=yes", because as far as I can tell, IP Firewall only gets applied to bridged traffic that is being forwarded (actually bridged from one port to another), not traffic that is originating from the router itself (that is, when the router is performing the PPPoE encapsulation, not merely bridging already-encapsulated frames). I did try it anyway "just in case", but of course it didn't work. I also tried creating a bridge filter rule that "set priority" "from ingress" on the "output" bridge filter chain, but that made no difference.

I *CAN* get this to work by putting a second router PHYSICALLY IN FRONT OF the PPPoE server that terminates the VLAN, bridges that VLAN to another VLAN on a different ethernet port, and has "use-ip-firewall-for-pppoe=yes" set along with the appropriate IP Mangle rule. But I don't want to have to throw a dedicated piece of hardware at the problem *JUST* to do this one thing. I want the PPPoE server *itself* to apply the correct CoS to the VLAN frame.

What am I missing?

Thanks much!

-- Nathan

Re: Setting CoS from DSCP on PPPoE server

Posted: Tue Jun 25, 2019 1:12 am
by mducharme
What am I missing?
I haven't tried this myself, but is your setup a bridge with a VLAN on it, or is it a bridge where there is a VLAN interface as the port of the bridge? There is sometimes a difference in behavior between the two - we do some QoS stuff with bridge filters that works only with the VLAN interface as a bridge port and doesn't work when the VLAN is on the bridge.

Re: Setting CoS from DSCP on PPPoE server

Posted: Tue Jun 25, 2019 4:03 am
by NathanA
I haven't tried this myself, but is your setup a bridge with a VLAN on it, or is it a bridge where there is a VLAN interface as the port of the bridge? There is sometimes a difference in behavior between the two - we do some QoS stuff with bridge filters that works only with the VLAN interface as a bridge port and doesn't work when the VLAN is on the bridge.
`
I tried it both ways. Nada tostada.

-- Nathan

Re: Setting CoS from DSCP on PPPoE server

Posted: Wed Jun 26, 2019 12:29 pm
by NathanA
NEVER MIND.

This actually works exactly the way you would expect it to: create the appropriate mangle rule on the router running the PPPoE server and voila. No need to involve bridges at all.

The reason I was not seeing what I expected to see is because there is apparently a device somewhere in the path between the PPPoE client and the PPPoE server that is resetting VLAN CoS to 0 in either direction. I haven't yet figured out what or why.

But when I decided to mirror a port for packet capturing that was closer in the path to the PPPoE server, I saw that the PPPoE server actually is setting VLAN CoS appropriately. And I then noticed the opposite problem: CoS was now zero for all VLAN frames originating from the PPPoE client, even though I was seeing appropriate CoS values at the location I had previously been conducting my packet captures from.

So RouterOS DSCP-to-CoS copying is working appropriately, even when you throw a PPPoE server in the mix. I'm just an idiot and didn't think to double-check my packet capture results from a different point on the network before crying for help.

MIKROTIK SUPPORT: If the e-mail that I sent you ever surfaces (strangely, I got no automated return e-mail this time with a ticket # as usual), you can safely ignore it... :oops:

...however, if I find that it's another MikroTik on the network that is responsible for stripping the CoS value as VLANs pass through, I'll be sure to hit you up again. :mrgreen:

Thanks,

-- Nathan

Bridge is resetting CoS to 0

Posted: Wed Jul 03, 2019 10:33 am
by NathanA
...however, if I find that it's another MikroTik on the network that is responsible for stripping the CoS value as VLANs pass through, I'll be sure to hit you up again. :mrgreen:
`
So. Turns out the device responsible for stripping the CoS was another MikroTik on the network.

After poking at this other MikroTik for some time, and doing some experimentation in a lab setting, I discovered that if a VLAN (either 1q-type 0x8100 or 1ad-type 0x88a8!) is forwarded between two ports by a bridge, and if bridge Fast Path is not active, then RouterOS will reset CoS to 0 in the VLAN header before forwarding the frame! But if Fast Path (or, by extension, Fast Forward) is active, then the CoS is not touched!

I also figured out that I could preserve the CoS by writing a bridge filter rule of "action=set-priority new-priority=from-ingress" in the forward chain.

As far as I can tell, RouterOS has behaved this way since at least 6.0, and all the way through the most current 6.45.1. However, I cannot find any place where this behavior is officially documented. So, is this a longstanding bug, or is this intended behavior? And if it is intended behavior, what is the reasoning behind this? Wouldn't you expect a bridge, unless otherwise explicitly configured via filter rules or NAT rules, to maintain the integrity of the frames it is forwarding, rather than to secretly change bits of them behind my back without telling me? I mean, now that I've discovered this is happening, how can I trust that MikroTik bridges aren't secretly screwing with other bits as well as they forward frames?

After scouring the wiki, I did eventually run across this page, where the author talks about this and even presents the exact same bridge filter rule I managed to come up with as a workaround, but this wiki page is NOT written by a MikroTik employee. At https://wiki.mikrotik.com/wiki/Vlans_on ... nvironment it says "By default bridges always set the CoS to 0. If we want the CoS to remain through all the network, we should set this rule..."

This seems crazy to me.

-- Nathan

Re: Bridge is resetting CoS to 0 (was: Setting CoS from DSCP on PPPoE server)

Posted: Fri Jul 05, 2019 7:54 am
by scampbell
Hi Nathan,

While counter-intuitive it seems Clearing Priority of a packet as it passes through a bridge has been a "default" in RoS for a long time.

Logically setting priority from ingress would make more sense as a default action to me too......

For any QoS structure we use both ingress priority as well as checking DSCP (that is carried as part of the packet and not stripped) :-)

Re: Bridge is resetting CoS to 0 (was: Setting CoS from DSCP on PPPoE server)

Posted: Fri Jul 05, 2019 11:15 am
by NathanA
Logically setting priority from ingress would make more sense as a default action to me too......
`
I'd go a step farther. "Setting priority from ingress" shouldn't be a default action. How about NOT TOUCHING THE CONTENTS OF THE FRAME AT ALL UNLESS OTHERWISE REQUESTED is the "default action".

That you have to explicitly tell the system "don't #$@% with bits of my frames as they are bridged" strikes me as *completely broken*. And like I said, the fact that this happens at all just raises the question about what OTHER bits of the frame are being screwed with as it passes through a bridge. NOT COOL. How am I supposed to be reassured that "it's okay, only VLAN CoS gets touched when a frame gets bridged"? Since this isn't OFFICIALLY DOCUMENTED anywhere, how can I know for certain that this is the only garbage that's unknowingly happening to bridged traffic? That this is undocumented does NOT give me confidence that similar things are not happening elsewhere without my knowledge.
`
For any QoS structure we use both ingress priority as well as checking DSCP (that is carried as part of the packet and not stripped) :-)
`
That's great as long as your IP packet is not itself further encapsulated (by a PPPoE frame, by a VLAN header, by MPLS, etc.).

As you can see from the first post in this thread, we are setting VLAN CoS on a VLAN-tagged PPPoE frame *from the DSCP value*. We are doing this precisely because we know that there is equipment in the path between the PPPoE server and the PPPoE client that cannot peer into PPPoE frames.

Another thing I didn't mention is that after this VLAN-tagged PPPoE frame leaves the PPPoE NAS, it gets shoved into a VPLS tunnel. And the thing that kills me is that there is a CLEAR inconsistency between how ROS treats priority when it comes to MPLS EXP and how it treats priority when it comes to VLAN CoS. The official documentation states that "When RouterOS switches MPLS packet, 'ingress priority' is automatically copied to 'priority', this way regular MPLS switching communicates priority info over whole label switched path" (https://wiki.mikrotik.com/wiki/Manual:M ... _behaviour). The difference here -- that with CoS you have to explicitly copy it over, but with MPLS you don't -- seems completely arbitrary.

Re: Bridge is resetting CoS to 0 (was: Setting CoS from DSCP on PPPoE server)

Posted: Sun Jul 07, 2019 9:20 pm
by mducharme
Another thing I didn't mention is that after this VLAN-tagged PPPoE frame leaves the PPPoE NAS, it gets shoved into a VPLS tunnel. And the thing that kills me is that there is a CLEAR inconsistency between how ROS treats priority when it comes to MPLS EXP and how it treats priority when it comes to VLAN CoS. The official documentation states that "When RouterOS switches MPLS packet, 'ingress priority' is automatically copied to 'priority', this way regular MPLS switching communicates priority info over whole label switched path" (https://wiki.mikrotik.com/wiki/Manual:M ... _behaviour). The difference here -- that with CoS you have to explicitly copy it over, but with MPLS you don't -- seems completely arbitrary.
Hi,

The reason for the difference in behavior is because you can use IP firewall mangle for CoS with IP packets but not for MPLS. Suppose you have a router with two routed (not bridged) interfaces, each of which has a VLAN. If an IP packet arrives on one interface with CoS, you can use an IP firewall mangle rule to copy ingress-priority to priority so that when the packet leaves on the other interface the tag has the CoS set to the same thing. It is not possible to use IP firewall mangle with MPLS frames because they bypass the firewall. It is therefore impossible to have a rule to set priority to ingress priority for MPLS switched packets because there is no MPLS firewall to mangle. Because it is impossible for a user to create this rule, they have the router do this by default.

Theoretically with MPLS you don't have to copy it over but there is a problem that we found where this behavior does not work as intended. When an MPLS labelled packet arrives at a router, and the packet leaves on a VLAN tagged interface the router is supposed to set the CoS automatically based on the MPLS experimental bits. The problem is that this only actually works as intended on CHR - on any hardware router it doesn't work. The only workaround we found is to create a single port bridge attached to the port that the MPLS packet arrives at the router on, and then add a bridge filter rule - any bridge filter rule works, even a passthrough rule. After doing those two things, the router properly copies MPLS EXP bits to CoS.

I suspect the problem is due to MPLS fastpath handler, and there is no possibility to manually disable MPLS fastpath. When it is active it seems to bypass reading EXP bits to ingress priority, and so ingress priority remains at zero, resulting in VLAN priority of 0 when the packet leaves the router. I believe the existence of the bridge plus the bridge filter rule on the ingress interface causes MPLS fastpath to be switched off, allowing for proper reading of MPLS EXP bits into ingress priority.

The problem with this workaround for us is that we are then forced to create a large number of bridges - if we have four MPLS interfaces on a router, we need four bridges, one for each, just to fix this behavior. Our network is fully routed, none of these routers would be bridging traffic, but now we need many one-port bridges.

Re: Bridge is resetting CoS to 0 (was: Setting CoS from DSCP on PPPoE server)

Posted: Tue Sep 03, 2019 11:17 am
by CoMMyz
The solution is to just switch the packets using the switch chip instead of the bridge.
Anything that passes through a bridge in a mikrotik that is CPU bound will remove the CoS priority !