Community discussions

MikroTik App
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

QoS prioritization only, without shaping?

Wed Jun 12, 2019 6:57 am

How is QoS configured on MikroTik for just packet prioritization and no shaping? i.e. just making sure high priority packets that are received immediately get pushed to the front of the queue and transmitted as soon as possible, retransmissions for those packets take priority etc. Not any form of shaping and slowing other traffic

This is for QoS on wireless links with variable bandwidth, so setting a fixed bandwidth value to carve out a portion of bandwidth for QoS doesn't work
Does MikroTik automatically prioritize packets with DSCP/TOS bits set? Is it done just with the priority setting in queue's? Does the queue type need to be set to anything in particular? etc
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 7:04 am

What wireless links are you using? In most cases, you will need to use a "set priority" mangle rule or bridge filter rule to prioritize the traffic.
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 7:42 am

So for all our routers just add a rule at the top of mangle with passthrough ticked
'set priority'
new priority: from dscp

And that's all thats needed? (Assuming DSCP is already set, otherwise add more mangle rules to set DSCP bits)

No queue's added?
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 8:20 am

So for all our routers just add a rule at the top of mangle with passthrough ticked
'set priority'
new priority: from dscp

And that's all thats needed? (Assuming DSCP is already set, otherwise add more mangle rules to set DSCP bits)

No queue's added?
Yes, *but* whatever you are using for wireless needs to be able to read that. If your wireless is MikroTik, and you are doing "set priority" on the radio itself, it will work as long as you are using NV2 or WMM. If the wireless radio is not MikroTik, it will need to support QoS. Most wireless radios support QoS using CoS (vlan priority) which "set priority" on the router also sets. So if you are using a non-MikroTik radio your router will need to set priority with a mangle rule and tag the packet with a VLAN tag in order for the radio's QoS to kick in and prioritize the packet.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 8:35 am

Also you should use new priority from dscp high 3 bits, not just from dscp. The mapping from-dscp is probably not what you want. DSCP high 3 bits results in a more useful mapping.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 11:01 am

If the wireless radio is not MikroTik, it will need to support QoS. Most wireless radios support QoS using CoS (vlan priority) which "set priority" on the router also sets. So if you are using a non-MikroTik radio your router will need to set priority with a mangle rule and tag the packet with a VLAN tag in order for the radio's QoS to kick in and prioritize the packet.
Well, when the radio is UBNT (quite common as they operate in the same market segment as MikroTik), the whole QoS thing will work automatically, also without VLAN tagging.
It uses the WMM defined queue mapping based on DSCP high 3 bits with 4 queues.

In the MikroTik equipment you need to manually configure that by:

- setting wmm-support=enabled on the wireless interface, likely you also want ampdu-priorities=0,1,2
- adding the mangle rule to set priority based on DSCP high 3 bits
- when it is operating in bridge mode: adding the use-ip-firewall=yes setting to the global bridge configuration

I think MikroTik should enable these by default in their quick set wizard, as it is not widely known and it makes the wifi links perform less than optimal for e.g. VoIP, and makes the competing product look better.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 5:01 pm

Well, when the radio is UBNT (quite common as they operate in the same market segment as MikroTik), the whole QoS thing will work automatically, also without VLAN tagging.
It uses the WMM defined queue mapping based on DSCP high 3 bits with 4 queues.
This is not true for all of their radios. We have mostly AirFiber and they do not read DSCP so we use the VLAN priority. Even their radios that do read DSCP (AirMax, AirMax ac, LTU) are unable to read the DSCP if there are extra L3 headers on the packet besides just the IP header (ex. PPPoE frame, MPLS label). Those same radios will however also read VLAN priority so it remains the most reliable way to do QoS that works across the Ubiquiti line. I do not like the Ubiquiti radios that read DSCP without being able to disable that feature (and have no QoS config options) because in a PtMP setup there is the potential for a single end customer to abuse that by tagging all of their packets as priority to get better rates in a congestion situation. With VLAN priority if the radio is adding the VLAN tag, the customer has no control over the priority set in the tag and therefore cannot use that to prop up their service to the detriment of other customers.
- setting wmm-support=enabled on the wireless interface, likely you also want ampdu-priorities=0,1,2
- adding the mangle rule to set priority based on DSCP high 3 bits
- when it is operating in bridge mode: adding the use-ip-firewall=yes setting to the global bridge configuration
You don't need use-ip-firewall=yes if you set priority using a bridge filter. I prefer working with bridge filters to set priority with MikroTik wireless bridges instead of "use IP firewall" for performance reasons - if I am already using firewall rules for other purposes on the same device, I shouldn't need all bridged wireless traffic to pass through those same rules unnecessarily.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 7:02 pm

With the UBNT devices, do you use VLAN tagging only on ethernet and then strip it in the radio, or extend VLAN all over the WiFi link?
I have not-so-good experience with the latter when it is not in PtP mode. Sometimes it works fine, sometimes it fails in strange ways.

The problem with using "bridge filter" for setting the priority is there is no "set priority from DSCP high 3 bits" option so it requires custom matching and possibly many rules.

Problems with customers abusing DSCP (fortunately I don't have that experience... my customers are not primarily wanting performance) could be overcome with a queue tree with some carefully selected limit-at and max-limit values, so it is not possible to monopolize the link from a single DSCP level. That is also the reason I like queue trees more than priority queuing. I have queue trees in the routers and priority queues in the APs.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 9:30 pm

With the UBNT devices, do you use VLAN tagging only on ethernet and then strip it in the radio, or extend VLAN all over the WiFi link?
I have not-so-good experience with the latter when it is not in PtP mode. Sometimes it works fine, sometimes it fails in strange ways.
We extend the VLAN over the WiFi link - we have hundreds of such setups and haven't had problems.

The problem with using "bridge filter" for setting the priority is there is no "set priority from DSCP high 3 bits" option so it requires custom matching and possibly many rules.

This is again not a problem if things are set up correctly.

Have the router set priority from dscp high 3 bits with a mangle rule and apply the VLAN tag. Then on the radio (that is doing the bridging), have the bridge filter rule set priority to ingress priority - only one rule needed. This will result in the radio prioritization being read from the VLAN priority of the incoming frame, and the VLAN priority was in turn set by the preceding router based on DSCP. This is how we do it and it works great. No need for "Use IP firewall" anywhere, and no need for custom matching and many rules.

Problems with customers abusing DSCP (fortunately I don't have that experience... my customers are not primarily wanting performance) could be overcome with a queue tree with some carefully selected limit-at and max-limit values, so it is not possible to monopolize the link from a single DSCP level. That is also the reason I like queue trees more than priority queuing. I have queue trees in the routers and priority queues in the APs.
This doesn't help when customers abuse with UDP on upload for traffic that has no kind of congestion control mechanism. We would have to use the traffic shaping settings on the Ubiquiti radio which is awkward to manage. Even if you did that, the problem would still be that latency for other customers is higher because the one abusive customer is marking everything with DSCP EF or whatever instead of just actual VoIP traffic.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 10:49 pm

With the UBNT devices, do you use VLAN tagging only on ethernet and then strip it in the radio, or extend VLAN all over the WiFi link?
I have not-so-good experience with the latter when it is not in PtP mode. Sometimes it works fine, sometimes it fails in strange ways.
We extend the VLAN over the WiFi link - we have hundreds of such setups and haven't had problems.
Ok, my experience with UBNT radios that are not in "WDS" mode has been that tagged VLAN traffic over the link does not always work correctly.
As we have a mix of UBNT/MikroTik in het network (both at the AP and client side) we often cannot run in "WDS" mode.
This should be the same difference as between "bridge" and "pseudobridge" mode in MikroTik but still they do not appear to be compatible.
(I think the operation of the non-WDS mode relies on ARP spoofing which does not work for VLAN tagged traffic across the link)
Have the router set priority from dscp high 3 bits with a mangle rule and apply the VLAN tag. Then on the radio (that is doing the bridging), have the bridge filter rule set priority to ingress priority - only one rule needed. This will result in the radio prioritization being read from the VLAN priority of the incoming frame, and the VLAN priority was in turn set by the preceding router based on DSCP. This is how we do it and it works great. No need for "Use IP firewall" anywhere, and no need for custom matching and many rules.
Ok that is a nice setup, I could try that. With MikroTik AP I normally run the user traffic over a VLAN on ethernet, with UBNT AP usually not (I feel less comfortable with separating the management and data on those, fearing to lock myself out at some time).

This doesn't help when customers abuse with UDP on upload for traffic that has no kind of congestion control mechanism. We would have to use the traffic shaping settings on the Ubiquiti radio which is awkward to manage. Even if you did that, the problem would still be that latency for other customers is higher because the one abusive customer is marking everything with DSCP EF or whatever instead of just actual VoIP traffic.
Fortunately until now it has not happened. In our network, there are not really "customers", we are all part of a co-operating club.
When people want to destroy the network for others, we can talk to them and when it does not improve we can always kick them off.
When you serve paying customers of course you can end up in all kinds of discussions, including even that you are not allowed to find out how they are abusing the system because that would be illegal wiretapping on their secret traffic, or not allowed to do any prioritization as that would be violation of net neutrality...
Functioning DSCP has the advantage that users of the network can enjoy voice traffic, normal traffic, and background traffic (e.g. rsync) all working together.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 10:57 pm

Ok, my experience with UBNT radios that are not in "WDS" mode has been that tagged VLAN traffic over the link does not always work correctly.
As we have a mix of UBNT/MikroTik in het network (both at the AP and client side) we often cannot run in "WDS" mode.
This should be the same difference as between "bridge" and "pseudobridge" mode in MikroTik but still they do not appear to be compatible.
(I think the operation of the non-WDS mode relies on ARP spoofing which does not work for VLAN tagged traffic across the link)

FYI have used a MikroTik subscriber unit in station WDS mode to connect to a Ubiquiti AirMax AP before (with AirMax disabled) and it did work. All I had to do was change station mode to station WDS, no other changes needed from the default. However I'm not sure if it works with security enabled.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 11:32 pm

I'm quite (but not completely) sure that there is no way of getting a UBNT client (no WDS) connected to a MikroTik AP (ap bridge) passing VLAN tagged traffic, and no config on the MikroTik AP side to be compatible with UBNT client WDS.
What UBNT calls WDS is different from MikroTik WDS, it is more like MikroTik "bridge" (vs the normal "pseudobridge"). WDS is supposed to be a form of "mesh" but UBNT appears to use it for "transparent".

The other way around, UBNT AP no WDS with MikroTik client connected can often pass VLAN tagged traffic but sometimes it stops and needs to be re-associated to continue.
Of course between 2 UBNT devices in WDS mode, and between 2 MikroTik devices in bridge mode there is no issue.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Wed Jun 12, 2019 11:50 pm

The other way around, UBNT AP no WDS with MikroTik client connected can often pass VLAN tagged traffic but sometimes it stops and needs to be re-associated to continue.
Of course between 2 UBNT devices in WDS mode, and between 2 MikroTik devices in bridge mode there is no issue.
In our case the UBNT APs have WDS enabled and the MikroTik client connected is in "station WDS" mode and it works. I have not tried the other way.
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 3:21 am

Ok so i'm a bit confused as to which method to use here. So lets step it back and i'll give a couple of different scenario's that may need different methods

Background:
We primarily use Cambium radio's but do use some Ubiquiti and a few Mikrotik
I'm going to talk about our backbone infrastructure and ignore customer facing radio's, as all of our customer radio links exceed their provisioned bandwidth so theoretically shouldn't need QoS on those links. All customers connect with PPPoE back to a central PPPoE concentrator. I'm strongly considering changing this design to instead terminate the PPPoE session as close to the customer as possible because it solves some other issues, QoS should be one of them (as I know some radio's won't read QoS tags if there is a PPPoE header)

First Scenario:
So our backbone wireless links are all run as a L2 bridge. MikroTik routers on each side do all the routing
At the moment due to the central PPPoE concentrator. Customer connections all come in on a VLAN and are bridged to a VPLS interface that goes back to the concentrator. So I actually don't know if we can QoS their traffic between these 2 end points because it's all encapsulated in VPLS then in PPPoE, so tags assigned by the customer router are most likely going to be ignored? no way to prioritize their VoIP over data through our backbone right? This is another reason i'd like to move PPPoE termination closer to the customer, cause then their traffic can transit our backbone as normal IP traffic, no need for VPLS or PPPoE running through the backbone, we can mark traffic as close to the customer as possible
So the question is, with our current setup (PPPoE over VPLS) does infrastructure QoS sit in the 'too hard/not viable' basket and so we shouldn't bother with it and instead wait until moving away from PPPoE?

Second Scenario
Lets ignore the customer traffic and focus on 'our' traffic (SNMP, ICMP, OSPF/BGP/MPLS, Keepalives etc between routers and for monitoring)
Currently we don't use a separate VLAN for router-router connections so we can't set VLAN priority. But that doesn't mean we can't change this design if its beneficial
As above we have different equipment connecting these routers together, Cambium PMP/ePMP/PTP series, Ubiquiti Airfiber and some MikroTik 60ghz. And we occasionally change out equipment. So it would be nice if we can have a 1 solution fits all QoS implementation so we don't have to make further changes in each router if we are replacing the radio link
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 4:15 am

Ok so i'm a bit confused as to which method to use here. So lets step it back and i'll give a couple of different scenario's that may need different methods

Background:
We primarily use Cambium radio's but do use some Ubiquiti and a few Mikrotik
I'm going to talk about our backbone infrastructure and ignore customer facing radio's, as all of our customer radio links exceed their provisioned bandwidth so theoretically shouldn't need QoS on those links. All customers connect with PPPoE back to a central PPPoE concentrator. I'm strongly considering changing this design to instead terminate the PPPoE session as close to the customer as possible because it solves some other issues, QoS should be one of them (as I know some radio's won't read QoS tags if there is a PPPoE header)
We have a similar setup to you only we don't have Cambium - we only have Ubiquiti and MikroTik. We have a central PPPoE concentrator as well with MPLS.

All radios will read VLAN priority from a VLAN tag regardless of what other headers are on the packet (ex. MPLS label), so run all traffic tagged across radio links. Have your routers apply the tags and then the radios can read them.
First Scenario:
So our backbone wireless links are all run as a L2 bridge. MikroTik routers on each side do all the routing
At the moment due to the central PPPoE concentrator. Customer connections all come in on a VLAN and are bridged to a VPLS interface that goes back to the concentrator. So I actually don't know if we can QoS their traffic between these 2 end points because it's all encapsulated in VPLS then in PPPoE, so tags assigned by the customer router are most likely going to be ignored? no way to prioritize their VoIP over data through our backbone right? This is another reason i'd like to move PPPoE termination closer to the customer, cause then their traffic can transit our backbone as normal IP traffic, no need for VPLS or PPPoE running through the backbone, we can mark traffic as close to the customer as possible
So the question is, with our current setup (PPPoE over VPLS) does infrastructure QoS sit in the 'too hard/not viable' basket and so we shouldn't bother with it and instead wait until moving away from PPPoE?
It is hard to do QoS for PPPoE traffic certainly. We do QoS for PPPoE sometimes over VPLS but it is an all-or-nothing situation - either we prioritize all of a customer's packets or none of them. You would need to "set priority" on the traffic before it has the MPLS label applied, then the priority would be carried across the network via MPLS experimental bits all the way to the destination. Do you really need to run VoIP over PPPoE though? Why not put the VoIP on a different VLAN? Assuming you have central VoIP servers, you only need a few firewall rules to allow the traffic to the VoIP servers and vice versa. The phones would not need complete access to the Internet, so it shouldn't require too many firewall rules on the side of the other routers. Then you can tag just the VoIP packets quite easily - either they could run over a different VPLS tunnel (and then you could prioritize everything in the VPLS tunnel with EXP bits) or they could be locally routed. We are planning to roll out VoIP for our customers in the next year and are not planning to run the VoIP over the PPPoE tunnel.
Second Scenario
Lets ignore the customer traffic and focus on 'our' traffic (SNMP, ICMP, OSPF/BGP/MPLS, Keepalives etc between routers and for monitoring)
Currently we don't use a separate VLAN for router-router connections so we can't set VLAN priority. But that doesn't mean we can't change this design if its beneficial
As above we have different equipment connecting these routers together, Cambium PMP/ePMP/PTP series, Ubiquiti Airfiber and some MikroTik 60ghz. And we occasionally change out equipment. So it would be nice if we can have a 1 solution fits all QoS implementation so we don't have to make further changes in each router if we are replacing the radio link
Yes - tag all packets from router to router where they go over a radio link. VLAN priority should be your 1 solution fits all QoS solution. For packets that are not MPLS, you can use set priority on the routers when the VLAN tag is applied and all radios can read the VLAN priority. MikroTik will copy MPLS EXP bits to VLAN priority so if you set priority on your MPLS packets before the label is applied, the VLAN priority will be set when that packet has a VLAN tag added. This transfer of VLAN priority is supposed to be fully automatic, but I found that this only works out of the box with CHR (cloud hosted router). With hardware routers I needed extra steps - I had to create extra single-port bridges on any ports that an MPLS packet can arrive or leave on with STP turned off and a bridge filter rule (chain=output action=passthrough ingress-priority=!0) in order for the automatic setting of VLAN priority from MPLS EXP bits to work. I believe it is due to a bug in the MPLS fastpath handler, and there is no config option to turn off MPLS fastpath. Although adding these extra bridges and the bridge filter is annoying, it works around the issue so that QoS works. I think what happens is MPLS fastpath is switched off by adding this config, which allows the automatic copying of MPLS EXP bits to VLAN priority to work as originally intended.

So in other words, if you have a router with three interfaces that you run MPLS LDP on, you would add a VLAN interface to each of those three interfaces (for traffic), then for each VLAN interface, then you would add three new bridges with STP turned off, each with a single port - the VLAN interface. So physical interface x3, vlan x3, bridge x3. The bridge would become your new interface for LDP and OSPF (you will need to reconfigure the LDP and OSPF settings to match). It still won't work due to MPLS fastpath, but now if you add the bridge filter rule (chain=output, action=passthrough, ingress-priority=!0) the MPLS exp bits will get automatically copied to VLAN priority and used by the radios. The VLAN interface is optional if the port does not connect to a radio, but you still need the bridge - in this case, the bridge would connect to the main interface instead of the VLAN interface.
Last edited by mducharme on Mon Jun 17, 2019 5:20 am, edited 1 time in total.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 4:52 am

Here is an example setup from memory:

concentrator <-cable-> P-1 <-radio-> P-2 <-radio-> PE <-cable-> CE (customer router)

In this example VPLS tunnel runs from concentrator to PE router, so concentrator and PE apply MPLS labels.

VPLS tunnel on concentrator would terminate on a bridge (running the PPPoE server). The only port of this bridge would be the VPLS tunnel. On PE router, VPLS tunnel would terminate on bridge with two ports - the VPLS tunnel itself and the port going to the CE router (running PPPoE client).

Concentrator:

We have only one LDP interface - the link to P-1 which is over a cable (no radio) so we don't need to worry about prioritizing over the link to P1. Use "set priority" action on bridge output chain to set priority to 5 - this is the bridge has the VPLS interface as its only port. PPPoE frame will go through the VPLS tunnel which will have MPLS label applied. MPLS experimental bits will be set to the value from "set priority" action. No extra bridges or VLANs needed on this device.

P-1 router:

VLAN not required on link to concentrator. VLAN required on link to P-2 since it goes over a radio - create it. Add two bridges, a bridge-to-concentrator with only one port (the ethernet link going to the concentrator) and a bridge-to-P2 with only one port (the traffic VLAN to P-2). Add bridge filter rule: chain=output action=passthrough ingress-priority=!0. Change LDP and OSPF interfaces to bridge-concentrator and bridge-P2. Result: MPLS experimental bits set by concentrator will be copied to VLAN priority across the link to P-2.

Radio between P-1 and P-2 routers:

If it is a mikrotik radio, you will need to a bridge filter rule: chain=forward ingress-priority=!0 action=set-priority new-priority=from-ingress and change the number of NV2 queues to 8. You will see the counter on this bridge filter rule going up when priority packets (non-zero) arrive. AirFiber needs no changes, it will read VLAN priority by default. Not sure if you have to enable reading VLAN priority on cambium - it might do it out of the box. VLAN priority is also called CoS and 802.1p.

Result: the MPLS exp bits that were originally set by the concentrator and were copied to VLAN priority by P-1 will be read by the radio and used for QoS.

P-2 router:

VLANs needed on link to P-1 router and on link to PE router. Add two bridges, bridge-to-P1 and bridge-to-PE, each of which has one port (the VLAN on the link going to that router). Add bridge filter rule (chain=output action=passthrough ingress-priority=!0). Change LDP and OSPF interfaces to the bridges.

Result: an MPLS packet arriving from either P-1 or PE will have VLAN priority set to equal the MPLS EXP bits

PE router:

There would be a bridge with two ports - one connecting to the VPLS tunnel and one connecting to the client router. Bridge filter rule to set priority to set MPLS experimental bits for traffic from PE to concentrator (chain=forward action=set-priority new-priority=5). No bridge necessary on port facing P-2 because the set priority action was done on this router so the bridge there is not required, although you would have a VLAN interface because traffic coming from P-2 is tagged.
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 5:16 am

The router-router links don't use VLAN's though
They just speak to each other on the ethernet link i.e. ether5 on RouterA connects to PTP670 link connects to ether7 on RouterB
So using the set priority mangle rule wouldn't do anything? Or would it still tag packets with native VLAN id so that priority is applied?
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 5:26 am

The router-router links don't use VLAN's though
They just speak to each other on the ethernet link i.e. ether5 on RouterA connects to PTP670 link connects to ether7 on RouterB
So using the set priority mangle rule wouldn't do anything? Or would it still tag packets with native VLAN id so that priority is applied?
You will need to configure your router-router links to use VLANs by placing VLAN interfaces on the ether5 (RouterA) and ether7 (RouterB) ports. The router needs to apply a VLAN tag in order to have somewhere to store the priority.

The "priority" of the packet is a software "tag" associated with the packet as it transits the router. It is similar to a DSCP tag, except priority is from 0-7, and since it is stored only in the router's memory, it is lost once the packet leaves the router. The "priority" tag has only one purpose: when the packet leaves the router, the router uses the priority value to try to set these four items: MPLS exp bits, VLAN priority, NV2 priority, and WMM priority. All four of those survive transit across the network and can be read by the receiving device. However, those four items can only be set if the packet has a place for that priority when it leaves the router - for MPLS exp bits it would need an MPLS label, for VLAN priority it would need a VLAN tag, for NV2 priority it would have to exit the router via a wireless interface running NV2, and for WMM priority it would have to exit the router via an 802.11 wireless interface with WMM enabled.

In other words, if you run set priority on a normal IP packet with no MPLS label going out an ethernet port with no VLAN tag, set priority will not have any effect. It can't set the NV2 or WMM priority because the packet is going out an ethernet port not a wireless port, it can't set the VLAN priority because there is no tag, and it can't set MPLS EXP bits because there is no MPLS label being applied. So in that case set-priority doesn't really do anything.
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 5:54 am

Ok but I have heard its best practice to use QoS tags at Layer3 as opposed to Layer2 so why not use DSCP tags instead of CoS?

And does a MikroTik router actually do anything with DSCP tagged packets by default or does it need to configured with mangle or queue's to apply prioritization to traffic?
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 6:04 am

Ok but I have heard its best practice to use QoS tags at Layer3 as opposed to Layer2 so why not use DSCP tags instead of CoS?

And does a MikroTik router actually do anything with DSCP tagged packets by default or does it need to configured with mangle or queue's to apply prioritization to traffic?
The router will not do anything with DSCP tagged packets by default.

I recommend using QoS at both layers for different purposes. Layer 3 QoS is used for end-to-end transit, remaining with the packet from point A to point Z. I configure QoS in such a way that the layer 3 QoS value is transferred to layer 2 (VLAN Priority / CoS) and then the layer 2 value is used by the wireless links. When the packet arrives at the next router, the Layer 3 QoS value is again transferred to CoS and read by the next wireless link, and the process repeats until it arrives at the destination.

There are two Layer 3 QoS Options: DSCP (which works for packets only with no MPLS label), and MPLS EXP bits (which works only for packets with an MPLS label). Both of those survive network transit across multiple hops.

Most radios cannot read MPLS EXP bits and may not be able to read DSCP depending on what additional headers the packet has. So the solution to standardize things better is to copy DSCP and MPLS EXP bits at each hop to CoS, and have the radios use only CoS. The advantage of this approach is that you can use both MPLS EXP bits and DSCP on your network, and since you are copying both to CoS, your radios only have to look at CoS to determine how to prioritize the packets
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 6:15 am

Also, AirFiber (except for the AF5XHD) can only read CoS, so you have to copy DSCP to CoS for the AirFiber devices to be able to read it.

CoS is really the most universal priority tag available - basically everything supports it, even if DSCP or MPLS EXP bits are not supported. The only downside of it is that it doesn't survive being routed (since that removes the VLAN tag), which is why setting it based on DSCP or MPLS EXP bits makes sense (since those do survive being routed).
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 9:57 am

Ok i'm liking this QoS implementaiton, its quite simple to implement and elegant. However couple more questions

I've been labbing this just with a couple of routers connected via 100mbit ethernet to first get the queueing correct then i'll test with radio's in between
Baseline was just to have no QoS, run UDP bandwidth test to entirely saturate the link, then run a ping with timeout=100 so it sends 10 pings a second
Baseline shows approx 30% packet loss and latency at 12ms. I've found it doesn't matter AT ALL what I set DSCP or priority to, I get exactly the same result until I go into
Queue->Queue Types and change 'only-hardware-queue' to something other than 'none' as this is the default.
bfifo and sfq work well, the rest all result in packet loss (at default settings anyway)
bfifo = +1-3ms increase (average 1ms) but 0% packet loss
sfq = +1-5ms increase (average 3ms) also 0% packet loss
(note i'm not actually creating a queue, just changing the hardware queue type)

Radio links in between are likely to be slower than the ethernet interface so I don't know if it matters too much, but I know that MikroTik 60ghz for instance can exceed 1gbit. And it is possible for a link to drop to 100mbit (usually rain ingress into cable or a poor termination) which is where the hardware queue's would be useful

First question: Are you changing the hardware queue type on the MikroTik's? What are you using and what settings?
Second question: Are you using a common template for QoS settings and would you care to share it?

The way i'm doing this in the lab setup is as follows...
/ip firewall mangle
add action=jump chain=postrouting jump-target=QoS
add action=change-dscp chain=QoS comment="QoS: 56 - VRRP" new-dscp=56 passthrough=yes protocol=vrrp
add action=change-dscp chain=QoS comment="QoS: 56 - OSPF" new-dscp=56 passthrough=yes protocol=ospf
add action=change-dscp chain=QoS comment="QoS: 56 - BFD" dst-port=3784,3785 new-dscp=56 passthrough=yes protocol=udp
add action=change-dscp chain=QoS comment="QoS: 56 - MPLS LDP" dst-port=646 new-dscp=56 passthrough=yes protocol=udp
add action=change-dscp chain=QoS comment="QoS: 46 - VoIP" dst-address-list=KnownVoIPServers new-dscp=46 passthrough=yes
add action=change-dscp chain=QoS comment="QoS: 43 - BGP" dst-port=179 new-dscp=43 passthrough=yes protocol=tcp
add action=change-dscp chain=QoS comment="QoS: 43 - ICMP" new-dscp=43 passthrough=yes protocol=icmp
add action=set-priority chain=QoS dscp=!0 new-priority=from-dscp-high-3-bits passthrough=yes
add action=return chain=QoS
I intend to use a similar but more complete template with all the important protocols and traffic in there that I can just paste into all our routers. Separate queue with a jump to it (more specific jump if i.e. it's a customer facing router, they'll get a different queue)
As well as a similar one for bridge filter (but with lowered priorities and forced overrides for anything above 45) for the customer traffic
 
pe1chl
Forum Guru
Forum Guru
Posts: 10197
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 11:47 am

When you are using DSCP for priority it often is not required to have mangle rules like that, because most applications already set DSCP.
E.g. a VoIP telephone and a VoIP server will already set the DSCP on their voice data to 46. No need to do that again in a mangle rule, unless something between the device and your router fouls it up. (e.g. some ISP clearing DSCP on all packets)
When this works OK it is better not to touch the DSCP as VoIP devices also have other traffic which may not require that priority, e.g. a download of a new software image. When required, set the DSCP only close to the affected device/service and let it travel through the network unmodified.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Mon Jun 17, 2019 9:30 pm

First question: Are you changing the hardware queue type on the MikroTik's? What are you using and what settings?
Second question: Are you using a common template for QoS settings and would you care to share it?
Answer to First question: No we aren't. One thing you need to realize is that, at least on MikroTik routers, ethernet interfaces do not consider the packet priority at all when deciding what packets to drop - only NV2 and WMM wireless interfaces do this. Changing the hardware queue type does not change this behavior. I generally do not bother doing QoS on two routers cabled together because if it is a cable and it is at 1G and it is maxed out, you can always bring it up to 10gig by moving to an SFP+ port or increase it by combining two links in a LAG pair, and remove the possibility of congestion.

Where we do need to have the MikroTik routers queue things themselves is sometimes we go over a rate-limited link offered by another provider - ex. we may buy 100Mbps or 200Mbps from someone else to connect to one of our towers. In this case we do need QoS and this is accomplished by setting up a queue tree structure on the interface facing the provider with a parent queue and 8 children, one for each priority 0-7. We set up this queue tree on both ends of the link to do QoS over that link. On these devices, after the "set priority" action, we have seven different packet mark rules to mark the packet "scavenger" if priority=1, mark it "background" if priority=2, basically marks for all priorities except priority 0 since no-mark works nicely for that. Doing this results in the router dropping the low priority packets before the high priority packets if the queue tree is maxed out.

Here is an example of the type of queue tree setup that we would have on both sides of a 40M link through another provider:

/queue tree
add max-limit=40M name=To-Tower1 parent=ether3
add name=0_best_effort_tower1 packet-mark=no-mark parent=To-Tower1 priority=6
add limit-at=2M max-limit=2M name=7_monitoring_and_routing_tower1 packet-mark=monitoring-and-routing parent=To-Tower1 priority=1
add limit-at=2M max-limit=2M name=6_mgmt_traffic_tower1 packet-mark=mgmt parent=To-Tower1 priority=2
add name=5_ent_priority_tower1 packet-mark=ent-priority parent=To-Tower1 priority=3
add name=4_ent_tower1 packet-mark=ent parent=To-Tower1 priority=4
add name=3_retail_priority_tower1 packet-mark=retail-priority parent=To-Tower1 priority=5
add name=2_background_tower1 packet-mark=background parent=To-Tower1 priority=7
add name=1_scavenger_tower1 packet-mark=scavenger parent=To-Tower1

Note: the priority values above are the priorities for the queues, not the packets.. the mapping goes like this: packet priority 7 is queue priority 1, packet priority 6 is queue priority 2, packet priority 5 is queue priority 3, packet priority 4 is queue priority 4, packet priority 3 is queue priority 5, packet priority 2 is queue priority 7, packet priority 1 is queue priority 8 (the default), and packet priority 0 is queue priority 6.

The reason for this is that, while there are 8 queue priorities and 8 packet priorities, the scale is different: for queue priorities (allowed values 1-8), 1 is the highest priority and 8 is the lowest priority. In terms of packet priority (VLAN priority, NV2 priority, etc., allowed values 0-7), they are highest to lowest: 7,6,5,4,3,0,2,1 (*not* 7,6,5,4,3,2,1,0)

Then you would apply the packet marks depending on the packet priority (ex. if priority is 1, apply packet mark "scavenger", if priority is 2, apply packet mark "background").

We only do this queue tree setup on links from 3rd party connectivity vendors where they guarantee us a certain bandwidth amount where we are at risk of actually maxing out that amount. It doesn't make sense to set up these queue trees and packet marks if the router is only connected to radio links and perhaps direct cables to other routers, because the radios will automatically do the QoS based on CoS (and the maximum rate can be variable depending on modulation), and the cables to other routers generally won't be maxed out so the queueing becomes an unnecessary complexity with those links.

Answer to Second question: We did have a common template, but some of our routers might be missing a few things so I would have to check to see which is the best before sharing an example. However, our setup looks similar to your lab setup - the biggest difference is that we use an interface list on each router called Trust-DSCP. If a packet is coming from an interface in this interface list, we trust the DSCP value. If it is not, we reset the DSCP to zero (if we want it to be best effort) or change it to what we want (if we don't want it to be best effort). That way the customer cannot set the DSCP. If the packet is coming from another router that has already set the DSCP and arrives via an interface in the Trust-DSCP interface list, we assume the DSCP on that packet is correct and simply set the priority based on that. That way you address pe1chl's concern that you don't really need to keep changing the DSCP over and over again on each router.
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Tue Jun 18, 2019 11:51 am

We only do this queue tree setup on links from 3rd party connectivity vendors where they guarantee us a certain bandwidth amount where we are at risk of actually maxing out that amount. It doesn't make sense to set up these queue trees and packet marks if the router is only connected to radio links and perhaps direct cables to other routers, because the radios will automatically do the QoS based on CoS (and the maximum rate can be variable depending on modulation), and the cables to other routers generally won't be maxed out so the queueing becomes an unnecessary complexity with those links.
This is the primary reason for this post. We want better QoS for backhaul wireless links that we own, and the bandwidth varies it cannot be guaranteed. Real world is not perfect, radio frequencies get crowded, new constructions go up and partially block signal, a bin chicken fly's into the radio and knocks it out of alignment etc
So for this you are just using mangle rules to set L2 CoS/VLAN priority from L3 DSCP values?

Most radio links will be below the throughput of ethernet, but 60ghz and 80ghz can exceed 1gbit. And/or it is possible for a cable issue to drop the connection from 1gbit to 100mbit. Both scenario's result in the ethernet being the bottleneck not the radio. So i'm thinking it may be worthwhile using SFQ on those. Or have you found that it really doesn't matter?
One thing you need to realize is that, at least on MikroTik routers, ethernet interfaces do not consider the packet priority at all when deciding what packets to drop - only NV2 and WMM wireless interfaces do this. Changing the hardware queue type does not change this behavior
Are you sure? Because my tests with bfifo show packet loss on the saturated link drops from 30% to 0%. My thoughts are that if it's not dropping low priority packets (since I don't think bfifo/pfifo does any reordering) then packet loss should remain at 30% if it's sustained saturation, since the bucket would remain full. But this doesn't seem to be the case
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS prioritization only, without shaping?

Tue Jun 18, 2019 4:40 pm

This is the primary reason for this post. We want better QoS for backhaul wireless links that we own, and the bandwidth varies it cannot be guaranteed. Real world is not perfect, radio frequencies get crowded, new constructions go up and partially block signal, a bin chicken fly's into the radio and knocks it out of alignment etc
So for this you are just using mangle rules to set L2 CoS/VLAN priority from L3 DSCP values?
Yes, that is all we are doing. Ubiquiti uses CoS by default, no config changes needed, dropping lower priority packets if necessary to make room for high priority. MikroTik radios need an extra bridge filter rule, chain=forward ingress-priority=!0 action=set-priority new-priority=from-ingress, and then it drops lower priority packets over the radio link too. Cambium probably uses it by default, but you should check to see if there is a setting that needs to be turned on, just in case.
Most radio links will be below the throughput of ethernet, but 60ghz and 80ghz can exceed 1gbit. And/or it is possible for a cable issue to drop the connection from 1gbit to 100mbit. Both scenario's result in the ethernet being the bottleneck not the radio. So i'm thinking it may be worthwhile using SFQ on those. Or have you found that it really doesn't matter?
I think the only way that you could handle a cable issue dropping an ethernet port to 100Mbps is to do the aforementioned queue tree setup (8 child queues, 1 parent) and use a script to automate lowering the parent queue limit to 100M in the event of the ethernet link dropping to 100Mbps.
Are you sure? Because my tests with bfifo show packet loss on the saturated link drops from 30% to 0%. My thoughts are that if it's not dropping low priority packets (since I don't think bfifo/pfifo does any reordering) then packet loss should remain at 30% if it's sustained saturation, since the bucket would remain full. But this doesn't seem to be the case
Yes - I think the problem is you are doing your testing with btest UDP. This does not really flood the link. Use a specialized UDP packet flooder, they are used to simulate DDoS situations. We do our QoS testing in DDoS simulation situations. In these situations the wireless radios will not drop higher priority packets (they will drop lower priority as necessary in congestion situations).
 
millenium7
Long time Member
Long time Member
Topic Author
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: QoS prioritization only, without shaping?

Thu Jul 04, 2019 4:29 am

Have started getting setup for this but its quite a long process adding VLAN tags to all router links - especially when there's a switch like a Netonix in between as their default policy is drop all unknown VLAN's, takes a fair bit more time per link to change

But this is also a good opportunity to redo the IP addressing on all these links. Most links weren't using any management VLAN for the radio's, so radio's and routers were in the same subnet i.e.
RouterA->RadioA->RadioB->Switch->RouterB
All those devices would be in the same subnet, say 10.0.115.0/27
Since customer traffic is on a separate VLAN and bridged into VPLS back to a PPPoE concentrator, security was never a problem as we only need firewall rules there to drop any access to internal IP ranges. And there was no real need to do it any differently
However I want to move PPPoE as close as possible to the customer, hence all routers will need firewall rules to prevent access to any equipment such as the radio's along the path

So by readdressing using /30 addresses (pity MikroTik doesn't work properly with /31) that only the routers reside in, this allows much neater address assignments for easy route summarization. As well as very easy firewall rules, only 2 rules as a minimum to allow access to the /30 link-net addresses for traceroutes to work etc and then another to drop to any other private IP range to keep the network secure
Radio's and switches can stay on untagged/native and keep their existing IP's (might change it later) as QoS to them for management isn't overly important. If we need to login to a highly saturated link we can do it through the neighboring router itself either via SSH or a NAT rule in a worst case scenario, as that traffic will be CS7

Who is online

Users browsing this forum: mada3k, massinia, tangent and 86 guests