Community discussions

MikroTik App
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 12:46 pm

Greetings, fellow community members!

We are glad to announce the beginning of a new project - Quality of Service Hardware Offloading (QoS-HW), introduced in RouterOS v7.10. The goal of the project is to perform QoS packet marking (VLAN PCP, IP DSCP, and in the future - MPLS EXP), traffic shaping, congestion avoidance/resolution, lossless forwarding, etc. - on the hardware level, which, in turn, means near-to-wire-speed performance.

Documentation: https://help.mikrotik.com/docs/pages/vi ... =189497483

The target devices are those based on Marvell Prestera® DX switch chips: MikroTik CRS3xx, CRS5xx series, CCR2116, and CCR2216. In other words, the devices that support L3HW will eventually support QoS-HW.

We are currently at the beginning of the project: RouterOS v7.10 can perform QoS packet marking on the hardware level. Other QoS features are still in development. The documentation will be available shortly. I'll put the link here once ready.

Your feedback is more than welcome! Please share your vision of QoS enforcement in RouterOS, use-cases, and setups. While the project is in the Beta phase, it is very flexible to adjust to community demand. Also, our QA engineers would like to perform QoS testing based on real setups rather than artificial test cases.

This topic will be updated with recent info regarding QoS HW.
 
User avatar
loloski
Member Candidate
Member Candidate
Posts: 277
Joined: Mon Mar 15, 2021 9:10 pm

Re: QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 1:18 pm

while we are at it could you please confirm if 802.1ad tag stacking offloading in the switch chip is in the pipeline?, great news by the way
 
ib254254
just joined
Posts: 23
Joined: Sat Nov 21, 2020 10:15 pm

Re: QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 2:43 pm

thanks you
Last edited by ib254254 on Wed May 10, 2023 10:27 pm, edited 1 time in total.
 
ib254254
just joined
Posts: 23
Joined: Sat Nov 21, 2020 10:15 pm

Re: QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 2:49 pm

thanks you
Last edited by ib254254 on Wed May 10, 2023 10:27 pm, edited 1 time in total.
 
User avatar
slackR
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Sat May 23, 2009 1:46 pm
Location: Buffalo, New York, USA

Re: QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 3:48 pm

Very nice! The Marvell 90DX chips have QoS feature that will make a great feature for ROS. Is there a page in documentation for the setup? I didn't see one.

Being able to classify traffic and leverage the switch chip queues would be beneficial. Prioritizing voice/video traffic is a good use case. I noticed the CRS309 has 8 hardware TX queues (tx-queue0-packet) but I didn't see how to classify traffic to use each queue.
 /interface/ethernet/switch/port> print stats        
                 name:  sfp-sfpplus1  sfp-sfpplus2 sfp-sfpplus3 sfp-sfpplus4 sfp-sfpplus5 sfp-sfpplus6 sfp-sfpplus7  sfp-sfpplus8  ether1 switch1-cpu
       driver-rx-byte:             0   452 976 691            0            0            0            0            0 1 086 224 642 130 375
     driver-rx-packet:             0       773 965            0            0            0            0            0       945 811     689
       driver-tx-byte:             0 1 188 556 474            0            0            0            0            0   453 023 932  39 675
     driver-tx-packet:             0     1 055 283            0            0            0            0            0       675 260     214
             rx-bytes:             0   527 365 820            0            0            0            0            0 1 369 898 664 137 181           0
         rx-too-short:             0             0            0            0            0            0            0             0       0           0
          rx-too-long:             0             0            0            0            0            0            0             0       0           0
           rx-unicast:             0       912 412            0            0            0            0            0     1 185 318       7           0
         rx-broadcast:             0         1 120            0            0            0            0            0         1 218     209           0
             rx-pause:             0             0            0            0            0            0            0             0       0           0
         rx-multicast:             0         7 748            0            0            0            0            0         1 908     518           0
      rx-error-events:             0             0            0            0            0            0            0             0       0           0
         rx-fcs-error:             0             0            0            0            0            0            0             0       0           0
          rx-fragment:             0             0            0            0            0            0            0             0       0           0
          rx-overflow:             0             0            0            0            0            0            0             0       0           0
            rx-jabber:             0             0            0            0            0            0            0             0       0           0
             tx-bytes:             0 1 473 609 464            0            0            0            0            0   523 059 415  40 531           0
           tx-unicast:             0     1 295 077            0            0            0            0            0       818 688      12           0
         tx-broadcast:             0           115            0            0            0            0            0             3      99           0
             tx-pause:             0             0            0            0            0            0            0             0       0           0
         tx-multicast:             0         1 967            0            0            0            0            0             2     103           0
          tx-underrun:             0             0            0            0            0            0            0             0       0           0
         tx-collision:             0             0            0            0            0            0            0             0       0           0
    tx-late-collision:             0             0            0            0            0            0            0             0       0           0
              tx-drop:             0             0            0            0            0            0            0             0       0           0
             tx-rx-64:             0        39 293            0            0            0            0            0       101 104      63           0
         tx-rx-65-127:             0       714 147            0            0            0            0            0       432 308     101           0
        tx-rx-128-255:             0        71 389            0            0            0            0            0       152 504     647           0
        tx-rx-256-511:             0        29 741            0            0            0            0            0        23 809     137           0
       tx-rx-512-1023:             0        26 816            0            0            0            0            0        19 827       0           0
       tx-rx-1024-max:             0     1 337 053            0            0            0            0            0     1 277 585       0           0
     tx-queue0-packet:             0     1 297 159            0            0            0            0            0       807 319     214           0
     tx-queue1-packet:             0             0            0            0            0            0            0        11 374       0           0
     tx-queue2-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue3-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue4-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue5-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue6-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue7-packet:             0             0            0            0            0            0            0             0       0           0

 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 11:11 pm

I noticed the CRS309 has 8 hardware TX queues (tx-queue0-packet) but I didn't see how to classify traffic to use each queue.
i was tracking that for a while, now we know this is real

keep in mind
The current implementation is for QoS Phase 1 - QoS Marking (introduced in RouterOS v7.10).
so maybe we will have to wait some versions for that queues to be available
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 11:13 pm

Greetings, fellow community members!

We are glad to announce the beginning of a new project - Quality of Service Hardware Offloading (QoS-HW), introduced in RouterOS v7.10. The goal of the project is to perform QoS packet marking (VLAN PCP, IP DSCP, and in the future - MPLS EXP), traffic shaping, congestion avoidance/resolution, lossless forwarding, etc. - on the hardware level, which, in turn, means near-to-wire-speed performance.

Documentation: https://help.mikrotik.com/docs/pages/vi ... =189497483

The target devices are those based on Marvell Prestera® DX switch chips: MikroTik CRS3xx, CRS5xx series, CCR2116, and CCR2216. In other words, the devices that support L3HW will eventually support QoS-HW.

We are currently at the beginning of the project: RouterOS v7.10 can perform QoS packet marking on the hardware level. Other QoS features are still in development. The documentation will be available shortly. I'll put the link here once ready.

Your feedback is more than welcome! Please share your vision of QoS enforcement in RouterOS, use-cases, and setups. While the project is in the Beta phase, it is very flexible to adjust to community demand. Also, our QA engineers would like to perform QoS testing based on real setups rather than artificial test cases.

This topic will be updated with recent info regarding QoS HW.

QoS Hardware Offloading (QoS-HW) will work in concur with L3 Hw Offload ?? or only in L2 setups?
 
User avatar
slackR
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Sat May 23, 2009 1:46 pm
Location: Buffalo, New York, USA

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 7:42 am

From what I understand QoS-HW will work with both L2 and L3.
/interface ethernet switch qos map
add name=classify

/interface ethernet switch qos profile
add dscp=8 name=scavanger
add name=best-effort priority=1
add name=class1 priority=1
add name=class2 priority=2
add name=class3 priority=3
add name=class4 priority=4
add name=class5 priority=5
add name=class6 priority=6
add name=class7 priority=7

/interface ethernet switch qos map vlan
add qos-map=classify qos-profile=scavanger
add priority=1 qos-map=classify qos-profile=class1
add priority=2 qos-map=classify qos-profile=class2
add priority=3 qos-map=classify qos-profile=class3
add priority=4 qos-map=classify qos-profile=class4
add priority=5 qos-map=classify qos-profile=class5
add priority=6 qos-map=classify qos-profile=class6
add priority=7 qos-map=classify qos-profile=class7

/interface ethernet switch qos map ip
add dscp=0 qos-map=classify qos-profile=best-effort
add dscp=2 qos-map=classify qos-profile=class1
add dscp=4 qos-map=classify qos-profile=class1
add dscp=6 qos-map=classify qos-profile=class1
add dscp=8 qos-map=classify qos-profile=scavanger
add dscp=10 qos-map=classify qos-profile=scavanger
add dscp=12 qos-map=classify qos-profile=class1
add dscp=14 qos-map=classify qos-profile=class1
add dscp=16 qos-map=classify qos-profile=class1
add dscp=18 qos-map=classify qos-profile=class2
add dscp=20 qos-map=classify qos-profile=class2
add dscp=22 qos-map=classify qos-profile=class2
add dscp=24 qos-map=classify qos-profile=class2
add dscp=26 qos-map=classify qos-profile=class3
add dscp=28 qos-map=classify qos-profile=class3
add dscp=30 qos-map=classify qos-profile=class3
add dscp=32 qos-map=classify qos-profile=class3
add dscp=34 qos-map=classify qos-profile=class3
add dscp=36 qos-map=classify qos-profile=class4
add dscp=38 qos-map=classify qos-profile=class4
add dscp=40 qos-map=classify qos-profile=class4
add dscp=42 qos-map=classify qos-profile=class4
add dscp=44 qos-map=classify qos-profile=class4
add dscp=46 qos-map=classify qos-profile=class5
add dscp=48 qos-map=classify qos-profile=class5
add dscp=50 qos-map=classify qos-profile=class5
add dscp=52 qos-map=classify qos-profile=class5
add dscp=54 qos-map=classify qos-profile=class5
add dscp=56 qos-map=classify qos-profile=class6
add dscp=58 qos-map=classify qos-profile=class7
add dscp=60 qos-map=classify qos-profile=class7
add dscp=62 qos-map=classify qos-profile=class7

/interface/ethernet/switch
port/set sfp-sfpplus1 qos-map=classify qos-trust-l2=keep qos-trust-l3=keep
port/set sfp-sfpplus2 qos-map=classify qos-trust-l2=keep qos-trust-l3=keep
...

 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 7:57 am

I tested in a CRS 317 Version 7.6 with L3 HW Offload, then Ingress ACL for rate limiting does not work, but in L2 Bridge Vlan Filtering Ingress ACL for rate limiting works OK

that's why i'm asking
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 9:46 am

QoS-HW is compatible with L3HW. You can use both features together.

Every supported device has 8 TX queues per port, and users will be able to assign QoS profiles to TX queues: either grant a QoS profile exclusive access to a queue or share a queue (or group of queues) between multiple profiles. The feature is still in development and not available yet. Meanwhile, all QoS profiles share all TX queues, i.e., there is no QoS enforcement yet.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 9:59 am

What about the ability to do hardware QoS at arbitrary rates less than line rate? I'm thinking specifically about situations where one might buy a circuit from an upstream provider that is something that isn't a normal Ethernet speed like 200Mbps or something.
 
ofca
Member Candidate
Member Candidate
Posts: 228
Joined: Fri Aug 20, 2004 7:18 pm

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 2:58 pm

and what about VXLAN en/decapsulation in hardware? Marvell chips support it.
 
glueck05
newbie
Posts: 37
Joined: Fri Jan 26, 2018 12:49 pm

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 6:40 pm

Hello, this is interesting news. Are there plans to use the new feature for dynamic queues, e.g. with a PPPoE server on a CCR2216?

Thanks
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 7:06 pm

When you are working on this anyway, please consider the following:
1. allow to set software queue priority directly from DSCP. now, there is a detour required: set "priority" from DSCP, then set "packet mark" from priority, then assign queue priority based on packet mark. However, Linux can directly assign queue priority from the "priority" field, possibly gaining speed and also freeing the packet mark for other use.
2. implement a table for DSCP to priority mapping. "set priority from DSCP high 3 bits" usually is OK, but due to the strange design of DSCP it is not always correct. especially the AFxx DSCP values often map incorrectly.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 8:43 pm

Hello, this is interesting news. Are there plans to use the new feature for dynamic queues, e.g. with a PPPoE server on a CCR2216?

Thanks

i think this features are more focused towards industry switching qos functionalities

most equipment is not able to inspect or remark encapsulated PPPoE traffic, yes is another disadvantage of PPPoE

aditionally i am not sure PPPoE Server can benefit of L3 Hardware Offload at all
 
ib254254
just joined
Posts: 23
Joined: Sat Nov 21, 2020 10:15 pm

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 10:56 pm

Hello, this is interesting news. Are there plans to use the new feature for dynamic queues, e.g. with a PPPoE server on a CCR2216?

Thanks
Hello,

I'm also curious about the answer to this question.

What are your plans in this regard? This is very important.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS Hardware Offloading (QoS-HW)

Thu May 11, 2023 11:28 pm

aditionally i am not sure PPPoE Server can benefit of L3 Hardware Offload at all
Maybe not this hardware, but actually some of the SoC do implement PPPoE hardware offloading, as it is a common use case for consumer routers.
However, it does not look like RouterOS uses it.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: QoS Hardware Offloading (QoS-HW)

Fri May 12, 2023 12:32 pm

Using hardware QoS for bandwidth limitation is a natural next step for the project, so the feature will likely be implemented in the future. However, the current goals must be met first. The main goal of QoS HW is to provide lossless audio/video switching and (together with L3HW) routing at near-to-wire-speed rates. However, we want to think "out of the box" and make QoS HW as flexible as possible to cover a wider range of possible usage scenarios.

Keep in mind that Hardware QoS enforcement has nothing in common with software queues! That's why we specifically put the entire QoS section under the Switch menu (currently - in CLI, later - in WinBox/WebFig) as a hint that the feature is hardware-only. QoS HW has limited functionality but provides near-to-wire-speed performance, while software queues have greater flexibility at the cost of CPU power. While you can use both features (by redirecting some traffic to the CPU for software processing), those are two entirely different things.

P.S. Please stay on the topic! If you have a question about other hardware features, create a separate thread.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS Hardware Offloading (QoS-HW)

Fri May 12, 2023 12:39 pm

P.S. Please stay on the topic!
If only MikroTik would do that themselves. I.e. finish the unfinished features in v7 before starting a new one.
 
glueck05
newbie
Posts: 37
Joined: Fri Jan 26, 2018 12:49 pm

Re: QoS Hardware Offloading (QoS-HW)

Fri May 12, 2023 3:02 pm

Using hardware QoS for bandwidth limitation is a natural next step for the project, so the feature will likely be implemented in the future. However, the current goals must be met first. The main goal of QoS HW is to provide lossless audio/video switching and (together with L3HW) routing at near-to-wire-speed rates. However, we want to think "out of the box" and make QoS HW as flexible as possible to cover a wider range of possible usage scenarios.
that's great news. Thanks for the information.
 
Railander
Frequent Visitor
Frequent Visitor
Posts: 85
Joined: Thu Jun 16, 2016 11:30 pm

Re: QoS Hardware Offloading (QoS-HW)

Sat May 13, 2023 9:21 pm

don't know if this could be supported but it would be great if we could prioritize traffic based on TCP/UDP port.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: QoS Hardware Offloading (QoS-HW)

Mon May 15, 2023 10:38 am

don't know if this could be supported but it would be great if we could prioritize traffic based on TCP/UDP port.

We have plans to implement QoS profile assignment via Switch ACL rules, where you can match almost any L2/L3/L4 fields. The command will be something like this:
# NOT IMPLEMENTED YET! IT MAY BE A SUBJECT TO CHANGE.
/interface/ethernet/switch/rule add protocol=tcp dst-port=1337 qos-profile=my-prioritized-traffic
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: QoS Hardware Offloading (QoS-HW)

Mon May 15, 2023 11:58 am

Would it be possible to omit "protocol=???" to catch UDP and TCP in one rule?
 
User avatar
loloski
Member Candidate
Member Candidate
Posts: 277
Joined: Mon Mar 15, 2021 9:10 pm

Re: QoS Hardware Offloading (QoS-HW)

Mon May 15, 2023 11:59 am

Please finish Q-in-Q,MACSEC and VXLAN processing/offloading in switch chip and stabilize Ros v7 along side with this new endeavor of your MT
 
un9edsda
Frequent Visitor
Frequent Visitor
Posts: 76
Joined: Sun Mar 15, 2020 11:11 pm

Re: QoS Hardware Offloading (QoS-HW)

Tue May 16, 2023 1:43 am

First of all thanks for the documentation @raimondsp . Upon checking it I have come across a thing that would be great if clarified:
in
Port settings
...
By default, ports are untrusted and receive the lowest (0, best-effort) priority, where priority fields are cleared from the egress packets.
Is this for the
dscp
or the
priority
parameter? As in neither case marks 0 the lowest priority since in case of RFC4594 DiffServ Service Classes (DSCP)
dscp=8
should mark the lowest priority (CS1 Low-Priority Data which is below DF (CS0) Standard in the pecking order). In case of Priority Code Point (PCP) governed by 802.1Q-2022 (IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks) currently the lowest priority 0 (for background traffic) has PCP value 1 and the default PCP value 0 is for the one notch higher (priority 1) best effort traffic. So is the above quoted part of the documentation refers to PCP or DSCP?
If the above refers to PCP, than PCP value or PCP defined priority as it does make difference since PCP value 0 has priority 1, while PCP value 1 has priority 0.
If the above refers to DSCP, than the same question rises: does it refer to the value, in which case 0 has higher priority (DF (CS0)) than 8 (CS1)?

While we are at it in the same documentation regarding the PCP values the
QoS Profile
...
priority (integer: 0..7; Default: 0) - VLAN priority value (IEEE 802.1q PCP - Priority Code Point). ...
refer to the PCP value or the priority (as unlike higher values in the case of the first two PCP values of 0 and 1 these don't match)?

Thanks in advance for the explanation (and for clearing this up in the documentation).
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: QoS Hardware Offloading (QoS-HW)

Tue May 16, 2023 9:50 am

First of all thanks for the documentation @raimondsp . Upon checking it I have come across a thing that would be great if clarified:
in
Port settings
...
By default, ports are untrusted and receive the lowest (0, best-effort) priority, where priority fields are cleared from the egress packets.
Is this for the
dscp
or the
priority
parameter? As in neither case marks 0 the lowest priority since in case of RFC4594 DiffServ Service Classes (DSCP)
dscp=8
should mark the lowest priority (CS1 Low-Priority Data which is below DF (CS0) Standard in the pecking order). In case of Priority Code Point (PCP) governed by 802.1Q-2022 (IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks) currently the lowest priority 0 (for background traffic) has PCP value 1 and the default PCP value 0 is for the one notch higher (priority 1) best effort traffic. So is the above quoted part of the documentation refers to PCP or DSCP?
If the above refers to PCP, than PCP value or PCP defined priority as it does make difference since PCP value 0 has priority 1, while PCP value 1 has priority 0.
If the above refers to DSCP, than the same question rises: does it refer to the value, in which case 0 has higher priority (DF (CS0)) than 8 (CS1)?

While we are at it in the same documentation regarding the PCP values the
QoS Profile
...
priority (integer: 0..7; Default: 0) - VLAN priority value (IEEE 802.1q PCP - Priority Code Point). ...
refer to the PCP value or the priority (as unlike higher values in the case of the first two PCP values of 0 and 1 these don't match)?

Thanks in advance for the explanation (and for clearing this up in the documentation).

Hi,

You are right - the "Scavenger" class (CS1, DSCP=8) is the lowest DiffServ class, lower than Best-Effort (DSCP=0). I didn't know that the same applies to PCP, where PCP=1 means the lowest priority, followed by Best-Effort (PCP=0). It makes sense since PCP values are usually obtained from DSCP by shifting the three least significant bits. We will update the documentation to clarify that the default QoS profile is Best-Effort (PCP=0, DSCP=8), but it isn't the lowest priority. Also, we will consider renaming the QoS profile's priority field to pcp to avoid further confusion.

An example to assign the lowest priority to the port:
/interface/ethernet/switch/qos/profile add name=lowest-priority priority=1 dscp=8
/interface/ethernet/switch/port set ether1 qos-profile=lowest-priority

Thank you for the feedback!
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: QoS Hardware Offloading (QoS-HW)

Tue May 16, 2023 10:44 am

The most often used priority sequence for the 3 bit field is: 1,2,0,3,4,5,6,7
So indeed 1 is lowest, 2 is above that, and then 0 (default) is higher and 3-7 are above that (7 is highest).
 
lazywizard
just joined
Posts: 2
Joined: Mon Jul 17, 2023 4:34 pm

Re: QoS Hardware Offloading (QoS-HW)

Mon Jul 17, 2023 4:57 pm

Thanks for your work on this. I'm looking forward to taking advantage of QoS-HW. If I understand, we're now able to mark in hardware but enforcement still requires queues, which are mostly incompatible with fasttrack. On the CRS309, we still depend on fasttrack above ~400mbps so a hardware solution will be a big improvement.

In addition to prioritizing voice and other real-time traffic (gaming, Teams calls, etc. who can tag DSCP), if it is possible I would like to see:
1. fq_codel-like mechanism that controls bufferbloat and ensures users' traffic does not get choked out by noisy neighbours
2. Shaping/queueing for media changes, e.g. 10G-1G flow, 10G-1G/2.5G-WiFi flow, etc. where traffic is repeatedly dropped/retried and throughput suffers as a result. I am struggling with this on my network where I have my NAS and virtualization hosts connected at 10G and a 10G link to the otherwise-1G switch where my access points, workstations, and other devices are connected.
 
cyayon
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Wed Aug 24, 2022 9:39 am

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 18, 2023 3:26 pm

Hello,

I don't know if it related or not. Sorry if i made a mistake to post here.

I am currently using switch rules on a CRS305 in front of CCR2116 tp change COS on DHCP (ipv6 ipv4) requests. These DHCP requests with COS 6 are a requirement to authenticate to my ISP.
I decided to remove the CRS305 and use only the CCR2116. Of course i moved the switch rules from the CRS305 to the CCR.

But the rules do not work at all. Switch rules to change COS are never applied.

I am not sure, but i think i have understand that the switch are applied to ingress traffic, not egress. I my case, the DHCP client requests are of course egress from the CCR to my ISP.

Am i right or i make a mistake ?
Is there a workaround to change COS for DHCP egress requests ?

thanks.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 18, 2023 3:43 pm

@cyayon

are you using L3 HW offload? or any other Switch Function?
 
cyayon
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Wed Aug 24, 2022 9:39 am

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 18, 2023 3:51 pm

Hello,

I do not change this params from default (upgraded to 7.10.2).
Switch > Switch1 > L3HW Offloading is DISABLED
Switch > Port > sfpX (wan port) > L3HW Offloading is ENABLED (i tried to disable but the web interface give an error for Storm Rate which is to 10000% per default).
/interface/ethernet/switch/print
Columns: NAME, TYPE, L3-HW-OFFLOADING, QOS-HW-OFFLOADING
# NAME     TYPE              L3-HW-OFFLOADING  QOS-HW-OFFLOADING
0 switch1  Marvell-98DX3255  no                no
[admin@router1] > /interface/ethernet/switch/print  detail
Flags: I - invalid
 0   name="switch1" type=Marvell-98DX3255 mirror-source=none mirror-target=none mirror-egress-target=none l3-hw-offloading=no qos-hw-offloading=no
[admin@router1] > /interface/ethernet/switch/port print detail
Flags: I - invalid
 0   name="sfp1.lan" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 1   name="sfp2.wan1" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 2   name="sfp-sfpplus3" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 3   name="sfp-sfpplus4" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 4   name="ether1.wan2" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 5   name="ether2" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 6   name="ether3" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 7   name="ether4" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 8   name="ether5" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 9   name="ether6" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

10   name="ether7" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

11   name="ether8" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

12   name="ether9" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

13   name="ether10" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

14   name="ether11" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

15   name="ether12" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

16   name="switch1-cpu" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes


/interface/ethernet/switch/rule/print detail
Flags: X - disabled, I - invalid; D - dynamic
 0 X  ;;; orange dhcp4 COS6
      switch=switch1 ports=sfp2.wan1 mac-protocol=ip vlan-id=832 protocol=udp dst-port=67 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

 1 X  ;;; orange dhcp6 COS6
      switch=switch1 ports=sfp2.wan1 mac-protocol=ipv6 vlan-id=832 protocol=udp dst-port=547 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

 2 X  ;;; orange icmp6 COS6 (fe00::/7 = fe80::/10 + ff02::/16)
      switch=switch1 ports=sfp2.wan1 mac-protocol=ipv6 vlan-id=832 protocol=icmp dst-address6=fe00::/7 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

 3 X  ;;; orange arp COS6
      switch=switch1 ports=sfp2.wan1 mac-protocol=arp vlan-id=832 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6
The switch rules are currently disabled (i have just disabled them because they do not work at all)
Is this related ? should i disable ?
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 18, 2023 5:20 pm

L3HW offload has to be enabled on the switch before enabling it on the ports makes any difference. If it's not enabled at the switch level, then none of the HW QoS marking will apply.
 
cyayon
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Wed Aug 24, 2022 9:39 am

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 18, 2023 5:27 pm

Thanks.

Do i need to enable on the switch ?

/interface/ethernet/switch/set l3-hw-offloading=yes

Does my current switch rules will work or do i need to rewrite them ?

thanks
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 25, 2023 4:14 pm

Thanks.

Do i need to enable on the switch ?

/interface/ethernet/switch/set l3-hw-offloading=yes

Does my current switch rules will work or do i need to rewrite them ?

thanks

Hi,

Enabling l3-hw-offloading on the switch level globally turns on L3HW (i.e., starts l3hw driver). Other l3hw configuration options do not matter when l3-hw-offloading=no (RouterOS still keeps showing them for diagnostic purposes).

Switch ACL Rules (/interface/ethernet/switch/rule) work with L3HW. Moreover, Switch Rules have a higher priority than L3HW. For example, you can create a rule to drop traffic or redirect it to the CPU - in both cases, the respective packets will not reach L3HW.
 
cyayon
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Wed Aug 24, 2022 9:39 am

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 25, 2023 5:08 pm

Thanks.

As i understand, switch ACL are only for inbound traffic, not outbound (my case).
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Tue Jul 25, 2023 6:03 pm

is a little off topic but i have the following question:

is MikroTik considering in the long term to have some Eggress ACL (switch rules) support on CRS Switches ?
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2095
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Wed Jul 26, 2023 8:23 am

is a little off topic but i have the following question:

is MikroTik considering in the long term to have some Eggress ACL (switch rules) support on CRS Switches ?
I am interested to know this too.

In my opinion, the current switch ACL language is extremely prohibitive. Mikrotik needs to develop an all new Switch ACL language that can handle more use-cases, a few off the top of my head:

- Ingress + Egress traffic flows
- Multiple actions, e.g. Rewrite + Count
- Ability to match on different and multiple parts of the packet header e.g. Outer and Inner VLAN tag
- Ability to rewrite outer and inner VLAN tags
- Ability to strip and add multiple VLAN tags

Ideally this would provide an abstraction layer allowing it to be used on current and future hardware, even if it has different switch ASIC's.
 
syadnom
Forum Veteran
Forum Veteran
Posts: 794
Joined: Thu Jan 27, 2011 7:29 am

Re: QoS Hardware Offloading (QoS-HW)

Sun Aug 06, 2023 8:50 pm

I would like this to have some fast-path option in the firewall. This would allow doing content matching by source for use in traffic shaping appliances and QoS on some radio hardware. ie, match zoom's ASN IPs and mark those voice on ingress.

interface rate setting for QoS. Most backhaul radios are not as fast as their interface, Ie we have 10G ports on 1.5Gbps radios. It would be nice to set that to the expected rate.

identify and drop ECN marks in hardware when rates approach/exceed the rate set on the interface as mentioned above.

Would be really nice to get fq_codel in there if the hardware can handle it. cake even better. Ability to handle a queue tree even better.

reporting. netflow or similar data.

much of this can be done in software, but at huge performance costs. If any of it can be done in hardware that would be great for the *isp market.
 
hicountrynet
just joined
Posts: 4
Joined: Wed Sep 20, 2023 6:32 am

Re: QoS Hardware Offloading (QoS-HW)

Wed Sep 27, 2023 3:25 pm

I'm wondering as well, does this QoS-HW mean fq-codel traffic shaping in hardware as opposed to a CPU only thing?

Cheers
Paul
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: QoS Hardware Offloading (QoS-HW)

Wed Sep 27, 2023 7:15 pm

I'm wondering as well, does this QoS-HW mean fq-codel traffic shaping in hardware as opposed to a CPU only thing?

Cheers
Paul
No
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: QoS Hardware Offloading (QoS-HW)

Thu Oct 19, 2023 9:16 am

I just stumbled across this thread. 6 comments:

cake's diffserv4 model is the closest we could come to matching multiple mutually conflicting rfcs.

because of confusion over the cs1 codepoint as background, the LE codepoint was created: RFC8622.

There is a new codepoint pending with basically the same priority as cs5 called NQB with many nuances: https://datatracker.ietf.org/doc/html/d ... vwg-nqb-19

I would dearly like the various states of the ecn bits to be counted and tracked. There is a lot more of it out there than most realize and

Lastly I really hate strict priority queues as a major footgun against deployment of diffserv at all. Others like this.

I have been soliciting input on a new version of CAKE here: https://docs.google.com/document/d/1tTY ... bnfu73wlpp
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: QoS Hardware Offloading (QoS-HW)

Thu Oct 19, 2023 9:29 am

QoS-HW is compatible with L3HW. You can use both features together.

Every supported device has 8 TX queues per port, and users will be able to assign QoS profiles to TX queues: either grant a QoS profile exclusive access to a queue or share a queue (or group of queues) between multiple profiles. The feature is still in development and not available yet. Meanwhile, all QoS profiles share all TX queues, i.e., there is no QoS enforcement yet.
In general I prefer bql + cake split evenly between all hardware tx queues vs QoS & diffserv applied this way because it leverages the scarcest resource, CPU. diffserv is a fiddly distraction compared to aqm and fq in the first place. A little diffserv on top of that can help, but first you gotta keep the queues short and the packets well mixed.

It doesn't help much for me to say that, 'cause people love these fiddly bits, but the qosify project mostly got it right on top of CAKE. All I can say is go ahead, offload bandwidth management in hw next, and measure latencies and throughput with flent, packet captures, and pay especial attention to the tcp rtt.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: QoS Hardware Offloading (QoS-HW)

Thu Oct 19, 2023 5:33 pm

Over here is an implementation of BQL for the mvpp2 chip, which may be similar to the chip you are using, with test results.

https://github.com/wojtas-marcin/Linux- ... 80057a70fb

Having BQL is a godsend, it lets you run fq_codel at line rate at about 1/20th the CPU of shaping, at the same level of smoothness as shaping. You can further add support for a hardware shaper down there, while retaining all the benefits of fq_codel or cake slightly up the stack. I think the mt79 and several intel chips support this nowadays.

Marcin (the author) I believe still does some marvell contract work.

For a simple guide to getting BQL going, and why it is important, please see: http://www.taht.net/~d/broadcom_aug9_2018.pdf

After you get something like that going, adding some qos can help.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: QoS Hardware Offloading (QoS-HW)

Sun Oct 22, 2023 8:18 pm

@raimondsp any feedback
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Topic Author
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: QoS Hardware Offloading (QoS-HW)

Tue Nov 07, 2023 9:56 am

Hi there, and sorry for the late reply,

  • Unfortunately, the current switch chips do not support fq_codel or CAKE. However, RouterOS supports those AQM models on the software level. If the bandwidth is not too high, consider redirecting the traffic to the CPU for FQ-CoDel or CAKE usage.
  • Most Marvell switch chip models that we use in our CCR2116, CCR2216, CRS3xx, and CRS5xx series support WRED (Weighted Random Early Detection). While not being as robust against butterbloat as some other AQM, WRED is performed on the hardware level at wire speed. We have plans to implement WRED offloading into RouterOS in the near future.
  • Some switch chip models support hardware ECN marking and processing (e.g., CCR2216). The feature is on our roadmap.
  • Enhancing Switch ACL rules for QoS Profile selection (and therefore, QoS marking and hw queue selection) is also on the roadmap.
  • Currently, RouterOS does not support Egress ACL rules. Unfortunately, at the moment of writing, I'm unaware if this feature is planned.
 
dtaht
Member Candidate
Member Candidate
Posts: 209
Joined: Sat Aug 03, 2013 5:46 am

Re: QoS Hardware Offloading (QoS-HW)

Sun Nov 12, 2023 12:11 am

My point was that BQL support helps first and most.
 
mobyfab
just joined
Posts: 6
Joined: Tue Jul 03, 2018 4:45 pm
Location: France

Re: QoS Hardware Offloading (QoS-HW)

Wed Jan 17, 2024 12:34 pm

Hello,

I don't know if it related or not. Sorry if i made a mistake to post here.

I am currently using switch rules on a CRS305 in front of CCR2116 tp change COS on DHCP (ipv6 ipv4) requests. These DHCP requests with COS 6 are a requirement to authenticate to my ISP.
I decided to remove the CRS305 and use only the CCR2116. Of course i moved the switch rules from the CRS305 to the CCR.

But the rules do not work at all. Switch rules to change COS are never applied.

I am not sure, but i think i have understand that the switch are applied to ingress traffic, not egress. I my case, the DHCP client requests are of course egress from the CCR to my ISP.

Am i right or i make a mistake ?
Is there a workaround to change COS for DHCP egress requests ?

thanks.
It never worked.
I have a similar setup, CCR2116 + CRS317.
Switch rules on cpu port are broken on CCR2116, I have a ticket open for almost 2 years...
So i'm using rules on the CRS317 which is no convenient at all.

Who is online

Users browsing this forum: anav, Andrey05, ivicask, Valerio5000 and 98 guests