QoS Hardware Offloading (QoS-HW)

Wed May 10, 2023 12:46 pm

Greetings, fellow community members!

We are glad to announce the beginning of a new project - Quality of Service Hardware Offloading (QoS-HW), introduced in RouterOS v7.10. The goal of the project is to perform QoS packet marking (VLAN PCP, IP DSCP, and in the future - MPLS EXP), traffic shaping, congestion avoidance/resolution, lossless forwarding, etc. - on the hardware level, which, in turn, means near-to-wire-speed performance.

Documentation: https://help.mikrotik.com/docs/pages/vi ... =189497483

The target devices are those based on Marvell Prestera® DX switch chips: MikroTik CRS3xx, CRS5xx series, CCR2116, and CCR2216. In other words, the devices that support L3HW will eventually support QoS-HW.

Your feedback is more than welcome! Please share your vision of QoS enforcement in RouterOS, use-cases, and setups. While the project is in the Beta phase, it is very flexible to adjust to community demand. Also, our QA engineers would like to perform QoS testing based on real setups rather than artificial test cases.

RouterOS v7.15 UPDATE: many QoS features have been implemented, including QoS enforcement, scheduling, active queue management, traffic shaping, etc. Check the documentation for details.

Current limitations and known issues:

WRED requires use-shared-buffers=yes. It will be forced on RouterOS v7.16. Meanwhile, avoid setting queues with: "wred=yes use-shared-buffers=no"
PFC works only on some switch ports and is unreliable in general. We are investigating the issue.
Do NOT change wred-* settings in /in/eth/sw/qos/settings.
Do NOT change queue-buffers in /in/eth/sw/qos/tx/queue.

loloski · Wed May 10, 2023 1:18 pm

while we are at it could you please confirm if 802.1ad tag stacking offloading in the switch chip is in the pipeline?, great news by the way

ib254254 · Wed May 10, 2023 2:43 pm

thanks you

ib254254 · Wed May 10, 2023 2:49 pm

thanks you

slackR · Wed May 10, 2023 3:48 pm

Very nice! The Marvell 90DX chips have QoS feature that will make a great feature for ROS. Is there a page in documentation for the setup? I didn't see one.

Being able to classify traffic and leverage the switch chip queues would be beneficial. Prioritizing voice/video traffic is a good use case. I noticed the CRS309 has 8 hardware TX queues (tx-queue0-packet) but I didn't see how to classify traffic to use each queue.

 /interface/ethernet/switch/port> print stats        
                 name:  sfp-sfpplus1  sfp-sfpplus2 sfp-sfpplus3 sfp-sfpplus4 sfp-sfpplus5 sfp-sfpplus6 sfp-sfpplus7  sfp-sfpplus8  ether1 switch1-cpu
       driver-rx-byte:             0   452 976 691            0            0            0            0            0 1 086 224 642 130 375
     driver-rx-packet:             0       773 965            0            0            0            0            0       945 811     689
       driver-tx-byte:             0 1 188 556 474            0            0            0            0            0   453 023 932  39 675
     driver-tx-packet:             0     1 055 283            0            0            0            0            0       675 260     214
             rx-bytes:             0   527 365 820            0            0            0            0            0 1 369 898 664 137 181           0
         rx-too-short:             0             0            0            0            0            0            0             0       0           0
          rx-too-long:             0             0            0            0            0            0            0             0       0           0
           rx-unicast:             0       912 412            0            0            0            0            0     1 185 318       7           0
         rx-broadcast:             0         1 120            0            0            0            0            0         1 218     209           0
             rx-pause:             0             0            0            0            0            0            0             0       0           0
         rx-multicast:             0         7 748            0            0            0            0            0         1 908     518           0
      rx-error-events:             0             0            0            0            0            0            0             0       0           0
         rx-fcs-error:             0             0            0            0            0            0            0             0       0           0
          rx-fragment:             0             0            0            0            0            0            0             0       0           0
          rx-overflow:             0             0            0            0            0            0            0             0       0           0
            rx-jabber:             0             0            0            0            0            0            0             0       0           0
             tx-bytes:             0 1 473 609 464            0            0            0            0            0   523 059 415  40 531           0
           tx-unicast:             0     1 295 077            0            0            0            0            0       818 688      12           0
         tx-broadcast:             0           115            0            0            0            0            0             3      99           0
             tx-pause:             0             0            0            0            0            0            0             0       0           0
         tx-multicast:             0         1 967            0            0            0            0            0             2     103           0
          tx-underrun:             0             0            0            0            0            0            0             0       0           0
         tx-collision:             0             0            0            0            0            0            0             0       0           0
    tx-late-collision:             0             0            0            0            0            0            0             0       0           0
              tx-drop:             0             0            0            0            0            0            0             0       0           0
             tx-rx-64:             0        39 293            0            0            0            0            0       101 104      63           0
         tx-rx-65-127:             0       714 147            0            0            0            0            0       432 308     101           0
        tx-rx-128-255:             0        71 389            0            0            0            0            0       152 504     647           0
        tx-rx-256-511:             0        29 741            0            0            0            0            0        23 809     137           0
       tx-rx-512-1023:             0        26 816            0            0            0            0            0        19 827       0           0
       tx-rx-1024-max:             0     1 337 053            0            0            0            0            0     1 277 585       0           0
     tx-queue0-packet:             0     1 297 159            0            0            0            0            0       807 319     214           0
     tx-queue1-packet:             0             0            0            0            0            0            0        11 374       0           0
     tx-queue2-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue3-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue4-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue5-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue6-packet:             0             0            0            0            0            0            0             0       0           0
     tx-queue7-packet:             0             0            0            0            0            0            0             0       0           0

Wed May 10, 2023 11:11 pm

I noticed the CRS309 has 8 hardware TX queues (tx-queue0-packet) but I didn't see how to classify traffic to use each queue.

i was tracking that for a while, now we know this is real

keep in mind

The current implementation is for QoS Phase 1 - QoS Marking (introduced in RouterOS v7.10).

so maybe we will have to wait some versions for that queues to be available

Wed May 10, 2023 11:13 pm

Greetings, fellow community members!

We are glad to announce the beginning of a new project - Quality of Service Hardware Offloading (QoS-HW), introduced in RouterOS v7.10. The goal of the project is to perform QoS packet marking (VLAN PCP, IP DSCP, and in the future - MPLS EXP), traffic shaping, congestion avoidance/resolution, lossless forwarding, etc. - on the hardware level, which, in turn, means near-to-wire-speed performance.

Documentation: https://help.mikrotik.com/docs/pages/vi ... =189497483

The target devices are those based on Marvell Prestera® DX switch chips: MikroTik CRS3xx, CRS5xx series, CCR2116, and CCR2216. In other words, the devices that support L3HW will eventually support QoS-HW.

We are currently at the beginning of the project: RouterOS v7.10 can perform QoS packet marking on the hardware level. Other QoS features are still in development. The documentation will be available shortly. I'll put the link here once ready.

Your feedback is more than welcome! Please share your vision of QoS enforcement in RouterOS, use-cases, and setups. While the project is in the Beta phase, it is very flexible to adjust to community demand. Also, our QA engineers would like to perform QoS testing based on real setups rather than artificial test cases.

This topic will be updated with recent info regarding QoS HW.

QoS Hardware Offloading (QoS-HW) will work in concur with L3 Hw Offload ?? or only in L2 setups?

slackR · Thu May 11, 2023 7:42 am

From what I understand QoS-HW will work with both L2 and L3.

/interface ethernet switch qos map
add name=classify

/interface ethernet switch qos profile
add dscp=8 name=scavanger
add name=best-effort priority=1
add name=class1 priority=1
add name=class2 priority=2
add name=class3 priority=3
add name=class4 priority=4
add name=class5 priority=5
add name=class6 priority=6
add name=class7 priority=7

/interface ethernet switch qos map vlan
add qos-map=classify qos-profile=scavanger
add priority=1 qos-map=classify qos-profile=class1
add priority=2 qos-map=classify qos-profile=class2
add priority=3 qos-map=classify qos-profile=class3
add priority=4 qos-map=classify qos-profile=class4
add priority=5 qos-map=classify qos-profile=class5
add priority=6 qos-map=classify qos-profile=class6
add priority=7 qos-map=classify qos-profile=class7

/interface ethernet switch qos map ip
add dscp=0 qos-map=classify qos-profile=best-effort
add dscp=2 qos-map=classify qos-profile=class1
add dscp=4 qos-map=classify qos-profile=class1
add dscp=6 qos-map=classify qos-profile=class1
add dscp=8 qos-map=classify qos-profile=scavanger
add dscp=10 qos-map=classify qos-profile=scavanger
add dscp=12 qos-map=classify qos-profile=class1
add dscp=14 qos-map=classify qos-profile=class1
add dscp=16 qos-map=classify qos-profile=class1
add dscp=18 qos-map=classify qos-profile=class2
add dscp=20 qos-map=classify qos-profile=class2
add dscp=22 qos-map=classify qos-profile=class2
add dscp=24 qos-map=classify qos-profile=class2
add dscp=26 qos-map=classify qos-profile=class3
add dscp=28 qos-map=classify qos-profile=class3
add dscp=30 qos-map=classify qos-profile=class3
add dscp=32 qos-map=classify qos-profile=class3
add dscp=34 qos-map=classify qos-profile=class3
add dscp=36 qos-map=classify qos-profile=class4
add dscp=38 qos-map=classify qos-profile=class4
add dscp=40 qos-map=classify qos-profile=class4
add dscp=42 qos-map=classify qos-profile=class4
add dscp=44 qos-map=classify qos-profile=class4
add dscp=46 qos-map=classify qos-profile=class5
add dscp=48 qos-map=classify qos-profile=class5
add dscp=50 qos-map=classify qos-profile=class5
add dscp=52 qos-map=classify qos-profile=class5
add dscp=54 qos-map=classify qos-profile=class5
add dscp=56 qos-map=classify qos-profile=class6
add dscp=58 qos-map=classify qos-profile=class7
add dscp=60 qos-map=classify qos-profile=class7
add dscp=62 qos-map=classify qos-profile=class7

/interface/ethernet/switch
port/set sfp-sfpplus1 qos-map=classify qos-trust-l2=keep qos-trust-l3=keep
port/set sfp-sfpplus2 qos-map=classify qos-trust-l2=keep qos-trust-l3=keep
...

Thu May 11, 2023 7:57 am

I tested in a CRS 317 Version 7.6 with L3 HW Offload, then Ingress ACL for rate limiting does not work, but in L2 Bridge Vlan Filtering Ingress ACL for rate limiting works OK

that's why i'm asking

Thu May 11, 2023 9:46 am

QoS-HW is compatible with L3HW. You can use both features together.

Every supported device has 8 TX queues per port, and users will be able to assign QoS profiles to TX queues: either grant a QoS profile exclusive access to a queue or share a queue (or group of queues) between multiple profiles. The feature is still in development and not available yet. Meanwhile, all QoS profiles share all TX queues, i.e., there is no QoS enforcement yet.

mducharme · Thu May 11, 2023 9:59 am

What about the ability to do hardware QoS at arbitrary rates less than line rate? I'm thinking specifically about situations where one might buy a circuit from an upstream provider that is something that isn't a normal Ethernet speed like 200Mbps or something.

ofca · Thu May 11, 2023 2:58 pm

and what about VXLAN en/decapsulation in hardware? Marvell chips support it.

glueck05 · Thu May 11, 2023 6:40 pm

Hello, this is interesting news. Are there plans to use the new feature for dynamic queues, e.g. with a PPPoE server on a CCR2216?

Thanks

pe1chl · Thu May 11, 2023 7:06 pm

When you are working on this anyway, please consider the following:
1. allow to set software queue priority directly from DSCP. now, there is a detour required: set "priority" from DSCP, then set "packet mark" from priority, then assign queue priority based on packet mark. However, Linux can directly assign queue priority from the "priority" field, possibly gaining speed and also freeing the packet mark for other use.
2. implement a table for DSCP to priority mapping. "set priority from DSCP high 3 bits" usually is OK, but due to the strange design of DSCP it is not always correct. especially the AFxx DSCP values often map incorrectly.

Thu May 11, 2023 8:43 pm

Hello, this is interesting news. Are there plans to use the new feature for dynamic queues, e.g. with a PPPoE server on a CCR2216?

Thanks

i think this features are more focused towards industry switching qos functionalities

most equipment is not able to inspect or remark encapsulated PPPoE traffic, yes is another disadvantage of PPPoE

aditionally i am not sure PPPoE Server can benefit of L3 Hardware Offload at all

ib254254 · Thu May 11, 2023 10:56 pm

Hello, this is interesting news. Are there plans to use the new feature for dynamic queues, e.g. with a PPPoE server on a CCR2216?

Thanks

Hello,

I'm also curious about the answer to this question.

What are your plans in this regard? This is very important.

pe1chl · Thu May 11, 2023 11:28 pm

aditionally i am not sure PPPoE Server can benefit of L3 Hardware Offload at all

Maybe not this hardware, but actually some of the SoC do implement PPPoE hardware offloading, as it is a common use case for consumer routers.
However, it does not look like RouterOS uses it.

Fri May 12, 2023 12:32 pm

Using hardware QoS for bandwidth limitation is a natural next step for the project, so the feature will likely be implemented in the future. However, the current goals must be met first. The main goal of QoS HW is to provide lossless audio/video switching and (together with L3HW) routing at near-to-wire-speed rates. However, we want to think "out of the box" and make QoS HW as flexible as possible to cover a wider range of possible usage scenarios.

Keep in mind that Hardware QoS enforcement has nothing in common with software queues! That's why we specifically put the entire QoS section under the Switch menu (currently - in CLI, later - in WinBox/WebFig) as a hint that the feature is hardware-only. QoS HW has limited functionality but provides near-to-wire-speed performance, while software queues have greater flexibility at the cost of CPU power. While you can use both features (by redirecting some traffic to the CPU for software processing), those are two entirely different things.

P.S. Please stay on the topic! If you have a question about other hardware features, create a separate thread.

pe1chl · Fri May 12, 2023 12:39 pm

P.S. Please stay on the topic!

If only MikroTik would do that themselves. I.e. finish the unfinished features in v7 before starting a new one.

glueck05 · Fri May 12, 2023 3:02 pm

Using hardware QoS for bandwidth limitation is a natural next step for the project, so the feature will likely be implemented in the future. However, the current goals must be met first. The main goal of QoS HW is to provide lossless audio/video switching and (together with L3HW) routing at near-to-wire-speed rates. However, we want to think "out of the box" and make QoS HW as flexible as possible to cover a wider range of possible usage scenarios.

that's great news. Thanks for the information.

Railander · Sat May 13, 2023 9:21 pm

don't know if this could be supported but it would be great if we could prioritize traffic based on TCP/UDP port.

Mon May 15, 2023 10:38 am

don't know if this could be supported but it would be great if we could prioritize traffic based on TCP/UDP port.

We have plans to implement QoS profile assignment via Switch ACL rules, where you can match almost any L2/L3/L4 fields. The command will be something like this:

# NOT IMPLEMENTED YET! IT MAY BE A SUBJECT TO CHANGE.
/interface/ethernet/switch/rule add protocol=tcp dst-port=1337 qos-profile=my-prioritized-traffic

Mon May 15, 2023 11:58 am

Would it be possible to omit "protocol=???" to catch UDP and TCP in one rule?

loloski · Mon May 15, 2023 11:59 am

Please finish Q-in-Q,MACSEC and VXLAN processing/offloading in switch chip and stabilize Ros v7 along side with this new endeavor of your MT

un9edsda · Tue May 16, 2023 1:43 am

First of all thanks for the documentation @raimondsp . Upon checking it I have come across a thing that would be great if clarified:
in

Port settings
...
By default, ports are untrusted and receive the lowest (0, best-effort) priority, where priority fields are cleared from the egress packets.

Is this for the

dscp

or the

priority

parameter? As in neither case marks 0 the lowest priority since in case of RFC4594 DiffServ Service Classes (DSCP)

dscp=8

should mark the lowest priority (CS1 Low-Priority Data which is below DF (CS0) Standard in the pecking order). In case of Priority Code Point (PCP) governed by 802.1Q-2022 (IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks) currently the lowest priority 0 (for background traffic) has PCP value 1 and the default PCP value 0 is for the one notch higher (priority 1) best effort traffic. So is the above quoted part of the documentation refers to PCP or DSCP?
If the above refers to PCP, than PCP value or PCP defined priority as it does make difference since PCP value 0 has priority 1, while PCP value 1 has priority 0.
If the above refers to DSCP, than the same question rises: does it refer to the value, in which case 0 has higher priority (DF (CS0)) than 8 (CS1)?

While we are at it in the same documentation regarding the PCP values the

QoS Profile
...
priority (integer: 0..7; Default: 0) - VLAN priority value (IEEE 802.1q PCP - Priority Code Point). ...

refer to the PCP value or the priority (as unlike higher values in the case of the first two PCP values of 0 and 1 these don't match)?

Thanks in advance for the explanation (and for clearing this up in the documentation).

Tue May 16, 2023 9:50 am

First of all thanks for the documentation @raimondsp . Upon checking it I have come across a thing that would be great if clarified:
in
Port settings
...
By default, ports are untrusted and receive the lowest (0, best-effort) priority, where priority fields are cleared from the egress packets.
Is this for the
Code: Select all
dscp
or the
Code: Select all
priority
parameter? As in neither case marks 0 the lowest priority since in case of RFC4594 DiffServ Service Classes (DSCP)
Code: Select all
dscp=8
should mark the lowest priority (CS1 Low-Priority Data which is below DF (CS0) Standard in the pecking order). In case of Priority Code Point (PCP) governed by 802.1Q-2022 (IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks) currently the lowest priority 0 (for background traffic) has PCP value 1 and the default PCP value 0 is for the one notch higher (priority 1) best effort traffic. So is the above quoted part of the documentation refers to PCP or DSCP?
If the above refers to PCP, than PCP value or PCP defined priority as it does make difference since PCP value 0 has priority 1, while PCP value 1 has priority 0.
If the above refers to DSCP, than the same question rises: does it refer to the value, in which case 0 has higher priority (DF (CS0)) than 8 (CS1)?

While we are at it in the same documentation regarding the PCP values the
QoS Profile
...
priority (integer: 0..7; Default: 0) - VLAN priority value (IEEE 802.1q PCP - Priority Code Point). ...
refer to the PCP value or the priority (as unlike higher values in the case of the first two PCP values of 0 and 1 these don't match)?

Thanks in advance for the explanation (and for clearing this up in the documentation).

Hi,

You are right - the "Scavenger" class (CS1, DSCP=8) is the lowest DiffServ class, lower than Best-Effort (DSCP=0). I didn't know that the same applies to PCP, where PCP=1 means the lowest priority, followed by Best-Effort (PCP=0). It makes sense since PCP values are usually obtained from DSCP by shifting the three least significant bits. We will update the documentation to clarify that the default QoS profile is Best-Effort (PCP=0, DSCP=8), but it isn't the lowest priority. Also, we will consider renaming the QoS profile's priority field to pcp to avoid further confusion.

An example to assign the lowest priority to the port:

/interface/ethernet/switch/qos/profile add name=lowest-priority priority=1 dscp=8
/interface/ethernet/switch/port set ether1 qos-profile=lowest-priority

Thank you for the feedback!

pe1chl · Tue May 16, 2023 10:44 am

The most often used priority sequence for the 3 bit field is: 1,2,0,3,4,5,6,7
So indeed 1 is lowest, 2 is above that, and then 0 (default) is higher and 3-7 are above that (7 is highest).

lazywizard · Mon Jul 17, 2023 4:57 pm

Thanks for your work on this. I'm looking forward to taking advantage of QoS-HW. If I understand, we're now able to mark in hardware but enforcement still requires queues, which are mostly incompatible with fasttrack. On the CRS309, we still depend on fasttrack above ~400mbps so a hardware solution will be a big improvement.

In addition to prioritizing voice and other real-time traffic (gaming, Teams calls, etc. who can tag DSCP), if it is possible I would like to see:
1. fq_codel-like mechanism that controls bufferbloat and ensures users' traffic does not get choked out by noisy neighbours
2. Shaping/queueing for media changes, e.g. 10G-1G flow, 10G-1G/2.5G-WiFi flow, etc. where traffic is repeatedly dropped/retried and throughput suffers as a result. I am struggling with this on my network where I have my NAS and virtualization hosts connected at 10G and a 10G link to the otherwise-1G switch where my access points, workstations, and other devices are connected.

cyayon · Tue Jul 18, 2023 3:26 pm

Hello,

I don't know if it related or not. Sorry if i made a mistake to post here.

I am currently using switch rules on a CRS305 in front of CCR2116 tp change COS on DHCP (ipv6 ipv4) requests. These DHCP requests with COS 6 are a requirement to authenticate to my ISP.
I decided to remove the CRS305 and use only the CCR2116. Of course i moved the switch rules from the CRS305 to the CCR.

But the rules do not work at all. Switch rules to change COS are never applied.

I am not sure, but i think i have understand that the switch are applied to ingress traffic, not egress. I my case, the DHCP client requests are of course egress from the CCR to my ISP.

Am i right or i make a mistake ?
Is there a workaround to change COS for DHCP egress requests ?

thanks.

Tue Jul 18, 2023 3:43 pm

@cyayon

are you using L3 HW offload? or any other Switch Function?

cyayon · Tue Jul 18, 2023 3:51 pm

Hello,

I do not change this params from default (upgraded to 7.10.2).
Switch > Switch1 > L3HW Offloading is DISABLED
Switch > Port > sfpX (wan port) > L3HW Offloading is ENABLED (i tried to disable but the web interface give an error for Storm Rate which is to 10000% per default).

/interface/ethernet/switch/print
Columns: NAME, TYPE, L3-HW-OFFLOADING, QOS-HW-OFFLOADING
# NAME     TYPE              L3-HW-OFFLOADING  QOS-HW-OFFLOADING
0 switch1  Marvell-98DX3255  no                no
[admin@router1] > /interface/ethernet/switch/print  detail
Flags: I - invalid
 0   name="switch1" type=Marvell-98DX3255 mirror-source=none mirror-target=none mirror-egress-target=none l3-hw-offloading=no qos-hw-offloading=no
[admin@router1] > /interface/ethernet/switch/port print detail
Flags: I - invalid
 0   name="sfp1.lan" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 1   name="sfp2.wan1" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 2   name="sfp-sfpplus3" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 3   name="sfp-sfpplus4" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 4   name="ether1.wan2" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 5   name="ether2" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 6   name="ether3" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 7   name="ether4" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 8   name="ether5" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

 9   name="ether6" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

10   name="ether7" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

11   name="ether8" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

12   name="ether9" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

13   name="ether10" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

14   name="ether11" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

15   name="ether12" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes l3-hw-offloading=yes qos-profile=default qos-map=default qos-trust-l2=ignore qos-trust-l3=ignore

16   name="switch1-cpu" switch=switch1 storm-rate=100 limit-unknown-unicasts=no limit-unknown-multicasts=no limit-broadcasts=yes


/interface/ethernet/switch/rule/print detail
Flags: X - disabled, I - invalid; D - dynamic
 0 X  ;;; orange dhcp4 COS6
      switch=switch1 ports=sfp2.wan1 mac-protocol=ip vlan-id=832 protocol=udp dst-port=67 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

 1 X  ;;; orange dhcp6 COS6
      switch=switch1 ports=sfp2.wan1 mac-protocol=ipv6 vlan-id=832 protocol=udp dst-port=547 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

 2 X  ;;; orange icmp6 COS6 (fe00::/7 = fe80::/10 + ff02::/16)
      switch=switch1 ports=sfp2.wan1 mac-protocol=ipv6 vlan-id=832 protocol=icmp dst-address6=fe00::/7 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

 3 X  ;;; orange arp COS6
      switch=switch1 ports=sfp2.wan1 mac-protocol=arp vlan-id=832 copy-to-cpu=no redirect-to-cpu=no mirror=no new-vlan-priority=6

The switch rules are currently disabled (i have just disabled them because they do not work at all)
Is this related ? should i disable ?

sirbryan · Tue Jul 18, 2023 5:20 pm

L3HW offload has to be enabled on the switch before enabling it on the ports makes any difference. If it's not enabled at the switch level, then none of the HW QoS marking will apply.

cyayon · Tue Jul 18, 2023 5:27 pm

Thanks.

Do i need to enable on the switch ?

/interface/ethernet/switch/set l3-hw-offloading=yes

Does my current switch rules will work or do i need to rewrite them ?

thanks

Tue Jul 25, 2023 4:14 pm

Thanks.

Do i need to enable on the switch ?

/interface/ethernet/switch/set l3-hw-offloading=yes

Does my current switch rules will work or do i need to rewrite them ?

thanks

Hi,

Enabling l3-hw-offloading on the switch level globally turns on L3HW (i.e., starts l3hw driver). Other l3hw configuration options do not matter when l3-hw-offloading=no (RouterOS still keeps showing them for diagnostic purposes).

Switch ACL Rules (/interface/ethernet/switch/rule) work with L3HW. Moreover, Switch Rules have a higher priority than L3HW. For example, you can create a rule to drop traffic or redirect it to the CPU - in both cases, the respective packets will not reach L3HW.

cyayon · Tue Jul 25, 2023 5:08 pm

Thanks.

As i understand, switch ACL are only for inbound traffic, not outbound (my case).

Tue Jul 25, 2023 6:03 pm

is a little off topic but i have the following question:

is MikroTik considering in the long term to have some Eggress ACL (switch rules) support on CRS Switches ?

Wed Jul 26, 2023 8:23 am

is a little off topic but i have the following question:

is MikroTik considering in the long term to have some Eggress ACL (switch rules) support on CRS Switches ?

I am interested to know this too.

In my opinion, the current switch ACL language is extremely prohibitive. Mikrotik needs to develop an all new Switch ACL language that can handle more use-cases, a few off the top of my head:

- Ingress + Egress traffic flows
- Multiple actions, e.g. Rewrite + Count
- Ability to match on different and multiple parts of the packet header e.g. Outer and Inner VLAN tag
- Ability to rewrite outer and inner VLAN tags
- Ability to strip and add multiple VLAN tags

Ideally this would provide an abstraction layer allowing it to be used on current and future hardware, even if it has different switch ASIC's.

syadnom · Sun Aug 06, 2023 8:50 pm

I would like this to have some fast-path option in the firewall. This would allow doing content matching by source for use in traffic shaping appliances and QoS on some radio hardware. ie, match zoom's ASN IPs and mark those voice on ingress.

interface rate setting for QoS. Most backhaul radios are not as fast as their interface, Ie we have 10G ports on 1.5Gbps radios. It would be nice to set that to the expected rate.

identify and drop ECN marks in hardware when rates approach/exceed the rate set on the interface as mentioned above.

Would be really nice to get fq_codel in there if the hardware can handle it. cake even better. Ability to handle a queue tree even better.

reporting. netflow or similar data.

much of this can be done in software, but at huge performance costs. If any of it can be done in hardware that would be great for the *isp market.

hicountrynet · Wed Sep 27, 2023 3:25 pm

I'm wondering as well, does this QoS-HW mean fq-codel traffic shaping in hardware as opposed to a CPU only thing?

Cheers
Paul

Wed Sep 27, 2023 7:15 pm

I'm wondering as well, does this QoS-HW mean fq-codel traffic shaping in hardware as opposed to a CPU only thing?

Cheers
Paul

No

dtaht · Thu Oct 19, 2023 9:16 am

I just stumbled across this thread. 6 comments:

cake's diffserv4 model is the closest we could come to matching multiple mutually conflicting rfcs.

because of confusion over the cs1 codepoint as background, the LE codepoint was created: RFC8622.

There is a new codepoint pending with basically the same priority as cs5 called NQB with many nuances: https://datatracker.ietf.org/doc/html/d ... vwg-nqb-19

I would dearly like the various states of the ecn bits to be counted and tracked. There is a lot more of it out there than most realize and

Lastly I really hate strict priority queues as a major footgun against deployment of diffserv at all. Others like this.

I have been soliciting input on a new version of CAKE here: https://docs.google.com/document/d/1tTY ... bnfu73wlpp

dtaht · Thu Oct 19, 2023 9:29 am

QoS-HW is compatible with L3HW. You can use both features together.

Every supported device has 8 TX queues per port, and users will be able to assign QoS profiles to TX queues: either grant a QoS profile exclusive access to a queue or share a queue (or group of queues) between multiple profiles. The feature is still in development and not available yet. Meanwhile, all QoS profiles share all TX queues, i.e., there is no QoS enforcement yet.

In general I prefer bql + cake split evenly between all hardware tx queues vs QoS & diffserv applied this way because it leverages the scarcest resource, CPU. diffserv is a fiddly distraction compared to aqm and fq in the first place. A little diffserv on top of that can help, but first you gotta keep the queues short and the packets well mixed.

It doesn't help much for me to say that, 'cause people love these fiddly bits, but the qosify project mostly got it right on top of CAKE. All I can say is go ahead, offload bandwidth management in hw next, and measure latencies and throughput with flent, packet captures, and pay especial attention to the tcp rtt.

dtaht · Thu Oct 19, 2023 5:33 pm

Over here is an implementation of BQL for the mvpp2 chip, which may be similar to the chip you are using, with test results.

https://github.com/wojtas-marcin/Linux- ... 80057a70fb

Having BQL is a godsend, it lets you run fq_codel at line rate at about 1/20th the CPU of shaping, at the same level of smoothness as shaping. You can further add support for a hardware shaper down there, while retaining all the benefits of fq_codel or cake slightly up the stack. I think the mt79 and several intel chips support this nowadays.

Marcin (the author) I believe still does some marvell contract work.

For a simple guide to getting BQL going, and why it is important, please see: http://www.taht.net/~d/broadcom_aug9_2018.pdf

After you get something like that going, adding some qos can help.

dtaht · Sun Oct 22, 2023 8:18 pm

@raimondsp any feedback

Tue Nov 07, 2023 9:56 am

Hi there, and sorry for the late reply,

Unfortunately, the current switch chips do not support fq_codel or CAKE. However, RouterOS supports those AQM models on the software level. If the bandwidth is not too high, consider redirecting the traffic to the CPU for FQ-CoDel or CAKE usage.
Most Marvell switch chip models that we use in our CCR2116, CCR2216, CRS3xx, and CRS5xx series support WRED (Weighted Random Early Detection). While not being as robust against butterbloat as some other AQM, WRED is performed on the hardware level at wire speed. We have plans to implement WRED offloading into RouterOS in the near future.
Some switch chip models support hardware ECN marking and processing (e.g., CCR2216). The feature is on our roadmap.
Enhancing Switch ACL rules for QoS Profile selection (and therefore, QoS marking and hw queue selection) is also on the roadmap.
Currently, RouterOS does not support Egress ACL rules. Unfortunately, at the moment of writing, I'm unaware if this feature is planned.

dtaht · Sun Nov 12, 2023 12:11 am

My point was that BQL support helps first and most.

mobyfab · Wed Jan 17, 2024 12:34 pm

Hello,

I don't know if it related or not. Sorry if i made a mistake to post here.

I am currently using switch rules on a CRS305 in front of CCR2116 tp change COS on DHCP (ipv6 ipv4) requests. These DHCP requests with COS 6 are a requirement to authenticate to my ISP.
I decided to remove the CRS305 and use only the CCR2116. Of course i moved the switch rules from the CRS305 to the CCR.

But the rules do not work at all. Switch rules to change COS are never applied.

I am not sure, but i think i have understand that the switch are applied to ingress traffic, not egress. I my case, the DHCP client requests are of course egress from the CCR to my ISP.

Am i right or i make a mistake ?
Is there a workaround to change COS for DHCP egress requests ?

thanks.

It never worked.
I have a similar setup, CCR2116 + CRS317.
Switch rules on cpu port are broken on CCR2116, I have a ticket open for almost 2 years...
So i'm using rules on the CRS317 which is no convenient at all.

Fri May 31, 2024 10:02 pm

With v7.15 i think It's time to take up this topic again, please share your experiences with QoS-HW

i am very glad with CRS-317 showing 4MByte of buffer

lashguti · Fri May 31, 2024 10:13 pm

Hello,

Where Can I find manual exactly for the feature of hw-qos Per-queue traffic shaping, which was introduced in 7.15?
Could not find this topic in official manual about HW-qos at help.mikrotik.com

Fri May 31, 2024 10:20 pm

Hello,

Where Can I find manual exactly for the feature of hw-qos Per-queue traffic shaping, which was introduced in 7.15?
Could not find this topic in official manual about HW-qos at help.mikrotik.com

i think this is what you are searching for

https://help.mikrotik.com/docs/pages/vi ... nforcement

However i sugest Reading Thoroughly all the page, as far as i inderstand until now QoS-HW configures in a very different way from anything seen before on MikroTik, because is something new

lashguti · Fri May 31, 2024 10:38 pm

Hello,

Thanks for your answer, still can not find-out, how I can offload the simple-queues, configured on my ccr2116 (

Fri May 31, 2024 10:42 pm

Hello,

how I can offload the simple-queues, configured on my ccr2116

please read the whole page, you will find that is not the way QoS-HW works

lashguti · Fri May 31, 2024 10:52 pm

Hi,

Can you tell me shortly, If in Ros 7.15 on ccr2116, it is possible to have per ip down/up shaper in hardware?
for example
1000 subscribers on 10mb/s down/up
800 subscribers on 20mb/s
2000 subscribers on 30mb/s?

Mon Jun 03, 2024 12:03 pm

Hi,

Can you tell me shortly, If in Ros 7.15 on ccr2116, it is possible to have per ip down/up shaper in hardware?
for example
1000 subscribers on 10mb/s down/up
800 subscribers on 20mb/s
2000 subscribers on 30mb/s?

Hi, the short answer is - yes, it is possible. But I guess there will be a follow-up question "how?" - that I'll answer a bit later.

Mon Jun 03, 2024 1:29 pm

Hi,

Can you tell me shortly, If in Ros 7.15 on ccr2116, it is possible to have per ip down/up shaper in hardware?
for example
1000 subscribers on 10mb/s down/up
800 subscribers on 20mb/s
2000 subscribers on 30mb/s?

Hi, the short answer is - yes, it is possible. But I guess there will be a follow-up question "how?" - that I'll answer a bit later.

Starting from v7.15, RouterOS allows setting per-queue shaper. One option to assign traffic to a queue is by setting a QoS Profile via a Switch Rule.

For simplicity, let's imagine that all clients are connected to sfp-sfpplus1, forwarding all traffic to sfp-sfpplus2. Feel free to expand the example based on your needs.

Create QoS profiles. We can leave the default profile for 10M subscribers, but redirect 20M and 30M subscribers to queue2 and queue3, respectively.
Code: Select all
```
/interface/ethernet/switch/qos/profile
add comment="20M subs" name=tc2 traffic-class=2
add comment="30M subs" name=tc3 traffic-class=3
```

I guess we don't want 30M traffic to completely block 20M and 10M subscribers, so let's schedule all three traffic classes under the same priority group, adjusting the weight based on speed:

/interface/ethernet/switch/qos/tx-manager/queue
set [find where traffic-class=1] schedule=low-priority-group weight=1
set [find where traffic-class=2] schedule=low-priority-group weight=2
set [find where traffic-class=3] schedule=low-priority-group weight=3

Limit egress rate of the respective queues:

/interface/ethernet/switch/qos/port
set sfp-sfpplus1,sfp-sfpplus2 egress-rate-queue1=10M
set sfp-sfpplus1,sfp-sfpplus2 egress-rate-queue2=20M
set sfp-sfpplus1,sfp-sfpplus2 egress-rate-queue3=30M

Define ACL rules to match traffic by IP. For example, if all 20M subs are in 192.168.24.0/21subnet, and 30M - 192.168.32.0/21:

/interface/ethernet/switch/rule
add new-qos-profile=tc2 ports=sfp-sfpplus1 src-address=192.168.24.0/21 switch=switch1
add new-qos-profile=tc2 ports=sfp-sfpplus2 dst-address=192.168.24.0/21 switch=switch1
add new-qos-profile=tc3 ports=sfp-sfpplus1 src-address=192.168.32.0/21 switch=switch1
add new-qos-profile=tc3 ports=sfp-sfpplus2 dst-address=192.168.32.0/21 switch=switch1

Here is the print output for reference:

[admin@MikroTik] /interface/ethernet/switch/qos/profile> print 
Flags: H - HW-OFFLOADED
Columns: NAME, PCP, DSCP, TRAFFIC-CLASS, COLOR
#   NAME     PCP  DSCP  TRAFFIC-CLASS  COLOR
;;; default QoS profile
0 H default    0     0              1  green
;;; 20M subs
1 H tc2        0     0              2  green
;;; 30M subs
2 H tc3        0     0              3  green

[admin@MikroTik] /interface/ethernet/switch/qos/tx-manager/queue> print where hw-offloaded 
Flags: H - HW-OFFLOADED
Columns: TX-MANAGER, TRAFFIC-CLASS, SCHEDULE, WEIGHT, QUEUE-BUFFERS, USE-SHARED-BUFFERS, SHARED-POOL-INDEX
#   TX-MANAGER  TRAFFIC-CLASS  SCHEDULE            WEIGHT  QUEUE-BUFFERS  USE-SHARED-BUFFERS  SHARED-POOL-INDEX
1 H default                 1  low-priority-group       1  auto           yes                                 0
2 H default                 2  low-priority-group       2  auto           yes                                 0
3 H default                 3  low-priority-group       3  auto           yes                                 0

[admin@MikroTik] /interface/ethernet/switch/qos/port> print proplist=name,egress-rate-queue1,egress-rate-queue2,egress-rate-queue3 where name=sfp-sfpplus1 or name=sfp-sfpplus2
Columns: NAME, EGRESS-RATE-QUEUE1, EGRESS-RATE-QUEUE2, EGRESS-RATE-QUEUE3
# NAME          EGRESS-RATE-QUEUE1  EGRESS-RATE-QUEUE2  EGRESS-RATE-QUEUE3
0 sfp-sfpplus1  10.0Mbps            20.0Mbps            30.0Mbps          
1 sfp-sfpplus2  10.0Mbps            20.0Mbps            30.0Mbps   

[admin@MikroTik] /interface/ethernet/switch/rule> print 
Flags: X - disabled, I - invalid; D - dynamic 
 0    switch=switch1 ports=sfp-sfpplus1 src-address=192.168.2.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc2 keep-qos-fields=no 

 1    switch=switch1 ports=sfp-sfpplus2 dst-address=192.168.2.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc2 keep-qos-fields=no 

 2    switch=switch1 ports=sfp-sfpplus1 src-address=192.168.3.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc3 keep-qos-fields=no 

 3    switch=switch1 ports=sfp-sfpplus2 dst-address=192.168.3.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc3 keep-qos-fields=no

P.S. We'll add "/in/eth/sw/qos/port print rates" for more convenient output in RouterOS v7.16.

Larsa · Mon Jun 03, 2024 2:38 pm

@Raimondsp, great example! Someone should add this to the docs.

cyayon · Mon Jun 03, 2024 3:42 pm

Hi,

My ISP (Orange France) require that ICMP and DHCP (clients) requests marked COS 6 and DSCP 6.

Currently, I am using another switch (CRS310) with the GPON SFP and some switch rules on it, because switch rules only work with input requests (not output).

Today (with 7.15), is it possible to use switch rules for outgoing ?
In this case, I could use switch rules directly on my CCR2116 (with the GPON SFP inside) and avoid using another switch.

Thanks.

Mon Jun 03, 2024 4:51 pm

Hi,

My ISP (Orange France) require that ICMP and DHCP (clients) requests marked COS 6 and DSCP 6.

Currently, I am using another switch (CRS310) with the GPON SFP and some switch rules on it, because switch rules only work with input requests (not output).

Today (with 7.15), is it possible to use switch rules for outgoing ?
In this case, I could use switch rules directly on my CCR2116 (with the GPON SFP inside) and avoid using another switch.

Thanks.

Hi, unfortunately, egress switch rules are not implemented yet, and I'm unaware of estimates.

Regarding your case, who is generating ICMP and DHCP requests?

If the requests are generated by CCR2116 itself, then I believe you can create an /ip/firewall/mangle rule to change DSCP of the packet.
If CCR2116 is forwarding those requests, then you can match the respective packets on the ingress side (via ACL switch rules) and assign a QoS Profile, which, in turn, may modify PCP/DSCP values on the egress side.

Am I missing something?

cyayon · Mon Jun 03, 2024 5:03 pm

Hi,

thanks for your answer.

The CCR2116 is generating the request, in particular DHCP-client to authenticate to ISP.

I am aware that I could use mangle rules, at least on DHCP ipv6 requests. But not on DHCP ipv4.

Moreover, it could be cleaner to be able to use directly switch rules or even better an option in DHCP clients to mark COS/DSCP 6.

lashguti · Mon Jun 03, 2024 11:07 pm

Hi, the short answer is - yes, it is possible. But I guess there will be a follow-up question "how?" - that I'll answer a bit later.

Starting from v7.15, RouterOS allows setting per-queue shaper. One option to assign traffic to a queue is by setting a QoS Profile via a Switch Rule.

For simplicity, let's imagine that all clients are connected to sfp-sfpplus1, forwarding all traffic to sfp-sfpplus2. Feel free to expand the example based on your needs.

Create QoS profiles. We can leave the default profile for 10M subscribers, but redirect 20M and 30M subscribers to queue2 and queue3, respectively.
Code: Select all
```
/interface/ethernet/switch/qos/profile
add comment="20M subs" name=tc2 traffic-class=2
add comment="30M subs" name=tc3 traffic-class=3
```

I guess we don't want 30M traffic to completely block 20M and 10M subscribers, so let's schedule all three traffic classes under the same priority group, adjusting the weight based on speed:

/interface/ethernet/switch/qos/tx-manager/queue
set [find where traffic-class=1] schedule=low-priority-group weight=1
set [find where traffic-class=2] schedule=low-priority-group weight=2
set [find where traffic-class=3] schedule=low-priority-group weight=3

Limit egress rate of the respective queues:

/interface/ethernet/switch/qos/port
set sfp-sfpplus1,sfp-sfpplus2 egress-rate-queue1=10M
set sfp-sfpplus1,sfp-sfpplus2 egress-rate-queue2=20M
set sfp-sfpplus1,sfp-sfpplus2 egress-rate-queue3=30M

Define ACL rules to match traffic by IP. For example, if all 20M subs are in 192.168.24.0/21subnet, and 30M - 192.168.32.0/21:

/interface/ethernet/switch/rule
add new-qos-profile=tc2 ports=sfp-sfpplus1 src-address=192.168.24.0/21 switch=switch1
add new-qos-profile=tc2 ports=sfp-sfpplus2 dst-address=192.168.24.0/21 switch=switch1
add new-qos-profile=tc3 ports=sfp-sfpplus1 src-address=192.168.32.0/21 switch=switch1
add new-qos-profile=tc3 ports=sfp-sfpplus2 dst-address=192.168.32.0/21 switch=switch1

Here is the print output for reference:

[admin@MikroTik] /interface/ethernet/switch/qos/profile> print 
Flags: H - HW-OFFLOADED
Columns: NAME, PCP, DSCP, TRAFFIC-CLASS, COLOR
#   NAME     PCP  DSCP  TRAFFIC-CLASS  COLOR
;;; default QoS profile
0 H default    0     0              1  green
;;; 20M subs
1 H tc2        0     0              2  green
;;; 30M subs
2 H tc3        0     0              3  green

[admin@MikroTik] /interface/ethernet/switch/qos/tx-manager/queue> print where hw-offloaded 
Flags: H - HW-OFFLOADED
Columns: TX-MANAGER, TRAFFIC-CLASS, SCHEDULE, WEIGHT, QUEUE-BUFFERS, USE-SHARED-BUFFERS, SHARED-POOL-INDEX
#   TX-MANAGER  TRAFFIC-CLASS  SCHEDULE            WEIGHT  QUEUE-BUFFERS  USE-SHARED-BUFFERS  SHARED-POOL-INDEX
1 H default                 1  low-priority-group       1  auto           yes                                 0
2 H default                 2  low-priority-group       2  auto           yes                                 0
3 H default                 3  low-priority-group       3  auto           yes                                 0

[admin@MikroTik] /interface/ethernet/switch/qos/port> print proplist=name,egress-rate-queue1,egress-rate-queue2,egress-rate-queue3 where name=sfp-sfpplus1 or name=sfp-sfpplus2
Columns: NAME, EGRESS-RATE-QUEUE1, EGRESS-RATE-QUEUE2, EGRESS-RATE-QUEUE3
# NAME          EGRESS-RATE-QUEUE1  EGRESS-RATE-QUEUE2  EGRESS-RATE-QUEUE3
0 sfp-sfpplus1  10.0Mbps            20.0Mbps            30.0Mbps          
1 sfp-sfpplus2  10.0Mbps            20.0Mbps            30.0Mbps   

[admin@MikroTik] /interface/ethernet/switch/rule> print 
Flags: X - disabled, I - invalid; D - dynamic 
 0    switch=switch1 ports=sfp-sfpplus1 src-address=192.168.2.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc2 keep-qos-fields=no 

 1    switch=switch1 ports=sfp-sfpplus2 dst-address=192.168.2.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc2 keep-qos-fields=no 

 2    switch=switch1 ports=sfp-sfpplus1 src-address=192.168.3.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc3 keep-qos-fields=no 

 3    switch=switch1 ports=sfp-sfpplus2 dst-address=192.168.3.0/24 copy-to-cpu=no redirect-to-cpu=no mirror=no 
      new-qos-profile=tc3 keep-qos-fields=no

P.S. We'll add "/in/eth/sw/qos/port print rates" for more convenient output in RouterOS v7.16.

Hello,

THanks for your answer, I tested it and it works

1. I have 2 following questions: can I make more then 8 internet packages? ( as I noticed it is bound with traffic classes )
2. what if subscribers in the same ip subnet have different internet packages?

Tue Jun 04, 2024 9:22 am

Hello,

THanks for your answer, I tested it and it works

1. I have 2 following questions: can I make more then 8 internet packages? ( as I noticed it is bound with traffic classes )
2. what if subscribers in the same ip subnet have different internet packages?

Hi again, and I'm glad that the solution works for you.

Each port has 8 Tx queues and one shaper, so we can set up to 8 different egress rates per port due to hardware limitations. However, you can extend this by using multiple egress ports and setting different queue rates for each port. For example, you can set egress-rate-queue2=20M on sfp1 but 50M on sfp2.

Switch Rules are executed in strict order. If a client needs a different traffic class than the rest of the subnet, the specific rule should be placed before the subnet rule. The next example assigns the "tc3" profile to host 192.168.10.15 while the rest of the subnet gets "tc2":

/interface/ethernet/switch/rule
add new-qos-profile=tc3 ports=sfp-sfpplus1 src-address=192.168.10.15/32 switch=switch1
add new-qos-profile=tc2 ports=sfp-sfpplus1 src-address=192.168.10.0/24 switch=switch1

lashguti · Tue Jun 04, 2024 10:16 am

Hi,

Now I am doing all this stuff via 3 thousand of dynamic simple queues( bound to dhcp rate limit) targeting subscribers per ip /32 address.
But ccr2116 began to have problems and queues are out of synch from dhcp leases and cpu cores
are at 30-40% already. I have 2 such CCRs where I have distributed the load.
Searchung for alternative approach to handle this issue.

P.s. in switch rules I guess I will need to transfer all 3000+ ip addresses from simple queues. What can you advice?
Will hardware queueing answer my requirements or go with CHR, maybe queue tree with pcq for better performance and lower cpu? Maybe simple queues desynch will not be issue
on another platform?

Tue Jun 04, 2024 10:28 am

CCR2116 can offload only 512 ACL rules (see Switch Chip Features). Hence, you cannot offload 3000 rules. Consider grouping same-service-level clients into a subnet, so you can define one rule for the entire subnet.

hzdrus · Tue Jun 04, 2024 1:30 pm

Hi,

Can you tell me shortly, If in Ros 7.15 on ccr2116, it is possible to have per ip down/up shaper in hardware?
for example
1000 subscribers on 10mb/s down/up
800 subscribers on 20mb/s
2000 subscribers on 30mb/s?
Starting from v7.15, RouterOS allows setting per-queue shaper. One option to assign traffic to a queue is by setting a QoS Profile via a Switch Rule.

For simplicity, let's imagine that all clients are connected to sfp-sfpplus1, forwarding all traffic to sfp-sfpplus2. Feel free to expand the example based on your needs.
...
/interface/ethernet/switch/rule
add new-qos-profile=tc2 ports=sfp-sfpplus1 src-address=192.168.24.0/21 switch=switch1
add new-qos-profile=tc2 ports=sfp-sfpplus2 dst-address=192.168.24.0/21 switch=switch1

This shaper will limit on per IP basis as lashguti requested? Or its one common 20 mbps pool for all IPs in 192.168.24.0/21?

If its per-IP shaper, how can I configure one common shaper for entire subnet? Thanks.

ajgnet · Wed Jun 12, 2024 6:09 pm

I need to achieve two main goals with my QoS setup on a CCR2116 and would like to use L3HW if possible:

1. De-prioritize all traffic on TCP port 21 (FTP).
2. Prioritize all VoIP traffic by matching DSCP values.

Is this possible with QoS-HW? What would the configuration look like, as an example?

Fri Jun 14, 2024 10:19 am

This shaper will limit on per IP basis as lashguti requested? Or its one common 20 mbps pool for all IPs in 192.168.24.0/21?

If its per-IP shaper, how can I configure one common shaper for entire subnet? Thanks.

It is actually a per-port, per-queue shaper. Connecting each client to an individual physical port will give you per-client shaping while connecting an entire subnet to one router port will give per-subnet shaping. Using QoS HW enforcement for IP traffic shaping is a temporary workaround rather than a solution. Top-end switch chips provide metering and policing features for per-flow QoS, which we will eventually implement.

Fri Jun 14, 2024 10:36 am

I need to achieve two main goals with my QoS setup on a CCR2116 and would like to use L3HW if possible:

1. De-prioritize all traffic on TCP port 21 (FTP).
2. Prioritize all VoIP traffic by matching DSCP values.

Is this possible with QoS-HW? What would the configuration look like, as an example?

Hi, please follow our Dante example for traffic prioritization based on DSCP.

Regarding FTP traffic, make a switch rule to downgrade traffic class via a QoS profile. The following example assumes the FTP server is connected to ether1, and the clients are on ether2-ether4:

/interface/ethernet/switch/qos/profile add name=lowest-priority traffic-class=0
/interface/ethernet/switch/rule add switch=switch1 ports=ether1 protocol=tcp src-port=21 new-qos-profile=lowest-priority
/interface/ethernet/switch/rule add switch=switch1 ports=ether2,ether3,ether4 protocol=tcp dst-port=21 new-qos-profile=lowest-priority

ajgnet · Fri Jun 14, 2024 1:52 pm

Thank you very helpful. But what if the uplink port has a 10G link but ISP bandwidth is only 5G? How can I make sure the buffer is not saturated at the ISP and thereby nullifying the priority queues? For example, should I set the sfp1 interface to 95% of 5Gbps so that the switch QoS takes effect? How would that look with WRED and ECN. Thank you

Fri Jun 14, 2024 5:00 pm

Thank you very helpful. But what if the uplink port has a 10G link but ISP bandwidth is only 5G? How can I make sure the buffer is not saturated at the ISP and thereby nullifying the priority queues? For example, should I set the sfp1 interface to 95% of 5Gbps so that the switch QoS takes effect? How would that look with WRED and ECN. Thank you

You can limit the egress rate of the port (or even a particular queue):

/interface/ethernet/switch/port set sfp1 egress-rate=5G

ajgnet · Mon Jun 17, 2024 1:55 pm

You can limit the egress rate of the port (or even a particular queue):
Code: Select all
/interface/ethernet/switch/port set sfp1 egress-rate=5G

For some reason when I do this, max bandwidth gets capped around 800Mbps not 5G

Also, I followed the Dante instructions but it does not seem like the below configuration has any effect. Additionally, traffic on port 563 does not appear to be deprioritized. Am I missing something? Suggestions? Thanks in advance

# 2024-06-17 06:50:40 by RouterOS 7.16beta2
# software id = L70L-YK87
#
# model = CCR2116-12G-4S+
/interface bridge
add name=bridge1 protocol-mode=none
add name=containers
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] name=sfp-sfpplus1-Hotwire
set [ find default-name=sfp-sfpplus4 ] name=sfp-sfpplus4-LAN
/interface ethernet switch
set 0 l3-hw-offloading=yes qos-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
/interface ethernet switch qos port
set sfp-sfpplus1-Hotwire trust-l3=keep
set sfp-sfpplus2 trust-l3=keep
set sfp-sfpplus3 trust-l3=keep
set sfp-sfpplus4-LAN trust-l3=keep
set ether1 trust-l3=keep
set ether2 trust-l3=keep
set ether3 trust-l3=keep
set ether4 trust-l3=keep
set ether5 trust-l3=keep
set ether6 trust-l3=keep
set ether7 trust-l3=keep
set ether8 trust-l3=keep
set ether9 trust-l3=keep
set ether10 trust-l3=keep
set ether11 trust-l3=keep
set ether12 trust-l3=keep
set switch1-cpu trust-l3=keep
/interface ethernet switch qos profile
add dscp=56 name=dante-ptp pcp=7 traffic-class=7
add dscp=46 name=dante-audio pcp=5 traffic-class=5
add dscp=8 name=dante-low pcp=1 traffic-class=0
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 fast-leave=yes interface=ether13
add bridge=bridge1 fast-leave=yes interface=ether1
add bridge=bridge1 fast-leave=yes interface=ether2
add bridge=bridge1 fast-leave=yes interface=ether3
add bridge=bridge1 fast-leave=yes interface=ether4
add bridge=bridge1 fast-leave=yes interface=ether5
add bridge=bridge1 fast-leave=yes interface=ether6
add bridge=bridge1 fast-leave=yes interface=ether7
add bridge=bridge1 fast-leave=yes interface=ether8
add bridge=bridge1 fast-leave=yes interface=ether9
add bridge=bridge1 fast-leave=yes interface=ether10
add bridge=bridge1 fast-leave=yes interface=ether11
add bridge=bridge1 fast-leave=yes interface=ether12
add bridge=bridge1 fast-leave=yes interface=sfp-sfpplus2
add bridge=bridge1 fast-leave=yes interface=sfp-sfpplus3
add bridge=bridge1 fast-leave=yes interface=sfp-sfpplus4-LAN
add bridge=containers fast-leave=yes interface=veth1-kuma
/interface ethernet switch l3hw-settings
set ipv6-hw=yes
/interface ethernet switch qos map ip
add dscp=56 profile=dante-ptp
add dscp=46 profile=dante-audio
add dscp=8 profile=dante-low
/interface ethernet switch qos tx-manager queue
set 1 weight=1
set 2 schedule=strict-priority
set 3 schedule=strict-priority
set 4 schedule=strict-priority
set 5 schedule=strict-priority
/interface ethernet switch rule
add dst-port=563 new-qos-profile=dante-low ports=sfp-sfpplus1-Hotwire protocol=tcp switch=switch1
/interface list member
add interface=sfp-sfpplus1-Hotwire list=WAN
add interface=bridge1 list=LAN

Mon Jun 17, 2024 4:51 pm

You can limit the egress rate of the port (or even a particular queue):
Code: Select all
/interface/ethernet/switch/port set sfp1 egress-rate=5G
For some reason when I do this, max bandwidth gets capped around 800Mbps not 5G

Also, I followed the Dante instructions but it does not seem like the below configuration has any effect. Additionally, traffic on port 563 does not appear to be deprioritized. Am I missing something? Suggestions? Thanks in advance

The config looks file. Please check if bridge1 has the H flag in "/interface/bridge print". The switch chip supports only one hardware bridge, let's double-check if the system picked the right one.

Use the following command to print per-queue port stats and see if there any traffic on queue0 and queue5. The default traffic goes through queue1.

/interface/ethernet/switch/qos/port print stats where running

ajgnet · Mon Jun 17, 2024 5:15 pm

The config looks file. Please check if bridge1 has the H flag in "/interface/bridge print". The switch chip supports only one hardware bridge, let's double-check if the system picked the right one.

[adam@gw01] /interface/bridge> /interface/bridge print
Flags: X - disabled, R - running
 0 R name="bridge1" mtu=auto actual-mtu=1500 l2mtu=1584 arp=enabled arp-timeout=auto mac-address=D4:01:C3:0E:BC:7E protocol-mode=none
     fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m vlan-filtering=no dhcp-snooping=no port-cost-mode=long mvrp=no
     forward-reserved-addresses=no max-learned-entries=auto

 1 R name="containers" mtu=auto actual-mtu=1500 l2mtu=65535 arp=enabled arp-timeout=auto mac-address=FE:45:01:62:6A:6B protocol-mode=rstp
     fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=20s forward-delay=15s transmit-hold-count=6
     vlan-filtering=no dhcp-snooping=no port-cost-mode=long mvrp=no max-learned-entries=auto

Use the following command to print per-queue port stats and see if there any traffic on queue0 and queue5. The default traffic goes through queue1.
Code: Select all
/interface/ethernet/switch/qos/port print stats where running

[adam@gw01] /interface/bridge> /interface/ethernet/switch/qos/port print stats where running
                  name:  sfp-sfpplus1-Hotwire sfp-sfpplus4-LAN
             tx-packet:           166 504 678      725 159 119
               tx-byte:        81 999 221 054  970 145 466 206
           drop-packet:                                      6
             drop-byte:                                  8 724
      tx-queue0-packet:                22 922       49 774 095
      tx-queue1-packet:           166 460 861      675 338 235
      tx-queue5-packet:                20 884           43 230
      tx-queue7-packet:                    11            3 559
        tx-queue0-byte:             3 921 218   62 196 321 974
        tx-queue1-byte:        81 991 322 136  907 936 713 946
        tx-queue5-byte:             3 976 820       11 259 634
        tx-queue7-byte:                   880        1 170 652
    drop-queue0-packet:                                      6
      drop-queue0-byte:                                  8 724

I don't see the H flag set in bridge1 - how do I change that? The only other bridge that I have is for my docker containers.

Mon Jun 17, 2024 5:21 pm

I'm very sorry for misinformation. Bridges do not have H flags. Bridge ports are the ones to check:

/in/bridge/port print

According to your QoS port stats, 4 queues are active and scheduling traffic - as expected.

loloski · Mon Jun 17, 2024 5:30 pm

[adam@gw01] /interface/bridge> /interface/bridge print
Flags: X - disabled, R - running
 0 R name="bridge1" mtu=auto actual-mtu=1500 l2mtu=1584 arp=enabled arp-timeout=auto mac-address=D4:01:C3:0E:BC:7E protocol-mode=none
     fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m vlan-filtering=no dhcp-snooping=no port-cost-mode=long mvrp=no
     forward-reserved-addresses=no max-learned-entries=auto

Does qos-hw needs bridge vlan filtering set to "yes"?

donkeyKong · Tue Jun 18, 2024 12:33 am

Hello,

I’m using a CRS305 to mark PCP=6/DSCP=CS6 as required by my ISP (Orange France). For reference, an ONT is connected to a CRS305 which is then connected to a CCR2004. The CRS305 is only present to change PCP/DSCP values.

At the moment DHCPv4/6and ARP packets are marked on the CRS305, and the other traffic requiring this special marking (icmpv6 types 133,135,136) are done on a router using mangle rules.

I’d like to confirm that the following setup will not remove/replace the PCP/DSCP values as set by the router on icmpv6 traffic. Will setting trust-l2 and trust-l3 to keep make the switch respect already set QoS values and only modify those for which a ACL rule is defined?


interface ethernet switch
set 0 qos-hw-offloading=yes

/interface ethernet switch qos port
set Router trust-l2=keep trust-l3=keep
set ONT trust-l2=keep trust-l3=keep

/interface ethernet switch qos profile
add comment="Orange BNC reqs - PCP=6, DSCP=48 (CS6)" dscp=48 name=orange-prio-bng pcp=6 \
    traffic-class=6

/interface ethernet switch rule
add comment="Modify QoS profile for Orange ISP - ARP traffic" mac-protocol=arp new-qos-profile=\
    orange-prio-bng ports=Router switch=switch1 vlan-id=832
add comment="Modify QoS profile for Orange ISP - DHCPv4" dst-port=67 mac-protocol=ip \
    new-qos-profile=orange-prio-bng ports=Router protocol=udp switch=switch1 vlan-id=832
add comment="Modify QoS profile for Orange ISP - DHCPv6" dst-port=547 mac-protocol=ipv6 \
    new-qos-profile=orange-prio-bng ports=Router protocol=udp switch=switch1 vlan-id=832

ncr · Sat Aug 03, 2024 11:18 pm

Hi @raimondsp,

Can you please help clarify how these QoS features relate to RoCEv1/v2? Does Mikrotik plan to cover the needed features?
It seems the new QoS support is aligned, but not sure of the full coverage or if some areas are lacking still.

I have use cases where Mikrotik 25Gb/100Gb solutions could make a lot of sense for non-mission critical datacenter and lab use cases (Fortune 100 company, thousands of prototype systems in a datacenter environment), and I'm unable to recommend the products due to the gaps with QoS/RoCE and also a bit of concerns around software quality (feature breakage/stability issues/bugs).

*edit* I'm really excited about this work and surprised more people aren't commenting in the patch notes or here. This is a major gap.

Larsa · Sun Aug 04, 2024 12:27 am

Maybe you should study how RoCE and its different versions work. V1 operates at L2 with various helpers according to DCB (e.g., PFC, ETS, etc.) which sometime is called lossless Ethernet. But you can also manage without it by using other means like standard switches and handling the flow controll in the device driver endpoints. V2 operates at L3 (UDP).

ncr · Sun Aug 04, 2024 7:07 am

Maybe you should study how RoCE and its different versions work. V1 operates at L2 with various helpers according to DCB (e.g., PFC, ETS, etc.) which sometime is called lossless Ethernet. But you can also manage without it by using other means like standard switches and handling the flow controll in the device driver endpoints. V2 operates at L3 (UDP).

Hi @Larsa, Thank you for your reply. My intention is to ask the product owner at Mikrotik about the coverage level of these new QoS features regarding RoCE, specifically the RoCEv1 Layer 2 / QoS features. I'm also interested in whether they think these features are mature enough or need more time. While I understand that PFC and ETS are crucial, I'm aware there might be additional L2 features involved, hence the 'etc.' in your response. Leveraging the switch chips' hardware RoCE features would be fantastic.

*edit* I felt that the first sentence of your response came across as a bit condescending, but I appreciate the context you provided. My goal is to understand the bounds of the new QoS features better. It's worth noting that others, like those at ServeTheHome, have pointed out the lack of RoCE support in the 100Gb product line, so this concern is shared by others in the community.

*edit2* Upon reviewing the 7.15 change log and the 7.16 beta, it seems that support for DCB (specifically ETS and QCN) and DCBX is missing.

Larsa · Sun Aug 04, 2024 8:12 am

As I mentioned, these are just L2 helpers for the old v1 which you can manage without using other methods but as most installations are running v3 (UDP) it really doesn’t matter.

Anyhow, since this is just a user forum you'll probably get better answers by emailing Mikrotik at 'support@mikrotik.com' or by opening a ticket at their service desk: https://help.mikrotik.com/servicedesk/servicedesk.

hknet · Sun Aug 11, 2024 6:56 am

Do we need to set explicit the egress/ingress rates per switch port or is it automatically derived from the negotiated interface attributes?

eg. do we need:

/interface ethernet switch port set ether1 ingress-rate=1G egress-rate=1G

manually or is this implicit if the port negotiates 1G link-speed?

and woule the the same be true for example for sfp-plus ports with only 1G links?

Who is online