Hey

* your tree should be a tree, not a list!

<some main queue linked to interface max=4M>
<q1 priority=1>
<q2 priority=2>
....

Currently each queue can transmit at 4M...-> no QoS

* you should also prioritise download, at least giving prio to VOIP
* don't use "bucket-size=0"
* "out-interface=all-ppp" (mangle) != "parent=pppoe-out1-UP" (queue)

* "tree"
You need one parent queue linked to an interface, with a number of child queues connected to that parent queue, ex:

Code: Select all

* "no qos"
Since all queues are linked to interface with limit of 4M, they are all allowed to consume the total upload bandwidth, independent of usage of other queues.

* mangle
The mangle is applied to packets leaving on interface "all-ppp", but the queue is attached to interface "pppoe-out1-UP".

* priority in download
If you don't, the quality of download is not maintained, and you're listening experience will very depending on the load.

* "no bottleneck"
If there is a heavy downloader, the 50M pipe will be filled, easily.

* "bucket"
bucket means "how much can I use BEFORE asking parent for more. But in your case there is no parent: parent is interface for all separate queues.

add bucket-size=0 limit-at=4500k max-limit=4500k name=pppoe-out1-UP parent=\
    pppoe-out1

1. that is better, it should point to pppoe

3. didn't encounter a "ppp-all" interface before, and it's not treated as a list "out-interface=all-ppp". Just to make sure I would use pppoe-out1 for now.
Once confirmed working, you can try with ppp-all, and verify.

/ip firewall mangle

Didn't verify it all, but for VOIP it should be fine. packets are marked on output and matching packet-mark is defined on queue for that output interface.

As mentioned before, on the inbound (download) side, I would recommend to also do mangling & class-based queueing, but if you're happy for the moment...Leave as is.

So what you say is that heavy download at your side doesn't affect how you hear the remote party (which would sound more logical) but how the remote party hears you, i.e. the suffering VoIP packets go in the opposite direction than the heavy load, correct?

If so, I'm not familiar with the ISP habits in Greece, but could it be that there is a Wi-Fi link between your place and the rest of the internet? The point is that Wi-Fi links are half-duplex (Rx and Tx cannot run simutaneously) so on low speed links this can generate jitter (small uplink VoIP packets have to wait until a large download TCP packet finishes transport). And some phones tolerate less jitter than others.

the other side hear me with gaps

Now the other side is always the same single VoIP phone or the situation is always the same when you call any remote number via some VoIP provider?

Can you sniff the traffic on the outgoing interface during such situation into a file or stream it to a machine running tcpdump or dumpcap (from the Wireshark suite) via an interface of your Mikrotik dedicated to the streaming, to see what you really send out?

Assuming you've identified the RTP stream by direction properly, it means that your Mikrotik is not just delaying the VoIP packets but dropping them, and even does that in chunks of up to three packets. So the first thing to do is to run /tool profile in that situation, to see whether the CPU load doesn't stay above 25 % (as there are spikes sometimes so 1/4 of the total CPU throughput is the maximum base load you can deem safe, maybe even less).

If the CPU load is higher, a complete export of your firewall (not just the mangle table) may or may not reveal some space for optimisation. If there is none, the hardware is insufficient for the purpose.

if i disable all mangles, filters and queues... the usage is the same.

My cpu in normal usage shows 10-15%. When downloading a file, 50-70% usage. What kind of hardware must I buy to prioritize traffic? I bought a CRS109 only for QoS. Ι am very dissatisfied.

You dont have to spend all that money for a 4011, just keep the CRS109 as a "switch only" and add a Mikrotik hEX for the "Routing".

I haven't got the 4011 so I can only extrapolate.

A few months ago I did a throughput measurement of hAP ac² with PPPoE on WAN; as QoS is essentially incompatible with fasttracking (you could fasttrack only the VoIP packets but there are so few of them that doing so wouldn't save you much CPU power), the throughput measurement with bi-directional traffic and no fasttracking on client side is most relevant for your case (without fasttracking, every single packet has to pass through all stages of the firewall). The summary value of 1180 Mbit/s in both directions in total is the maximum you would achieve with a hAP ac² as I had no packet marking rules in place during the test. 4011 uses the same CPU architecture and number of cores like the hAP ac² but almost double the clock rate, so you should get around 2300 Mbit/s total throughput on full-MTU-sized packets. Further throughput decrease in Mbits/s comes from handling of short packets, as smaller packets mean less payload bytes transported while spending the same CPU effort for routing, firewalling etc.

In real life, there is very roughly one small packet per two large ones for one-way file transfers (a small TCP ACK is usually sent for every second data packet received and the data packets are as large as the path permits). VoIP calls use only small to mid-size packets, except the call control (signalling) which is a few packets' exchange in the beginning of a call, but the share of the VoIP on the total traffic will likely be negligible unless you provide the VoIP service for the whole neighbourhood.

So all in all, 200/20 Mbit/s will surely be fine with 4011 even with QoS and PPPoE handling, 1000/1000 may or may not be depending on the traffic pattern. Another performance drop would happen if you'd deploy bridging to provide advanced L2 functionalities on the LAN ports (VLAN filtering, MSTP, ...)

The official test results are much more optimistic on large packets but they don't take into account the PPPoE processing.

The 4011 is also more powerful than hEX, although here the comparison would be more complex as the CPU architecture is different.

It's a good idea to pass the voip, ack and syn traffic from fast track?
In this way... have better priority from other traffic of QoS?

It's a good idea to pass the voip, ack and syn traffic from fast track?
In this way... have better priority from other traffic of QoS?

Fasttracking means that most packets of a fasttracked connection are only processed by the connection tracking stage of the firewall and are not processed by the various rule chains and queues. ACK-only packets belong to a connection, and fasttracking by its very principle handles either all packets of a connection or none, so within the same connection, you cannot selectively fasttrack only ACK-only packets and process the large packets the regular way even if it would make any sense (and it actually doesn't, see below).

Also, as even fasttracked packets still need the CPU to get moving, fasttracking a few packets out of the total (the VoIP flow volume is negligible as compared to your torrent traffic) will not significantly lower the CPU load, so even if you fasttrack VoIP packets and keep queueing the rest, there will be losses in your scenario.

The goal is to make every packet pass through as little firewall rules as necessary, and let these few rules' match conditions be as simple as possible. To fulfil this goal, it is best to translate complex match conditions into connection-marks when handling the first packet of each connection, so that you don't need to re-evaluate all those conditions for every single packet, and to translate the connection-mark into the packet-mark as the first thing to be done for packets which already have got a connection-mark because the connection tracker has found them to match a marked connection. So what you get is something like:

Code: Select all

/ip firewall mangle

This is the rough skeleton. Real life may be more complex, and I don't say this will surely save your CRS109 from trash because the optimisation won't do a dramatic change , but you may give it a try. I'd recommend to start from just two priorities - VoIP and the rest. If doing so saves the VoIP calls, it makes sense to spend effort on fine-graining the other priorities.

# maybe not need the next

...

1. In connections mark, why i need to have passthrough=yes?
In this way when some connection mark as C1 and next check all the others rules.

2. Why I need to have jmp for mark connections at the end of connections mark.
When finish with connections mark, do not continue to packet mark?

3. In packet mark you say to accept the download packets.
Why?

But i can't see any traffic in accept rule for connection-mark=!no-mark.
Why?

Any comments for the new configuration?

But i can't see any traffic in accept rule for connection-mark=!no-mark.
Why?

Because your chain=pktmark works well and does not leave any packets unhandled :-) That action=accept rule was there just to catch packets for which the translation of connection-mark to packet-mark did not happen for any reason in the chain=pktmark and not let them get further. However,

Any comments for the new configuration?

yes, there is an important one related to the above - as you have added out-interface=pppoe to the first two rules as compared to my suggestion, all packets in the download direction do pass through all the connection-marking rules followed by passing through all the chain=pktmark rules, because neither of the first two rules matches on them (as they have a connection-mark assigned but don't have out-interface=pppoe). So if you don't want to enqueue download packets, remove the out-interface=pppoe from the accept rule with connection-mark=!no-mark, so it will accept all download packets belonging to already marked connections and thus will not let them pass all the connection-marking rules (as doing so generates an unnecessary CPU load). So all download packets (except the initial ones of connections initiated from WAN side which probably don't exist) will be handled by just two rules - the first one which won't match on them, and the second one which will accept them. In my initial suggestion, the first rule in the chain=connmark2pktmark was responsible for the same, but it caused the upload packets to pass through that one extra rule, so the way described just above is more efficient.

Other than that, it seems fine to me. So if after implementing the change above you still experience RTP packet loss, there is no more optimisation I could suggest.

add action=jump chain=postrouting comment=Packets connection-mark=!no-mark \
   

But if i remove from first and second rule the out interface, packet mark count additional the dw traffic.

Before... i catch only upload in connections and packet mark.
Even if all dw traffic pass from all connections mark, i have last rule to jump to packet mark with out-interface.
So the dw traffic don't pass to packet mark.

But if i remove from first and second rule the out interface, packet mark count additional the dw traffic.

Don't remove out-interface=pppoe from the first rule, remove it only from the second one.
First rule will send only connection-marked UL packets to the mark translating chain (and they will never return from there as they get packet-marked with passthrough=no there), and it will ignore all DL packets so they won't be sent to the mark translating chain.
Only all the DL packets and the UL packets with no connection-mark will reach the second rule, and that rule will match on all of the DL packets belonging to already connection-marked connections and will thus accept them, so only packets in either direction which have no connection-mark will get past this second rule (and eventually get a connection-mark assigned followed by getting assigned a packet-mark, but there are so few of them that it doesn't matter).

Before... i catch only upload in connections and packet mark.
Even if all dw traffic pass from all connections mark, i have last rule to jump to packet mark with out-interface.
So the dw traffic don't pass to packet mark.

Sorry, can you re-word this? I'm lost a bit in what is "before" and which "last rule" you have in mind. But as it is the last one, it means to me that there are some rules before it, through which the DL packets had to pass. And even though none of those rules before matched those packets, the evaluation may take the same amount of CPU time when it doesn't match as when it matches - although I suppose that the mismatch of the whole rule is declared as soon as the first mismatch of one of the conditions is encountered, the first condition failing to match may be the last one evaluated.

/ip firewall mangle
o

add action=jump chain=postrouting comment=Packets connection-mark=no-mark \
    jump-target=conmark out-interface=pppoe-out1

If i change the accept rule with:

Code: Select all

add action=jump chain=postrouting comment=Packets connection-mark=no-mark \
    jump-target=conmark out-interface=pppoe-out1

Is this ok?

If i change the accept rule with:

Code: Select all

add action=jump chain=postrouting comment=Packets connection-mark=no-mark \
    jump-target=conmark out-interface=pppoe-out1

Is this ok?

Well, there is little difference whether you use a separate chain=conmark and jump to it from chain=postrouting for packets which need connection-marking, or whether you do the connection-marking within chain=postrouting itself and bypass that part of it for packets which do not require connection-marking. So sorry if I mislead you, I didn't go deep into your configuration in post #29 because I've understood from what you wrote there that it was an excerpt from the one you've published earlier and in those ones there were no dedicated chains for connection marking and packet marking.

/ip firewall mangle
add action=jump chain=postrouting comment=MrkConJmp2PckMark connection-mark=\
   

/queue simple

To me it seems fine, just double-check whether you have the connection-mark=no-mark condition in all action=mark-connection rules except the first one, to avoid accidentally rewriting an already assigned connection-mark. You obviously realized such need on oyur own, but as you haven't shown all these rules for easier readability (good!), you have to double-check yourself.

What are the practical results?

add action=jump chain=postrouting comment=MrkConJmp2PckMark connection-mark=\
s

Dο you know some book for MT QoS analysis?

Dο you know some book for MT QoS analysis?

Haven't needed one yet so no. What do you mean by QoS analysis?

Some book to explain how work QoS (packet flow etc).