What happens when there’s more flows than pcq-total-limit/pcq-limit? Do dynamic queues shrink automatically to accomodate more?
PCQ algorithm is very simple - at first it uses selected classifiers to distinguish one sub-stream from another, then applies individual FIFO queue size and limitation on every sub-stream, then groups all sub-streams together and applies global FIFO queue size and limitation.
…
pcq-limit (number) : queue size of one sub-stream in packets
pcq-total-limit (number) : queue size of global FIFO queue
http://wiki.mikrotik.com/wiki/PCQ
PFIFO and BFIFO
These queuing disciplines are based on the FIFO algorithm (First-In First-Out). The difference between PFIFO and BFIFO is that one is measured in packets and the other one in bytes.
\
- pfifo-limit (number) : Maximum number of packets that the PFIFO queue can hold
- bfifo-limit (number) : Maximum number of bytes that the BFIFO queue can hold
Every packet that cannot be enqueued (if the queue is full), is dropped. > Large queue sizes can increase latency, but utilize channel better.
Thanks fewi for a detailed relpy but that is not what I’m asking. Let me explain better:
PCQ forms number of pfifo queues dynamically for each flow that it identifies. If each pfifo is pcq-limit long - PCQ can only allocate pcq-max-limit/pcq-limit pfifo queues of pcq-limit size. My question is what happens if there’s not enough pcq-max-limit packet queues to setup another pfifo queue? Do pfifo queues shrink in size or what?
Question is based on information from:
http://mum.mikrotik.com/presentations/CZ09/QoS_Megis.pdf page 29:
50/2000 PCQ can only take 40 flows…
If all PCQ sub-queues together are holding more packets than pcq-max-limit and a new packet arrives, it gets dropped. At least that’s my understanding of it. All the substreams form one big FIFO queue with a packet limit of pcq-max-limit, and FIFO behaviour is to drop packets if the queue is full.
So what you are saying is that with 50/2000 settings pcq will service up to 2000 flows as long as each flow is not taking more than 1 queue? If so what is the 50 parameter for?
My understanding was that each flow is allocated 50 packet fifo…or is 50 the maximum size beyond which packets will be tail dropped for the particular flow?
Both parameter aren’t for flows, they are for packets.
If all substreams combined are holding pcq-total-limit packets (2000 in your example) in their FIFO queues and a new packet arrives, it is dropped. If there’s still room the packet can be accepted by the global PCQ queue and it looks how the packet fits into a substream.
Each substream (however it is created, it might be a flow if you’re using all of src-address, src-port, dst-address and dst-port as PCQ classifiers) is a FIFO queue of size pcq-limit (50 in your example). If a packet arrives for that substream and the FIFO queue of that substream already holds pcq-limit packets, the packet is dropped. If at the same time another substream’s FIFO queue still has room and a packet arrives for it the packet is accepted but might be shaped according to the substreams data rate.
Edited for clarity.
FIFO queue of size pcq-limit (50 in your example). If a packet arrives for that substream and the FIFO queue of that substream already holds pcq-limit packets, the packet is dropped. If at the same time another substream’s FIFO queue still has room and a packet arrives for it the packet is accepted but might be shaped according to the substreams data rate.
So we agree that “sub-queues” are of size pcq-limit.
If parameters are 50/2000 that gives 40 sub-queues - correct?
My qyestion is what happens if new traffic arrives that is not part of any current sub-queue (it’s a new substream) and there are 40 already allocated? Followup question: are sub-queues hardcoded in size ie so when new substream is identified 50 slots are allocated for it from max-limit queue?
Edit: added “current” to make it clearer.
Sort of, but not quite. You can max out 40 substreams with those parameters, and if 40 substreams are indeed maxed out and traffic arrives that would create another substream that traffic will be dropped due to the pcq-total-limit. If you have 40 substreams but none of them (or only some of them) are maxed out and packets arrive that would require a new substream to be created, a 41st substream would be created because there’s still room in the global FIFO queue. Which also answers the follow up question.
In the document you’re referring to, the author is calculating how large his pcq-total-limit needs to be based on knowledge of how many substreams there are. He knows he’s creating substreams based on client IP address, so there’s one substream per client. He knows he has 40 clients. Since the per client pcq-limit is 50, he needs a pcq-total-limit of 2000 since that’s the absolute maximum he would ever see, and he wants to make sure client packets are only dropped based on their substream and never because of a global limit.
In other deployments you may not know how many substreams there are going to be.
OK, I suspected that this could be the case. I see a trouble with such behavior because:
Say I have 50/2000 PCQ with 80 substreams - this means that some substreams will never be 50 packets long. When more packets come to existing substreams their packets will be dropped even for substreams with queue 1 packet long. In such scenario high rate streams overpower (cause premature packet loss, jitter due to irregular queue length) low rate substreams.
EDIT: to recap, past limit/max-limit substreams PCQ seems to favor high rate streams over low rate - exactly opposite of what should (or, I hopped for) happen.
PS.: Thank you fewi for guiding me through this 
Reading this topic with interest, even though it might be relative old. But still valuable. And I hope to learn more about this complicated material.
I would like to continue on this specific topic in relation to my situation.
I have rb1000 with 436 MiB free Memory according Resources.
I have approx 250 client IP’s passing router.
Bulk of client have 3Mb download and 300Kb up and recently starting to separate VOIP/Skype traffic by means of giving these data streams their own IP address per client.
I use simple queues with Queue Type = pcq_up and pcq-dwn and classifier only src or dst address.
Both for these: Rata=0, Limit=40 and Total Limit=2000 (=default setting of ROS)
I understand that increasing the Limit=xx the more guaranteed delivery of packages while the lesser creates better (=smaller) latency.
Given that it seems to me there is plenty of memory (Memory = RAM in Resources?) I could easy increase Total Limit to =10000?
(Take calc example from referred doc 10000/40=250 queues available? Also 10000*2200bytes=22MB memory needed?. Or is the Total limit back calculated 430.000.000/2200=+/-19.500??
Does it also mean a rb1000 can have approx 19.500/40=+/- 490 queues silmoultanously? Taken that you probably never see more then 50% at the same time online and also that of the online users average usage will probably only 25% of their theoratical combined maximum usage we could say a rb1000 can serve sort of 4000 users, pipe limitation not taken in respect?)
Also, could it now be advisable to set a pcq type for my Skype/Voip queues with Limit=30 to improve latency? And set rest of traffic to 50 or 60 to get better throughput?
Or am I completely missing the concept of the working of queues and their types here?
I have been reading all related manuals and presentations but to be honest I am still not confident in how it actually should be done…