Hello,

I need to configure my queues in such a manner as to achieve an equal distribution of available bandwidth between a bunch of vlans.

I have 15 vlans that each represent a residential unit, plus one more vlan that is a mikrotik hotspot. Current I am using a simple queue with pcq to evenly distribute bandwidth amongst every device across all networks, but I want the queues to be per vlan rather than per device.

I have a basic understanding of queues and am trying to gain a proper understanding. Can anyone point me in the right direction to achieve the above?

Thank you

The queueing works at L3, so if you use routing on the device, you have to set a mind map between VLAN IDs and IP subnets and mark the packets for queueing up to their source/destination IP addresses. You can use bridge rules to translate the VLAN IDs into packet marks (the bridge rules even allow you to mark packets on the CoS field of the 802.1Q tag if necessary), but for the WAN -> LAN direction, the VLAN ID is a function of the destination address anyway.

Thanks Sindy, unfortunately that went over my head.

I understand how to packet mark. All vlans are configured like 10.200.112.0/27 = vlan 112 etc. Can you break it down more or give me an example?

Thank you

Apologies, I haven't realized what PCQ means in detail. As the PCQ alghoritm has its own embedded classifiers, external marking is not only not necessary but also not possible. So if you would insist on using PCQ in particular, you would have to all traffic from each VLAN to a distinct single address, and then set the PCQ to classify on alone (as everything else has to remain different for each connection). Doing of inbound traffic is both resource-intensive and not exactly intuitive to configure, but you may still deem it easier than to configure the classical tree of queues in HTB with child queues chosen up to packet marks assigned by mangle rules which would translate the customer subnets to packet marks. So the decision is up to you. I can help you with the for the PCQ way, not with the HTB one.

Hi Sindy,

Sorry I never responded. I am revisiting this now as I must find a solution.

Would you consider the method you speak of regarding src-nat to be a work-around or best practice? I must do it the 'proper' way as I will be rolling it out across many sites soon and can't let my lack of knowledge in this area dictate what method my client ends up with.

If building a complex queue tree is best practice then that is what I must learn. And vice vera for the src-nat method. I am keen to understand both. If you can help with the src-nat method that would be appreciated - I do wonder the implications of src-natting all the different vlans to one subnet/vlan and its effects on firewall rules etc..?

If anyone else has a method for evenly sharing available bandwidth amongst VLANs i'd like to hear your thoughts please.

Thank you

PCQ is a pre-cooked solution for simple cases where you want to grant a fair share of bandwidth to equal "targets" (means bandwidth limitation targets) - as the manual states, "PCQ was introduced to optimize massive QoS systems, where most of the queues are exactly the same for different sub-streams.".

So the first thing you need to think well about is the overall concept - are you OK with fair distribution of the bandwidth among the VLANs=subnets as a whole and let the hosts inside each VLAN= subnet fight for bandwidth without control, or whether you want both a fair distribution between the hosts within each VLAN=subnet and fair distribution between VLANs=subnets, or even something more complex (such as preference of VoIP traffic over the fair distribution rules, which I feel to rule out PCQ at once but I may be wrong).

Depending on that, choose the method to learn and implement. Generically, it is useful to learn how to configure hierarchical queues; for your particular case, it may be simpler to use the "NAT each subnet" approach for both (possibly) easier learning and (possibly) simpler configuration if your goal matches one (or two stacked) PCQ-like handlings.

So to answer your question, the NAT method is definitely a workaround, but the hidden question is what exactly it works around If it works around an unbearable complexity of the alternative configuration, it definitely is the best practice

And bear in mind that your CPU may start glowing red depending on the complexity of the task no matter which particular way you set it up.

Thank you Sindy.

I think it would be OK to just balance the bandwidth between VLANs and then let the hosts on each VLAN fight for available bandwidth, as the goal is to replicate a typical one circuit per apartment setup so , just like it would be if each had their own connection/router.

Having said that, I wonder if I do actually need to go through with this. I assumed it needed to be done to stop one apartment from taking more than its fair share of the bandwidth by having multiple hosts all torrenting/downloading but, having watched traffic at these sites for quite a few hours it became evident this just doesn't happen. Even in the student accom where I expected it to be really bad, they just don't seem to queue up loads of torrents and download 24/7/365 - has behaviour changed?

Anyway, I would like to understand the src-nat method you speak of so could you possibly write in words the way to achieve this? The fields I currently have in the simple queue is target (0.0.0.0/0) and destination (WAN Port), plus the actual pct settings within queue types. There is also a 300/100 limit on the overall queue. Where to from here to succeed with the src-nat method you speak of?

Thank you

All the child queues of a single parent queue with PCQ behaviour are created automatically and get the same treatment. The behaviour depends on the classifier which says what connections to aggregate into one child queue. If you set that classifier to client-side IP address, all connections of each client device will be handled by a single child queue; if you set it to client address and port, each individual connection will have its own child queue.

If each client gets from you several individual IP addresses, one per each end device, with a single common parent queue for all clients and a single auto-created child queue per client IP address (PCQ classifier = client IP address), then if client A has a single device connected and client B has 9, client A will get 10 % of the available bandwidth and client B will get 90 %.

the goal is to replicate a typical one circuit per apartment setup, just like it would be if each had their own connection/router.
...
Anyway, I would like to understand the src-nat method you speak of so could you possibly write in words the way to achieve this?

Below is a picture of the conventional "one circuit per apartment" case where you assign a single IP address to each client and it is their choice whether they use it for a single end device or place there a NATing router and connect as many devices as they wish to it; you can see traffic of all client devices as a traffic of that single IP address, so each such client connection gets the same share of the total bandwidth if they are treated using PCQ with local side IP address used as classifier.

multi-router setup code

                  internet
                     |           provider rtr
      ---------------------------------
     |            p.p.p.23             |
     |              NAT                |
     |             DHCPs               |
     |            a.a.a.1              |
      ---------------------------------
          |                       |
          |  client rtr C         |  client rtr D
    ------------            ------------
   |  a.a.a.c   |          |  a.a.a.d   |
   |    NAT     |          |    NAT     |
   |   DHCPs    |          |   DHCPs    |
   |  c.c.c.1   |          |  d.d.d.1   |
    ------------            ------------
       |  |  |                 |  |  |
 c.c.c.2  |  c.c.c.4     d.d.d.2  |  d.d.d.4
          |                       |
       c.c.c.3                 d.d.d.3

I assume that in your actual setup, your router performs the functionality of the clients' routers in terms that you have created one IP subnet per client, each in its own VLAN, and you deliver the VLANs to the clients by a network of switches, where each client has got several access ports to a single VLAN as follows:

single-router multi-subnet setup code

                  internet
                     |           provider rtr
      ---------------------------------
     |            p.p.p.23             |
     |              NAT                |
     |   VLAN C              VLAN D    |
     |  DHCPs C              DHCPs D   |
     |  c.c.c.1              d.d.d.1   |
     |             trunk               |
      ---------------------------------
                     ||
                     ||    client switch(es)
    ------------------------------------
   |               trunk                |
   |                                    |
   | ..........              .......... |
   | : VLAN C :              : VLAN D : |
    ------------------------------------
       |  |  |                 |  |  |
 c.c.c.2  |  c.c.c.4     d.d.d.2  |  d.d.d.4
          |                       |
       c.c.c.3                 d.d.d.3

So at L3, your router handles a handful of IP addresses, one per each of the clients' end devices.


The convenience of the PCQ is that the child queues are generated automatically and dynamically using the classifier. However, you can only choose from predefined values of the classifier, and the most coarse resolution step available is an IP address - you cannot say that traffic of a whole subnet of a given size should be treated as a single stream (and thus by a single child queue). So to treat all the clients' subnets equally, each by its own child queue, you have to create these child queues yourself, using the subnet address as a target (or translating it into a packet mark, I keep saying I haven't needed to dive into the queue trees yet).

The other approach to it is to translate each client's subnet to a single IP address, just like the traditional architecture does, and use PCQ on these addresses, but do that on a single router. There is no big deal in NATing every client subnet to its own IP address on a single machine; the complex part is to make the router further process the packet after it got src-nated.

virtual multi-router setup code

				  internet
                     |           provider rtr
      ---------------------------------
     |            p.p.p.23             |
     |              NAT                |
     |            a.a.a.1              |
     ?                                 ?
     ?                                 ?
     ?                                 ?
     |  a.a.a.c              a.a.a.d   |
     |    NAT                 NAT      |
     |   VLAN C              VLAN D    |
     |  DHCPs C              DHCPs D   |
     |  c.c.c.1              d.d.d.1   |
     |             trunk               |
      ---------------------------------
                     ||
                     ||    client switch(es)
    ------------------------------------
   |               trunk                |
   |                                    |
   | ..........              .......... |
   | : VLAN C :              : VLAN D : |
    ------------------------------------
       |  |  |                 |  |  |
 c.c.c.2  |  c.c.c.4     d.d.d.2  |  d.d.d.4
          |                       |
       c.c.c.3                 d.d.d.3

The whole trick how to fill in the question-marked part of the picture is described in the post to which I gave a link in my previous post - you have to use an IPIP tunnel in order to create an interface through which you can send the packet "out of the router" after NATing it, because normally src-nat is only done as the packet is leaving the device, but to still be able to treat the packet further because the other end of the IPIP tunnel is another interface on the same device. So each packet from the client passes through the routing and firewall twice. In the first pass, where it comes in via the VLAN interface, you only do the "virtual client router" src-nat and route it "out" via the IPIP tunnel; in the second pass where it comes "in" via the IPIP tunnel, you do the PCQ handling with the tunnel interface as PCQ target and local side IP address as PCQ classifier, and then do the "provider router" NAT and route it out your WAN uplink.

So the whole picture then looks as follows:

virtual multi-router setup code

				  internet
                     |           provider rtr
      ---------------------------------
     |            p.p.p.23             |
     |              NAT                |
     |            a.a.a.1              |
     |         IPIP-to-client          |
     |...............|.................|
     |        IPIP-to-provider         |
     |  a.a.a.c              a.a.a.d   |
     |    NAT                 NAT      |
     |   VLAN C              VLAN D    |
     |  DHCPs C              DHCPs D   |
     |  c.c.c.1              d.d.d.1   |
     |             trunk               |
      ---------------------------------
                     ||
                     ||    client switch(es)
    ------------------------------------
   |               trunk                |
   |                                    |
   | ..........              .......... |
   | : VLAN C :              : VLAN D : |
    ------------------------------------
       |  |  |                 |  |  |
 c.c.c.2  |  c.c.c.4     d.d.d.2  |  d.d.d.4
          |                       |
       c.c.c.3                 d.d.d.3

The link I gave before addresses much more than that, so it is probably repelling a bit, but for your case neither the IPsec/L2TP related information nor the script taking care of dynamic change of to-addresses value in the src-nat rule are relevant, so it boils down just to establishing an IPIP tunnel whose both endpoint interfaces are on the same router and a tiny bit of in-interface based policy routing.

Hi Sindy,

Thanks for such a detailed explanation. I think I actually understand you now, correct me if I am wrong - each vlan (with multiple hosts) will be presented (to queuing) as one IP address, then all the vlan's IPs will be queued in the same way the hosts IPs currently are PCQ'ed? The IPIP tunnel sounds work aroundy but I get why its necessary.

I hate to ask after such a great explanation but, could you explain in words (or code) the other way of doing it, with complex parent and child queues? I really need to learn this and your explanations are great.

BTW you are right about my network topology.

Thank you

you cannot say that traffic of a whole subnet of a given size should be treated as a single stream

I'm not sure but it seems that it is possible to use subnets as a sub-stream.
There are pcq-dst-address-mask and pcq-src-address-mask parameters and by default they are set to /32 to refer to a single address.

it seems that it is possible to use subnets as a sub-stream

Yes, @xvo, I've completely missed this part (too low on the PCQ page). So the whole workaround consisting in translating a subnet to a single IP address using NAT becomes useless, good.

correct me if I am wrong - each vlan (with multiple hosts) will be presented (to queuing) as one IP address, then all the vlan's IPs will be queued in the same way the hosts IPs currently are PCQ'ed?

Yes, this was the idea, but the remark of @xvo makes it obsolete, as there is no point any more (since long ago actually) in complicating things this much. Just assign all tenants the same subnet size, change the pcq-src-address-mask (or maybe the pcq-dst-address-mask , I belive that src and dst are seen from the the target perspective but I may be wrong) to the mask size of those subnets, and you're done with distributing the bandwidth between the subnets (vlans) in a fair manner. The only thing which could complicate it would be if the system attempted to map the target IP subnet (which must be a superset of all tenants' subnets) to a single interface and was unable to work if it would not be possible, but it is unlikely.

could you explain in words (or code) the other way of doing it, with complex parent and child queues?

I strongly prefer to explain only things I at least believe to understand myself, which is clearly not the case of queue tree. I've tried to set it up once and reality has hit me hard, I hazily remember that it was that you could prioritize between several queues at the same hierarchical level only if it was the bottom-most level, which has made me go away from it. So for this explanation, I'm not your man, at least within a useful time frame.

Thanks for your time Sindy, and thanks xvo for the possible solution. I will do testing and report back.

Thank you

XVO was right, I just changed the masks from 32 to 27 in queue types and it worked as desired, sweet.

As much as i'd like to say solved, not quite yet. I have one subnet (hotspot) that is a /23 and the rest are all /27. Yes I could make them all /23 but that's yuk and not proper.

Anyone have any ideas?

Thank you XVO

One possibility - use an individual /23 network for each tenant as well, probably actually assigning only addresses from a /27 subset of it. The 192.168.0.0/16 range can be split into 128 /23 subnets, the 172.16.0.0/12 can be split into 512 /23 subnets.

Another possibility - use my workaround with IPIP tunnel and NAT only for the hotspot network and let the tenants be handled transparently. You can use action=same and to-addresses=so.me.thi.ng/27 in the chain=srcnat rule so that the 510 hotspot users' addresses would be NATed to 32 different IP addresses instead of just a single one. The /23 subnet must not match the PCQ's target parameter, all the tenants' /27 subnets and the /27 subnet to which the hotspot network will be translated must match it.

Yet another possibility is to find a tutorial on how to configure all queues manually if you cannot get it from the official Mikrotik manual better than I can.

XVO was right, I just changed the masks from 32 to 27 in queue types and it worked as desired, sweet.

As much as i'd like to say solved, not quite yet. I have one subnet (hotspot) that is a /23 and the rest are all /27. Yes I could make them all /23 but that's yuk and not proper.

Anyone have any ideas?

Thank you XVO

Glad to hear.
You are welcome.

If you have only one client with /23 and you don't expect to add new /27 very often (not to reconfigure the things below every once and then), I would suggest this:
1) Create one parent queue of pfifo type
2) Create a child pcq queue with limit-at=(n-1)/n of max-limit of the parent (where n is total number of clients) - for your /27 clients
3) Create a child pfifo queue with limit-at=1/n of max-limit of the parent - for your /23 client
4) Max-limit of both children queues a little less than max limit of the parent

I think that should work pretty close to one pcq for all. A little not in favour of /23 client in situations when only few of /27 clients are active.
This can be further balanced by lowering the limit-at in (2) from (n-1)/n to 3/4, 2/3, 1/2 (depending on what is your actual n and average number of active clients are).

But still the best solution will be to give everybody his own /23.

A /23 for everyone is must be then - my understanding of queuing is limited at best.

Thanks again XVO and Sindy