I’ve been combing the Wiki and searching the forum for days. The only thing I can find is that the WiFi drivers will use the Priority marking to make WMM decisions. Do any of the Queue Types available in ROS respect the Priority mark in any way? I need to be able to demote the priority of a certain class of traffic as it passes through the queues. Right now I add Packet Marks and use little HTB trees for each customer. This works for static IP customers, but not for PPPoE customers because their queues are created dynamically. If one of the Queue Types respected Priority, then I could set the Priority and tell the PPPoE server to create Queues of that type.
The Linux default Queue, pfifo_fast, has 3 bands that are emptied in a strict priority fashion. Packets are queued to the bands based on a Priority Map. Although I would rather have something more intelligent than a FIFO queue, it’s more important to squash the obnoxious traffic. So, just getting access to pfifo_fast would do the job.
Sorry but no, you have to tell it: http://forum.mikrotik.com/t/default-priority-dscp-and-tos/105632/2
OK. So I’ve seen multiple mentions of creating a queue tree for priority, and then shaping in simple queues. As an ISP, I fail to see how this could work. I have identified an obnoxious service (MSDO) that will forcibly consume all of the bandwidth for a given customer when it fires up. But since MSDO is used for updates, I can’t just ban it. If I demote it so all other traffic has priority, everything is fine.
I’m having trouble seeing how I can make two layers of queues work correctly. Lets say I have customer A and customer B. Both connect through PPPoE, so their bandwidth is handled by dynamically created simple queues. A new update comes out, so MSDO fires up. Customer A is streaming video, so MSDO should only get whatever portion of customer A’s bandwidth the video stream doesn’t need. Customer B is not using the internet at the moment, so MSDO should be allowed to consume all of customer B’s bandwidth. How can a queue tree in HTB Global restrict the flow of MSDO packets into customer A’s simple queue while letting customer B’s MSDO packets flow freely?
Just curious, what does “MSDO” stand for?
As regards the other, just set the minimal queue-rate for A and the rest of the bandwith will be available for B. If nothing happens at A, then B will get it all. If the bandwidth requirement is changing during the day you probably need some scripting to alter the values..
Something like that?
MSDO - Microsoft Delivery Optimization
Customer A has a 3Mbps account and customer B has a 2Mbps account. Their bandwidth is not shared. They each get a simple queue created dynamically when their PPPoE Connection is authenticated through RADIUS. Because they are dynamic, I can’t do more than pick what Queue Type will be used.
You can probably see my conundrum here. How can the HTB Global Queue Tree prioritize packets if it doesn’t know the customer’s speed? As far as I can tell, the only solution that doesn’t require a major kludge of a headache is to find a Queue Type that respects the Priority marks (SKB->Priority).
Well, you know that to have priority work it needs a reference, meaning queue1 has higher priority to queue2. That’s why the queues need to be related as in queue tree. When they re not related, it is not possible to apply priority.
In case of pppoe users, do not set limits on the profile of users, instead put them on an address-list. By not setting limits, it will not create a simple queue. Then using address-list you can create mangle rules to use them in queue tree.
Or, in the profile of the pppoe user you have the option to set its parent queue. I don’t know to tell radius about this, you should contact who built that radius and ask if such thing can be made from radius.
Good idea, worth checking out! And if you are lucky it might be possible to apply some kind of RADIUS custom attributes.
Links to MikroTik Specific RADIUS Attributes
http://wiki.mikrotik.com/wiki/Manual:RADIUS_Client
http://wiki.mikrotik.com/wiki/Manual:RADIUS_Client/vendor_dictionary
I assume you mean Windows Update Delivery Optimization. Then it might be worth examine if the customer is channeling all the updates through their own Windows Server Update Services (WSUS) and if it’s properly configured to download just necessary updates. Windows 10 is much smarter nowadays but Window 7/8 can be pretty stupid to handle Windows updates thus I strongly recommend to use WSUS when the bandwidth is limited. With WSUS it’s also easy to apply bandwidth throttling and limit active download hours. WSUS is included in Windows Server but it’s quite easy to actually tweak it to be installed on a windows client.
I do know that Queue Priority is relative to other queues under a parent queue, and that is how I manage this for static IP customers. Identify and mark MSDO Connection - Apply packet marks to all packets in the marked connection - create 2 sub-queues of the customers simple queue, one for marked packets at queue priority 8 and one for all else at queue priority 6. But you can’t have the Mikrotik dynamically create these little queue trees. I do not wish to have any kind of priority relationship between Customer A and Customer B. I only want a priority relationship between Customer A’s MSDO packets and all of Customer A’s other packets, completely independent of Customer B’s packets.
There is another Priority that is attached to each packet via the packets SKB structure. See IP/Firewall/Mangle action set-priority for how it’s set. Linux, which ROS is based off of, respects this Packet Priority value in it’s pfifo-fast classless queuing discipline. So if I could tell ROS to create dynamic simple queues using pfifo-fast for the queue type, then instead of marking the packets with Packet Marks, I could set the Packet Priority and get the demotion of the MSDO packets that I’m after. If PCQ queues used a Packet Priority aware internal queue, then I could potentially use your address-list suggestion as recommended in the http://mum.mikrotik.com/presentations/CZ09/QoS_Megis.pdf. But even MicroTik doesn’t seem to have a solution for my problem. In the QoS_Megis.pdf it just says “Create packet marks in the mangle chain ‘Prerouting’ for traffic prioritization in the global-in queue” without even giving a hint as to how to integrate it into the queuing structure they just recommended ISPs use.
So I do appreciate your trying to help, but I can’t see daylight in any of your suggestions so far.
You mentioned that customers are dynamic and thus you can’t apply the queue to your needs, so I offered an idea how to avoid dynamic queues and apply what you need. If you further need to prioritize MSDO connection you can identify it by mangle rules if it can be separated from other traffic.
I don’t know how linux works, so if you are looking for exact feature as in linux I will leave it here since I am not knowledgeable at all on that.
If I can figure out the queuing issue, those URLs could be valuable. Thanks.
Nope, it’s Microsoft Delivery Optimization, check out the description of the MSDO service. It replaces the Background Intelligent Transfer Service(BITS) that was used by every version of windows since XP, until Windows 10. In the first release of Windows 10 they were still using BITS for updates and only used MSDO for the Windows Store. But in preparation for the Anniversary update they switched to MSDO for Windows Updates. On Windows 10 Pro and higher you can tell the computer to go back to BITS, but not on Home. The majority of our customers are residential, and there’s hundreds of them, so WSUS is not a feasible solution.
Understand that MSDO is a P2P network client ala the old Napster. It can be told not to share or download from other clients, but even then it will still open dozens of connections to the Microsoft download servers. It can’t be told to limit it’s bandwidth or number of connections. And just like the old Napster, when it fires up it crowds out all the other connections on the customers network. I’ve even seen Windows 10 machines that were downloading updates claim that there is no Internet connectivity in the task bar at the same time that Performance in the Task Manager shows the computer is downloading at the customers full rate. MSDO has lost us customers, because it makes the customer think our Internet is unreliable. It has to be tamed. If Microsoft would tame it, then it would actually be a boon as it can share updates with other machines on the local LAN. But as it is, its firmly in the necessary evil category, like Blizzard choosing to use BitTorrent to distribute their updates. Only modern BitTorrent is a lot smarter than MSDO.
Lets ignore Linux for now and talk about your suggestions. Perhaps you have discovered something I’m unaware of. I’ve only mentioned Customer A and Customer B, but there’s actually hundreds of customers I need to fix this for. I have no problems identifying the traffic that needs to be demoted. It’s how to actually demote it that’s frustrating me. For our static IP based customers I have:
add max-limit=0/10M name="Customer H" queue=default-small/synchronous-default target=172.29.236.40/32
add limit-at=0/256k max-limit=0/10M name="Customer H-Bulk" packet-marks=ToS-Bulk parent="Customer H" \
queue=default-small/ethernet-default target=172.29.236.40/32
add limit-at=0/768k max-limit=0/10M name="Customer H-Other" parent="Customer H" priority=6/6 \
queue=default-small/synchronous-default target=172.29.236.40/32
Notes: I mark the MSDO packets with ToS-Bulk, in case I later find other traffic that needs demoting. Upload is shaped at the customer’s end, this is from our head end. Queue Types are the MicroTik defaults. Name and IP changed for privacy reasons, but the original IP is a real world routable IP. Each customer gets a duplicate of this modified for their rate plan.
The problem I have is how to do the equivalent for customers who are on dynamic IP services like PPPoE. Each customer’s bandwidth needs to be independent of the other customers. For each customer, all their other packets need to come before ToS-Bulk packets.
That is a lot of simple queues you would need to create for each customer. Mikrotik says that they have improved a lot the performance of simple queues, but I haven’t tried it in real world since I am a lot more comfortable with queue tree. The good thing about queue tree is that all queues are treated at the same time, while with a simple queue the packet must check them all in their order until it matches the one which deals with it.
I think that there must be customers who are on the same bandwidth plan, say you got to have 10 customers who are on 10/10Mbps. If you have such groups, you can set the profile of pppoe with no limits and add them to address-list. Then use address-list to identify the MSDO coming-going from this address-list, and the other traffic. Once you have marked packets, you can apply them on queue-tree with one parent equivalent at the total of the group and then children queues with pcq to distribute it equally to the customers of the group. You can save a lot of queues this way, freeing up resources of the router.
I see you have no problem to identify the traffic, so can’t see why not use address-list generated from the pppoe profile. Once a customers connects through pppoe, no matter what IP will be assigned, he will be added to the specified address-list, and with no limits on pppoe profile, no dynamic simple queue will be created, thus moving them on the queue tree.
Ok, I was totally wrong there who assumed you meant a company that was hooked up but now I see your point. Yeah, that kind of p2p traffic might be a beast in that regards. Is it possible at all to identify any kind of patterns or port numbers that relates to MSDO so it might be down-prioritized? It seem that you gonna need a pretty complex kind of L7 content analysis in order to find out when MSDO kicks in…
Oh, you’ve already found a way how to identify MSDO traffic. Do you mind sharing how it’s done?
a two seconds search in RavenWing71 posts would have told you that: http://forum.mikrotik.com/t/mark-ms-services-updates/105098/2
To my knowledge, this is totally wrong. The current implementation if Simple Queues uses hash-table lookup, which is very fast and scales up really well.
Well, as I said, I haven’t used simple queues in very long time since I understood how queue tree works. But, for clarification, here is what Mikrotik wiki says (unless they haven’t updated it):
Simple queues have a strict order - each packet must go through every queue until it reaches one queue which conditions fits packet parameters or until the end of queues list is reached. (In case of 1000 queues, packet for last queue will need to proceed through 999 queues before it will reach the destination)
Well, I’m not that well educated forum user but thank you anyhow! 
Just a thought - will that also capture the actual p2p traffic that is initiated in a step further on in the update process?
Secondly, last time I checked it was ssl/tls that was utilized for most of the ms related services. This makes it hard to manage L7 filters and isn’t a long term solution since it will become more or less impossible in the future when everything has been moved over to ssl. The update process will also continue to evolve with new application protocols.
It will probably be necessary for Microsoft to offer some way to cap service traffic in order to handle this type of obstacles. In any case, I would suggest that someone opens a support case at some of the microsoft forums to make them aware that not everyone is sitting on a high speed wan link…
My packet sniffing, crossed with internet research, many years of internet servicing, and dabbling in programming, gives me this rough framework for MSDO:
It uses HTTPS for the command and control aspects and downloading the catalog with it’s security checksums. The actual P2P downloading of the files is done through HTTP, allowing the L7 matcher to identify the P2P connections. Microsoft realized HTTPS incurs too much overhead, and since MSDO only downloads fragments of the file from any one source, someone sniffing in the middle would not get anything useful. The final assembled download gets checked against the checksum in the catalog to verify it’s authenticity.
So with my L7 filter and queue structure I don’t end up demoting the command and control, just the actual file downloads.
Yes it is a lot of queues. 3 per customer. I have nearly 900 simple queue entries and I haven’t finished adding the bulk and other queues to all the customers yet.
I’m glad to hear this. It settles my worry about all these queues.
This sounds like a variation on what is recommended by the QoS_Megis.pdf. There must be something I’m missing about how queue trees work. I could mark packets as 10/10 and 10/10-Bulk and then they would go into different branches of the tree, but if I make a PCQ for each mark how would it know that 172.29.236.40 only gets 10Mbps total and not 10Mbps for each mark since they are separate PCQs? Can I branch off of a PCQ, making two fifo queues beyond the PCQ, one for each mark? What do I really get if I do? Would it then make 2 fifo sub-queues for each dynamically generated queue within the PCQ? If it did, then this would work, otherwise it’s just a mess. Perhaps you could mock up a queue tree that does what you are suggesting?