Thx for the quick answer NetworkPro. Here is my response to your remarks, I'm just putting in my answer, without quoting your reply to avoid making a long email:
Q1: I also think he is doing PCQ or something elsewhere which is not in this config.
Q2: Yes, I have tested this. Actually I was burnt very badly around 4 weeks ago, when I divided my NAT + BWMgr box into two MT Boxes, and forgot to migrate the pcq-total-limit values. We have around 3,000 customers in one particular category, where the users used to constantly complain during peak hours, that their BW would fluctuate between the 0 and max-assigned value. Took me 7 painful days, and a lot of re-configs in lots of places, before I noticed this small thing. And, since then, this has been singing like nothing. To repeat the behavior, bring up the (RX & TX)_Queued_Packets colums in simple queues, and will definitely see a problem if any category has more than the default 2000 packets in their queues. At peak time, this particular category had 7000+ queued packets!
Q3: I re-read the Wiki, and i says clearly in the first line, that he is trying to put QoS in Download Traffic. Or is this from the router's perspective, rather than his network's? I'm confused! Could you please correct me on this if I'm wrong?
Q5: This is indeed a major headache. To me it appears more like a balancing act, between how long we make each connection-bytes bucket, and the PCQ which caps the maximum download. Will have to play around and see. But, need answer to Q4 first!
Ques 5 [should be 6
]: While I'm not quite sure what fatonk is looking to do QoS on, I need to do QoS on downloads on Eth_1 only, as that is what we pay for by the Mbps. So, I'll play around a bit this week and find out!
Thank you once again NetworkPro for your quick answer. Looking forward to share more ideas!