Some traffic not seen on PCQ Queues

RouterOS General
1

I have just had something weird. I observed it earlier this week on the Mikrotik 1100 AHx2. Have just seen this happen again.

I ran a torch on the port and it showed only 1 mbps to this device.
I ran a torch on the wan interface to the source of the traffic and was seeing 5 mbps coming in from that external address. I can see on my internal interface the traffic passing to the device and it is visible in accounting. The traffic was over port 80. I am suspecting a vpn application but his should still not bypass the simple queues. I have tested this of ROS verson 6.32.4 and 6.35.2
I am running Simple queues as follows

 0    name="LAN" target=172.17.0.0/16 dst=172.17.0.0/16 parent=none packet-marks="" priority=8/8 
      queue=synchronous-default/synchronous-default limit-at=0/0 max-limit=1G/1G burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

 1    name=" VOIP 1" target=172.17.0.0/16 dst=aaa.bbb.ccc.ddd/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-down-4M/pcq-upload-4M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      bucket-size=0.1/0.1 

 2    name="VOIP 2" target=172.17.0.0/16 dst=aaa.bbb.ccc.ddd/32 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-down-4M/pcq-upload-4M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      bucket-size=0.1/0.1 

 3    name="queue1" target=172.17.8.199/32 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-download-default limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 total-queue=default 

 4 XI  name="queue2" target=172.17.24.56/32 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-download-default limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 total-queue=default 

 5 XI  name="Satellite connection" target=172.17.0.0/16 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-256k/pcq-down-512k limit-at=0/0 max-limit=2500k/4500k burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 total-queue=default 

 6    name="TV" target=172.17.19.224/28 parent=WAN packet-marks="" priority=8/8 queue=pcq-upload-1M/pcq-down-4>
      limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s bucket-size=0.1/0.1 
      total-queue=default 

 7 XI  name="reception" target=172.17.18.199/32 parent=WAN packet-marks="" priority=1/1 
      queue=pcq-upload-default/pcq-download-default limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 total-queue=default 

 8 XI  name="Priotrity" target=172.17.19.176/28 parent=WAN packet-marks="" priority=1/1 
      queue=pcq-upload-default/pcq-download-default limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 total-queue=default 

 9 XI  name="reception" target=172.17.18.198/32 parent=WAN packet-marks="" priority=1/1 
      queue=pcq-upload-default/pcq-download-default limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 total-queue=default 

10    name="Backbone" target=172.17.0.0/24 parent=WAN packet-marks="" priority=8/8 queue=pcq-upload-4M/pcq-down-4M 
      limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s bucket-size=0.1/0.1 
      total-queue=default 

11    name="Vlan8" target=172.17.8.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

12    name="Vlan9" target=172.17.9.0/24 parent=WAN packet-marks="" priority=8/8 queue=pcq-upload-default/pcq-down-1M 
      limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s bucket-size=0.1/0.1 

13    name="Vlan10" target=172.17.10.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

14    name="Vlan11" target=172.17.11.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

15    name="Vlan12" target=172.17.12.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

16    name="Vlan16" target=172.17.16.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

17    name="Vlan17" target=172.17.17.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

18    name="Vlan18" target=172.17.18.0/23 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1.5M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

19    name="Vlan24" target=172.17.24.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

20    name="Vlan25" target=172.17.25.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

21    name="Vlan26" target=172.17.26.0/24 parent=WAN packet-marks="" priority=8/8 
      queue=pcq-upload-default/pcq-down-1M limit-at=0/0 max-limit=0/0 burst-limit=0/0 burst-threshold=0/0 
      burst-time=0s/0s bucket-size=0.1/0.1 

22    name="WAN" target=172.17.0.0/16 parent=none packet-marks="" priority=8/8 
      queue=synchronous-default/synchronous-default limit-at=21M/21M max-limit=21M/21M burst-limit=0/0 
      burst-threshold=0/0 burst-time=0s/0s bucket-size=0.1/0.1
      0 * name="default" kind=pfifo pfifo-limit=50 

 1 * name="ethernet-default" kind=pfifo pfifo-limit=50 

 2 * name="wireless-default" kind=sfq sfq-perturb=5 sfq-allot=1514 

 3 * name="synchronous-default" kind=red red-limit=60 red-min-threshold=10 red-max-threshold=50 red-burst=20 
     red-avg-packet=1000 

 4 * name="hotspot-default" kind=sfq sfq-perturb=5 sfq-allot=1514 

 5   name="pcq-upload-64k" kind=pcq pcq-rate=64k pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

 6   name="pcq-down-128k" kind=pcq pcq-rate=128k pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

 7   name="pcq-down-384k" kind=pcq pcq-rate=384k pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=6s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

 8   name="pcq-down-1M" kind=pcq pcq-rate=1M pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

 9   name="pcq-upload-128k" kind=pcq pcq-rate=128k pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

10   name="pcq-upload-1M" kind=pcq pcq-rate=1M pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

11   name="pcq-down-4M" kind=pcq pcq-rate=4M pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

12   name="pcq-down-768k" kind=pcq pcq-rate=768k pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=6s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

13   name="pcq-upload-512k" kind=pcq pcq-rate=512k pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

14   name="pcq-down-512k" kind=pcq pcq-rate=512k pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=6s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

15   name="pcq-upload-256k" kind=pcq pcq-rate=256k pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

16   name="pcq-down-1.5M" kind=pcq pcq-rate=1500k pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

17   name="pcq-upload-4M" kind=pcq pcq-rate=4M pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

18   name="pcq-down-2M" kind=pcq pcq-rate=2M pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

19   name="pcq-down-8M" kind=pcq pcq-rate=8M pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

20 * name="pcq-upload-default" kind=pcq pcq-rate=0 pcq-limit=50KiB pcq-classifier=src-address pcq-total-limit=4000KiB 
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

21 * name="pcq-download-default" kind=pcq pcq-rate=0 pcq-limit=50KiB pcq-classifier=dst-address pcq-total-limit=4000Ki>
     pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-src-address-mask=32 pcq-dst-address-mask=32 
     pcq-src-address6-mask=128 pcq-dst-address6-mask=128 

22 * name="only-hardware-queue" kind=none 

23 * name="multi-queue-ethernet-default" kind=mq-pfifo mq-pfifo-limit=50 

24 * name="default-small" kind=pfifo pfifo-limit=10
2

You didn’t mention in your post, but I’m assuming that you’re looking at the gold band of the graph (Host is 172.17.24.56)

Was queue #4 active at the time? It is set to allow ~4Mbps and would only have that exact host using it…

Do you have fasttrack enabled? Fasttrack traffic bypasses queues.

3

Hey zero byte
Queue 4 was not enabled. The queue print was done later after I had spent some time troubleshooting. I was using queue 4 to try and see if I could limit the traffic and or monitor how much bandwidth was being used. The queue was not able to limit that is or see all the traffic. I am not running fast track either.
The device in question was an android phone. Earlier in the week it was a lenovo laptop. Suspect they may have been running vpn software over port 80

4

I have a similar problem. 10 Mbps simple queue consistently shows 14 Mbps on queue traffic tab. But torch shows exactly 10 Mbps. The client’s wireless bridge shows 10 Mbps.
queue.png

5

And the torch on the wan interface for the same IP as queue shows consistently about 4 Mbps more.
queue2.png

6

You’d think that wouldn’t matter though - unless your queue is tailored to port 80 specifically…

I wonder what you would see if you torched the user’s MAC instead of their IP.

7

My Queues are operating on a per connection basis irrelevant of the traffic passing through which is why I was concerned. If I see it again will torch the MAC and see if there is any difference. Weird thing is Ip accounting saw the traffic but not Simple queue.

8

chiefbmr
When you carried out the torch did you locate where the traffic was coming from and to. Next time run a torch showing the ports etc. Try and track it to an internal device.

Mine there was one computer going out to one remote IP and there was extra traffic coming back from that IP that the Simple Queue was not seeing.

9

One more thing I’ve thought of (did I already mention this before? if so, sorry for being repetitive) is that fasttrack will jump over queues - could any traffic be getting fasttracked?

10

On My side I am not using fast track.

11

Majority of it was going to a microsoft windows update IP. I’m pretty sure my problem was how the traffic is accounted with NAT in the new ROS. Maybe it’s a coincidence that each time I torch the value would change by 4 mbps through the NAT. (Double NAT in this case.) And your example is also 4 mbps difference.

But the wireless bridge at customers location using ROS 6.33.3 showed 10 mbps like the queue settings. So I am trusting this bridge for the actual throughput to the customer.

I haven’t had time to look into. But will let you know if I find my problem.

12

I don’t fully grasp the fasttrack feature. But it doesn’t show any fasttrack traffic under IP Settings. I don’t see how that would be the problem when the CPE shows 10 mbps and the queue shows 14 mbps.

I could see that being the problem with the queue and wan interface difference. But I don’t know how to check for fasttracked traffic.

NAT at Customer Router --------10 mbps at wireless CPE------ 14 mbps at queue----- NAT---- 18 mbps on wan interface

13

If the queue shows less-than-actual traffic the one of two things can be the cause:

1- some traffic is not being matched by the queue
2- the queue is reporting false information

Probably 2 isn’t the case.

Fast path traffic does not enter queues (currently)
Hence that’s a possible source of the discrepancy. If you have any firewall rules with action=fasttrack-connection then this would be a possible source. If you have no such rules then fasttrack cannot be the problem.

Simple queues match in order (similar to firewall chains) so if an early queue matches something the user is doing then that traffic will not be counted by the user’s own queue.

Those are your most-likely issues. There is also the possibility of non-ip traffic (eg IPv6) but I suspect you would know you’re using IPv6 or not. :wink:

14

chiefbmr can you post your sample of your queues. and where traffic was to and from .

15

As far as I can see the difference in numbers is proportionate to the number of connections open.

In my case I have a 10 Mb simple queue. It limits at 10 correctly, but the more connections that open, the simple queue traffic graph increases. But the traffic to the customer stays at 10.

And that traffic increases more than the simple queue on the WAN side of the router with the queues.
traffic.png

16

My first examples had more traffic and more connections.
traffic 2.png
As the connection number goes down things look right.
traffic 3.png

17

Can you post a copy of your simple queue in question and the queue types that go with it.

Thanks