Traffic Shapping from ISP

RouterOS General
1

We’re using a MikroTik RB3011UiAS router with a gigabit (1G) link connected to our ISP’s NTU through ether1
The router does not perfom any Firewalling or NAT

We have configured traffic shaping on the MikroTik using “simple queues.” However, the ISP reports they’re still seeing policer drops on their end. Below is the shaper we are using on the Mikrotik.

/queue simple
add burst-limit=980M/980M burst-threshold=940M/940M burst-time=20s/20s limit-at=940M/940M max-limit=940M/940M name=Shapeer priority=1/1 \
target=ether1

The ISP requires that our shaper conform to their requirements, which are

  • Shape average 980000 (M bits/s) 1035Kbytes (Bc) 125 bytes (Be)

I was just wondering if anyone can see any reasons why our current shaper policy is not conforming to the ISP’s requirements

The issue is that when we try to do file transfers to a Server in the DC (on the other end of the link) when get the following results :

  • When PC is plugged directly into NTU (bypassing Mikrotik) We get transfer speeds of 110 MB/s download


  • When PC is plugged into the Mikrotik, We get file transfer speeds of 60 MB/s download

Other config that might be relevant

/ip firewall connection tracking set udp-timeout=10s
2

How is the cabling done ? Do you use the SFP slot ?
Remember the RB3011 has an architecture where both groups of ports are interconnected to CPU-cores using a 1Gbits/sec channels. (1 for each CPU-core)
When you do queuing I’m pretty sure the CPU is involved so if you use 2 adjacent ports your probably have this result.

Packets travel from ingress port (your PC/server) → via CPU → back to egress port (NTU) so cutting speed in half worst case depending on how CPU is used for the queuing processes.
RB3011UiAS-160307123613_160313.png


So a RB5009 would be a better choice for your setup I think, this has a full-duplex 10Gbits/s channel between the Marvell-switch chip and CPU-complex.

https://cdn.mikrotik.com/web-assets/product_files/RB5009UGS_220852.png

3

A few things.

  1. The RB3011 isn’t really fast.
  2. It is quite a bit slower using Rosv7
  3. You can’t fast track simple queues. (not fast track is much slower)
  4. You need to set a lower bucket size on the queues.

You can show system resources and watch the cpu when you run traffic through the link.

Thoughts.

  1. Mostly want to limit outbound traffic correctly.
    Inbound less of an issue (It is the ISP throttling the traffic)

My thought would be to test something like the following:


Rather than using simple queues Create a queue in queue tree and have ether1 (Wan port) as its parent interface.
Interface queues work with fast tracked traffic.
Have its packet mark being no mark.
Set the bucket size of the queue to 0.01 (or better 0.005).
(Approximately the max bucket size is a time <= time for 1035k byte at 980Mbps, smaller is more CPU)

Enable fast track (if you have previously disabled it)
Unless you are running fastpath (No firewall rules at all).

See how this goes.
If still running out of cpu, try downgrade to latest V6, same config.

4

Thanks for the reply all.
@jvanhambelgium

  • We not using an SFP port. just using ether1 for WAN and ether2 for LAN so i could be an issue with the grouping?

@rplant
Would something like this work

/ip firewall mangle
add action=mark-packet chain=postrouting out-interface=ether1 new-packet-mark=shaped-packets passthrough=no

/queue tree
add name="WAN-Shaper" parent=ether1 packet-mark=shaped-packets max-limit=980M \
limit-at=980M burst-limit=980M burst-threshold=960M burst-time=20s burst-size=1035k

We are not using any NAT, IPSEC or Firewall rules

Im not entirely familiar with the Mikrotik so any suggestions on the config are welcome

Thanks again :slight_smile:

5

We not using an SFP port. just using ether1 for WAN and ether2 for LAN so i could be an issue with the grouping?

Sure it could be that, try moving “ether2” for example to port 6/7/8/9/10 and see what happens.
I’m pretty sure throughput will for sure increase as you have a 1Gbps “link” to each CPU-core.

6

Thanks a lot @jvanhambelgium. I will try that and see what happens.

Any thoughts or recommendations on this potential shapping config

/ip firewall mangle
add action=mark-packet chain=postrouting out-interface=ether1 new-packet-mark=shaped-packets passthrough=no

/queue tree
add name="WAN-Shaper" parent=ether1 packet-mark=shaped-packets max-limit=980M \
limit-at=980M burst-limit=980M burst-threshold=960M burst-time=20s burst-size=1035k
7

Sorry for the delay, forum didn't seem to work for me.

/ip firewall mangle

(remove/disable this rule) add action=mark-packet chain=postrouting out-interface=ether1 new-packet-mark=shaped-packets passthrough=no

/queue tree
add name="WAN-Shaper" parent=ether1 packet-mark=no-mark max-limit=980M
limit-at=980M burst-limit=980M burst-threshold=960M burst-time=20s burst-size=1035k bucket-size=0.005


On the 3011 you really don't have enough cpu to be marking packets at 1gbe.
(And when you do mark packets, you cannot then use fast track)

And annoyingly, it often only uses 1 core, so 1 core is flat out with the other core idling.

From the test results for the device:
https://mikrotik.com/product/RB3011UiAS-RM test results tab.

For 512 bytes no firewall (fastpath), will get around 1650M, but with a few rules,
will get 452M. Hopefully your scenario will be closer to no firewall.

But even 1650M is still well slower than 1Gb in both directions at the same time, so perhaps a device upgrade.

Apparently the switch chip will do some sort of ingress rate limiting in hardware, so you could maybe try that, but you will likely struggle to get
assistance with it.

8

Okay so update

Thanks you so much @rplant that new queue tree worked!!!
I am forever grateful

I do have one more issue though. we have another site with a similar issue. File transfers TO the DC (uploads) result in the following

  • 40MBps to 60MBs without any shaper


  • ~8MBs with the shaper

File transfers from the DC (downloads) are ~2MBps regardless of shaper/no shaper (but i suspect this is to do with a separate issue somewhere upstream)

Its the exact same model (3011), also with a 1Gig link to the ISP. The difference here is that there is considerably more latency to the DC (~25ms)

Shaper

/queue tree
 name="WAN-Shaper" parent=ether1 packet-mark=no-mark limit-at=980M queue=default-small priority=8 max-limit=980M burst-limit=980M 
     burst-threshold=960M burst-time=20s bucket-size=0.005

i tried increasing the bucket size to 0.1 (as per documentation) but that didnt help either.

Any ideas?