Hi, I work for an ISP and currently we deliver Internet service using Juniper EX2300 switches we install out at each customer’s location, however, going forward I’d like to be able to use MikroTik hardware as our CPE devices instead. The original CRS305 is especially attractive due to its low cost and so are the CRS309 and the CRS310 (both variants). The problem though is that I need to be able to rate-limit customer traffic on a single port - the one that hands off directly to the customer router. The obvious solution for this requirement is to set up a policer and a shaper on that handoff port. Unfortunately, I haven’t been able to get policers to work correctly on any of the MikroTik switch models I’ve experimented with, including CRS305, CRS309, and CRS310. When I create a policer via Switch → Ports → Ingress, I get wildly inaccurate throughput numbers. For instance, on a CRS310 I set the ingress speed for a port connected at 10 Gbps to 1100 Mbps. The result is ingress throughput that ranges from 38 to 60 Mbps. If I attempt to instead police the port to 1100 Mbps using a switch rule on this same CRS310, I get about 100 Mbps. Similar behavior is observed on the CRS305 and CRS309. Remove the policer and I’m back to getting 9.4 Gbps of traffic as expected.
Shaping on the other hand is working fine. If I go into Switch → QoS → Port and then set Queue0 to 1100 Mbps, the shaper delivers a very consistent 1100 Mbps (including overhead). Same if I go into Switch → Ports and then set “egress” to 1100 Mbps.
My guess is that the policing is messed up because we’re not able to define the policer burst sizes and whatever value the switch is using automatically might be goofed up. Has anyone else observed this behavior before? Is there any hope of getting this fixed? As I said at the beginning of my post, the CRS3XX models check off pretty much all the boxes for my requirements as CPE except for this one irritating limitation. Any insight anyone could provide would be greatly appreciated!
Note: All testing was performed using RouterOS 7.19.2 or 7.20 beta. Test traffic was TCP and UDP with normal sized frames. I normally run 9216 byte L2MTU on all ports but experimented with bumping them down to 1518 bytes with no impact on policer behavior. If I substitute a Juniper EX2300 with an 1100 Mbps policer applied to the port, I get my desired throughput consistently.
since previous versions i had the same problem, there are several topics about it, because of that i rely only on egress shaping which started working relatively good since newer QoS features since 7.15+
If ever MT fixed this issue, do you think this device is capable of doing it? e.g if you implement ingress policing of 5G can it do line rate?, this is very attractive if they do
edit: because of this limitation our deployment with this so called “special build circuit” is always OLT based and our opex get bigger if we need to handover more than > 2G of traffic
after reading this i was curious, made a simple test on LAB of an ingress ACL action rate on a CRS-317 7.20 b4 and worked OK, only tested up to 900mbps OK (only have 1g equiment)
Will do my own experimentation once the v7.20 become stable no spare switch atm thanks for the insight the last time we test this the CPU becomes the bottleneck and speed varies as well and ever since I personally don’t look back ahaha, thanks a ton for the insight
I tried both port based setting and switch rule on a CRS328 (v7.19.2) but in both cases the ingress rate was not matching at all the setting.
I tried 20M limit on a connection with 60Mbits peak, but I always got around 8Mbits instead.
/interface ethernet switch port
set ether1 ingress-rate=20M
in previous related topics somebody mentioned that policing rely on dropping frames without any queuing and that can lead to unstable behavior of application traffic when measuring, maybe that explains the issues
edit:
from docs:
The ingress policer controls the received traffic with packet drops. Everything that exceeds the defined limit will get dropped. This can affect the TCP congestion control mechanism on end hosts and achieved bandwidth can be actually less than defined.
i think this can be, maybe my testing with traffic generator is “too ideal” maybe with real world traffic small micro bursts of traffic get dropped and thats what create the problems, i will try to test with real traffic but my LAB is limited
Hi Chechito, thanks for your replies. When I initially searched the forums about this issue it was with the keyword “ingress” which returned page upon page of VLAN ingress filtering results, hence my post. Now I see that I should have kept going as there is a fair bit of relevant conversations I can read through (including many helpful comments you yourself have made).
It’s true that policing relies on dropping undesired traffic and it can indeed have a more pronounced effect on throughput for TCP streams vs rate-limiting via shaping, but mostly only in situations where you’re dealing with high round trip times. If your latency is low such as in a lab environment, TCP should ramp up again fairly quickly after being policed. But even with low latency, having an improperly tuned policer burst size (the amount of data that is allowed to pass at full line rate before being policed) can play havoc on your TCP performance. UDP on the other hand doesn’t care about being policed, it just happily carries on consuming all available bandwidth.
Can you elaborate on the ACL you created for your test please? Config snippets would be greatly appreciated so I can examine how what you’ve done is different from all the things I tried. I’d also be curious to hear what happens if you change your 900 Mbps policer to say, 110 Mbps - does throughput decrease as expected?
For my own testing, I’ve mostly used the RouterOS bandwidth test tool on two CHRs hosted on a fairly beefy Proxmox server to generate my traffic with either a CRS305, a CRS309, or a CRS310 sandwiched in between them. I’ve also substituted a pair of PCs with iperf3 instead of the CHRs to verify that my results were the same though. I could fire up a hardware traffic generator as well but I’d prefer not to since RFC2544 or BERT tests aren’t stateful and I don’t consider them to be a good representation of real world traffic.
as you mention i tested with traffic generator which is stateless, not a good representation of real world traffic, maybe because of that the issues does not arise
Your experience is essentially the same as mine, although unlike your second example, I didn’t bother defining src-mac-address and instead I applied my rule to the whole switch port. Regardless of what I set “rate=” to, I observed extremely poor performance.
I don’t see why not. In theory policing should be easier to implement than shaping, because unlike shaping you don’t necessarily need to classify each packet for placement into queues and you don’t need buffer space to store important bits of traffic until room opens up on your link. Instead, you’re just doing simple metering of how much traffic is passing through an interface during a given time interval and then dropping anything that exceeds your defined threshold. I’ve been reading through older forum posts about the issue and I have to admit, it’s discouraging to see that reports of the problem go back several years. But on the flip side, it doesn’t seem to be specific to any particular switch models, which suggests to me that the problem is in RouterOS itself, meaning it can probably be fixed if MikroTik throws enough resources at it.
When I create a policer via Switch → Ports → Ingress, I get wildly inaccurate throughput numbers. For instance, on a CRS310 I set the ingress speed for a port connected at 10 Gbps to 1100 Mbps. The result is ingress throughput that ranges from 38 to 60 Mbps. If I attempt to instead police the port to 1100 Mbps using a switch rule on this same CRS310, I get about 100 Mbps. Similar behavior is observed on the CRS305 and CRS309. Remove the policer and I’m back to getting 9.4 Gbps of traffic as expected.
the question is why would you shape nor polish ingress interface?
that won’t work. any traffic shaping nor polisher need classifiers to work. ie dropping or tag-untag so forth. and that classifiers need input from the ingress interface. in short - any queueing - shaping polishing etc should be done controlled on egress interface. while ingress interface is only for the classifiers.
note that ingress or egress is relative to from which direction the traffic came from and heading to which direction. just like you do firewalling.
other than that - you will messing up with the switch backplane capacity. or even the cpu resources.
I’m not sure what you mean here. Ingress policing at the client handoff port is perfectly valid - we do it all the time in the ISP world. You ingress police the incoming customer traffic down to whatever CIR they’re paying for and shape down to the CIR the traffic outgoing towards them, all on the same port as long as that functionality is available in your hardware. Otherwise for outgoing customer traffic you can do an egress policer on the handoff port (again, if supported) or an ingress policer for their specific service VLAN on the CPE uplink. You can of course use a shaper on the uplink port to rate-limit customer upload traffic instead of ingress policing at the handoff as long as your hardware allows you to assign different rates to different classes of traffic. It appears the CRS3XX devices do support multiclass shaping in addition to port-based shaping so I’m going to test that in my lab to see if it suits my needs.
thanks