_HUGE_ Packet loss on CRS125 :(((

RouterOS General
1

Hello,

I’m not sure it is software or hardware problem, but I still have a question to all of you guys.

I have CRS125-24G hardware and I have a packet loss on every traffic passed thru CPU (I think).

  1. Ports 17 (IP1.1) and 19 (IP1.2) are slave. Master port 21 for both. There is IP1 on port 21 (or on bridge1 which is port 21 only - tried both - no matter)
  • there is no packet loss between IP1.1 and IP1.2 (they are going thru switch directly)
  • there is a packet loss between IP1.2 and IP1 (3-6% on load)
  1. Port SFP is slave with port 2 master. There is IP2 on port 2.
  • there is a packet loss on this link also (the same 3-6%)

RouterOS 6.26 / Firmware 3.22 (the same with 6.25/3.19)
CPU at ~40% mark while running probe.

The probe is:

ping -i 0.001 -s 10000 -q -c 1000 192.168.2.200

PING 192.168.2.200 (192.168.2.200): 10000 data bytes

— 192.168.2.200 ping statistics —
1000 packets transmitted, 964 packets received, 3.6% packet loss
round-trip min/avg/max/stddev = 0.611/0.848/41.717/1.935 ms

Diagram attached.
IMG_1980.JPG

2

Ok. Even if I set IP address on port 19 running probe, take it out of switch (master port=none) when the problem still persist. So I assume the problem on SWITCH-CPU internal link. Wondering if it is hardware problem or it can be figured out by some software settings?

3

There is an increased rtt in case of packet loss also:
— 192.168.2.200 ping statistics —
1000 packets transmitted, 958 packets received, 4.2% packet loss
round-trip min/avg/max/stddev = 0.605/0.787/41.168/1.913 ms

— 192.168.2.200 ping statistics —
1000 packets transmitted, 1000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.600/0.685/1.111/0.041 ms

4

src - You could try to set interface queue to default small for interfaces which are being used in communication (also master port if ti is configured for used interfaces). “/queue interface set <interface_name> queue=default-small”

5

The same problem. No problem to ping another mikrotik via switched ports. The problem to ping CRS125 itself.

6

Maybe the problem in switch settings? I didn’t touch it, but there is a bunch of ROS updates placed in top of default (very old) settings.

7

upgraded to 6.27 - no change

8

Anyone?

9

Do you have any packet loss with -i 0.01?

Do you have any packet loss with -s 1472?

The CPU on the CRS is small. Watch cpu utilization while you run the ping command. You are doing a serious flood ping with your existing ping command.

Your existing size is forcing it to send 7 packets over the wire per ping request, unless you’ve increased the MTU and not told us about it. So, combined with your interval, you are sending 7 * 1000 packets per second to the CPU of the CRS. The CPU of the CRS, and your testing device, is having to perform reassembly actions on every incoming packet and fragmentation actions for every outgoing packet. I think you are just abusing the CPU of the CRS. I presume that the CPU of your test source device is faster than the CRS.

I have no idea why anyone would pick that size option for rapid ping test packets.

10

It’s a stress test, right. Because the packet loss is up and down, I prefer to do a comparison tests between different systems. So I run these settings because the loss is more constant on these packets. The other system running the same setup is based on the same CPU (mikrotik 2011) and have no packet loss with this probe. The other one connected to CRS port is mikrotik 912 with no packet loss too. Even while probe running thru CRS switch logics. So I assume there is no problem with CPU overloading, the load is not exceed 40% while running probe.

11

+1
same http://forum.mikrotik.com/t/packet-loss-in-crs125-24g-1s/76451/1

12

Thanks God I’m not alone! :slight_smile:

13

Ok. I reset config on CRS125.
Now what we have with no default config, just IP address on one of interfaces:
— x.x.x.66 ping statistics —
5000 packets transmitted, 4788 packets received, 4.2% packet loss
round-trip min/avg/max/stddev = 14.705/15.063/76.878/2.814 ms

The same subnet, mikrotik 2011:
— x.x.x.67 ping statistics —
5000 packets transmitted, 5000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.759/12.916/14.490/0.146 ms

So any thoughts?

The probe is:
ping -s 10000 -i 0.001 -q -c 5000 x.x.x.y

14

Also guys have performance issues:
http://forum.mikrotik.com/t/performance-issue-with-crs125-24g-1s-rm/74755/1


MIKROTIK, any answer on this?

15

Or even so:

— X.X.X.66 ping statistics —
1000 packets transmitted, 826 packets received, 17.4% packet loss
round-trip min/avg/max/stddev = 16.003/36.950/50.360/5.109 ms

— X.X.X.67 ping statistics —
1000 packets transmitted, 1000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.911/13.055/14.039/0.150 ms

16

UP. how to solve the problem?

17

Seems no way to solve it. Switched back to old rb2011 :frowning:

18

We have the same issue using a CRS125 on v6.27 with 16 ports bridging onto a VPLS tunnel and one port for MPLS/IP uplink - inexplicable packet loss when pinging to IPs that are both on and through the CRS. There is an absence of any L2 errors on any interfaces. CPU is only hitting 30-40%

I have not found a solution either, we are about to replace it with something else. :frowning:

19

Mikrotik, any response?!

20

Hello, I have exactly the same problem, I do not think that it is a weak CPU but something bad in RoS because how is it possible that in version 6.12 is quite low packet loss, and then in the amounts of 6.19 versions need is much greater? Not noticed that he was somehow overloaded switch.
For me, I pray also that mikrotik responded to this, obviously is a mistake on his part (mikrotik). Caution If you replace the CRS RouterBOARD RB2011 and the problem disappears, and is the same CPU!