CCR 1036. High load loses packets. Big problem.

RouterOS General
1

CCR 1036. High load loses packets.
Ticket#2014021766000422

[admin@MikroTik] > system resource print
             uptime: 5d9h3m25s
            version: 6.9
         build-time: Jan/31/2014 11:18:19
        free-memory: 3243.1MiB
       total-memory: 3969.0MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 18%
     free-hdd-space: 890.4MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-12G-4S
           platform: MikroTik

2.png
1.png

2

Maybe a problem somewhere down the road? Have you tried tracert?
We have a flow rate of 700Mbit and ping is 0ms 0% packet loss
ccr.jpg

3

Delete

4

I’m lying, then downgrade to 6.7 was a little better, but still very noticeable packet loss of 1-2%.

5

I think I found the cause of the loss. If after CCR let UDP traffic packages from 64 to 500 bytes, the CCR begins blunt and lose packets!
And so begin:

1. I will try to convey through CCR UDP packet size 64-500 bytes.
Running ping to the resource that is available through the CCR 1036.
Test off.

2. Test on. UDP packets of 64 bytes. Test makes to external mikrotik. Not on CCR.
The test showed that CCR not withstand the load and starts to drop packets.
2.png
1.png

6

I did a visual test demonstrating the problem. Please developers pay attention to it.
1.png
2.png

7

Next.
The problem occurs when testing UDP packets in bytes 750-1000.
We can conclude that the CCR1036 weak performance when passing UDP traffic. With TCP-packets such problem was not found.
Is it possible to fix this by optimizing ROS?
1000.png

8

Anyone? Anything?
Normis?

9

Maybe try disable/enable fast patch.
IP>Settings…

10

2jarek, thanks for the answer!
Sorry, did not help. Tried all options in IP-Settings->…

11

Ticket#2014021766000422

12

up!

13

I’m seeing the same problem with 4 of my CCR’s on 6.9. Please update when fixed.

14

I can not get an answer from the support. Ticket created and silence ..

15

Hi

if I read right this is just for UDP packet ?

16

Hello!
Yes, passing UDP packets is very unstable effect on CCR1036.
For example, now I have 500 Mbit real traffic. If I miss CCR1036 through UDP packets of 64-1000 Byte, then begin the hard packet loss.

17

hi

just to add my quick findings

so I have 3 sites all attached by 10G SFP+ (fibre runs)

site A
test box is centos 6.5 vm vmnext3 → dell 8024F 10G switch … doing routing as well → 10G SFP to Site C → dell 8132F switch router → another test box centos 6.5 vm vmnext3

Site O
test box centos 6.5 vm vmnext3 → 10G dell 8024F switch (switching only) → CCR1036-8G-2S+ (actually 2 in vrrp mode). → switch (same) → 10G SFP to site A or C [stp determines] @ site C → dell 8132F switch router → another test box centos 6.5 vm vmnext3 (only 1 test server at site C)


the CCR1036-8G-2S+ are connected via 2 SFP+ 10G cables to the 8024F’s I have 10G link speed. I have set them up as LACP and there are vlan trunked.
the underlying trunk is 9216 l2mtu but most of the vlans are 1500 mtu. 1 vlan is 9k but it is not traverse in these tests
the outbound hash is l2-l3 this makes it use different slave ports for inbound and outbound for the test


i use iperf

site C box is iperf server
iperf -s <<< tcp test
iperf -s -u -b 20480M -U <<< singe thread udp


clients
iperf -c 10.33.95.21 -u -b 20480M -U -i 2 -t 60 << udp
iperf -c 10.33.95.21 -t 60 << tcp


I have tuned the linux box’s 10000 txqueue length sack and window size (3M)


site a → site c 8-9Gb/s
site c → site c 9Gb/s … different ip network so its routed through the dell router/switch

site O → site c 800Mb/s is the max I can get with tcp
site O → site c 4-6G/b if I use UDP
if i setup a dual test its around 5G/b

I think my udp packets are not 1-1K… not sure how to set that right now and I am not sure it tests that…



just thought I would add my 2c.

My plan had been to remove the routing off the dell switchs. they have the functions but they are not very good at a lot of things.

I still have to investigate weather is the bonded SFPs…

edit:
I have firewall rules set in place on most interfaces

edit:
ran some TCP tests in parrallel from the same machine 10 streams got me 1.9G
20 tcp streams got me to 6.29Gb/s

again i think with full packets ~1500


edit..
tried the inbuild bandwithtest… most I could get out of it was 500Mb/s …

I bought 2 of these… going to be very disappointed if they can’t route at close to 10Gb/s …

18

up!

19

This is pretty sad… so this is a cross connect between 2 CCR1036 nothing inbetween ether1 to ether1 …

[admin@ybortr2] /tool> bandwidth-test 192.168.0.1 protocol=tcp
status: running
duration: 1m10s
rx-current: 51.8Mbps
rx-10-second-average: 49.5Mbps
rx-total-average: 61.1Mbps
random-data: no
direction: receive

[admin@ybortr2] /tool> bandwidth-test 192.168.0.1 protocol=tcp local-tx-speed=1024000000 remote-tx-speed=1024000000
status: running
duration: 39s
rx-current: 47.0Mbps
rx-10-second-average: 51.4Mbps
rx-total-average: 68.4Mbps
random-data: no
direction: receive


But I checked cpu
1cpu at 100% so not power CPU for doing speed test…

Really getting to not like these boxs…

edit …
just checked
/interface ethernet> print detail
Flags: X - disabled, R - running, S - slave
0 R ;;; cross connect to ybortr1
name=“ether1” default-name=“ether1” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5C orig-mac-address=D4:CA:6D:E1:CB:5C arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

1 name=“ether2” default-name=“ether2” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5D orig-mac-address=D4:CA:6D:E1:CB:5D arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

2 name=“ether3” default-name=“ether3” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5E orig-mac-address=D4:CA:6D:E1:CB:5E arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

3 name=“ether4” default-name=“ether4” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5F orig-mac-address=D4:CA:6D:E1:CB:5F arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

4 name=“ether5” default-name=“ether5” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:60 orig-mac-address=D4:CA:6D:E1:CB:60 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

5 name=“ether6” default-name=“ether6” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:61 orig-mac-address=D4:CA:6D:E1:CB:61 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

6 name=“ether7” default-name=“ether7” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:62 orig-mac-address=D4:CA:6D:E1:CB:62 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

7 name=“ether8” default-name=“ether8” mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:63 orig-mac-address=D4:CA:6D:E1:CB:63 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps


Seems like they are only 100M ports


http://routerboard.com/CCR1036-8G-2Splus states they are 1G/100/10Mb ports..

atleast the SFP+ are showing up as 10Gb

edit !

after setting the ports to 1G … better

/tool> bandwidth-test 192.168.0.1 protocol=udp local-tx-speed=1024000000 remote-tx-speed=1024000000
status: running
duration: 8s
rx-current: 975.2Mbps
rx-10-second-average: 966.2Mbps
rx-total-average: 966.2Mbps
lost-packets: 2775
random-data: no
direction: receive
rx-size: 1500


bandwidth-test 192.168.0.1 protocol=tcp
status: running
duration: 21s
rx-current: 47.4Mbps
rx-10-second-average: 55.9Mbps
rx-total-average: 83.6Mbps
random-data: no
direction: receive


So is there an issue with TCP over UDP ???

20

btest is unreliable. Get two good PC’s and test THROUGH the CCR using the tool called iperf.