Community discussions

MUM Europe 2020
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

CCR 1036. High load loses packets. Big problem.

Sun Feb 16, 2014 4:22 pm

CCR 1036. High load loses packets.
Ticket#2014021766000422
[admin@MikroTik] > system resource print
             uptime: 5d9h3m25s
            version: 6.9
         build-time: Jan/31/2014 11:18:19
        free-memory: 3243.1MiB
       total-memory: 3969.0MiB
                cpu: tilegx
          cpu-count: 36
      cpu-frequency: 1200MHz
           cpu-load: 18%
     free-hdd-space: 890.4MiB
    total-hdd-space: 1024.0MiB
  architecture-name: tile
         board-name: CCR1036-12G-4S
           platform: MikroTik
You do not have the required permissions to view the files attached to this post.
Last edited by morf on Wed Feb 19, 2014 3:47 pm, edited 1 time in total.
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2303
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: CCR 1036. High load loses packets. Big problem.

Sun Feb 16, 2014 4:54 pm

Maybe a problem somewhere down the road? Have you tried tracert?
We have a flow rate of 700Mbit and ping is 0ms 0% packet loss
You do not have the required permissions to view the files attached to this post.
LAN, FTTx, Wireless. ISP operator
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Sun Feb 16, 2014 5:06 pm

Delete
Last edited by morf on Mon Feb 17, 2014 10:32 am, edited 1 time in total.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Sun Feb 16, 2014 7:21 pm

I did downgrade to version 6.7. All normal. Version 6.9 with a big problem.
I'm lying, then downgrade to 6.7 was a little better, but still very noticeable packet loss of 1-2%.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Sun Feb 16, 2014 10:01 pm

I think I found the cause of the loss. If after CCR let UDP traffic packages from 64 to 500 bytes, the CCR begins blunt and lose packets!
And so begin:

1. I will try to convey through CCR UDP packet size 64-500 bytes.
Running ping to the resource that is available through the CCR 1036.
Test off.

2. Test on. UDP packets of 64 bytes. Test makes to external mikrotik. Not on CCR.
The test showed that CCR not withstand the load and starts to drop packets.
You do not have the required permissions to view the files attached to this post.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Mon Feb 17, 2014 10:19 am

I did a visual test demonstrating the problem. Please developers pay attention to it.
You do not have the required permissions to view the files attached to this post.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Mon Feb 17, 2014 7:37 pm

Next.
The problem occurs when testing UDP packets in bytes 750-1000.
We can conclude that the CCR1036 weak performance when passing UDP traffic. With TCP-packets such problem was not found.
Is it possible to fix this by optimizing ROS?
You do not have the required permissions to view the files attached to this post.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 19, 2014 12:55 pm

Anyone? Anything?
Normis?
 
2jarek
Member Candidate
Member Candidate
Posts: 136
Joined: Thu May 17, 2007 3:28 pm
Location: Poland

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 19, 2014 2:00 pm

Maybe try disable/enable fast patch.
IP>Settings.....
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 19, 2014 3:45 pm

Maybe try disable/enable fast patch.
IP>Settings.....
2jarek, thanks for the answer!
Sorry, did not help. Tried all options in IP-Settings->...
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 19, 2014 3:47 pm

Ticket#2014021766000422
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Fri Feb 21, 2014 9:30 pm

up!
 
bryans2k
just joined
Posts: 21
Joined: Fri Apr 26, 2013 6:10 am

Re: CCR 1036. High load loses packets. Big problem.

Fri Feb 21, 2014 10:30 pm

I'm seeing the same problem with 4 of my CCR's on 6.9. Please update when fixed.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Sat Feb 22, 2014 6:50 am

I'm seeing the same problem with 4 of my CCR's on 6.9. Please update when fixed.
I can not get an answer from the support. Ticket created and silence ..
 
AlexS
Member Candidate
Member Candidate
Posts: 259
Joined: Thu Oct 10, 2013 7:21 am

Re: CCR 1036. High load loses packets. Big problem.

Sat Feb 22, 2014 7:13 am

Hi

if I read right this is just for UDP packet ?
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Sat Feb 22, 2014 10:14 am

Hi

if I read right this is just for UDP packet ?
Hello!
Yes, passing UDP packets is very unstable effect on CCR1036.
For example, now I have 500 Mbit real traffic. If I miss CCR1036 through UDP packets of 64-1000 Byte, then begin the hard packet loss.
 
AlexS
Member Candidate
Member Candidate
Posts: 259
Joined: Thu Oct 10, 2013 7:21 am

Re: CCR 1036. High load loses packets. Big problem.

Sat Feb 22, 2014 11:07 pm

hi

just to add my quick findings

so I have 3 sites all attached by 10G SFP+ (fibre runs)

site A
test box is centos 6.5 vm vmnext3 -> dell 8024F 10G switch ... doing routing as well -> 10G SFP to Site C -> dell 8132F switch router -> another test box centos 6.5 vm vmnext3

Site O
test box centos 6.5 vm vmnext3 -> 10G dell 8024F switch (switching only) -> CCR1036-8G-2S+ (actually 2 in vrrp mode). -> switch (same) -> 10G SFP to site A or C [stp determines] @ site C -> dell 8132F switch router -> another test box centos 6.5 vm vmnext3 (only 1 test server at site C)


the CCR1036-8G-2S+ are connected via 2 SFP+ 10G cables to the 8024F's I have 10G link speed. I have set them up as LACP and there are vlan trunked.
the underlying trunk is 9216 l2mtu but most of the vlans are 1500 mtu. 1 vlan is 9k but it is not traverse in these tests
the outbound hash is l2-l3 this makes it use different slave ports for inbound and outbound for the test


i use iperf

site C box is iperf server
iperf -s <<< tcp test
iperf -s -u -b 20480M -U <<< singe thread udp


clients
iperf -c 10.33.95.21 -u -b 20480M -U -i 2 -t 60 << udp
iperf -c 10.33.95.21 -t 60 << tcp


I have tuned the linux box's 10000 txqueue length sack and window size (3M)


site a -> site c 8-9Gb/s
site c -> site c 9Gb/s .... different ip network so its routed through the dell router/switch

site O -> site c 800Mb/s is the max I can get with tcp
site O -> site c 4-6G/b if I use UDP
if i setup a dual test its around 5G/b

I think my udp packets are not 1-1K... not sure how to set that right now and I am not sure it tests that...



just thought I would add my 2c.

My plan had been to remove the routing off the dell switchs. they have the functions but they are not very good at a lot of things.

I still have to investigate weather is the bonded SFPs...

edit:
I have firewall rules set in place on most interfaces

edit:
ran some TCP tests in parrallel from the same machine 10 streams got me 1.9G
20 tcp streams got me to 6.29Gb/s

again i think with full packets ~1500


edit..
tried the inbuild bandwithtest.... most I could get out of it was 500Mb/s ...

I bought 2 of these... going to be very disappointed if they can't route at close to 10Gb/s ...
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Tue Feb 25, 2014 4:00 pm

up!
 
AlexS
Member Candidate
Member Candidate
Posts: 259
Joined: Thu Oct 10, 2013 7:21 am

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 26, 2014 2:24 am

This is pretty sad... so this is a cross connect between 2 CCR1036 nothing inbetween ether1 to ether1 ....

[admin@ybortr2] /tool> bandwidth-test 192.168.0.1 protocol=tcp
status: running
duration: 1m10s
rx-current: 51.8Mbps
rx-10-second-average: 49.5Mbps
rx-total-average: 61.1Mbps
random-data: no
direction: receive
-
[admin@ybortr2] /tool> bandwidth-test 192.168.0.1 protocol=tcp local-tx-speed=1024000000 remote-tx-speed=1024000000
status: running
duration: 39s
rx-current: 47.0Mbps
rx-10-second-average: 51.4Mbps
rx-total-average: 68.4Mbps
random-data: no
direction: receive


But I checked cpu
1cpu at 100% so not power CPU for doing speed test...

Really getting to not like these boxs.........

edit ....
just checked
/interface ethernet> print detail
Flags: X - disabled, R - running, S - slave
0 R ;;; cross connect to ybortr1
name="ether1" default-name="ether1" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5C orig-mac-address=D4:CA:6D:E1:CB:5C arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

1 name="ether2" default-name="ether2" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5D orig-mac-address=D4:CA:6D:E1:CB:5D arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

2 name="ether3" default-name="ether3" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5E orig-mac-address=D4:CA:6D:E1:CB:5E arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

3 name="ether4" default-name="ether4" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:5F orig-mac-address=D4:CA:6D:E1:CB:5F arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

4 name="ether5" default-name="ether5" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:60 orig-mac-address=D4:CA:6D:E1:CB:60 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

5 name="ether6" default-name="ether6" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:61 orig-mac-address=D4:CA:6D:E1:CB:61 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

6 name="ether7" default-name="ether7" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:62 orig-mac-address=D4:CA:6D:E1:CB:62 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps

7 name="ether8" default-name="ether8" mtu=1500 l2mtu=1590 mac-address=D4:CA:6D:E1:CB:63 orig-mac-address=D4:CA:6D:E1:CB:63 arp=enabled auto-negotiation=yes full-duplex=yes
speed=100Mbps


Seems like they are only 100M ports


http://routerboard.com/CCR1036-8G-2Splus states they are 1G/100/10Mb ports..

atleast the SFP+ are showing up as 10Gb

edit !

after setting the ports to 1G ... better

/tool> bandwidth-test 192.168.0.1 protocol=udp local-tx-speed=1024000000 remote-tx-speed=1024000000
status: running
duration: 8s
rx-current: 975.2Mbps
rx-10-second-average: 966.2Mbps
rx-total-average: 966.2Mbps
lost-packets: 2775
random-data: no
direction: receive
rx-size: 1500


bandwidth-test 192.168.0.1 protocol=tcp
status: running
duration: 21s
rx-current: 47.4Mbps
rx-10-second-average: 55.9Mbps
rx-total-average: 83.6Mbps
random-data: no
direction: receive


So is there an issue with TCP over UDP ???
 
onnoossendrijver
Member
Member
Posts: 418
Joined: Mon Jul 14, 2008 11:10 am
Location: The Netherlands

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 26, 2014 9:46 am

btest is unreliable. Get two good PC's and test THROUGH the CCR using the tool called iperf.
Linux/network engineer: ITIL, LPI1, CCNA R+S, CCNP R+S, JNCIA, JNCIS-SEC
 
AlexS
Member Candidate
Member Candidate
Posts: 259
Joined: Thu Oct 10, 2013 7:21 am

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 26, 2014 9:51 am

btest is unreliable. Get two good PC's and test THROUGH the CCR using the tool called iperf.
If you look 2 posts up you will see that's what I have done similar results
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 26, 2014 10:09 am

btest is unreliable. Get two good PC's and test THROUGH the CCR using the tool called iperf.
I did these tests. See post above with pictures. I wrote everything in detail.
 
morf
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 26, 2014 2:08 pm

I solved this problem. She was associated with the hotspot.
When you create a hotspot on an interface, you create a dynamic rule with queues simple (hs-<interface_name>) Queue Type = hotspot-default. I put Kind = none. Helped. Traffic began to pass unscathed.
 
AlexS
Member Candidate
Member Candidate
Posts: 259
Joined: Thu Oct 10, 2013 7:21 am

Re: CCR 1036. High load loses packets. Big problem.

Wed Feb 26, 2014 2:28 pm

Hmm I dont have any hotspots nor any queues
 
carcara
just joined
Posts: 13
Joined: Fri May 25, 2007 3:10 pm

Re: CCR 1036. High load loses packets. Big problem.

Thu Aug 21, 2014 4:53 pm

I have the a similar problem, i'm using pppoe, and i'm on 6.18 version.
 
laith43d
just joined
Posts: 16
Joined: Sun Jun 30, 2013 1:40 pm

Re: CCR 1036. High load loses packets. Big problem.

Mon Sep 08, 2014 2:33 pm

I have pppoe with many queues and nat, the same problem here with 350 Mbps and 1400 online pppoe sessions.

The problem is happening once the BW hits a high rate above 200 Mbps, below that seems to be normal.

OS version is 6.19, HW is CCR1036-8G-2S+ with 16 GB of ram

Thanks

Who is online

Users browsing this forum: jebz and 161 guests