CC1036 Performance issues

RouterOS General
1

Hi

I am starting a new thread from
CCR 1036. High load loses packets. Big problem. as the OP seems to have solved their problem.

I am having some issue pushing 1 TCP stream pas 1Gb/s through these devices.

env

2 x 8024F 10G switches stacked
2 x cc1036 cross connect to 2 10G ports / device. This is done as a LACP. I also use vrrp

/interface bonding
add mode=802.3ad mtu=9000 name=bond0 slaves=sfp-plus1,sfp-plus2 transmit-hash-policy=layer-2-and-3


interface port-channel X
description "rtr lag"
hashing-mode 6
switchport mode trunk
mtu 9216
exit

for sfp-plus1&2 I have l2mtu at 9216


I have previously used 3 test box's with iperf. I use ESX 5.5i and vmxnet3 with centos 6.5


2 clients and 1 server.

I have not been able to get above 1Gb/s for a single tcp stream, I can sometimes push UDP up to 6.5Gb/s . I have been able to push tcp by doing multiple streams.

I am now just testing with 2 VM

client to server same VLAN.... 10Gb's tcp and udp

if I route through the CCR, I get crappy performance. I have monitored interfaces and can see the packets are getting distributed.

I am at a loss, this it no where near the capacity talked about on the spec sheet







from previous post

so I have 3 sites all attached by 10G SFP+ (fibre runs)

site A
test box is centos 6.5 vm vmnext3 -> dell 8024F 10G switch ... doing routing as well -> 10G SFP to Site C -> dell 8132F switch router -> another test box centos 6.5 vm vmnext3

Site O
test box centos 6.5 vm vmnext3 -> 10G dell 8024F switch (switching only) -> CCR1036-8G-2S+ (actually 2 in vrrp mode). -> switch (same) -> 10G SFP to site A or C [stp determines] @ site C -> dell 8132F switch router -> another test box centos 6.5 vm vmnext3 (only 1 test server at site C)


the CCR1036-8G-2S+ are connected via 2 SFP+ 10G cables to the 8024F's I have 10G link speed. I have set them up as LACP and there are vlan trunked.
the underlying trunk is 9216 l2mtu but most of the vlans are 1500 mtu. 1 vlan is 9k but it is not traverse in these tests
the outbound hash is l2-l3 this makes it use different slave ports for inbound and outbound for the test


i use iperf

site C box is iperf server
iperf -s <<< tcp test
iperf -s -u -b 20480M -U <<< singe thread udp


clients
iperf -c 10.33.95.21 -u -b 20480M -U -i 2 -t 60 << udp
iperf -c 10.33.95.21 -t 60 << tcp


I have tuned the linux box's 10000 txqueue length sack and window size (3M)


site a -> site c 8-9Gb/s
site c -> site c 9Gb/s .... different ip network so its routed through the dell router/switch

site O -> site c 800Mb/s is the max I can get with tcp
site O -> site c 4-6G/b if I use UDP
if i setup a dual test its around 5G/b

I think my udp packets are not 1-1K... not sure how to set that right now and I am not sure it tests that...



just thought I would add my 2c.

My plan had been to remove the routing off the dell switchs. they have the functions but they are not very good at a lot of things.

I still have to investigate weather is the bonded SFPs...

edit:
I have firewall rules set in place on most interfaces

edit:
ran some TCP tests in parrallel from the same machine 10 streams got me 1.9G
20 tcp streams got me to 6.29Gb/s

again i think with full packets ~1500


edit..
tried the inbuild bandwithtest.... most I could get out of it was 500Mb/s ...

I bought 2 of these... going to be very disappointed if they can't route at close to 10Gb/s ...

2

[root@ybonas ~]# iperf -c 10.172.208.100 -t 60

Client connecting to 10.172.208.100, TCP port 5001
TCP window size: 3.00 MByte (default)

[ 3] local 10.172.203.34 port 56453 connected with 10.172.208.100 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.66 GBytes 811 Mbits/sec
[root@ybonas ~]# iperf -c 10.172.208.100 -t 60 -U -b 10240M -u
WARNING: option -b implies udp testing

Client connecting to 10.172.208.100, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 32.0 MByte (default)

[ 3] local 10.172.203.34 port 59620 connected with 10.172.208.100 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 36.7 GBytes 5.25 Gbits/sec
[ 3] Sent 26807630 datagrams
[ 3] Server Report:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-60.1 sec 27.9 GBytes 3.99 Gbits/sec 0.692 ms 6452939/26807629 (24%)
[ 3] 0.0-60.1 sec 18310497 datagrams received out-of-order

3

Bump

No one else having this problem

4

A single thread will never be faster than the speed of a single link.

5

Interesting when i remove the cc1036 and use the l3 routing of dell 8024f I get up around 9Gb/s
same iperf test…

if I vlan direct between the boxes I get around 9Gb/s

6

Bump

Pretty sad, I was ready to buy 4 more of these, but so far nothing back from support, nothing in the forums.

The box’s look like they could be quiet good, but it seems like they just can’t perform.

I am guessing for now its a software problem as I am push 8Gbs+ udp packets through them, just not tcp.

7

Hi,

I’d suggest doing a sup-out while the stream is running for support.

Also, when the process is running, do you see a single core 100% maxxed out?

Try kick up 10 different TCP streams at the same time (run iperf in screen), does this allow closer to the performance you were requesting?

8

Remember that with high-bandwidth links the latency becomes very important. Is latency higher with CCR instead of Dell?

9

Agreed, but I have


Vm → switch → CCR

they are in the same rack.

If I remove the CCR and use routing in the dell switches and choice a destination VM that is 15Km away I can still get 9.xGb/s with a single TCP stream.

I need/want high through put on single streams.

to me it seems like tcp is getting inspected and that is capping the speed at < 1Gb/s

A

10

I do not understand the practical application of this test - 2x 10G bonded together and single TCP connection. why? use 10G switch for this application. CCR suppose to have many users and many connections.

I never bonded 10G interfaces before, check if removing that solves the problem

11

I’d use em for mpls/vpls if it weren’t for that limitation. Iscsi from datacenter to datacenter, easy 3-4 gig single tcp stream. Vmotion with hosts heavy on ram. 1gig is nothing for network bandwidth these days.