Poor (ridiculously) performance on two CCR1072

RouterOS General
1

We have a poor performance between 2 CCR 1072… we are an FTTH ISP, we have 5000 PPPoE tunnels on each CCR and 5000 correspondent queues. But we think that no effects because CPU is near 5% - 10%.

WAT?!

Bandwith test with direct 10G connected, from one CCR to another, DIRECT CABLE, no switches no nothing.
Same configuration on both:
L2MTU: 9216
MTU: 9000

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword
status: running
duration: 11s
rx-current: 484.9Mbps
rx-10-second-average: 690.3Mbps
rx-total-average: 700.7Mbps
random-data: no
direction: receive

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword direction=transmit
status: running
duration: 9s
tx-current: 622.0Mbps
tx-10-second-average: 785.6Mbps
tx-total-average: 785.6Mbps
random-data: no
direction: transmit


Both supout.rif are uploaded to my Mikrotik account, login is nadeu

Regards

2

http://forum.mikrotik.com/t/v6-36rc-release-candidate-is-released-wireless-fp-package-is-discontinued/97337/32

Make a real test where traffic is forwarded and no BW test tool

3

I think bandwidth-test only uses one core of the tile processor, so that test isn’t valid. Chances are good that whatever core the test is running on is hitting 100% which limits how much traffic it can generate and respond to.

A better test is to get a pair of PCs that can handle 10Gbps throughput and run iperf between them. Test directly between the two computers first as a control, and then connect one to one CCR and the other to the other, and THEN run the test this way.

4

Our FTTH clients cannot reach more than 150Mbps of upload, we have the same setup with CCR1036 and run properly.
We don’t know why not run properly with 1072.

With CCR1036 our clients do a Speedtest in Speedtest.net and get 500/500, like queue indicates. And the clients that have a 1G/1G get 900/900.

Why now service is degradated?

Regards

5

With MikroTik:

root@cdn02:~# iperf -c 10.1.1.197 -i1 -t 10 -m

Client connecting to 10.1.1.197, TCP port 5001
TCP window size: 4.00 MByte (default)

[ 3] local 10.1.1.199 port 46395 connected with 10.1.1.197 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 29.1 MBytes 244 Mbits/sec
[ 3] 1.0- 2.0 sec 30.6 MBytes 257 Mbits/sec
[ 3] 2.0- 3.0 sec 33.5 MBytes 281 Mbits/sec
[ 3] 3.0- 4.0 sec 32.4 MBytes 272 Mbits/sec
[ 3] 4.0- 5.0 sec 32.9 MBytes 276 Mbits/sec
[ 3] 5.0- 6.0 sec 32.1 MBytes 269 Mbits/sec
[ 3] 6.0- 7.0 sec 30.4 MBytes 255 Mbits/sec
[ 3] 7.0- 8.0 sec 24.6 MBytes 207 Mbits/sec
[ 3] 8.0- 9.0 sec 31.9 MBytes 267 Mbits/sec
[ 3] 9.0-10.0 sec 29.9 MBytes 251 Mbits/sec
[ 3] 0.0-10.0 sec 308 MBytes 258 Mbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)


Without MikroTik, two 1G connected between servers throught Quanta switch.

root@cdn02:~# iperf -c 10.0.88.201 -i1 -t 10 -m

Client connecting to 10.0.88.201, TCP port 5001
TCP window size: 4.00 MByte (default)

[ 3] local 10.0.88.202 port 34982 connected with 10.0.88.201 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 116 MBytes 974 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 941 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 941 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 940 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 943 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 941 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 941 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 941 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 940 Mbits/sec
[ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
root@cdn02:~#


As requested. Between 2 E3-1245 V3 with 10G HotLava passing by router.

Regards

6

Okay - that rules out the bandwidth-test limitation.
Unfortunately, I don’t have any CCR to work with so I’m going to let someone else take a stab at this.

One thing I’ve seen mentioned, though, is something to do with the interface queues being set to hw-only. Perhaps fastpath is not working for you?

7

bad testing procedure i have obtained more than that on a ccr1009

here are test of ccr1072 reaching 80gpbs

http://www.stubarea51.net/2015/10/09/mikrotik-ccr1072-1g-8s-review-part-3-80-gbps-throughput-testing/

8

Hello,
The test is launched from Winbox, and if you look at iperf gives the same result, even worse, so I do not think it’s anything that this evil in our sense. I understand that in the video StubArea51 works perfect, so we buy two drives us, but the reality is that currently we are disappointing a lot and are generating a thousand problems.

Sincerly,
Pau

9

Hardware queues are enabled.

We disabled fastpath.

Regards,
Pau.

10

Maybe incompatible SFP module, what SFP modules you use?
Damaged fibre cable?
Damaged port? Try switching to different ports.

11

ICMAN1:

ICMAN2:

Regards

12

Aggreggated traffic (multiple connections) is OK, we got peaks of 1.3Gbps, I think that issue it’s with a single connection.

Regards

13

again you are doing things wrong

you are stating the device have a ridiculous performance

but you dont document your test

frits you make a test with btest

then iperf but you dont state how the test was made, and what configuration have the device at the moment of the test

off course you are having troubles

well deserved problems

please document your test properly

14

root@cdn02:~# iperf -c 10.1.1.197 -i1 -t 10 -m

Client connecting to 10.1.1.197, TCP port 5001
TCP window size: 4.00 MByte (default)

[ 3] local 10.1.1.199 port 46395 connected with 10.1.1.197 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 29.1 MBytes 244 Mbits/sec
[ 3] 1.0- 2.0 sec 30.6 MBytes 257 Mbits/sec
[ 3] 2.0- 3.0 sec 33.5 MBytes 281 Mbits/sec
[ 3] 3.0- 4.0 sec 32.4 MBytes 272 Mbits/sec
[ 3] 4.0- 5.0 sec 32.9 MBytes 276 Mbits/sec
[ 3] 5.0- 6.0 sec 32.1 MBytes 269 Mbits/sec
[ 3] 6.0- 7.0 sec 30.4 MBytes 255 Mbits/sec
[ 3] 7.0- 8.0 sec 24.6 MBytes 207 Mbits/sec
[ 3] 8.0- 9.0 sec 31.9 MBytes 267 Mbits/sec
[ 3] 9.0-10.0 sec 29.9 MBytes 251 Mbits/sec
[ 3] 0.0-10.0 sec 308 MBytes 258 Mbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)


Esto no parece claro? Es un test entre dos servidores pasando por 2 Mikrotiks con una ruta estatica, los dos Mikrotiks tienen un 10G anteriormente documentado.

Para responder sin sentidos no respondas; estoy pidiendo ayuda y solo haces que decirme que falta documentación. :slight_smile:

I follow the ticket with Mikrotik support team, you're not from Mikrotik.

Thank you,
Pau

15

FYI:

16

you have to look for some bottleneck somewhere on your testing environment

for your reference there are a test i conducted on a ccr1009, in this result i think my environment test are limiting the results but i dont have more hardware to test
iperf.jpg
the topology of the test
topology.jpg
the config of ccr was only a bridge between the two ports in fast path mode, mtu 1500 (virtualbox do not support jumbo frames) unfortunately i am limited to 1gbit testing because i dont have more pci express slots for more nics

17

Contact you via private message.

18

Yesterday we analyze it and get the same conclusions: CCR1072 is a unfinished product.
We are awaiting for a solution.

Regards

19

Up please! This is a nightmare.
Regards

20

Did you check RX and TX stats on both links? Inside the interface where errors are counted, look for errors, crc problems this would lead to a faulty line or faulty SFP.

Did you try a test with the router having no configurations whatsoever?

Did you try with different ports, different sfp?

I know people with this particular model. None of them presenting issues like this.