We have a poor performance between 2 CCR 1072... we are an FTTH ISP, we have 5000 PPPoE tunnels on each CCR and 5000 correspondent queues. But we think that no effects because CPU is near 5% - 10%.

WAT?!

Bandwith test with direct 10G connected, from one CCR to another, DIRECT CABLE, no switches no nothing.

Same configuration on both:
L2MTU: 9216
MTU: 9000

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword
status: running
duration: 11s
rx-current: 484.9Mbps
rx-10-second-average: 690.3Mbps
rx-total-average: 700.7Mbps
random-data: no
direction: receive

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword direction=transmit
status: running
duration: 9s
tx-current: 622.0Mbps
tx-10-second-average: 785.6Mbps
tx-total-average: 785.6Mbps
random-data: no
direction: transmit


Both supout.rif are uploaded to my Mikrotik account, login is nadeu

Regards

http://forum.mikrotik.com/viewtopic.php ... 65#p534265

Make a real test where traffic is forwarded and no BW test tool

I think bandwidth-test only uses one core of the tile processor, so that test isn't valid. Chances are good that whatever core the test is running on is hitting 100% which limits how much traffic it can generate and respond to.

A better test is to get a pair of PCs that can handle 10Gbps throughput and run iperf between them. Test directly between the two computers first as a control, and then connect one to one CCR and the other to the other, and THEN run the test this way.

Our FTTH clients cannot reach more than 150Mbps of upload, we have the same setup with CCR1036 and run properly.
We don't know why not run properly with 1072.

With CCR1036 our clients do a Speedtest in Speedtest.net and get 500/500, like queue indicates. And the clients that have a 1G/1G get 900/900.

Why now service is degradated?

Regards

With MikroTik:

root@cdn02:~# iperf -c 10.1.1.197 -i1 -t 10 -m
------------------------------------------------------------
Client connecting to 10.1.1.197, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ 3] local 10.1.1.199 port 46395 connected with 10.1.1.197 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 29.1 MBytes 244 Mbits/sec
[ 3] 1.0- 2.0 sec 30.6 MBytes 257 Mbits/sec
[ 3] 2.0- 3.0 sec 33.5 MBytes 281 Mbits/sec
[ 3] 3.0- 4.0 sec 32.4 MBytes 272 Mbits/sec
[ 3] 4.0- 5.0 sec 32.9 MBytes 276 Mbits/sec
[ 3] 5.0- 6.0 sec 32.1 MBytes 269 Mbits/sec
[ 3] 6.0- 7.0 sec 30.4 MBytes 255 Mbits/sec
[ 3] 7.0- 8.0 sec 24.6 MBytes 207 Mbits/sec
[ 3] 8.0- 9.0 sec 31.9 MBytes 267 Mbits/sec
[ 3] 9.0-10.0 sec 29.9 MBytes 251 Mbits/sec
[ 3] 0.0-10.0 sec 308 MBytes 258 Mbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)


Without MikroTik, two 1G connected between servers throught Quanta switch.

root@cdn02:~# iperf -c 10.0.88.201 -i1 -t 10 -m
------------------------------------------------------------
Client connecting to 10.0.88.201, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ 3] local 10.0.88.202 port 34982 connected with 10.0.88.201 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 116 MBytes 974 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 941 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec
[ 3] 3.0- 4.0 sec 112 MBytes 941 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 940 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 943 Mbits/sec
[ 3] 6.0- 7.0 sec 112 MBytes 941 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 941 Mbits/sec
[ 3] 8.0- 9.0 sec 112 MBytes 941 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 940 Mbits/sec
[ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
root@cdn02:~#


As requested. Between 2 E3-1245 V3 with 10G HotLava passing by router.

Regards

Okay - that rules out the bandwidth-test limitation.
Unfortunately, I don't have any CCR to work with so I'm going to let someone else take a stab at this.

One thing I've seen mentioned, though, is something to do with the interface queues being set to hw-only. Perhaps fastpath is not working for you?

We have a poor performance between 2 CCR 1072... we are an FTTH ISP, we have 5000 PPPoE tunnels on each CCR and 5000 correspondent queues. But we think that no effects because CPU is near 5% - 10%.

WAT?!

Bandwith test with direct 10G connected, from one CCR to another, DIRECT CABLE, no switches no nothing.

Same configuration on both:
L2MTU: 9216
MTU: 9000

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword
status: running
duration: 11s
rx-current: 484.9Mbps
rx-10-second-average: 690.3Mbps
rx-total-average: 700.7Mbps
random-data: no
direction: receive

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword direction=transmit
status: running
duration: 9s
tx-current: 622.0Mbps
tx-10-second-average: 785.6Mbps
tx-total-average: 785.6Mbps
random-data: no
direction: transmit


Both supout.rif are uploaded to my Mikrotik account, login is nadeu

Regards

bad testing procedure i have obtained more than that on a ccr1009

here are test of ccr1072 reaching 80gpbs

http://www.stubarea51.net/2015/10/09/mi ... t-testing/

We have a poor performance between 2 CCR 1072... we are an FTTH ISP, we have 5000 PPPoE tunnels on each CCR and 5000 correspondent queues. But we think that no effects because CPU is near 5% - 10%.

WAT?!

Bandwith test with direct 10G connected, from one CCR to another, DIRECT CABLE, no switches no nothing.

Same configuration on both:
L2MTU: 9216
MTU: 9000

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword
status: running
duration: 11s
rx-current: 484.9Mbps
rx-10-second-average: 690.3Mbps
rx-total-average: 700.7Mbps
random-data: no
direction: receive

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword direction=transmit
status: running
duration: 9s
tx-current: 622.0Mbps
tx-10-second-average: 785.6Mbps
tx-total-average: 785.6Mbps
random-data: no
direction: transmit


Both supout.rif are uploaded to my Mikrotik account, login is nadeu

Regards

bad testing procedure i have obtained more than that on a ccr1009

here are test of ccr1072 reaching 80gpbs

http://www.stubarea51.net/2015/10/09/mi ... t-testing/

Hello,
The test is launched from Winbox, and if you look at iperf gives the same result, even worse, so I do not think it's anything that this evil in our sense. I understand that in the video StubArea51 works perfect, so we buy two drives us, but the reality is that currently we are disappointing a lot and are generating a thousand problems.

Sincerly,
Pau

Hardware queues are enabled.

We disabled fastpath.

Regards,
Pau.

Maybe incompatible SFP module, what SFP modules you use?
Damaged fibre cable?
Damaged port? Try switching to different ports.

ICMAN1:


ICMAN2:


Regards

Aggreggated traffic (multiple connections) is OK, we got peaks of 1.3Gbps, I think that issue it's with a single connection.

Regards

We have a poor performance between 2 CCR 1072... we are an FTTH ISP, we have 5000 PPPoE tunnels on each CCR and 5000 correspondent queues. But we think that no effects because CPU is near 5% - 10%.

WAT?!

Bandwith test with direct 10G connected, from one CCR to another, DIRECT CABLE, no switches no nothing.

Same configuration on both:
L2MTU: 9216
MTU: 9000

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword
status: running
duration: 11s
rx-current: 484.9Mbps
rx-10-second-average: 690.3Mbps
rx-total-average: 700.7Mbps
random-data: no
direction: receive

[admin@ICMAN02] > /tool bandwidth-test protocol=tcp address=10.1.2.7 user=admin password=somepassword direction=transmit
status: running
duration: 9s
tx-current: 622.0Mbps
tx-10-second-average: 785.6Mbps
tx-total-average: 785.6Mbps
random-data: no
direction: transmit


Both supout.rif are uploaded to my Mikrotik account, login is nadeu

Regards

bad testing procedure i have obtained more than that on a ccr1009

here are test of ccr1072 reaching 80gpbs

http://www.stubarea51.net/2015/10/09/mi ... t-testing/

Hello,
The test is launched from Winbox, and if you look at iperf gives the same result, even worse, so I do not think it's anything that this evil in our sense. I understand that in the video StubArea51 works perfect, so we buy two drives us, but the reality is that currently we are disappointing a lot and are generating a thousand problems.

Sincerly,
Pau

again you are doing things wrong

you are stating the device have a ridiculous performance

but you dont document your test

frits you make a test with btest

then iperf but you dont state how the test was made, and what configuration have the device at the moment of the test

off course you are having troubles

well deserved problems

please document your test properly

again you are doing things wrong

you are stating the device have a ridiculous performance

but you dont document your test

frits you make a test with btest

then iperf but you dont state how the test was made, and what configuration have the device at the moment of the test

off course you are having troubles

well deserved problems

please document your test properly

root@cdn02:~# iperf -c 10.1.1.197 -i1 -t 10 -m
------------------------------------------------------------
Client connecting to 10.1.1.197, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ 3] local 10.1.1.199 port 46395 connected with 10.1.1.197 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 29.1 MBytes 244 Mbits/sec
[ 3] 1.0- 2.0 sec 30.6 MBytes 257 Mbits/sec
[ 3] 2.0- 3.0 sec 33.5 MBytes 281 Mbits/sec
[ 3] 3.0- 4.0 sec 32.4 MBytes 272 Mbits/sec
[ 3] 4.0- 5.0 sec 32.9 MBytes 276 Mbits/sec
[ 3] 5.0- 6.0 sec 32.1 MBytes 269 Mbits/sec
[ 3] 6.0- 7.0 sec 30.4 MBytes 255 Mbits/sec
[ 3] 7.0- 8.0 sec 24.6 MBytes 207 Mbits/sec
[ 3] 8.0- 9.0 sec 31.9 MBytes 267 Mbits/sec
[ 3] 9.0-10.0 sec 29.9 MBytes 251 Mbits/sec
[ 3] 0.0-10.0 sec 308 MBytes 258 Mbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)


Esto no parece claro? Es un test entre dos servidores pasando por 2 Mikrotiks con una ruta estatica, los dos Mikrotiks tienen un 10G anteriormente documentado.

Para responder sin sentidos no respondas; estoy pidiendo ayuda y solo haces que decirme que falta documentación.

I follow the ticket with Mikrotik support team, you're not from Mikrotik.

Thank you,
Pau

again you are doing things wrong

you are stating the device have a ridiculous performance

but you dont document your test

frits you make a test with btest

then iperf but you dont state how the test was made, and what configuration have the device at the moment of the test

off course you are having troubles

well deserved problems

please document your test properly

FYI:

you have to look for some bottleneck somewhere on your testing environment

for your reference there are a test i conducted on a ccr1009, in this result i think my environment test are limiting the results but i dont have more hardware to test
the topology of the test
the config of ccr was only a bridge between the two ports in fast path mode, mtu 1500 (virtualbox do not support jumbo frames) unfortunately i am limited to 1gbit testing because i dont have more pci express slots for more nics

you have to look for some bottleneck somewhere on your testing environment

for your reference there are a test i conducted on a ccr1009, in this result i think my environment test are limiting the results but i dont have more hardware to test

iperf.jpg

the topology of the test

topology.jpg

the config of ccr was only a bridge between the two ports in fast path mode, mtu 1500 (virtualbox do not support jumbo frames) unfortunately i am limited to 1gbit testing because i dont have more pci express slots for more nics

Contact you via private message.

you have to look for some bottleneck somewhere on your testing environment

for your reference there are a test i conducted on a ccr1009, in this result i think my environment test are limiting the results but i dont have more hardware to test

iperf.jpg

the topology of the test

topology.jpg

the config of ccr was only a bridge between the two ports in fast path mode, mtu 1500 (virtualbox do not support jumbo frames) unfortunately i am limited to 1gbit testing because i dont have more pci express slots for more nics

Contact you via private message.

Yesterday we analyze it and get the same conclusions: CCR1072 is a unfinished product.
We are awaiting for a solution.

Regards

Up please! This is a nightmare.
Regards

Up please! This is a nightmare.
Regards

Did you check RX and TX stats on both links? Inside the interface where errors are counted, look for errors, crc problems this would lead to a faulty line or faulty SFP.

Did you try a test with the router having no configurations whatsoever?

Did you try with different ports, different sfp?

I know people with this particular model. None of them presenting issues like this.

No errors in any interface. The issue is related with a single TCP connection.

Regards

No errors in any interface. The issue is related with a single TCP connection.

Regards

What about the other questions I asked?

Enviado desde mi MotoE2(4G-LTE) mediante Tapatalk

-Did you try a test with the router having no configurations whatsoever?
+We only have 2 CCR 1072, we cannot do this test.

-Did you try with different ports, different sfp?
+All ports has this issue.

Note that if we do test between 1036 and 1072 the results are better.

Regards,
Pau.

Up please

nadeu,
* I would recommend using traffic-generator instead of bandwidth test.
http://wiki.mikrotik.com/wiki/Manual:To ... _Generator
* check SFP modules and SFP cables, make sure they are working correct.
* run test again and if you are not satisfied with the results,
please contact MikroTik support with attached support output files (support@mikrotik.com)
We will see how we can help you.

nadeu,
* I would recommend using traffic-generator instead of bandwidth test.

the issue was validated using iperf between servers

Router are rebooted due kernel panic 30 minutes ago...

This is a nightmare. We have those routers and production, but we are planning rollback to 1036..

Please make sure you are running the latest MikroTik RouterOS version 6.35.1 and the latest RouterBOARD firmware (you can update it by /system routerboard upgrade and /system reboot).

Run the tests again and if you notice any issues, make support output files and send to us (support@mikrotik.com), it will be much more helpful.
Otherwise it is very difficult for us to help you.

Please make sure you are running the latest MikroTik RouterOS version 6.35.1 and the latest RouterBOARD firmware (you can update it by /system routerboard upgrade and /system reboot).

Run the tests again and if you notice any issues, make support output files and send to us (support@mikrotik.com), it will be much more helpful.
Otherwise it is very difficult for us to help you.

We downgraded to the bugfix only branch, this is more stable and quickly. But the forwarding speed are blocked on 200Mbps.

Regards,

Both test machines are running Linux/Unix?

Both test machines are running Linux/Unix?

Yes

Regards.

I had a similar situation with a MT router, but the problem was a missing "secret" update on Windows 7/2008 R2.
Only Windows machines were exhibiting the behavior, while Linux boxes were not.
So does not apply to you. Wish I had something to help you with.

I had a similar situation with a MT router, but the problem was a missing "secret" update on Windows 7/2008 R2.
Only Windows machines were exhibiting the behavior, while Linux boxes were not.
So does not apply to you. Wish I had something to help you with.

Waiting a resolution...I'm annoyed..

Regards

Waiting a resolution...I'm annoyed..

Have you contacted support already (as suggested by sergejs above)?
I'd like to remind (again) that this is a user forum, and not an official support platform.

Waiting a resolution...I'm annoyed..

Have you contacted support already (as suggested by sergejs above)?
I'd like to remind (again) that this is a user forum, and not an official support platform.

Yes.. but response time is too slow.

Regards

Not solved yet. We are planning another migration.

How to abandon a client: by MikroTik.

What is your support ticket number?

Well as far as i can see this topic is more about "how to ignore other user suggestions".

Bottom line - CCR is working just fine, many people have many deployments, and lately there are almost no complains.
Usually when there are actual problem there are much more that 1 person complaining in the forums

testing performance in the lab might be painful as you can't simply use bandwidth-test (single core program) and press one RUN button.
You need to use more complex programs like Traffic generator.
wiki.mikrotik.com/wiki/Manual:Performance_Testing_with_Traffic_Generator

What is your support ticket number?

My ticket ID is #2016032466000431

Well as far as i can see this topic is more about "how to ignore other user suggestions".

Bottom line - CCR is working just fine, many people have many deployments, and lately there are almost no complains.
Usually when there are actual problem there are much more that 1 person complaining in the forums

testing performance in the lab might be painful as you can't simply use bandwidth-test (single core program) and press one RUN button.
You need to use more complex programs like Traffic generator.
wiki.mikrotik.com/wiki/Manual:Performance_Testing_with_Traffic_Generator

You read the thread?
We run two iperf on Supermicro servers, when this passes by 1072 result is 200, else, result is full 10G.

I attach a network diagram:

Is it possible that your test through the CCR1072 is not following the path you expect?
I see that they could potentially choose to go through the Huawei switches or through the CPD-MAN-x routers....

I'm not saying that's what's happening, but is it possible?

Is it possible that your test through the CCR1072 is not following the path you expect?
I see that they could potentially choose to go through the Huawei switches or through the CPD-MAN-x routers....

I'm not saying that's what's happening, but is it possible?

Hello,
The IPs of servers are forced in both ICMANs with static route, i think that is impossible go through other link.

Nadeu
When you run a test can you view the inerfaces and ensure traffic is going through the CCR1072 routers.

The Huawei's do not support bonding-rr you need to run LACP with L3/L4 hash configured on both sides.

Also bonding is NOT fastpathed or fast tracked. You will never get full performance in a bonding configuration until Mikrotik add bonding fastpath TX and RX support.

I don't think Nadeu is using that bonding path though - at least not for the testing. According to the diagram, the link is a 10G-sfp directly between the two devices, and static routes pointing across the 10G link.

I would definitely agree with this though:

Nadeu
When you run a test can you view the inerfaces and ensure traffic is going through the CCR1072 routers.

I've definitely had some "oh, crap now I feel silly" type moments, and just making 100% sure that the traffic is doing what you think is a prudent step.

I don't think Nadeu is using that bonding path though - at least not for the testing. According to the diagram, the link is a 10G-sfp directly between the two devices, and static routes pointing across the 10G link.

I would definitely agree with this though:

Nadeu
When you run a test can you view the inerfaces and ensure traffic is going through the CCR1072 routers.

I've definitely had some "oh, crap now I feel silly" type moments, and just making 100% sure that the traffic is doing what you think is a prudent step.

Traffic are forwared as I indicated in network diagram. CDNs are directly connected to CCR1072 and each 1072-1072 throught SFP+ 10G.

MikroTik can't solve it, the same with CCR1036 run properly.

Regards,

nadeu, please post some more info. you are only posting accusations and statements without any concrete info. how can we help in this case?

re-read the topic above, there are many unanswered questions. as you can see from other replies, the performance you see is not normal, so there is absolutely something in the setup that should be changed/improved.

nadeu, please post some more info. you are only posting accusations and statements without any concrete info. how can we help in this case?

re-read the topic above, there are many unanswered questions. as you can see from other replies, the performance you see is not normal, so there is absolutely something in the setup that should be changed/improved.

Hi Normis,
i'm glad to you read this post. We have a opened ticket with all information that is required by MikroTik;

Ticket ID is #2016032466000431

Regards,

Hi, any news??


Have a good day!

Hi, any news??


Have a good day!

Unfortunately, no. Incredible but true.

when testing between 2 CCRs dont use bandwidth tester. Use the packet generator as explained on the wiki. bandwidth tester on routerboards are single threaded and dont give a true indicator of performance, they are only there to test link speeds.

when testing between 2 CCRs dont use bandwidth tester. Use the packet generator as explained on the wiki. bandwidth tester on routerboards are single threaded and dont give a true indicator of performance, they are only there to test link speeds.

We tested with two iperf servers with 8 cores E3-1240 v3;

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz

iperf 10G L2 throught switch: full 10G reached
iperf 10G L3 throught switch->bonding->ccr1072->switch->bonding->ccr1072: 200Mb/s

Why we need to run traffic-generator if test that we run is the most real test?

Regards,
Pau

@nadeu

Bonding does not use fastpath for TX. Fastpath is RX only!

Have you tried without bonding ?

@nadeu

Bonding does not use fastpath for TX. Fastpath is RX only!

Have you tried without bonding ?

We don't tried without bonding because we can't undo bonding. But... 200Mb/s? Really? ...
BTW, any ETA for implement Fastpath TX for bonding?

Regards,
Pau

i think the speed may be due to choice of bonding type. Try different ones till you find one that works well.