So I was testing my RB1009-8G-1S-1S+ and RB1009-8G-1S-1S+PC tonight and added IP addresses to both eth1 interfaces, plugged in Gigabit Ethernet cable and ran a Btest. Surprisingly I get Btests that start out at 350-450 Mbit (I was expecting more actually), but then it gradually drops down to 40-70 Mbit over the course of a couple of minutes and refuses to climb back up. Simply pushing the "Start" button again immediately gets me 350-450 Mbit, but then it slowly drops again. Does anyone know what causes this issue? Do I have some bad units? Oh, and the funny thing is the CPU never gets above 30%. I can't understand what the limiting factor is. I have basically no config on these things...

In the first screenshot I'm running 6.27, but I updated to 6.31 and it made it worse. In 6.31 it would start at around 150 Mbit, but almost immediately crash down to 14-40 Mbit.

Oh, and these tests are TCP tests as you can see in the images, but UDP tests results in near Gigabit speeds (of course).

Does anyone have any advice on troubleshooting methods or are my units just toast?

6.30.4 is the latest stable version. Learn how to make bandwidth tests first and do not use tested devices as traffic generators. Btest has some problems too so you cannot rely on its results. Keep them informative only and make proper test again with iperf for example. Watch the profiler to see where is the bottleneck if any.

With BTest you are overloading test Device itself, and Btest is limited to one core.

You can also use Traffic Generator
http://wiki.mikrotik.com/wiki/Manual:Pe ... _Generator

Thanks for the suggestions guys. I'll try the new Traffic Generator. I'm glad to see there is a next generation tool like that built into ROS. Thanks for putting it in v6. That's awesome! However, what still puzzles me is the fact that I am getting such low Btest speeds when I can do a similar test between two RB1200 boards and get 90+ Mbit. I have RB1200s at most of my towers and those towers are connected with P2P 5GHz radios and I am able to push 70-90 Mbit from tower to tower via Btest running on the RB1200s.

I realize that Btest is consuming resources on the CCR1009 that would otherwise be used to send network packets, but I was expecting the CCR1009 to be able to run a Btest that was at least as good as an RB1200. After all, it's CPU was never maxed out. I've done Btests on an RB750 and even RB450 that were several times more than this CCR1009. Heck, I think my old RB150s could outperform my CCR1009s' Btest numbers. I guess what I am trying to ask is if there is a bug in the Btest tool in ROS6 that would be causing this trouble. Or also what I am wondering is if a single one of the CCR1009 cores is less powerful than the RB1200 and since Btest is only capable of using 1 core that is why I am seeing such low speeds.

I know how to use iperf to do these tests. I'm not looking to do a true stress test nor do I have access to $2,000 iperf servers where I am. I just had these two CCR1009s in my office and got curious about how much more they could push via Btest than the RB1200s I am going to replace them with. Before I'll feel comfortable replacing my RB1200s I would like to find out why my CCR1009s can't do a Btest more than 40 Mbit and would eventually like to see them do a test over 100 Mbit. It may be a hardware limit, software limit, or a mistake on my part. I'd just like to know where the problem lies.

Ok guys, I think I made some progress in testing this out and have potentially found the problem. However, first let me start off by asking a question about the new Traffic Generator tool. Is this something that only tests via UDP connections? Tonight was the first time I had used it, so I am quite unfamiliar with it and it seems very complex, but from what I read on the wiki page and from the tests I ran with it tonight it seems like it is only testing via UDP. Is this correct or did I miss where I can set it to test with TCP connections instead? None of the traffic shows up in the Torch tool, so I don't know for sure what it's doing. Testing with UDP is fairly useless to me as the tests I normally run are to measure TCP speed capabilities to get the best real world performance estimates for my links. Is there any chance Btest will be written to use multiple CPU cores so that we can run TCP tests at near Gigabit speeds?

Anyway, after testing more tonight I think I've come to the conclusion that my RB1009s are fine (hardware wise), but there seems to be a bug in ROS. I've tested with 6.27, 6.30.4, and 6.31. They all have similar problems running Btests, but 6.27 seems to have less trouble. Just to be sure I don't have something messed up in my config I have reset both routers and only added an IP on ether1 to both routers. I have run Btests under the following circumstances with the following results:

(All test were run with the TCP protocol in both send or receive)

Btest from my PC with btest.exe - I can get over 200 Mbit reliably (bottleneck seems to be btest.exe taxing my PC's CPU)

Btest between 2 RB1009s - I can only get 20 - 40 Mbit

Btest between 2 RB1009s with local and remote TX speed in the Btest tool set to 380M - I get a solid 380 Mbit

Btest between 2 RB1009s with local and remote TX speed in the Btest tool set to any number below 380M - I get a solid speed corresponding exactly to what number I have set

Btest between 2 RB1009s with local and remote TX speed in the Btest tool set to any number above 380M - Speeds drop down to 20-40 Mbit

Side note: My RB2011 can do a Btest of about 250 Mbit to my RB1009 without needing to set any local or remote TX speed. It is running ROS 6.27.

So it seems like the single core that Btest is utilizing can push just under 400 Mbit, but any attempts to get speeds higher than that result in the test bugging out and reporting low speeds. So it looks like you need to set a speed cap in the Btest tool to make sure the speeds never go beyond about 400 Mbit so that you get accurate results. Is this a well known Btest bug for speeds above 400 Mbit or does this seem to be something new? Can anyone verify my results?

Any chance on getting some verification that this Btest glitch is being caused by a software bug or issue with the CCR1009? My only guess at this point is that the code for Btest was written before gigabit Ethernet was invented and the software just isn't capable of generating over 400 Mbit of traffic efficiently or even if it could the logic just can't display or process the results of the test correctly. Either that or multicore hardware is screwing stuff up. Any smart guys want to weigh in on this? I may test with a couple of RB1200s and see what speeds I get.

Same results from CCR1036-8G-2S (6.22) to CCR1009-8G-1S (6.31).

RX: 375.7Mbps stable
TX current: 44.4Mbps (99% time)
TX 10s: 77.2Mbps
Both: 255.2Mpbs / 255.5Mbps stable

Response time between routers are 0ms

MTU on Router A is: 1590, in router B is: 1500
L2MTU on Router A is: 1590, in router B is: 1590

Is MTU problem?

Regards.

Well I tested with an RB1200 and Btest didn't glitch out like it does on the CCR1009, but that is probably because the RB1200 couldn't send or receive more than 250 Mbit without maxing out its CPU. Here are the results:

Btest on a RB1200 connecting to a CCR1009
-Send test: ~250 Mbit throughput, RB1200 CPU 100%, CCR1009 CPU 95% on 1 core
-Receive test: ~190 Mbit throughput, RB1200 CPU 100%, CCR1009 CPU 100% on 1 core

Btest on a CCR1009 connecting to a RB1200
-Send test: ~190 Mbit throughput, RB1200 CPU 100%, CCR1009 CPU 100% on 1 core
-Receive test: ~240 Mbit throughput, RB1200 CPU 100%, CCR1009 CPU 95% on 1 core

I realize I'm just talking to myself at this point, but I just wanted to post my findings so that anyone that has this same issue and is searching these forums will find this thread and know that their CCR boards are probably ok and the problem is just caused by buggy or poorly designed Mikrotik software.

[If anyone is still reading this thread] Does anyone have any recommendations on what kind of setup to use to test speeds above 400 Mbit with client software on a PC/Mac? It is important that it is >>>client software<<< since I really don't want to lug around a CCR to run Traffic Generator tests with at every site. I realize iperf is good, but that would involve installing an iperf server at every tower we have and that is not cost effective or possible for us due to space and power restrictions. Has anyone had any luck with installing a Linux image on Metarouter and running an iperf server from there? I realize this will put a load on the router and won't show true speeds, but I am just looking for a quick and dirty way to test wireless links. For more in-depth testing I suppose I could use Traffic Generator between RouterBoards. I definitely don't want to move away from RouterOS since it is so awesome and I love it, but I'm wondering if I should just put a bunch of Ubiquiti EdgeRouters plugged into my RouterBoards at my towers to act as iperf servers. I have one lying around here somewhere, but I really hate the UI... I guess I'll dust it off and see how it performs... reluctantly...

nadeu, thanks for taking the time to run some tests to verify my results. I'm not sure how I didn't see your post before I posted my last findings. Sorry for overlooking your reply.

You didn't exactly specify in your results, but I'm assuming that when you ran your TX test that got 44.4 Mbps (99% of the time) the traffic was being generated by the CCR1009? Also, was the stable 375.5 Mbps being generated by the CCR1036? I'm assuming so. I'm guessing that maybe you got higher and more stable speed from the CCR1036 due to it being more powerful. What was the CPU load on the cores of the CCRs during the tests? Also, try and set the "Local TX speed" and "Remote TX speed" to 350000000 and see if you can get a stable 350 Mbps single duplex either way. I don't remember what results I got myself full duplex. Somewhere around 250 Mbps does seem about right.

I ended up deploying one of my CCR1009s into the field to see how it behaves so I could determine if it is faulty or not. I set it up last night, and so far no issues. It is performing much better than the RB1200 that it replaced. So I'm not going to be able to experiment with it any further and try different MTU settings. Perhaps I will order another one for a test unit in my office.

Meanwhile, can any Mikrotik personnel advise us on what is going on here?