I’ve just bought an RB1100 router that I intended to use as a replacement for my Intel server traffic shaper running RouterOS 5.0rc4. I’ve saved the settings on the Intel, uploaded them on the RB1100 and off I went to move the cables from the Intel to the new shaper. I was lazy enough not to use a dedicated out-of-band management interface (useless anyway after you read further) so it took me quite a while to load the prefs in the Winbox. I took a look at the interfaces, it took about a minute for it to load the 13 interfaces and when it did, it only indicated about 120Mbps going out while on the Intel there are about 220-240Mbps passing through at peak times according to the 95th percentile graph from the Cisco port it’s plugged into.
Next step was obviously to check the CPU load. 100% @ ~120Mbps out of the normal 240Mbps. I started to get emails from my Smokeping server in Germany, old respose time 35-40ms → 250-350ms.
It’s obvious not to expect much from an appliance using an 800Mhz single core CPU instead of the Intel’s X3440 Xeon Quad Core but I wasn’t expecting a 13 gigabit ports appliance not to be able to handle more than 120Mbps doing only that.
Some tech details about my old setup: About 10 drop or accept rules in the firewall, one L7 rule, about 500 simple pfifo queues all of these using about 15-20% of the Xeon. The intel has two NICs as ports of a bridge and it’s only purpose is shaping the traffic in a transparent bridge setup.
My questions are: are there any special requirements for using the RB1100 in this current setup that I was not aware of? Because maybe there are things that need to be done in a certain way for the 800 MHz CPU to be able to handle the traffic.
Why are there 1Gbps ports for a device that’s not able to handle at least 1Gbps. I would expect this to be oversubscribed like some Cisco switches are but I would also expect to be able to carry 1Gbps at least in half-duplex.
I was quite happy with the Intel except for the fact that it’s NICs were on PCI, thus limiting the bridge throughput to the PCI bus speed (from my experience up to 350-400Mbps traffic).
I’ve read about the queues optimizations. I haven’t found anything relevant anywhere, though I think I definitely read something about this.
Simple Queues and L7 inspection slaughter the CPU. Please do not use them, or at least don’t use hundreds or thousands of them.
No Routerboard will be able to cope with that kind of (useless, because redundant) work.
Try to convert your simple queues to a queue tree, or to PCQ simple queues. Drop the L7 rule, if it isn’t 100% necessary.
You may also try overclocking your CPU to 1066 MHz, but note that this may make the board unstable.
Also, disable connection tracking, if you are not using any NAT configuration or advanced firewall features which require them (like connection-bytes or L7).
Just for the sake of it, I already tried that even though it was useless to me. I’ve disabled all queues, I’ve disabled all firewall rules. I’ve also tried to overclock it, I don’t recall the command exactly, the point is that the command was valid but a “Your system does not allow CPU clock settings” or something alike was returned. Anyway, overclocking it from 800 → 1066 will mean what, bridging 200Mbps without loss but not more?
It just doesn’t add up. I don’t understand this: http://routerboard.com/pdf/routerboard_performance_tests.pdf
According to this benchmark Mikrotik says that the router can perform at it’s worst 121000 pps. OK. The normal traffic through the Intel shaper as observed after the rollback was 20000-25000 pps with a total throuhput of 200 Mbps. Doing some simple math it turns out that under normal usage the average packet size can be approximated at a size of 1000 bytes.
So Mikrotik measured a maximum 500.32Mbps throughput for a constant 512 bytes per packet and 1335.84 for a 1500 bytes packet, it would seem reasonable to estimate the throughput for 1000 bytes at (500.32+1335.84)/2=918Mbps. I only need 200-300Mbps but the CPU goes nuts without any fancy work.
And may I also remind you that the above mentioned values are worst case scenarios according to the benchmark.
What type of bus is using is irrelevant. What’s relevat here is that it can’t bridge more than 120Mbps through 2 gigabit ports from the same port group, nor between ports from two different groups.
Ehm, it is best case scenario - with almost no configuration. So as soon as you start configure, something numbers will go down.
I was refering to the fact that the numbers I took from the official Mikrotik benchmarks are the lowest possible values, under the worst case scenario. The performance is a lot better without any sort of routing/conntrack. The values I used as listed by Mikrotik are under the Firewall ON (no mention on how many rules) + Conntrack ON + RSTP Bridge Mode.
Let me see if I understand this correctly.
You are saying that you disabled all queues and all firewall rules but you still were not able to bridge 200 Mbps of traffic?
Did you make sure that connection tracking was off?
I already tried that even though it was useless to me. I’ve disabled all queues, I’ve disabled all firewall rules.
Besides the fact that the conntrack is something that I need, I disabled it anyway, it was roughly a 50% throughput gain, but then again, no shaping, no filters, no conntrack? Doesn’t this nice and shiny box look more and more like a rubbish TP-Link unmanaged switch? Only about 4 times more expensive.
And please don’t make me do this, but I’m seriously thinking of buying one of those, just to convince myself that the TP-Link will be able to handle 1Gbps.
@Mikrotik: If the sole purpose of 1Gbps port is to be able to handle more than 100Mbps per port (but not much above this value) I will completely understand and close this topic, I will just assume that your tests showed that the RB1100 CPU can handle more than 100Mbps per port and decided to go for gigabit ports, yet knowing that I will never be able to go as high as 1Gbps. Yet, the performance benchmarks are highly erroneous in respect to the advertised router performance, mine performs 4-5 times worse. And don’t take this the wrong way, I love RouterOS, in fact I just bought a PRO/1000 ET card as a replacement for my curent shaper (low speed PCI bus and no RX/TX queues on the built-in NICs) and I will always have only good words about RouterOS. My problem is with your top-notch Routerboard. I’m still waiting for an official response from you in respect to the discrepancies between the real router and the advertised performance.
Hey mihaialdea, could you please provide a capture of the traffic you need to bridge?
Maybe it’s some big size packet crazy thing.
I had ping time issues when bridging 802.1q, 802.1ad, EoIP and IPIP. I haven’t tested other types. The issue could be NOT IN THE MikroTik bridge, I haven’t had time to check more. I just untagged the traffic.
P.S. I once reloaded a router by reflashing from bootloader and then re-created all configs from withing WinBox without restoring a .backup from another platform. Then one problem went away.
Have you tried with v5 RC7 / RC8 ? There are improvements in RC7…
Hey if you can’t find the RB1100 usable - send it to me
I had similar problems with bridging and queues on my 1100s; I bridged Port 11-12 ,
enabled piping through fw, and added mangle rules and a complex queue tree for interface 12:
The result was a more or less completely unresponsible RB - the interfaces showed only a few Mbps of data and was hardly reachable on all other IFs.
I just changed the bridged ports to 6-7 and the queue on port 7; Everything is fine since then and i can easily get 200Mbps+ without exceeding 30% load.
This happens on all my 1100s; I thought it was an hardware issue and they fixed it and redesigned parts of the RBs (cause the RB1100 were unavailable for a couple of month).
But this still happens on my new ones i got last week.
I solve this using the rb 1100 has a gigaswitch and the queue, conntrack and firewall rules with a X86 server with 2 giganics, the 1100 cpu can´t handle all the traffic.
No, unfortunately I can’t provide you with that because that would mean for me to break the NDA I’ve signed with my employer. For the time being it’s only bridging the traffic for a single server which is a online game server. No more thant 30-40Mbps at any given time. The CPU ranges from from 40-80% usge. The client doesn’t complain about anything, but if I start adding servers things will go poof.
I’m not using any sort of encapsulation, everything is in access mode.
Pretty expensive gigabit switch, huh?