MikroTik

Posted: **Mon May 28, 2012 10:01 am**

Hey everyone,

I'm experiencing a strange problem with some of our gateways. It has already been here (unsolved) with some variations:

http://forum.mikrotik.com/viewtopic.php?f=7&t=60663

http://forum.mikrotik.com/viewtopic.php?f=7&t=60337

The general problem is following - Suppose you have three mikrotiks connected in line.

A <----> B <-----> C

A and C are "anything", B is a firewall with connection tracking and NAT turned on.

I do bandwidth test over TCP with a single connection (simulating real user http throughput).

A to B works okay, ~100Mbit of ethernet speed.
B to C works okay, same.
A to C gives only around 5-20Mbit.

There's no queueing on B's setup, nothing that could anyhow affect the performance of TCP. No layer7, no mangling, only the NAT+conntrack.

I have also tried with A and C being linux routers, single connection transfer also gives max. around 20Mbit.

I noticed that the limited speed scales exactly with TCP connection count, if you put 2 TCP connections, you got twice the speed.

I also noticed some dependency on CPU load (more like 5Mbit in the evening peaks, 20Mbit during early morning silence), but I cannot confirm this yet ('tis graphing right now).

I confirmed the problem when B was rb1100ah2x, rb1200, rb1000, with clean config except for a single SNAT rule, all recent versions (5.14-5.16). Problem doesn't happen when B is x86 PC and also on (some) RB1000's. I'm afraid this also happens on 711's and SXT's too (for me, it would mean fixing/replacing around 2000 of them), but wasn't able to confirm that yet.

Any idea what could cause such a limit?

Thanks
-exa

PS. This actually totally sucks for an ISP who is (sadly) deemed to rely on "speedtest.net" for showing/selling his service.

PPS. I cannot confirm this problem exists also with conntrack turned off - affected machines currently cannot be replaced, and I wasn't able to find a lab setup that would be affected by the problem yet (I don't have any spare 1100's to play with

)

Posted: **Mon May 28, 2012 10:09 am**

Paste and "/export compact" here - use the latest version.
Also what kind of traffic are yu using for your testing? Only speedtest?

Posted: **Mon May 28, 2012 10:28 am**

Paste and "/export compact" here - use the latest version.
Also what kind of traffic are yu using for your testing? Only speedtest?

This is the export of one affected live machine. I have XXXXxx'ed/deleted everything confidental, nothing interesting was lost.

# may/28/2012 09:16:03 by RouterOS 5.14
# software id = YANW-5BJ9
#
/interface ethernet
set 0 name=XXXXXXXX
set 1 name=ether5+
set 2 name=ether4+
set 3 name=ether3+
set 4 name=XXXXXXXX
set 7 name=ether6+
set 8 name=ether7+
set 9 name=ether8+
set 10 name=ether9+
set 11 name=XXXXXXXX
/interface vlan
add interface=XXXXXXX l2mtu=1594 name=XXXx vlan-id=101
add interface=XXXXXXX l2mtu=1594 name=XXXx vlan-id=100
add interface=XXXXXXX l2mtu=1594 name=XXXx vlan-id=2
/ip pool
add name=dhcp_pool1 ranges=10.12.100.2
add name=dhcp_pool2 ranges=10.12.100.6
add name=dhcp_pool3 ranges=10.12.100.10-10.12.100.14
add name=dhcp_pool4 ranges=10.12.100.18-10.12.100.22
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=XXXxXXXxXXXx name=\
    dhcp1
add address-pool=dhcp_pool4 disabled=no interface=XXXx name=dhcp2
/port
set 0 flow-control=hardware
/routing ospf instance
set [ find default=yes ] distribute-default=if-installed-as-type-1 \
    redistribute-connected=as-type-1 router-id=XxxxXx
/interface ethernet switch port
set 0 vlan-mode=fallback
set 1 vlan-mode=fallback
set 2 vlan-mode=fallback
set 3 vlan-mode=fallback
set 4 vlan-mode=fallback
/ip dhcp-server lease
add address=10.12.100.20 mac-address=00:25:90:56:B9:66 server=dhcp2
/ip dhcp-server network
add address=10.12.100.0/30 dns-server=8.8.8.8,8.8.4.4 gateway=10.12.100.1
add address=10.12.100.4/30 dns-server=8.8.8.8,8.8.4.4 gateway=10.12.100.5
add address=10.12.100.16/29 dns-server=XXXXXX gateway=\
    10.12.100.17
/ip dns
set max-udp-packet-size=512 servers=XXXXX
/ip firewall connection tracking
set generic-timeout=3m tcp-established-timeout=2h
/ip firewall nat
add action=jump chain=srcnat comment="masquerade private IPs" \
    jump-target=masquerade-spread src-address=10.0.0.0/8
add action=jump chain=srcnat jump-target=masquerade-spread src-address=\
    192.168.0.0/16
add action=src-nat chain=masquerade-spread per-connection-classifier=\
    src-address:2/0 to-addresses=PUB_IP_1_XXX
add action=src-nat chain=masquerade-spread per-connection-classifier=\
    src-address:2/1 to-addresses=PUB_IP_2_XXX
add action=src-nat chain=masquerade-spread comment=\
    "fallback" to-addresses=PUB_IP_2_XXX
/ip firewall service-port
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
/ip route
add check-gateway=ping distance=1 gateway=XXXXXXXX
/ip service
set ftp disabled=yes
set ssh port=2222
/queue interface
set XXXXXX queue=ethernet-default
set ether5+ queue=ethernet-default
set ether4+ queue=ethernet-default
set ether3+ queue=ethernet-default
set XXXXXX queue=ethernet-default
set XXXXXX queue=ethernet-default
/routing ospf interface
add authentication=md5 authentication-key=XXXXXXXXX network-type=broadcast
/routing ospf network
add area=backbone network=10.0.0.0/8
add area=backbone network=192.168.0.0/16
/system clock
set time-zone-name=Europe/Prague
/system identity
set name=XXXXXX
/system logging
add action=echo topics=info
/system ntp client
set enabled=yes mode=unicast primary-ntp=77.78.110.71 secondary-ntp=\
    212.96.160.147
/system resource irq rps
set XXXXXX disabled=no
set ether5+ disabled=no
set ether4+ disabled=no
set ether3+ disabled=no
set XXXXXX disabled=no

For traffic testing, I use bandwidth test and linux-to-linux HTTP using wget.

EDIT: no change when PCC or OSPF disappears.

Posted: **Mon May 28, 2012 12:16 pm**

interface queue - change to hardware queue only (if multi core then to multi-fifo queue)

Posted: **Mon May 28, 2012 12:28 pm**

interface queue - change to hardware queue only (if multi core then to multi-fifo queue)

Wow, seems that helped a lot. "only-hardware-queue" did nothing, but after setting multiqueue fifo, throughput was finally hitting line limits.

Do you have any idea what would the root cause be? I'm suspecting queue locking issues. Also, could routers without multi-queue get also affected?

anyway, thanks very much for help!

-exa

Posted: **Mon May 28, 2012 4:01 pm**

yep, you basically allowed your interfaces to use multiple CPU cores (i assume that it is multi-core router.)

Simple FIFO force all stream to use single CPU core.

Posted: **Mon Jul 09, 2012 1:45 am**

hi,
i think i have similar problem ...

first there is rb750up then there is wireless bridge on ubnt equipment then there is rb711UA-5HnD and again CPE on rb sextant

-tcp performance ( always 1 TCP session ) from rb750 to rb711 around 40-45 Mbit
-tcp performance from rb750 to sextant only 20-24 Mbit
-tcp performance from rb711 to sextant 35-40 Mbit

what more next to rb711UA there is another rb711UA connected via tplink switch and performance between them is ok about 45 Mbit

i tried to change queue's but nothing happened

im not quite sure if i have to reboot each time i change type of queue ?
any way i tried both with and without reboot

so i am still search for the answer why tcp performance drops over second or third wireless bridge?

any sugestions?

Posted: **Mon Jul 09, 2012 8:00 am**

hi,
i think i have similar problem ...

first there is rb750up then there is wireless bridge on ubnt equipment then there is rb711UA-5HnD and again CPE on rb sextant

-tcp performance ( always 1 TCP session ) from rb750 to rb711 around 40-45 Mbit
-tcp performance from rb750 to sextant only 20-24 Mbit
-tcp performance from rb711 to sextant 35-40 Mbit

what more next to rb711UA there is another rb711UA connected via tplink switch and performance between them is ok about 45 Mbit

i tried to change queue's but nothing happened im not quite sure if i have to reboot each time i change type of queue ?
any way i tried both with and without reboot

so i am still search for the answer why tcp performance drops over second or third wireless bridge?

any sugestions?

I've seen something like this with 2 Devices which have ethernet problems when connected together. Minimal packet drops caused tcp to slow down. Replaced one side and set fixed ethernet speed solved the problem.

Posted: **Mon Aug 13, 2012 8:04 pm**

Hello,

I'm having similar problem:

RB1000 <-----> RB1100AHx2 <-----> RB1200

While doing some bandwith test with TCP and single session, I have this results:

RB1000 <-----> RB1100AHx2 - working OK
RB1100AHx2 <-----> RB1200 - working OK

But when I try:

RB1000 <-----> RB1200 bandwith is poor. It goes up with additional sessions.

I've tried "mq pfifo" in the RB1100AHx2 ethernets, but same result.

Any ideas?

Posted: **Tue Oct 23, 2012 11:40 am**

Hello,

I'm having similar problem:

RB1000 <-----> RB1100AHx2 <-----> RB1200

While doing some bandwith test with TCP and single session, I have this results:

RB1000 <-----> RB1100AHx2 - working OK
RB1100AHx2 <-----> RB1200 - working OK

But when I try:

RB1000 <-----> RB1200 bandwith is poor. It goes up with additional sessions.

I've tried "mq pfifo" in the RB1100AHx2 ethernets, but same result.

Any ideas?

Have you solved this issue?
What about your ccq?

Posted: **Wed Oct 24, 2012 11:36 pm**

Hi Ciambot,

It's only ethernet, no Wireless..... and we're still with the same problem.

Regards

Posted: **Thu Feb 14, 2013 10:36 am**

Any news??

Thanks,

Michele Pietravalle

Posted: **Mon Jul 21, 2014 7:52 pm**

Hi!

Does anybody have any clue?

Regards

Posted: **Mon Jul 21, 2014 8:01 pm**

Does anybody have any clue?

Well, honestly:

We have closed this problem because we don't really know what caused it, and with disappearance of some devices that seemed to be "more harmful" it looks okay.

If you have the problem, make sure you have:

- version 6
- no powerPC, especially not rb1100/1200
- correctly set-up interface queues to accomodate at least (bandwidth*cpu_jitter) of packet data with some good reserve, using MQ variants on multi-CPU machines (e.g. mq_pfifo)
- on wireless, latest nv2 if using nv2, and correctly set-up 11n (all channels!) if using 11n
- firewall/conntrack off wherever possible
- bandwidth-testing machines with enough CPU power on both sides

After applying all those rules, problem usually disappears.

If it helps you and don't have any better solution, feel free to mark this closed. :]

-exa

Posted: **Thu Dec 20, 2018 10:35 am**

yep, you basically allowed your interfaces to use multiple CPU cores (i assume that it is multi-core router.)

Simple FIFO force all stream to use single CPU core.

Hello,
all 36 cores selected only-hardware-queue by default, change that interface queue to multi-queue-ethernet-default can affect traffic ?

MikroTik

TCP performance over Mikrotik

TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik

Re: TCP performance over Mikrotik