Community discussions

MikroTik App
 
exa
newbie
Topic Author
Posts: 37
Joined: Sat Jul 04, 2009 2:07 pm

TCP performance over Mikrotik

Mon May 28, 2012 10:01 am

Hey everyone,

I'm experiencing a strange problem with some of our gateways. It has already been here (unsolved) with some variations:

http://forum.mikrotik.com/viewtopic.php?f=7&t=60663

http://forum.mikrotik.com/viewtopic.php?f=7&t=60337

The general problem is following - Suppose you have three mikrotiks connected in line.

A <----> B <-----> C

A and C are "anything", B is a firewall with connection tracking and NAT turned on.

I do bandwidth test over TCP with a single connection (simulating real user http throughput).

A to B works okay, ~100Mbit of ethernet speed.
B to C works okay, same.
A to C gives only around 5-20Mbit.

There's no queueing on B's setup, nothing that could anyhow affect the performance of TCP. No layer7, no mangling, only the NAT+conntrack.

I have also tried with A and C being linux routers, single connection transfer also gives max. around 20Mbit.

I noticed that the limited speed scales exactly with TCP connection count, if you put 2 TCP connections, you got twice the speed.

I also noticed some dependency on CPU load (more like 5Mbit in the evening peaks, 20Mbit during early morning silence), but I cannot confirm this yet ('tis graphing right now).

I confirmed the problem when B was rb1100ah2x, rb1200, rb1000, with clean config except for a single SNAT rule, all recent versions (5.14-5.16). Problem doesn't happen when B is x86 PC and also on (some) RB1000's. I'm afraid this also happens on 711's and SXT's too (for me, it would mean fixing/replacing around 2000 of them), but wasn't able to confirm that yet.

Any idea what could cause such a limit?

Thanks
-exa


PS. This actually totally sucks for an ISP who is (sadly) deemed to rely on "speedtest.net" for showing/selling his service.

PPS. I cannot confirm this problem exists also with conntrack turned off - affected machines currently cannot be replaced, and I wasn't able to find a lab setup that would be affected by the problem yet (I don't have any spare 1100's to play with :( )
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1734
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: TCP performance over Mikrotik

Mon May 28, 2012 10:09 am

Paste and "/export compact" here - use the latest version.
Also what kind of traffic are yu using for your testing? Only speedtest?
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
exa
newbie
Topic Author
Posts: 37
Joined: Sat Jul 04, 2009 2:07 pm

Re: TCP performance over Mikrotik

Mon May 28, 2012 10:28 am

Paste and "/export compact" here - use the latest version.
Also what kind of traffic are yu using for your testing? Only speedtest?
This is the export of one affected live machine. I have XXXXxx'ed/deleted everything confidental, nothing interesting was lost.
# may/28/2012 09:16:03 by RouterOS 5.14
# software id = YANW-5BJ9
#
/interface ethernet
set 0 name=XXXXXXXX
set 1 name=ether5+
set 2 name=ether4+
set 3 name=ether3+
set 4 name=XXXXXXXX
set 7 name=ether6+
set 8 name=ether7+
set 9 name=ether8+
set 10 name=ether9+
set 11 name=XXXXXXXX
/interface vlan
add interface=XXXXXXX l2mtu=1594 name=XXXx vlan-id=101
add interface=XXXXXXX l2mtu=1594 name=XXXx vlan-id=100
add interface=XXXXXXX l2mtu=1594 name=XXXx vlan-id=2
/ip pool
add name=dhcp_pool1 ranges=10.12.100.2
add name=dhcp_pool2 ranges=10.12.100.6
add name=dhcp_pool3 ranges=10.12.100.10-10.12.100.14
add name=dhcp_pool4 ranges=10.12.100.18-10.12.100.22
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=XXXxXXXxXXXx name=\
    dhcp1
add address-pool=dhcp_pool4 disabled=no interface=XXXx name=dhcp2
/port
set 0 flow-control=hardware
/routing ospf instance
set [ find default=yes ] distribute-default=if-installed-as-type-1 \
    redistribute-connected=as-type-1 router-id=XxxxXx
/interface ethernet switch port
set 0 vlan-mode=fallback
set 1 vlan-mode=fallback
set 2 vlan-mode=fallback
set 3 vlan-mode=fallback
set 4 vlan-mode=fallback
/ip dhcp-server lease
add address=10.12.100.20 mac-address=00:25:90:56:B9:66 server=dhcp2
/ip dhcp-server network
add address=10.12.100.0/30 dns-server=8.8.8.8,8.8.4.4 gateway=10.12.100.1
add address=10.12.100.4/30 dns-server=8.8.8.8,8.8.4.4 gateway=10.12.100.5
add address=10.12.100.16/29 dns-server=XXXXXX gateway=\
    10.12.100.17
/ip dns
set max-udp-packet-size=512 servers=XXXXX
/ip firewall connection tracking
set generic-timeout=3m tcp-established-timeout=2h
/ip firewall nat
add action=jump chain=srcnat comment="masquerade private IPs" \
    jump-target=masquerade-spread src-address=10.0.0.0/8
add action=jump chain=srcnat jump-target=masquerade-spread src-address=\
    192.168.0.0/16
add action=src-nat chain=masquerade-spread per-connection-classifier=\
    src-address:2/0 to-addresses=PUB_IP_1_XXX
add action=src-nat chain=masquerade-spread per-connection-classifier=\
    src-address:2/1 to-addresses=PUB_IP_2_XXX
add action=src-nat chain=masquerade-spread comment=\
    "fallback" to-addresses=PUB_IP_2_XXX
/ip firewall service-port
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
/ip route
add check-gateway=ping distance=1 gateway=XXXXXXXX
/ip service
set ftp disabled=yes
set ssh port=2222
/queue interface
set XXXXXX queue=ethernet-default
set ether5+ queue=ethernet-default
set ether4+ queue=ethernet-default
set ether3+ queue=ethernet-default
set XXXXXX queue=ethernet-default
set XXXXXX queue=ethernet-default
/routing ospf interface
add authentication=md5 authentication-key=XXXXXXXXX network-type=broadcast
/routing ospf network
add area=backbone network=10.0.0.0/8
add area=backbone network=192.168.0.0/16
/system clock
set time-zone-name=Europe/Prague
/system identity
set name=XXXXXX
/system logging
add action=echo topics=info
/system ntp client
set enabled=yes mode=unicast primary-ntp=77.78.110.71 secondary-ntp=\
    212.96.160.147
/system resource irq rps
set XXXXXX disabled=no
set ether5+ disabled=no
set ether4+ disabled=no
set ether3+ disabled=no
set XXXXXX disabled=no

For traffic testing, I use bandwidth test and linux-to-linux HTTP using wget.

EDIT: no change when PCC or OSPF disappears.
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1734
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: TCP performance over Mikrotik

Mon May 28, 2012 12:16 pm

interface queue - change to hardware queue only (if multi core then to multi-fifo queue)
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
exa
newbie
Topic Author
Posts: 37
Joined: Sat Jul 04, 2009 2:07 pm

Re: TCP performance over Mikrotik

Mon May 28, 2012 12:28 pm

interface queue - change to hardware queue only (if multi core then to multi-fifo queue)
Wow, seems that helped a lot. "only-hardware-queue" did nothing, but after setting multiqueue fifo, throughput was finally hitting line limits.

Do you have any idea what would the root cause be? I'm suspecting queue locking issues. Also, could routers without multi-queue get also affected?

anyway, thanks very much for help!

-exa
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1734
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: TCP performance over Mikrotik

Mon May 28, 2012 4:01 pm

yep, you basically allowed your interfaces to use multiple CPU cores (i assume that it is multi-core router.)

Simple FIFO force all stream to use single CPU core.
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
User avatar
ringlet
newbie
Posts: 35
Joined: Sun Mar 07, 2010 9:24 pm
Location: Gdynia, Poland
Contact:

Re: TCP performance over Mikrotik

Mon Jul 09, 2012 1:45 am

hi,
i think i have similar problem ...

first there is rb750up then there is wireless bridge on ubnt equipment then there is rb711UA-5HnD and again CPE on rb sextant

-tcp performance ( always 1 TCP session ) from rb750 to rb711 around 40-45 Mbit
-tcp performance from rb750 to sextant only 20-24 Mbit
-tcp performance from rb711 to sextant 35-40 Mbit

what more next to rb711UA there is another rb711UA connected via tplink switch and performance between them is ok about 45 Mbit

i tried to change queue's but nothing happened :( im not quite sure if i have to reboot each time i change type of queue ?
any way i tried both with and without reboot

so i am still search for the answer why tcp performance drops over second or third wireless bridge?

any sugestions?
 
ste
Forum Guru
Forum Guru
Posts: 1837
Joined: Sun Feb 13, 2005 11:21 pm

Re: TCP performance over Mikrotik

Mon Jul 09, 2012 8:00 am

hi,
i think i have similar problem ...

first there is rb750up then there is wireless bridge on ubnt equipment then there is rb711UA-5HnD and again CPE on rb sextant

-tcp performance ( always 1 TCP session ) from rb750 to rb711 around 40-45 Mbit
-tcp performance from rb750 to sextant only 20-24 Mbit
-tcp performance from rb711 to sextant 35-40 Mbit

what more next to rb711UA there is another rb711UA connected via tplink switch and performance between them is ok about 45 Mbit

i tried to change queue's but nothing happened :( im not quite sure if i have to reboot each time i change type of queue ?
any way i tried both with and without reboot

so i am still search for the answer why tcp performance drops over second or third wireless bridge?

any sugestions?
I've seen something like this with 2 Devices which have ethernet problems when connected together. Minimal packet drops caused tcp to slow down. Replaced one side and set fixed ethernet speed solved the problem.
 
lbenzo
newbie
Posts: 28
Joined: Mon Feb 27, 2006 12:12 am

Re: TCP performance over Mikrotik

Mon Aug 13, 2012 8:04 pm

Hello,

I'm having similar problem:

RB1000 <-----> RB1100AHx2 <-----> RB1200

While doing some bandwith test with TCP and single session, I have this results:

RB1000 <-----> RB1100AHx2 - working OK
RB1100AHx2 <-----> RB1200 - working OK

But when I try:

RB1000 <-----> RB1200 bandwith is poor. It goes up with additional sessions.

I've tried "mq pfifo" in the RB1100AHx2 ethernets, but same result.

Any ideas?
 
Ciambot
Member Candidate
Member Candidate
Posts: 144
Joined: Thu Jan 22, 2009 5:22 pm

Re: TCP performance over Mikrotik

Tue Oct 23, 2012 11:40 am

Hello,

I'm having similar problem:

RB1000 <-----> RB1100AHx2 <-----> RB1200

While doing some bandwith test with TCP and single session, I have this results:

RB1000 <-----> RB1100AHx2 - working OK
RB1100AHx2 <-----> RB1200 - working OK

But when I try:

RB1000 <-----> RB1200 bandwith is poor. It goes up with additional sessions.

I've tried "mq pfifo" in the RB1100AHx2 ethernets, but same result.

Any ideas?
Have you solved this issue?
What about your ccq?
 
lbenzo
newbie
Posts: 28
Joined: Mon Feb 27, 2006 12:12 am

Re: TCP performance over Mikrotik

Wed Oct 24, 2012 11:36 pm

Hi Ciambot,

It's only ethernet, no Wireless..... and we're still with the same problem.

Regards
 
User avatar
MichelePietravalle
Trainer
Trainer
Posts: 99
Joined: Sun Apr 19, 2009 9:03 pm

Re: TCP performance over Mikrotik

Thu Feb 14, 2013 10:36 am

Any news??

Thanks,

Michele Pietravalle
 
lbenzo
newbie
Posts: 28
Joined: Mon Feb 27, 2006 12:12 am

Re: TCP performance over Mikrotik

Mon Jul 21, 2014 7:52 pm

Hi!

Does anybody have any clue?

Regards
 
exa
newbie
Topic Author
Posts: 37
Joined: Sat Jul 04, 2009 2:07 pm

Re: TCP performance over Mikrotik

Mon Jul 21, 2014 8:01 pm

Does anybody have any clue?
Well, honestly:

We have closed this problem because we don't really know what caused it, and with disappearance of some devices that seemed to be "more harmful" it looks okay.

If you have the problem, make sure you have:

- version 6
- no powerPC, especially not rb1100/1200
- correctly set-up interface queues to accomodate at least (bandwidth*cpu_jitter) of packet data with some good reserve, using MQ variants on multi-CPU machines (e.g. mq_pfifo)
- on wireless, latest nv2 if using nv2, and correctly set-up 11n (all channels!) if using 11n
- firewall/conntrack off wherever possible
- bandwidth-testing machines with enough CPU power on both sides

After applying all those rules, problem usually disappears.

If it helps you and don't have any better solution, feel free to mark this closed. :]

-exa
 
User avatar
amt
Long time Member
Long time Member
Posts: 527
Joined: Fri Jan 16, 2015 2:05 pm

Re: TCP performance over Mikrotik

Thu Dec 20, 2018 10:35 am

yep, you basically allowed your interfaces to use multiple CPU cores (i assume that it is multi-core router.)

Simple FIFO force all stream to use single CPU core.

Hello,
all 36 cores selected only-hardware-queue by default, change that interface queue to multi-queue-ethernet-default can affect traffic ?

Who is online

Users browsing this forum: eworm and 94 guests