MikroTik RouterOS • View topic - ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

lionspy · **Posted:** Sun Jan 29, 2012 4:19 pm

ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86. every 5-12 hours i see a a soft lockup on a random core of the cpu, after that whole system hangs. ROS 5.11 have much more performance on my BRAS (about 150+ Mbps, than it hangs). supout.rif does not creates in case of hangup.
ROS 5.6 seems to be very stable, uptime is over 2 weeks, generally.
but in pptp mode performance is wery low, below 130 Mbps.
Configuration - x86 core quad Q6600 + dualhead INTEL 82576 NIC + rtl8169 NIC for communication with billing server, (radius protocol). Billing - Abills 0.52b. Simple queue for each user, connected via pptp.
In case of 120+ users online i see that LA on 1 of cores CPU reaches 100% (98-100% by interrupts). This 100% LA moves from 1 core to another, and total throughput of BRAS limited near 130 Mbps. In this case other cores loaded on 5-20% (system-resources-cpu) in tools-profile i see that this core loaded by queuing & firewall (mostly).
Today i bind tx-rx-lanes of NICs to different cores of cpu, and increase ethernet-default queue to 400 and default-small queue to 50. Now i gathering statistics of LA.
On this forum i read that i should disable RPS on my ethernet interfaces. How can i do this? i can't find it.
Any ideas about increasing PPtP performance? (Linux Accel-ppp + quagga?)

P.S. Cracked ROS 3.30 was very stable, but it doesn't support 82576.
I bought L5 license for ROS 5.6 and i don't know why i should pay for such instability.
P.P.S. Sorry for my English, i'm from Ukraine.

lionspy · **Posted:** Sun Jan 29, 2012 8:28 pm

i found where i can disable RPS
System - Resources - RPS

lionspy · **Posted:** Mon Jan 30, 2012 9:21 pm

this is the 100% LA on a random core of CPU

lionspy · **Posted:** Mon Jan 30, 2012 9:29 pm

another one

lionspy · **Posted:** Mon Jan 30, 2012 9:36 pm

It's a daily loadings of the backbone interface. (ether4)
interface is limited at 200Mbps by our provider.

NetworkPro · **Posted:** Tue Jan 31, 2012 11:42 pm

Hey man I was bored to death and saw you had a problem that no one helped with.

I see you use RTL8169. have you tried removing and replacing that one with one with a totally different chipset?

nest · **Posted:** Tue Jan 31, 2012 11:51 pm

I will support NetworkPro's suggestion. I have tried various ROS versions, upgrade/downgrade but always a problem. Then, changed it for a NIC with an Intel based chipset, no more problems. Might be a problem with the hardware chipset, might be a ROS driver problem. Who cares, I now know to stay away from the Realtek chipset. Haven't got the time to investigate every reason why something doesn't work! :-)

lionspy · **Posted:** Wed Feb 01, 2012 10:36 am

"+ rtl8169 NIC for communication with billing server, (radius protocol)".
it's ether6 interface with traffic = 30-40 kBps (max 50 pps)

NetworkPro · **Posted:** Wed Feb 01, 2012 11:26 am

If you take two-three supouts when this problem happens and send them to support, what is their answer about what is causing 100% CPU there ?

lionspy · **Posted:** Mon Feb 06, 2012 8:27 pm

there is their answer!
Hello,

1) in /queue interface menu set all queues to
multi-queue-ethernet-default
>> i change ethernet interfaces queues to multi-queue-ethernet-default. doesn't affected on router behavior.
I don't know how i can change queue type from "default-small" to multi-queue-ethernet-default on my dynamic pptp user interfaces.

2)in /system resources rps menu disable all entries
>>i did it 3 days before sending supouts to support. doesn't affected on router behavior.

3) in /system resources irq try to allocate cores manually (by default it at least
one core per interface)
>>i did it 3 days before sending supouts to support. doesn't affected on router behavior.
Examine this presentation:
http://www.tiktube.com/index.php?video= ... wolLonKGo=
>> no more news for me. I did all optimisations.

Regards,
Janis Megis

seems to me, it's no necessary to attach supouts - MTK support don't use them when analyses user complaints.

NetworkPro · **Posted:** Tue Feb 07, 2012 4:44 am

I like these optimisation tips. Good info.

When you allocated IRQs to cores, can you tell if the core that goes to 100% is the one with the RTL8169 IRQ allocated to it?

Does mutli-queue-etherhet-default work with the RTL8169?

lionspy · **Posted:** Tue Feb 07, 2012 12:35 pm

NetworkPro wrote:

I like these optimisation tips. Good info.

When you allocated IRQs to cores, can you tell if the core that goes to 100% is the one with the RTL8169 IRQ allocated to it?

>>no, i can't tell that - 100% LA core moves randomly, and independently from core, serving for Realtek.

Does mutli-queue-etherhet-default work with the RTL8169?

I don't know. It's no errors displayed when i change RTL queue to mutli-queue-etherhet-default.
But in mutli-queue-etherhet-default kind ob buffer is "unknown" instead of "pfifo" on etherhet-default.

NetworkPro · **Posted:** Tue Feb 07, 2012 1:32 pm

use the multi-queue. Does it still happen, the problem?

Send one-two supouts when it is not happening and two-three supouts when it is happening. Maybe they could catch it by comparison.

lionspy · **Posted:** Wed Feb 08, 2012 10:38 pm

Yes, it still happen.
The problem exist when any pptp user (it's may be only 1 user, may be few users) exceeds speed of 20-25 MBps. 100% LA on 1 of cores of CPU, moving from 1 core to another.
when all users running at 6MBps it's all right, load normally balanced by CPU cores. But in this case overall load don't overs 100-110 MBps and i don't know which will be result, when total traffic exceeds,for example, 150 MBps.
Seems to me, it's limitation of userspace pptp server software in MTK ROS.

NetworkPro · **Posted:** Thu Feb 09, 2012 8:30 am

Can you make a Linux installation with the same router cofiguration and boot it with the same hardware to test ?

lionspy · **Posted:** Thu Feb 09, 2012 9:59 pm

i'm moved from slackware +2.6.23 kernel + accel-pptp 0.8.5
+ self-made billing (all settings and DB, statistics in mysql, lot of perl scripts) with NAT on this Q6600
+ 2 100 Mbit intel desktop NIC's and 80-95 MBps peak traffic load
to ROS 3.30 + Abills billing + 2 D-link DGE-530T NIC's (marvell?).+ BGP default route (to handle real IP per each user)
than i bought 82576 NIC $ ROS 5.6 to handle 82576.
Troubles began.
On attached interface utilisation diagram:
apr. 26, 2011 - linux replaced by ROS 3.30
sept. 17, 2011 - ros 3.30 upgraded to ROS 5.6, NIC's upgraded to 82576 dualhead pci-e.

I move to ROS because my self-made billing can't handle users with real IP's.(it has been build in far 2002-2004y).
Today i'm prepare my network to move to pppoe, than - to ipoe.(i have 4 routed subnets /24)

Chupaka · **Posted:** Fri Feb 10, 2012 2:56 pm

lionspy wrote:

I move to ROS because my self-made billing can't handle users with real IP's.(it has been build in far 2002-2004y).

what's difference between real IPs and... uhm... unreal (?) IPs?..

lionspy · **Posted:** Fri Feb 10, 2012 8:43 pm

That system has built-in NAT and hasn't BGP.
Self-made, not by me.

NetworkPro · **Posted:** Fri Feb 10, 2012 8:58 pm

Can you swap the motherboard with a different model to test? It is what I would do if I get stuck like this due to hunch about hardware+drivers issues.

lionspy · **Posted:** Sat Feb 11, 2012 8:38 pm

similar results on intel G33 (Gigabyte) & intel G41 (Asus).

seems to me - it's userlevel code of pptp server software on mikrotik.

I've tested this box in both pptp (all users) + 2-3 pppoe tunnels (test users, 1 mschapv2+mppe + 2 md5 chap without mppe), in pppoe throughput was near 100M, this load don't overload the CPU, i achieve (65-70 MBps download/ 25-60 MBps upload)/user (netbook on intel atom N450 CPU) in torrents and no softlockups on BRAS CPU.

NetworkPro · **Posted:** Sat Feb 11, 2012 9:01 pm

Cool that you tested with different motherboards. At the time the problem happens - grab a supout.rif and send to support. If it is truly a bug they should simply fix it in the next RouterOS release or sooner by providing you a custom version.

Cheers.

mixer · **Posted:** Mon Feb 13, 2012 3:36 am

I have a similar problem... BRAS in MT 4.x fine... upgrade to 5.8 - 5.12... trouble -> the (2 items) Intel D510MO (NIC-> realtek) has frozen

~80 and ~35 PPPoE sesion

Meybe trouble is in ROS diver NIC-> realtek?

Sorry for my English, i'm from Poland.

NetworkPro · **Posted:** Mon Feb 13, 2012 9:43 am

I have a Intel 510MO that is working with 5.12. The BIOS is updated though.

mixer · **Posted:** Mon Feb 13, 2012 2:58 pm

@NetworkPro 510MO i s a pppoe bras?

I have 4 x 510MO:
- 80 PPoE -> frozen 3 x
- 35 PPoE -> frozen 1x
- 28 PPoE -> no frozen
- only router BGP/OSPF -> no frozen

In MT 4.7 is no problem, uptime ~ 240 days. In MT 5.8 - 5.12 uptime is 2 - 20 days and frozen

NetworkPro · **Posted:** Mon Feb 13, 2012 3:45 pm

What do you do for cooling ? Its a hot board.

mixer · **Posted:** Mon Feb 13, 2012 4:51 pm

2 fans in the rack.
Is installed in the room where it is currently -5 ... -20 deg C

So it's not the heat. Especially because the MT 4.7 worked well. This is definitely a problem in MT 5.x

NetworkPro · **Posted:** Mon Feb 13, 2012 5:48 pm

at -20 stuff gets smaller and electrolitic capacitors may have different capacity at that point, maybe its this ?

lionspy · **Posted:** Mon Feb 13, 2012 9:27 pm

NetworkPro wrote:

Cool that you tested with different motherboards. At the time the problem happens - grab a supout.rif and send to support. If it is truly a bug they should simply fix it in the next RouterOS release or sooner by providing you a custom version.

Cheers.

it's no sense to sending supout.rif - I've send 3 supouts, they didn't analyze them - they proposed me to do that optimizations, that i made on that time when i send supouts.
or suppout doesn't include info about my configuration?

NetworkPro · **Posted:** Mon Feb 13, 2012 9:34 pm

Do the configs they want and send new supouts. They should eventually check em out. In more debth.

mixer · **Posted:** Mon Feb 13, 2012 11:16 pm

NetworkPro wrote:

at -20 stuff gets smaller and electrolitic capacitors may have different capacity at that point, maybe its this ?

OK, but why at 4.17 mt work ok? This is not the temperature ... because the other atom (not bras) in the same conditions work ok.

You did not answer me, do you work at 510MO as BRAS? If so, how many PPPoE sessions? Are you using the integrated NIC (Realtek), or another?

NetworkPro · **Posted:** Tue Feb 14, 2012 11:52 am

I am using it as a backup off-site PPPoE concentrator/main router serving a few clients while waiting for the main one to fail and bunch of PPPoE connections to come through (never happened so far, the main one is a good one

). I use the onboard gig eth for main Internet connection and the PCI slot with a MT RB44GV eth card (yes PCI is a bottleneck but traffic is low there).

Only problems so far were Silicon Motion USB controller inside USB Flash - incomatibility with Intel Chipset, I believe.
If there were other issues - they looked like USB Store problems.

mixer · **Posted:** Thu Feb 16, 2012 4:24 am

You do not have much traffic there? With less traffic for me no problems.

It seems to me that I have found the cause of problems with the NIC Realtek: viewtopic.php?f=2&t=59407

Is there another motherboard because I have it in the office (D510 is far away). But this is a similar motherboard.

NetworkPro · **Posted:** Thu Feb 16, 2012 7:50 am

Cool info. I will see if I can set up test for the D510MO with enormous amounts of small UDP packets soon.

Cheers.

mixer · **Posted:** Fri Feb 17, 2012 4:53 pm

@lionspy: have you tried to change Realtka on Intel? Możesz pokazać screen Queue Liste -> Interface Queues?

I see the Interface Queues at MT 5.12 PPPoE interfaces as a set only-hardware-queue.
It can also cause a problem?

Who is online