ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.
RouterOS general discussion

34 posts   •   Page 1 of 1
lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Sun Jan 29, 2012 5:19 pm

ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86. every 5-12 hours i see a a soft lockup on a random core of the cpu, after that whole system hangs. ROS 5.11 have much more performance on my BRAS (about 150+ Mbps, than it hangs). supout.rif does not creates in case of hangup.
ROS 5.6 seems to be very stable, uptime is over 2 weeks, generally.
but in pptp mode performance is wery low, below 130 Mbps.
Configuration - x86 core quad Q6600 + dualhead INTEL 82576 NIC + rtl8169 NIC for communication with billing server, (radius protocol). Billing - Abills 0.52b. Simple queue for each user, connected via pptp.
In case of 120+ users online i see that LA on 1 of cores CPU reaches 100% (98-100% by interrupts). This 100% LA moves from 1 core to another, and total throughput of BRAS limited near 130 Mbps. In this case other cores loaded on 5-20% (system-resources-cpu) in tools-profile i see that this core loaded by queuing & firewall (mostly).
Today i bind tx-rx-lanes of NICs to different cores of cpu, and increase ethernet-default queue to 400 and default-small queue to 50. Now i gathering statistics of LA.
On this forum i read that i should disable RPS on my ethernet interfaces. How can i do this? i can't find it.
Any ideas about increasing PPtP performance? (Linux Accel-ppp + quagga?)

P.S. Cracked ROS 3.30 was very stable, but it doesn't support 82576.
I bought L5 license for ROS 5.6 and i don't know why i should pay for such instability.
P.P.S. Sorry for my English, i'm from Ukraine.

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Sun Jan 29, 2012 9:28 pm

i found where i can disable RPS
System - Resources - RPS

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Mon Jan 30, 2012 10:21 pm

this is the 100% LA on a random core of CPU
Attachments
mtk-la100.JPG
100% LA
mtk-la100.JPG (221.03 KiB) Viewed 2754 times

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Mon Jan 30, 2012 10:29 pm

another one
Attachments
mtk3-la100.JPG
100% LA
mtk3-la100.JPG (192.77 KiB) Viewed 2752 times

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Mon Jan 30, 2012 10:36 pm

It's a daily loadings of the backbone interface. (ether4)
interface is limited at 200Mbps by our provider.
Attachments
c_3569_day.png
daily load
c_3569_day.png (32.8 KiB) Viewed 2751 times

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Wed Feb 01, 2012 12:42 am

Hey man I was bored to death and saw you had a problem that no one helped with.

I see you use RTL8169. have you tried removing and replacing that one with one with a totally different chipset?
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

User avatar
nest
Forum Veteran
Forum Veteran
 
Posts: 733
Joined: Tue Feb 27, 2007 2:52 am
Location: Guernsey and UK

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by nest » Wed Feb 01, 2012 12:51 am

I will support NetworkPro's suggestion. I have tried various ROS versions, upgrade/downgrade but always a problem. Then, changed it for a NIC with an Intel based chipset, no more problems. Might be a problem with the hardware chipset, might be a ROS driver problem. Who cares, I now know to stay away from the Realtek chipset. Haven't got the time to investigate every reason why something doesn't work! :-)
Ron Touw - Mikrotik Certified Trainer
LinITX.com - MultiThread Consultants
Get your MikroTik RBs and Training: http://linitx.com/category/166
Largest Official UK MikroTik Distributor
IRC channel: #routerboard on irc.z.je (IPv4), 6.irc.z.je (IPv6)

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Wed Feb 01, 2012 11:36 am

"+ rtl8169 NIC for communication with billing server, (radius protocol)".
it's ether6 interface with traffic = 30-40 kBps (max 50 pps)

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Wed Feb 01, 2012 12:26 pm

If you take two-three supouts when this problem happens and send them to support, what is their answer about what is causing 100% CPU there ?
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Mon Feb 06, 2012 9:27 pm

there is their answer!
Hello,

1) in /queue interface menu set all queues to
multi-queue-ethernet-default
>> i change ethernet interfaces queues to multi-queue-ethernet-default. doesn't affected on router behavior.
I don't know how i can change queue type from "default-small" to multi-queue-ethernet-default on my dynamic pptp user interfaces.

2)in /system resources rps menu disable all entries
>>i did it 3 days before sending supouts to support. doesn't affected on router behavior.

3) in /system resources irq try to allocate cores manually (by default it at least
one core per interface)
>>i did it 3 days before sending supouts to support. doesn't affected on router behavior.
Examine this presentation:
http://www.tiktube.com/index.php?video= ... wolLonKGo=
>> no more news for me. I did all optimisations.

Regards,
Janis Megis

seems to me, it's no necessary to attach supouts - MTK support don't use them when analyses user complaints.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Tue Feb 07, 2012 5:44 am

I like these optimisation tips. Good info.

When you allocated IRQs to cores, can you tell if the core that goes to 100% is the one with the RTL8169 IRQ allocated to it?

Does mutli-queue-etherhet-default work with the RTL8169?
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Tue Feb 07, 2012 1:35 pm

NetworkPro wrote:I like these optimisation tips. Good info.

When you allocated IRQs to cores, can you tell if the core that goes to 100% is the one with the RTL8169 IRQ allocated to it?

>>no, i can't tell that - 100% LA core moves randomly, and independently from core, serving for Realtek.


Does mutli-queue-etherhet-default work with the RTL8169?


I don't know. It's no errors displayed when i change RTL queue to mutli-queue-etherhet-default.
But in mutli-queue-etherhet-default kind ob buffer is "unknown" instead of "pfifo" on etherhet-default.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Tue Feb 07, 2012 2:32 pm

use the multi-queue. Does it still happen, the problem?

Send one-two supouts when it is not happening and two-three supouts when it is happening. Maybe they could catch it by comparison.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Wed Feb 08, 2012 11:38 pm

Yes, it still happen.
The problem exist when any pptp user (it's may be only 1 user, may be few users) exceeds speed of 20-25 MBps. 100% LA on 1 of cores of CPU, moving from 1 core to another.
when all users running at 6MBps it's all right, load normally balanced by CPU cores. But in this case overall load don't overs 100-110 MBps and i don't know which will be result, when total traffic exceeds,for example, 150 MBps.
Seems to me, it's limitation of userspace pptp server software in MTK ROS.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Thu Feb 09, 2012 9:30 am

Can you make a Linux installation with the same router cofiguration and boot it with the same hardware to test ? :)
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Thu Feb 09, 2012 10:59 pm

i'm moved from slackware +2.6.23 kernel + accel-pptp 0.8.5
+ self-made billing (all settings and DB, statistics in mysql, lot of perl scripts) with NAT on this Q6600
+ 2 100 Mbit intel desktop NIC's and 80-95 MBps peak traffic load
to ROS 3.30 + Abills billing + 2 D-link DGE-530T NIC's (marvell?).+ BGP default route (to handle real IP per each user)
than i bought 82576 NIC $ ROS 5.6 to handle 82576.
Troubles began.
On attached interface utilisation diagram:
apr. 26, 2011 - linux replaced by ROS 3.30
sept. 17, 2011 - ros 3.30 upgraded to ROS 5.6, NIC's upgraded to 82576 dualhead pci-e.

I move to ROS because my self-made billing can't handle users with real IP's.(it has been build in far 2002-2004y).
Today i'm prepare my network to move to pppoe, than - to ipoe.(i have 4 routed subnets /24)
Attachments
c_3569_year.png
yearly traffic consumption
c_3569_year.png (32.54 KiB) Viewed 2409 times

User avatar
Chupaka
Forum Guru
Forum Guru
 
Posts: 7229
Joined: Mon Jun 19, 2006 11:15 pm
Location: Home Network Ltd., Minsk, Belarus

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by Chupaka » Fri Feb 10, 2012 3:56 pm

lionspy wrote:I move to ROS because my self-made billing can't handle users with real IP's.(it has been build in far 2002-2004y).

what's difference between real IPs and... uhm... unreal (?) IPs?.. :)
For every complex problem, there is a solution that is simple, neat, and wrong.

¡ɹǝ|SOɹǝʇnoɹ ʞıʇoɹʞıW ɯ‚|

MikroTik. Your life. Your routing.

skype: pavel.skuratovich

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Fri Feb 10, 2012 9:43 pm

That system has built-in NAT and hasn't BGP.
Self-made, not by me.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Fri Feb 10, 2012 9:58 pm

Can you swap the motherboard with a different model to test? It is what I would do if I get stuck like this due to hunch about hardware+drivers issues.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Sat Feb 11, 2012 9:38 pm

similar results on intel G33 (Gigabyte) & intel G41 (Asus).

seems to me - it's userlevel code of pptp server software on mikrotik.

I've tested this box in both pptp (all users) + 2-3 pppoe tunnels (test users, 1 mschapv2+mppe + 2 md5 chap without mppe), in pppoe throughput was near 100M, this load don't overload the CPU, i achieve (65-70 MBps download/ 25-60 MBps upload)/user (netbook on intel atom N450 CPU) in torrents and no softlockups on BRAS CPU.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Sat Feb 11, 2012 10:01 pm

Cool that you tested with different motherboards. At the time the problem happens - grab a supout.rif and send to support. If it is truly a bug they should simply fix it in the next RouterOS release or sooner by providing you a custom version.

Cheers.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

mixer
just joined
 
Posts: 7
Joined: Mon Feb 13, 2012 4:18 am

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by mixer » Mon Feb 13, 2012 4:36 am

I have a similar problem... BRAS in MT 4.x fine... upgrade to 5.8 - 5.12... trouble -> the (2 items) Intel D510MO (NIC-> realtek) has frozen :( ~80 and ~35 PPPoE sesion

Meybe trouble is in ROS diver NIC-> realtek?



Sorry for my English, i'm from Poland.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Mon Feb 13, 2012 10:43 am

I have a Intel 510MO that is working with 5.12. The BIOS is updated though.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

mixer
just joined
 
Posts: 7
Joined: Mon Feb 13, 2012 4:18 am

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by mixer » Mon Feb 13, 2012 3:58 pm

@NetworkPro 510MO i s a pppoe bras?

I have 4 x 510MO:
- 80 PPoE -> frozen 3 x
- 35 PPoE -> frozen 1x
- 28 PPoE -> no frozen
- only router BGP/OSPF -> no frozen

In MT 4.7 is no problem, uptime ~ 240 days. In MT 5.8 - 5.12 uptime is 2 - 20 days and frozen :(

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Mon Feb 13, 2012 4:45 pm

What do you do for cooling ? Its a hot board.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

mixer
just joined
 
Posts: 7
Joined: Mon Feb 13, 2012 4:18 am

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by mixer » Mon Feb 13, 2012 5:51 pm

2 fans in the rack.
Is installed in the room where it is currently -5 ... -20 deg C

So it's not the heat. Especially because the MT 4.7 worked well. This is definitely a problem in MT 5.x

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Mon Feb 13, 2012 6:48 pm

at -20 stuff gets smaller and electrolitic capacitors may have different capacity at that point, maybe its this ?
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

lionspy
just joined
 
Posts: 17
Joined: Fri Jan 27, 2012 12:40 pm

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by lionspy » Mon Feb 13, 2012 10:27 pm

NetworkPro wrote:Cool that you tested with different motherboards. At the time the problem happens - grab a supout.rif and send to support. If it is truly a bug they should simply fix it in the next RouterOS release or sooner by providing you a custom version.

Cheers.

it's no sense to sending supout.rif - I've send 3 supouts, they didn't analyze them - they proposed me to do that optimizations, that i made on that time when i send supouts.
or suppout doesn't include info about my configuration?

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Mon Feb 13, 2012 10:34 pm

Do the configs they want and send new supouts. They should eventually check em out. In more debth.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

mixer
just joined
 
Posts: 7
Joined: Mon Feb 13, 2012 4:18 am

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by mixer » Tue Feb 14, 2012 12:16 am

NetworkPro wrote:at -20 stuff gets smaller and electrolitic capacitors may have different capacity at that point, maybe its this ?


OK, but why at 4.17 mt work ok? This is not the temperature ... because the other atom (not bras) in the same conditions work ok.

You did not answer me, do you work at 510MO as BRAS? If so, how many PPPoE sessions? Are you using the integrated NIC (Realtek), or another?

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Tue Feb 14, 2012 12:52 pm

I am using it as a backup off-site PPPoE concentrator/main router serving a few clients while waiting for the main one to fail and bunch of PPPoE connections to come through (never happened so far, the main one is a good one :P). I use the onboard gig eth for main Internet connection and the PCI slot with a MT RB44GV eth card (yes PCI is a bottleneck but traffic is low there).

Image

Only problems so far were Silicon Motion USB controller inside USB Flash - incomatibility with Intel Chipset, I believe.
If there were other issues - they looked like USB Store problems.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

mixer
just joined
 
Posts: 7
Joined: Mon Feb 13, 2012 4:18 am

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by mixer » Thu Feb 16, 2012 5:24 am

You do not have much traffic there? With less traffic for me no problems.

It seems to me that I have found the cause of problems with the NIC Realtek: viewtopic.php?f=2&t=59407

Is there another motherboard because I have it in the office (D510 is far away). But this is a similar motherboard.

User avatar
NetworkPro
Forum Guru
Forum Guru
 
Posts: 1345
Joined: Mon Jan 05, 2009 7:23 pm
Location: The World

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by NetworkPro » Thu Feb 16, 2012 8:50 am

Cool info. I will see if I can set up test for the D510MO with enormous amounts of small UDP packets soon.

Cheers.
wiki.mikrotik.com/wiki/UPnP_Multi-WAN <- CONTRIBUTORS WANTED
wiki.mikrotik.com/wiki/NetworkPro_on_Quality_of_Service <- contributors welcome

mixer
just joined
 
Posts: 7
Joined: Mon Feb 13, 2012 4:18 am

Re: ROS 5.10 - 5.12 very unstable as pptp BRAS on an x86.

by mixer » Fri Feb 17, 2012 5:53 pm

@lionspy: have you tried to change Realtka on Intel? Możesz pokazać screen Queue Liste -> Interface Queues?

I see the Interface Queues at MT 5.12 PPPoE interfaces as a set only-hardware-queue.
It can also cause a problem?

34 posts   •   Page 1 of 1

Who is online

Users browsing this forum: cbergerson, heislerb and 47 guests

It is currently Mon Nov 24, 2014 4:39 pm