Community discussions

MUM Europe 2020
 
Rivera
Member Candidate
Member Candidate
Posts: 105
Joined: Thu Jul 21, 2011 7:42 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Oct 03, 2012 4:33 pm

Does this patches (for mipsbe) included in 6.0rc1?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Oct 05, 2012 9:34 am

yes, all of the changes regarding MetaROUTER changes done for 5.21rc1 are in 6.0rc1
 
guy1with1mr1problems
newbie
Posts: 43
Joined: Wed Jun 13, 2012 12:34 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Oct 16, 2012 1:00 pm

I just got a crash on powerpc (rb800) with both 6.0rc1 and 5.21.

When I load the 'malicious' image, the board prints this on its serial port
BUG(1): unhandled unknown exception!
pc=0x30023064 cpc=0x80000f64 lr=0x10013750 ctr=0x30023108
regs=0xfc926f60 cr=0x30000093 xer=0xc000006b
r0=0x10013750 r1=0x7fffc640 r2=0x00000000 r3=0x7fffc710 
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x1001eb40 
r8=0x0000000c r9=0x10030000 r10=0x100324e8 r11=0x7fffc600 
r12=0x30000093 r13=0x10038018 r14=0x7dfdbf7b r15=0xe9ffe5ef 
r16=0xffda7cbc r17=0xffbfa775 r18=0x7ff7fe5d r19=0xcf7ffffe 
r20=0xffdb7eff r21=0xdfbb7dbf r22=0xbffbbef8 r23=0x00000000 
r24=0x7fffc958 r25=0x1001cd20 r26=0x7fffcfe7 r27=0x10001f08 
r28=0x10002420 r29=0x00000000 r30=0x7fffca14 r31=0x00000002 
dear=0x10032004 msr=0x0002d000(0002d000) esr=0x00000080
and then hangs. This happens every time.
by trial and error, we determined this happens immediately when busybox(uclibc) with hardfloat and '-march=powerpc' runs.

Also another problem: Metarouter sets kernel parameters on powerpc. Unfortunately, in regular kernels, this makes setting custom kernel parameters inside the image impossible, as they always get overriden by those Metarouter sends. We had to modify the kernel sources to force it to accept our boot parameters.
 
guy1with1mr1problems
newbie
Posts: 43
Joined: Wed Jun 13, 2012 12:34 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Oct 16, 2012 1:11 pm

I think I might have an idea. Looks like our code mistakenly uses binary incompatible fpu instructions, it should be using fpe. But still, it definitely shouldn't crash the host, maybe just the vm kernel.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Oct 16, 2012 2:52 pm

can you share the image, so i could get it?
 
guy1with1mr1problems
newbie
Posts: 43
Joined: Wed Jun 13, 2012 12:34 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Oct 16, 2012 2:57 pm

can you share the image, so i could get it?
no problem, but due to its proprietary nature, could we agree on exchange via e-mail or something like that? It's a single 3.3 MB .tar.gz file
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Oct 16, 2012 3:00 pm

then send it to support@mikrotik.com
 
guy1with1mr1problems
newbie
Posts: 43
Joined: Wed Jun 13, 2012 12:34 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Oct 16, 2012 3:05 pm

then send it to support@mikrotik.com
mail sent from my personal account, its subject is: 'powerpc metarouter image that crashes RB800' (without the quotes). Thanks.
 
alfil
just joined
Posts: 2
Joined: Tue Mar 09, 2010 1:27 pm

Re: MetaROUTER stability issues on certain MIPSBE boards

Thu Oct 18, 2012 11:20 am

...
We did various tests on RB450G boards, most of them were conducted by NathanA. The conclusions for this board so far are:
1.) The powersupply does have an influence but isn't the cause.
2.) Disabling hardware monitoring by a hack seems to improve the stability, assumably because of much lowered IRQ load.
3.) It seems as there has been no significant progress by the MT devs, maybe with point 2 there will be.
UPDATE: It more and more seems to narrow down to the GPIO ISR(s) as NathanA reports that high IRQ load from the switch chip doesn't seem to cause issues.
...
About work on the RG450G.
I have more than half year steadily working the OpenWrt Kamikaze - With X-Wrt Extensions 8.09.
And at first I had constant rebooting. But after I drill the case of RG450G and mount the fan all the problems were solved.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Oct 18, 2012 1:20 pm

there where different methods on how to delay the hangs, like using 12V PSU instead of 24V that reduced interrupt count.

Same as removing voltage.ko driver that removes the cause of the hang not the reason. But still with removed driver router still could hang, as that was not the only interrupt generator, but it does that at correct frequency to cause the problems.
 
Wazza
newbie
Posts: 39
Joined: Thu Oct 13, 2011 10:43 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Oct 18, 2012 2:15 pm

Has any progress actually been made with PPC systems?

I've recently upgraded 2 RB1100AH units, to 5.21, thinking that it "might help", and we're only running a couple (2 ROS metarouters), and I'm still seeing daily reboots, under very lightly loaded traffic conditions...

Wazza
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Oct 29, 2012 9:32 am

over last week a lot of work went into metarotuer on mips. It would be really nice if you could try and see if this still hangs/freezes for you. I ran tests over the weekend (55hours) and did not get eny problems.

So if any of you can put it somewhere, that would be highly appretiated.

Here the link for new download:
http://www.mikrotik.com/download/share/ ... .22rc1.npk

regarding ppc metarotuer - fixed crash when FPU exception was rised.
 
xbojer
just joined
Posts: 14
Joined: Thu Oct 25, 2012 7:00 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Oct 31, 2012 2:08 am

What changed in RouterOS v6 regarding metarouter on mipsbe? MR seems to work fine on my rb2011uas-2hnd-in on versions 5.xx but trying any rc of 6.0 causes router crash on adding metarouter...
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Oct 31, 2012 11:32 am

we will fix that
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Nov 12, 2012 6:24 pm

over last week a lot of work went into metarotuer on mips. It would be really nice if you could try and see if this still hangs/freezes for you. I ran tests over the weekend (55hours) and did not get eny problems.

So if any of you can put it somewhere, that would be highly appretiated.

Here the link for new download:
http://www.mikrotik.com/download/share/ ... .22rc1.npk

regarding ppc metarotuer - fixed crash when FPU exception was rised.
I will do some tests, one thing I noticed right after starting is that health vlaues now are totally bogus. I think this was to be expected
Lets see how it will behave in longterm testing.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 13, 2012 11:32 am

unfortunately due to how metarouter operates (one of my older posts in this thread) health monitoring values can be off the mark.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 13, 2012 11:38 am

unfortunately due to how metarouter operates (one of my older posts in this thread) health monitoring values can be off the mark.
Yes I know, that's why me and other people recommended disabling health monitoring completely. ;-)
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 13, 2012 2:53 pm

first ppc metarouter has to start work, and resolve issues there, maybe afterwards someone will make this cosmetic change.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 13, 2012 2:55 pm

first ppc metarouter has to start work, and resolve issues there, maybe afterwards someone will make this cosmetic change.
Understandable ;)
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 13, 2012 6:28 pm

I am sorry, but i have to report the 12 second hang as still NOT fixed.
Still exactly the same behaviour like in 5.21rc2 and 5.21, whole system freezes for 12 seconds and resumes afterwards like nothing happened.
Therefore I request MT to finally provide an image without hardware monitoring for testing, lets just test if my and other RB450Gs then work without those hangs...

EDIT: I did some math because I find those 12 seconds suspicious and discovered that 12,89 seconds is exactly the time it takes for a 32bit counter, which is incremented every 333µs (or with 3000 Hz), to roll over. You say the MR hypervisor ticks every 300Hz maybe your devs use a scaled 3000 Hz timer for this. This is to accurate to be a coincidence, don't you think?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Nov 14, 2012 3:55 pm

how often these happen? can you describe configuration, mine was like this:

1 metarouter running with 1 dynamic vif and 1 static vif interface. both bridged with physical port. in network only usual broadcast chatter. No freezes so far.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Nov 14, 2012 4:27 pm

how often these happen?
It now happened once in 24h .
can you describe configuration, mine was like this:

1 metarouter running with 1 dynamic vif and 1 static vif interface. both bridged with physical port. in network only usual broadcast chatter. No freezes so far.
The complete config without any steps left out was asd following:

On host
--------
1.) Fresh netinstall setup.
1a.) enable IPv6, disable hotspot and wireless packages.
2.) Create bridge with ether1 in it.
3.) Add dhcp client on host.
4.) Create 2 metarouters each with 32mb ram.
5.) Connect each metarouter with one dynamic vif to bridge.
6.) Start metarouters and add dhcp client to ether1 inside each metarouter.

On unrelated MT system
-----------------------
Ping host from above and both metarouters

On unrelated Mac OS X box
--------------------------
Ping host from above and both metarouters
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 15, 2012 9:00 am

off, to test some metarotuers. Thank you for details of your setup.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 15, 2012 9:10 am

off, to test some metarotuers. Thank you for details of your setup.
I wish you luck. The bug seems to be related to the scheduling of the hypervisor I think. An overflow (maybe with interrupt) seems to "wake" the scheduler up after the normal tick has been somehow blocked. Your devs should be able to diagnose this better then me or you, I guess. ;-)
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 15, 2012 11:01 am

there is more than this, as normal tick, happens every 333us (microsecond) in case of 12.88 second delay these are 38K missing ticks. and at least 10K missing ticks that would bring us back to time-slice of main guest of hypervisor with 2 guests running, as hypervisor sees them as 3 guests where one of them has bit more privileges.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 15, 2012 11:10 am

Hmm your are right with your 38k ticks, I made a mistake in my calculation. Never do math aber 10:00pm ;-)
I will come back to you, as soon as get this 32bit counter thing figured out correctly.

EDIT: Is there any chance, that you use a 32bit integer as a timestamp with microsecond accuracy? Cause what I originally calculated was 2^32us / 333us which gives 12,89s. I can't figure out the rest, as I didn't implement your scheduler or timing logik.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 15, 2012 11:25 am

i will relay this to devs. Maybe they can come with something useful.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 16, 2012 11:06 pm

EDIT: Merged post with next post.
Last edited by timberwolf on Sat Nov 17, 2012 10:15 am, edited 1 time in total.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Sat Nov 17, 2012 9:55 am

In the meantime I started conducting the exact same tests with a RB2011UAS-RM but with the following modifications:
1.) ROS 5.21
2.) Host an MRs use static IPs
3.) Host runs NTP client

On first sight, I got the same freezes after just 4 hours but with differing packet losses and timeouts,to rule out any errors in the test setup I restarted the test and setup a dedicated RB450G for monitoring in addition to the Mac OS X machine. I will report back, but please be so kind and also setup the same tests with a RB2011UAS if your ressources allow.

--

Ok here are the results of running this setup for one night, which don't look very good:
192.168.10.100 ist the host, .101 is the first and .102 the second MetaRouter.

Pings from RB450G
----------------
HOST: IP 192.168.10.100
sent=37938 received=37924 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=191ms

MR1: IP 192.168.10.101
sent=37936 received=36951 packet-loss=2% min-rtt=0ms avg-rtt=2ms max-rtt=485ms

MR2: IP 192.168.10.102
sent=37939 received=37925 packet-loss=0% min-rtt=0ms avg-rtt=3ms max-rtt=82ms

Pings from Mac OS X box
---------------------
HOST: IP 192.168.10.100
39006 packets transmitted, 39006 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.255/3.841/13489.331/152.880 ms

MR1: IP 192.168.10.101
39003 packets transmitted, 38076 packets received, 2.4% packet loss
round-trip min/avg/max/stddev = 0.757/382.016/757900.732/15178.624 ms

MR2: IP 192.168.10.102
38997 packets transmitted, 38997 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.805/5.513/13495.470/152.992 ms

As ROS discards pings after a timeout and Mac OS X doesn't the results differ. The Host and MR2 seemed to freeze for the same amount of time 13.5seconds(+/- 1s), MR1 showed a different behaviour of dropping pings totally.

I will restart the tests with ROS5.22rc1.

--

Same behaviour with 5.22rc1, this time it happened twice within just 20 minutes.

Pings from RB450G
----------------
HOST: IP 192.168.10.100
sent=1263 received=1237 packet-loss=2% min-rtt=0ms avg-rtt=1ms max-rtt=995ms

MR1: IP 192.168.10.101
sent=1261 received=1234 packet-loss=2% min-rtt=0ms avg-rtt=2 max-rtt=753ms

MR2: IP 192.168.10.102
sent=1262 received=1235 packet-loss=2% min-rtt=0ms avg-rtt=2ms max-rtt=672ms

Pings from Mac OS X box
---------------------
HOST: IP 192.168.10.100
1241 packets transmitted, 1241 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.311/167.836/14015.345/1252.018 ms

MR1: IP 192.168.10.101
1241 packets transmitted, 1241 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.827/166.600/14138.749/1238.100 ms

MR2: IP 192.168.10.102
1242 packets transmitted, 1242 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.786/161.610/13734.863/1209.999 ms

So MetaRouters on RB450G and RB2011UAS is still broken. I doubt that if I would tester further MIPS-BE boards I would find one which is working flawless.
As this issue even affects the RB2011 boards with a new CPU, there really must be a bug hidden inside MetaRouter (hypervisor) code which causes trouble on any platform. And as it always is the same timing behaviour it all points to an error inside some timing routine.
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Nov 19, 2012 11:41 pm

I have a problem with MetaROUTER on RB2011UAS-2HnD v6rc3, too. I have set up 3 MetaRouters for testing:
 #   NAME                 MEMORY-SIZE     DISK-SIZE     USED-DISK STATE        
 0   OpenWRT-1                  20MiB      32768kiB       8132kiB running         
 1   RouterOS-1                 24MiB      32768kiB        219kiB running         
 2   OpenWRT-2                  20MiB      32768kiB       4790kiB running       
OpenWRT-1 is r18199, downloaded from: http://openwrt.pantele.com.ua/trunk/metarouter/
and OpenWRT-2 is r30812, from: http://mirrors.muarf.org/openwrt-custom ... r/mr-mips/
RouterOS-1 is just regular MR as name says.

Their interfaces are all dynamic connected to one bridge:
 #   VIRTUAL-MACHINE         TYPE    STATIC-INTERFACE         VM-MAC
 0 A OpenWRT-1               dynamic                          02:89:
 1 A RouterOS-1              dynamic                          02:80:
 2 A OpenWRT-2               dynamic                          02:9D:
Configuration on all of them is just basic: IP numbers and gateway on interfaces. I set up Tools\Netwatch for every of them (ping every 5 seconds) and every time MetaRouter starts or stops responding, it is logged. Example of Netwatch script that I created for every MR:
add down-script=":log warning \"- - - - - - - RouterOS is not responding\"" host=192.168.0.3 interval=5s up-script=\
    ":log warning \"- - - - - - - RouterOS is OK\""
And now my log is:
19:58:25 script,warning - - - - - - - RouterOS is OK 
20:05:58 script,warning - - - - - - - RouterOS is not responding 
20:13:31 script,warning - - - - - - - RouterOS is OK 
20:15:28 script,warning - - - - - - - RouterOS is not responding 
20:22:20 script,warning - - - - - - - RouterOS is OK 
20:24:18 script,warning - - - - - - - RouterOS is not responding 
20:32:50 script,warning - - - - - - - RouterOS is OK 
20:33:03 script,warning - - - - - - - RouterOS is not responding 
20:34:21 script,warning - - - - - - - RouterOS is OK 
20:34:28 script,warning - - - - - - - RouterOS is not responding 
20:34:55 script,warning - - - - - - - RouterOS is OK 
20:35:33 script,warning - - - - - - - RouterOS is not responding 
20:36:56 script,warning - - - - - - - RouterOS is OK 
20:37:03 script,warning - - - - - - - RouterOS is not responding 
20:41:15 script,warning - - - - - - - RouterOS is OK 
20:41:23 script,warning - - - - - - - OpenWRT-1 is not responding 
20:43:10 script,warning - - - - - - - OpenWRT-1 is OK 
20:43:28 script,warning - - - - - - - OpenWRT-2 is not responding 
20:46:45 script,warning - - - - - - - OpenWRT-2 is OK 
20:47:08 script,warning - - - - - - - RouterOS is not responding 
20:47:58 script,warning - - - - - - - RouterOS is OK 
20:48:23 script,warning - - - - - - - RouterOS is not responding 
20:50:26 script,warning - - - - - - - RouterOS is OK 
20:51:33 script,warning - - - - - - - OpenWRT-2 is not responding 
20:53:22 script,warning - - - - - - - OpenWRT-2 is OK 
20:58:33 script,warning - - - - - - - RouterOS is not responding 
21:01:18 script,warning - - - - - - - RouterOS is OK 
22:01:43 script,warning - - - - - - - RouterOS is not responding 
22:01:51 script,warning - - - - - - - RouterOS is OK 
22:12:08 script,warning - - - - - - - RouterOS is not responding 
22:13:15 script,warning - - - - - - - RouterOS is OK 
And on night time:
02:25:28 script,warning - - - - - - - RouterOS is not responding 
02:25:35 script,warning - - - - - - - RouterOS is OK 
02:35:58 script,warning - - - - - - - OpenWRT-2 is not responding 
02:36:10 script,warning - - - - - - - OpenWRT-2 is OK 
02:37:53 script,warning - - - - - - - RouterOS is not responding 
02:38:17 script,warning - - - - - - - RouterOS is OK 
04:08:38 script,warning - - - - - - - OpenWRT-2 is not responding 
04:09:17 script,warning - - - - - - - OpenWRT-2 is OK 
04:24:13 script,warning - - - - - - - RouterOS is not responding 
04:26:32 script,warning - - - - - - - RouterOS is OK 
06:19:43 script,warning - - - - - - - RouterOS is not responding 
06:26:06 script,warning - - - - - - - RouterOS is OK 
07:56:08 script,warning - - - - - - - RouterOS is not responding 
07:57:58 script,warning - - - - - - - RouterOS is OK 
08:44:13 script,warning - - - - - - - RouterOS is not responding 
08:44:35 script,warning - - - - - - - RouterOS is OK 
10:42:33 script,warning - - - - - - - OpenWRT-2 is not responding 
10:44:10 script,warning - - - - - - - OpenWRT-2 is OK 
12:39:23 script,warning - - - - - - - OpenWRT-2 is not responding 
12:41:13 script,warning - - - - - - - OpenWRT-2 is OK 
13:32:08 script,warning - - - - - - - OpenWRT-2 is not responding 
13:32:47 script,warning - - - - - - - OpenWRT-2 is OK 
15:27:28 script,warning - - - - - - - RouterOS is not responding 
15:29:03 script,warning - - - - - - - RouterOS is OK 
Definitely it happens more often when there is load on the router (e.g. downloading files from internet) and when MetaRouter is not responding to pings, also it's console is not responding - it freezes.
I have red this thread and I don't see any solution for this isue. Will it be fixed? Or am I doing something wrong?
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 20, 2012 8:03 am

Will it be fixed? Or am I doing something wrong?
To keep the answer short, we all hope it will be fixed anytime soon as it is an open issue since 2 years(if I'm not mistaken). And no I'am quite sure you aren't doing anything wrong.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 20, 2012 9:54 am

payday - use of 5.22[rc1] will yield a bit better results. however it is not yet completely fixed.

timberwolf - during weekend and national holiday i have generated quite a lot of useful stuff for devs to work on regarding metarouter.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 20, 2012 10:31 am

timberwolf - during weekend and national holiday i have generated quite a lot of useful stuff for devs to work on regarding metarouter.
Thanks, I must admit, right now I'm quite disappointed that RB2011 isn't working either. :-(
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 8:52 am

anyone running MetaROUTER on MIPSbe routers - your OS version of choice at the moment of release is 5.22
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 8:57 am

Are you indicating, that you might have fixed it?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 9:14 am

atm - I'd say - less likely to happen. No problematic events on debug metarouter have happened for a long time. At le
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 9:55 am

atm - I'd say - less likely to happen. No problematic events on debug metarouter have happened for a long time. At le
Hmm "less likely", what's the developers point of view? No offense, but this sounds like "we have no idea if we did fix it", don't you agree? ;-)
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 10:37 am

switch from one guest to another does not happen over one instruction, so, when guest gets distracted in the switch event it could go bananas. Now this issue has been addressed and router containing pached hypervisor did not crash in over 18h+ period.

So, all i can say now is - it is likely to be fixed as there where periods in time that router did not crash even without a patch. Anything more conclusive i can say only after weekend.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 10:44 am

Thank you for the additional information.
I have one question left for now, you talk about crashes, what about those freezes which I've even seen on a RB2011UAS?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 10:52 am

these actually are freezes, router will continue to work after some time. Called them crashes due to way this was changed in debug version - i get registry dump and stacktrace after each event and guest is restarted each time this happens, sometimes host was also restarted. So, control router just freezes and continues as if nothing has happened. Last real reboots was in 5.20, at least for routers i am testing on.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 1:02 pm

Good sounds promising. I will retest with 5.22 as soon as it is available, as I happen to have some spare time at the moment.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 23, 2012 11:25 pm

Ok janisk here come the results for the 5.22 release version.

RB450G
Got a freeze after about 5 hours, this time 11,9seconds, affecting the host and both MR instances, so no significant change, just -1second(luck?).

RB2011UAS
Still the same strange behaviour I got with 5.22rc1. Host froze for about 10seconds, one MR instance froze for about 12,6seconds and the other MR instance lost some packet totally and/or was unresponsible for about 330seconds(!).

Sorry, not fixed and not even really better. :-(
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Nov 26, 2012 12:09 am

I have just downgraded my RB2011UAS-2HnD from 6rc3 to 5.22, started two metarouters (RouterOS and OpenWRT) and set up Netwatch. I will report, tomorrow.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Nov 26, 2012 10:51 am

thanks for the updates. I also got a freeze. So we are continuing on this issue.
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Nov 26, 2012 9:36 pm

03:59:08 script,warning - - - - - - - OpenWRT-2 is not responding 
04:03:36 script,warning - - - - - - - OpenWRT-2 is OK 
04:23:58 script,warning - - - - - - - OpenWRT-2 is not responding 
04:27:01 script,warning - - - - - - - OpenWRT-2 is OK 
08:42:53 script,warning - - - - - - - OpenWRT-2 is not responding 
08:44:57 script,warning - - - - - - - OpenWRT-2 is OK 
09:19:08 script,warning - - - - - - - RouterOS is not responding 
09:23:35 script,warning - - - - - - - RouterOS is OK 
10:31:28 script,warning - - - - - - - OpenWRT-2 is not responding 
10:32:20 script,warning - - - - - - - OpenWRT-2 is OK 
10:32:33 script,warning - - - - - - - RouterOS is not responding 
10:33:23 script,warning - - - - - - - RouterOS is OK 
11:24:23 script,warning - - - - - - - OpenWRT-2 is not responding 
11:27:38 script,warning - - - - - - - OpenWRT-2 is OK 
11:32:33 script,warning - - - - - - - RouterOS is not responding 
11:33:10 script,warning - - - - - - - RouterOS is OK 
15:09:23 script,warning - - - - - - - RouterOS is not responding 
15:15:15 script,warning - - - - - - - RouterOS is OK 
16:37:08 script,warning - - - - - - - RouterOS is not responding 
16:37:17 script,warning - - - - - - - RouterOS is OK 
17:03:13 script,warning - - - - - - - RouterOS is not responding 
17:10:03 script,warning - - - - - - - RouterOS is OK 
19:18:58 script,warning - - - - - - - RouterOS is not responding 
19:19:20 script,warning - - - - - - - RouterOS is OK 
19:39:43 script,warning - - - - - - - OpenWRT-2 is not responding 
19:45:35 script,warning - - - - - - - OpenWRT-2 is OK
There is no change on mine, also. Problem persists.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Nov 27, 2012 9:14 am

There is no change on mine, also. Problem persists.
Thanks for your verification effort! :)
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 29, 2012 10:28 pm

I have also noticed that short (about 10 seconds) freeze of whole device (not only MetaRouters), many times yesterday, when using console a lot. Traffic is not forwarded during that break and console does not respond.
I monitor some hosts behind MT router. Normally they are all green and when freeze happens all go red for few seconds.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Dec 03, 2012 8:27 am

I have also noticed that short (about 10 seconds) freeze of whole device (not only MetaRouters), many times yesterday, when using console a lot. Traffic is not forwarded during that break and console does not respond.
I monitor some hosts behind MT router. Normally they are all green and when freeze happens all go red for few seconds.
Yes, this is the "expected" behaviour for now, freezes are arround 12-14 seconds and affect the whole system running MR instances.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Dec 18, 2012 8:22 am

@MT
Any christmas presents planned? ;-)
 
PastuhMedvedey
newbie
Posts: 40
Joined: Fri Jan 13, 2012 1:42 pm
Location: Ukraine

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Dec 19, 2012 1:28 pm

Metarouter freezes when loading the base system more than 40-50 per cent.
When using metarouter on RB2011, metarouter hangs on version 5.22.
After a while, when the load on the base system fall below 10 percent, metarouter restores normal operation.
After you upgrade to 6.0rc5, the problem remained.
RB751U-2HnD, RB2011LS, RB1100AH, RB433, RB411GL, RBSXT, CCR1016
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Dec 19, 2012 4:15 pm

Metarouter freezes when loading the base system more than 40-50 per cent.
...
We are not even at the point, where Metarouter works in an idle situation. So I guess it's no suprise that a freezes with load. ;-)
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Jan 09, 2013 10:17 pm

From http://forum.mikrotik.com/viewtopic.php?f=15&t=55286 :
yes, of course we are working on these issues. Unfortunately MetaROUTER feature is neither simple to debug or simple to fix issues found.
Can we help you in solving this issue? I am a former Assembler programmer with some OS programming experience, and maybe there are some more MetaROUTER users that can halp analyze the problem. Don't get me wrong. I'm not saying your developers are poor but some fresh look on the problem may be helpful. :)

MetaROUTER functionality is a big selling point in RouterOS, greatly expanding abilities of MikroTik devices. This is very important feature that makes me choose MikrotTik over other manufacturers. I am at the point where I have to decide whether I can rely on MetaROUTER functionality or not - in future implementations. I am working on replacing HP AP's and HP MSM Wireless controllers with MikroTik devices (as a cheaper alternative) and as long as MR feature is not reliable I can not "invest" in this idea. I am afraid that problems mentioned in this thread will persist for another half of the year or maybe one day you will say that it can't be fixed at all.
I was talking to my manager about great features of RouterOS (trying to convince him) and finally I tried to show him MetaROUTER and it was not responding. :)

For me it is kinda funny: If you are that good that you developed MetaROUTER then it should be easy for you to fix some small issues. :)
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Jan 10, 2013 10:04 pm

+1 on payday
----------------------------
Want to learn more and more...
 
infused
Member
Member
Posts: 308
Joined: Fri Dec 28, 2012 2:33 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Jan 14, 2013 11:33 am

Just to chime in. I bought Mikrotik because of Metarouter. Waiting for it to be ported to dual cores (CCR). Otherwise I will eventually have to go with a Juniper, which I'd rather not do.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Jan 17, 2013 9:01 am

Well some fresh optimism in this thread can't do any harm. ;-)
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Feb 07, 2013 2:44 pm

I guess this topic is an general hold again? Mikrotik, care to comment on current status?
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Feb 25, 2013 11:52 am

Has anyone tried to change power supply for RB2011UAS-2HnD-IN and tested its impact to MetaROUTER stability? Is it even possible to change it without losing warranty?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Feb 25, 2013 3:24 pm

yes, it is possible. It is allowed to open cases of RouterBOARD devices. Just use quality 12V adapter to power the router.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Feb 25, 2013 4:00 pm

Has anyone tried to change power supply for RB2011UAS-2HnD-IN and tested its impact to MetaROUTER stability? Is it even possible to change it without losing warranty?
This isn't even worth a try, as the MR issues clearly aren't powersupply dependent. But if you want to waste your time, like we all did in the past, you are welcome. ;-)
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Feb 25, 2013 4:42 pm

there where difference in intrrupts when used different PSUs, so that guess was not without merit, just for some bad stuff happened less frequent.
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Feb 26, 2013 5:55 pm

Ok, I stole PSU from TP-Link TL-WR1043ND (12V 1,5A) and RB2011 is again up and running. Next I'll set up my test-OpenWRT MetaROUTER again and maybe generate some load for that and we'll see what happens..
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Feb 26, 2013 9:00 pm

Ok, I stole PSU from TP-Link TL-WR1043ND (12V 1,5A) and RB2011 is again up and running. Next I'll set up my test-OpenWRT MetaROUTER again and maybe generate some load for that and we'll see what happens..
Well, not a significant difference.. I think freeze periods are a bit shorter than earlier, but anyway them still exists.

Edit: My RB2011 was while ago rebooted by Watchdog (first time ever). I think it was overly good timing to be just coincidence, so I quickly changed back to original PSU.
 
barkas
Member Candidate
Member Candidate
Posts: 260
Joined: Sun Sep 25, 2011 10:51 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Feb 27, 2013 11:45 am

Please don't go the powersupply way again, that has been thoroughly tested and been found to not be the problem.
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Feb 27, 2013 5:03 pm

When the Metarouter will stable? Please MT guys, say something shiny. I`ve very good solutions and ideas using Metarouter. But stability issue doesn`t allow me to do cool things...... :(
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Feb 27, 2013 9:51 pm

I wonder why I'm not seeing this issue so far? :?: Not only have all of my own phone calls been going through an RB450G for the past few months, but we also have a few customers now that we have installed similar systems for (RB450G that handles internet routing -- PPPoE client + NAT -- and runs Asterisk inside of MetaROUTER with all phone calls being trunked via VoIP) and so far they have no complaints either. If this were happening to them, they would be getting random periods of dead air > 10s in duration in the middle of phone calls, and I expect their PPPoE tunnel would drop, too, after timing out on our side (11 seconds is an eternity!). But neither of these things are happening...

Full disclosure: I still distrust the interrupt-crazy health monitoring subsystem on the 450G, so even on 5.22 and 5.23, I have been ripping out voltage.ko on any systems we deploy.

I'll see if we have a 2011 kicking around somewhere that I can run some tests on; I haven't bothered to play with MetaROUTER on one of those.

-- Nathan
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Feb 27, 2013 10:09 pm

I didn't tried that voltage.ko -trick, it felt too arduous to find out how to do that and give it a shot.
ROS version was 5.24.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Mar 06, 2013 10:27 am

I wonder why I'm not seeing this issue so far? ...
... Full disclosure: I still distrust the interrupt-crazy health monitoring subsystem on the 450G, so even on 5.22 and 5.23, I have been ripping out voltage.ko on any systems we deploy....
Well that is because it is most likely a faulty ISR which kills the system, and ripping out voltage.ko lowers the IRQ load significantly. So what are you wondering about again, I simply don't get it?
I'll see if we have a 2011 kicking around somewhere that I can run some tests on; I haven't bothered to play with MetaROUTER on one of those
Testet it with the same setup I've been using for the RB450G, result was the same with slightly different timing behaviour.
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 08, 2013 10:29 am

Hey MT, I know you have a lot of work with ROSv6.0, but how about Metarouter stability issue?
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 11, 2013 5:43 am

Well that is because it is most likely a faulty ISR which kills the system, and ripping out voltage.ko lowers the IRQ load significantly. [...] Testet it with the same setup I've been using for the RB450G, result was the same with slightly different timing behaviour.
I finally got around to testing on a 2011 (L-IN).

MetaROUTER isn't even close to being stable on it. I will note the following:

Interrupt load on a 2011 isn't nearly as high as it is on a 450G. voltage.ko monitoring is not done on a 2011L-IN (/system health print == blank). Only things generating interrupts are the 2 switch chips and "beeper".

I ran my tests on a brand-new RB2011L-IN that I then upgraded to 5.24. I also tried it on the stock 5.20 install before upgrading. I was running my Asterisk MetaROUTER image, the same one I run on 450Gs without a problem.

The problems I observed are these:

1) At random intervals (usually after being up for only a few minutes), the MR guest stops responding for a long, long time (a few minutes usually). No IP traffic, no console response. Eventually it will start responding again, but usually it only works again for a few seconds before it hangs for several more minutes. I have seen it eventually break out of this death spiral and then work for a long period of time again.

2) The metafs driver acts...weird. Sometimes certain branches of the filesystem tree will just outright disappear, and only come back after rebooting the guest. I had the contents of /usr/share disappear on me twice while testing my image on this 2011. If you did an 'ls -la /usr/share', it wouldn't even show the '.' and '..' directories.

3) I tried to generate a supout while the hanging problem (#1) was occurring. On the RB2011L-IN, the RouterOS host never hangs: only the guest does. However, if the guest is hung, and you try to make a supout, the supout process itself hangs and never completes. (The rest of the router continues to be responsive. CPU goes to 100% for a while during supout generation, then eventually settles back into single-digits. Supout progress, though, fails to advance.)

Thus, I cannot recommend anybody use MetaROUTER on the 20xx models. I'm mystified why the behavior is so startlingly different from the 450Gs, but obviously nobody at MikroTik bothered to actually test MetaROUTER on this model before shipping it.

-- Nathan
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 11, 2013 11:22 am

Hi Nathan,

thanks for also testing this. Using only MT Metarouter instances wasn't as bad as you described, but also practically unuasable. I also would have expected MT to test this...
And as this topic gets ignored by MT again...well, I won't repeat myself.
 
gamespb
just joined
Posts: 15
Joined: Mon Nov 03, 2008 5:32 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 11, 2013 12:20 pm

Hi, I just RB2011UAS-2HnD 6.0rc13 OpenWRT - Linux version 2.6.31.10 (bencoh@procyon) (gcc version 4.6.3 20120201 (prerelease) (L
inaro GCC 4.6-2012.02) ) #4 Mon Mar 5 12:16:12 CET 2012 and freez.
In openwrt "top" shows the CPU utilization on the lan interface, not the underlying processes.
if /metarouter interface leave empty, MR have only loopback. No Freez!(top show - CPU: 0% usr 1% sys 0% nic 98%) .
This may be due to the user interface to switch openwt promiscuous mode? tell me how to check this?
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 11, 2013 6:48 pm

uhhhh... I strongly wish that NathanA`s Asterisk image be worked stable. If I`m correctly remember, my RB450G hanged too. But it`s nice to hear that RB450G has no issue. I will test it on RB450G again and will post here.
----------------------------
Want to learn more and more...
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 17, 2013 11:07 am

Tested with RB450G ROSv5.20. Result is same. :-( How to solve this issue...^*(&()$^^%$%^#$##$%%$#&*
[admin@PBXTest450G] > log print
00:00:06 system,info router rebooted 
00:00:15 interface,info ether5 link up (speed 1000M, full duplex) 
00:00:59 system,info,account user admin logged in from 192.168.123.149 via winbox 
00:02:56 system,info monitoring new host by admin 
00:02:57 script,warning Metarouter UP 
00:58:05 system,info,account user admin logged out from 192.168.123.149 via winbox 
02:37:32 script,warning Metarouter Down 
02:41:06 script,warning Metarouter UP 
02:56:14 system,info,account user admin logged in from 192.168.123.149 via winbox 
02:56:25 system,info,account user admin logged out from 192.168.123.149 via winbox 
02:57:00 script,warning Metarouter Down 
03:01:51 script,warning Metarouter UP 
03:12:34 system,info,account user admin logged in from 192.168.123.149 via winbox 
03:26:31 script,warning Metarouter Down 
03:38:06 script,warning Metarouter UP 
03:38:37 system,info,account user admin logged out from 192.168.123.149 via winbox 
04:04:57 system,info,account user admin logged in from 192.168.123.149 via winbox 
04:05:08 system,info,account user admin logged in from 192.168.123.149 via telnet 
----------------------------
Want to learn more and more...
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 17, 2013 1:15 pm

please try these new buiilds:
http://www.mikrotik.com/download/share/ ... .25rc1.npk
http://www.mikrotik.com/download/share/ ... .25rc1.zip


http://www.mikrotik.com/download/share/ ... .0rc14.npk
http://www.mikrotik.com/download/share/ ... .0rc14.zip

they contain metarouter fixes.
corresponing cangelog entry:
*) metarouter: fixed occasional lockups on mipsbe boards;
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 17, 2013 1:27 pm

Wooooow!!!
I will test right now and report here back.
----------------------------
Want to learn more and more...
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 17, 2013 2:09 pm

Wooooow!!!
I will test right now and report here back.
I can't test it until friday, so I am waiting for your reports.
Try to put heavy load on router - this makes lockups happen more often.
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 18, 2013 6:40 am

:D Pretty good results. That`s very very satisfied for me:
RB2011 with v5.25rc1 - No interruption during 14 hours. Average CPU utilization is 36%. At this rate, on v5.24, interruption was 5-6 times.
RB450G with v5.24 - No interruption during 16 hours. Tomorrow I will test v5.25rc1 on 450G.

Unfortunately I have only 1 SIP client. Hope result is same using many SIP clients.
----------------------------
Want to learn more and more...
 
gamespb
just joined
Posts: 15
Joined: Mon Nov 03, 2008 5:32 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 18, 2013 10:00 am

set up - http://www.mikrotik.com/download/share/ ... .0rc14.npk

ethernet ports have stopped working(reset to default does not help, try eth1), just wifi worked.
check metarouter not.

autosupout.rif created how i try
http://kuznecova.net/autosupout.rif
http://kuznecova.net/autosupout.old.rif

RB2011UAS-2HnD
Last edited by gamespb on Thu Apr 18, 2013 10:15 am, edited 1 time in total.
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 18, 2013 10:11 am

please try these new buiilds:
http://www.mikrotik.com/download/share/ ... .25rc1.npk
http://www.mikrotik.com/download/share/ ... .25rc1.zip


http://www.mikrotik.com/download/share/ ... .0rc14.npk
http://www.mikrotik.com/download/share/ ... .0rc14.zip

they contain metarouter fixes.
corresponing cangelog entry:
*) metarouter: fixed occasional lockups on mipsbe boards;
Will eventually test this today or tomorrow evening, many thanks for your update. :)
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 8:39 am

RB2011withv5.25rc1 works 2 days without Metarouter issue. Starting to test RB450G using v5.25rc1.
----------------------------
Want to learn more and more...
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 10:42 am

RB2011withv5.25rc1 works 2 days without Metarouter issue. Starting to test RB450G using v5.25rc1.
Thanks for great info. I can't wait to test it myself. Today, in the evening I will test v6.0rc14.
Last edited by payday on Sat Apr 20, 2013 1:39 am, edited 1 time in total.
 
gamespb
just joined
Posts: 15
Joined: Mon Nov 03, 2008 5:32 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 11:38 am

Hello.
i have RB2011UAS-2HnD
downgrade to 5.25rc1
New firmware has changed the MAC address on the other!!!
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 11:46 am

downgraded from 6.0rc? and how your interfaces where configured? static vif interfaces?
 
telepro
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Sun Apr 03, 2011 7:50 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 3:56 pm

Have tested v5.25rc1 on 3 RB951G units for ~ 40 hours each. Would have expected to see 6 or more failures on each during that interval. Have found none. Thanks much.
 
gamespb
just joined
Posts: 15
Joined: Mon Nov 03, 2008 5:32 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 9:57 pm

downgraded from 6.0rc? and how your interfaces where configured? static vif interfaces?
Yep from 6.0rc14
interface vif1 dynamic
[admin@Kuznecova3] > interface bridge port pr
Flags: X - disabled, I - inactive, D - dynamic 
 #    INTERFACE               BRIDGE               PRIORITY  PATH-COST    HORIZON
 0    eth1-welcome-to      Welcome                  0x80         10       none
 1    eth5-welcome-from    Welcome                  0x80         10       none
 2  I wlan1                   Welcome                  0x80         10       none
 3  D vif1                    Welcome                  0x80         10       none
Now i try MetaRouter 25rc1
2 hours - good fly :)
 
gamespb
just joined
Posts: 15
Joined: Mon Nov 03, 2008 5:32 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 19, 2013 10:04 pm

OpenWrt r30812
[ 0.000000] Linux version 2.6.31.10 (bencoh@procyon) (gcc version 4.6.3 20120201 (prerelease) (Linaro GCC 4.6-2012.02) ) #4 Mon Mar 5 12:16:12 CET 2012

top show :
Mem: 7336K used, 14736K free, 0K shrd, 0K buff, 2412K cached
CPU:   0% usr   1% sys   0% nic  98% idle   0% io   0% irq   0% sirq
nic 98% - i only ping use
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Sat Apr 20, 2013 2:16 am

:D Wow!
6 hours of heavy testing and no freeze at all! It is ROCK SOLID! :D
I think it's occasion for party! :)
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Sat Apr 20, 2013 4:30 pm

I started tests with 5.25rc1 on RB2011 a few hours ago, I am using the same test setup like before, so time will tell. It didn't hnag in the last 5h but I got that result before. So lets wait a little before we declare this fixed. Right now I was brave enough to also update a more important RB450G to 5.25rc1, so lets see what we will get there.
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Sat Apr 20, 2013 9:02 pm

Mine (6.0rc14) is working fine since yesterday. When using previous version I had about 20 hangs a day.
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 22, 2013 5:45 am

OpenWrt r30812
[ 0.000000] Linux version 2.6.31.10 (bencoh@procyon) (gcc version 4.6.3 20120201 (prerelease) (Linaro GCC 4.6-2012.02) ) #4 Mon Mar 5 12:16:12 CET 2012

top show :
Mem: 7336K used, 14736K free, 0K shrd, 0K buff, 2412K cached
CPU:   0% usr   1% sys   0% nic  98% idle   0% io   0% irq   0% sirq
nic 98% - i only ping use
Seems it`s not. As your top show, the NIC is 0% and IDLE is 98%. It`s normal.


RB450G with ROSv5.25rc1, without Metarouter interruption last 2 days. Hope V5.25rc1 all other functions are same as Metarouter without any issue. :D
----------------------------
Want to learn more and more...
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 22, 2013 9:15 am

Still looking good for me too, will upgrade another system soon. Keeping my fingers crossed, but it looks very promising this time.
 
Wazza
newbie
Posts: 39
Joined: Thu Oct 13, 2011 10:43 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 22, 2013 2:04 pm

All,

So I don't suppose anyone has had any luck with PPC based systems?
This all sounds very promising, but I'd really really like to see this working in a PPC environment too.

I've updated a couple of test RB1100AH's. I'm not going to hold my breath, but here's hoping...

Thanks.

Wazza
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 22, 2013 2:23 pm

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.
 
gamespb
just joined
Posts: 15
Joined: Mon Nov 03, 2008 5:32 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 22, 2013 7:50 pm

Hello All.
Point to the working mirrors Metarouter?

After change from http://downloads.openwrt.org/snapshots/ ... ps/package
to: src/gz snapshots http://downloads.openwrt.org/snapshots/ ... s/packages

Now error: Package zsh version 5.0.2-1 has no valid architecture, ignoring

or could share newest image OpenWrt to MetaROUTER?
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 6:58 am

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.
Hi Janisk, any expected date for fixing issue of PPC? Also do you have any plan to implement metarouter for multicore PPC? tile?
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 8:00 am

A lot of credit appears to go to majbthrd, who came up with a reliable way of reproducing the hangs on mipsbe: http://forum.mikrotik.com/viewtopic.php?f=15&t=58936

I find it interesting, though, that in that thread, RB2011 are talked about as if they don't have any problems with MetaROUTER, even though I can easily reproduce the problem myself on my RB2011L-IN running 5.24.

I tried 5.25rc1, and I can confirm that even on the RB2011L-IN, there are no more freezes! Awesome! Also, I'll have to run some tests, but I think CPU utilization as shown by the host (/system resource print) when the guests are busy is more accurate now. In the past, I could load down the guests with tasks that would bring the system to a crawl, but CPU utilization would not be showing anything near 100%.

However, I am still seeing the same bizarre problems with MikroTik's MetaFS. I don't know whether the bug is in the host or the guest code. Testing on my RB2011L-IN after several reboots of my Asterisk guest, I saw...

1) the contents of /usr/share completely disappear almost every time, shortly after bootup.
2) the contents of /usr/lib/asterisk/static-http disappear (and thus the Asterisk GUI was unable to find its files)
3) the contents of /etc/init.d disappear in the middle of bootup (so it did not complete booting properly)

And once, I was in /etc/init.d trying to change the file mode of one of the startup scripts (I wanted to prevent Asterisk from starting on next boot to test a theory I had), and although the files were listing, they could not be accessed; chmod told me "chmod: asterisk: Invalid argument". Worked fine after another reboot.

This almost never happens on my RB450G boards, *although* I have seen similar symptoms on an RB450G after using SCP to copy a large file over to the MetaROUTER guest (the file was larger than the amount of RAM I assigned to the guest, and 'top' showed that during the copy, all of either the page cache or the filesystem cache had been consumed). Whenever I did this, the guest would begin to act very...wonky, and similar things would occur (entire directories would disappear, trying to execute commands would result in a system hang or "Segmentation fault", and so on). This made me wonder if the problem was memory-related (after all, 450G has 256MB of memory and I always assign my Asterisk guest half of that), and that's why I wanted to prevent Asterisk from starting on boot-up on my 2011. What I found is that it does help to reduce the occurrence of the problem on the RB2011, but doesn't eliminate it completely (when /etc/init.d disappeared while init was in the middle of running through the boot-up scripts, it disappeared before it reached the Asterisk one, so Asterisk never even managed to start that time).

On the 450G, whenever I would copy a large file to it over the network, I would carefully watch the consumption of available RAM, and periodically issue an 'echo 3 > /proc/sys/vm/drop_caches' to free up any memory that could be freed. That did seem to help. Interestingly, on the 2011 board, when I do that, it almost immediately causes the guest to go sideways...from that moment on, executing any command will either hang the guest or produce "Segmentation fault" (the guest is still responsive, though...console prints typed characters, it responds to pings, etc., so it's not the normal "hanging" problem, which I really do think has finally been fixed for good). It must be a problem with the pagecache because I get the same behavior if I echo '1' instead of '3' (see http://linux-mm.org/Drop_Caches).

Whatever is causing this (bug in Linux kernel in guest, bug in guest MetaFS driver, bug in RouterOS host), it's really annoying.

-- Nathan
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 8:28 am

I find it interesting, though, that in that thread, RB2011 are talked about as if they don't have any problems with MetaROUTER, even though I can easily reproduce the problem myself on my RB2011L-IN running 5.24.
Different CPU? Im not into MIPS assembly but locking/pipelining might be a totally different beast when you compare 24K to 74K MIPS...
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 8:47 am

2) the contents of /usr/lib/asterisk/static-http disappear (and thus the Asterisk GUI was unable to find its files)
That issue occurs in my case also timezone, hostname configurations lose after non-proper reboot. So I added custom script into rc.local:
echo GMT-8 > /etc/TZ

ntpclient -c 1 -s -h 202.131.224.136

hostname RouterBoard_PBX

echo "[general]
enabled = yes
bindaddr = 0.0.0.0
bindport = 8088
enablestatic = yes
redirect = / /static/config/index.html
[post_mappings]
backups = /usr/lib/asterisk/gui_backups
moh = /usr/lib/asterisk/moh" > /etc/asterisk/http.conf

udhcpc
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 8:57 am

Quick update:

Whenever the contents of a directory "go missing", it appears that I can fix the problem without rebooting by 'echo 2 > /proc/sys/vm/drop_caches'. Sometimes I need to run a 'sync' before doing so.

On RB2011, telling the kernel to flush the pagecache by echoing '1' or '3' (which is essentially an instruction to perform both 1 and 2) to drop_caches immediately makes the guest unusable. This doesn't happen on 450G, even when the guest is configured to have the same amount of RAM (in this case, 32MB).

Here's another weird one: when I import an image into MetaROUTER, and then reboot the RouterBOARD, RouterOS creates a bunch of directories in the root directory of all my guests. They are called /cm, /logs, /pckg, /run, /store, and /vstore. They are all empty. Those directories do not exist in my original compressed tarball, and are not being made by any program included in the image. If I check the guest immediately after importing and before rebooting the router, they are not present, and they do not show up simply by my shutting down and rebooting the guest. If I remove the directories and then reboot RouterOS, they will come back.

-- Nathan

EDIT: I deleted the VM and re-imported it on the 2011. All the same problems remained. However, after performing a Netinstall of 5.25rc1 on my RB2011, the problems seemed to have gone away. Could this be a YAFFS problem??

EDIT 2: Spoke too soon. Reconfigured the router the way it was before, and same problems are all back. Probably going to take me a little bit to track down the culprit.
 
luminescentsimian
just joined
Posts: 6
Joined: Fri Nov 02, 2012 7:41 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 9:59 am

Looking good on RB750GL running some openwrt image running unbound, openconnect, and quagga to do DNSSEC validation and OSPF routing for my Cisco work VPN. I haven't noticed any stalling or reboots and quagga doesnt seem to be crashing and generating gibberish OSPF information like it had been.

[admin@Mikrobob] > /system resource print
uptime: 2d3h44m51s
version: 6.0rc14
build-time: Apr/19/2013 15:00:13
....
board-name: RB750GL
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 2:55 pm

i would say that all of the credit goes to majbthrd as he created image that what it does is wait for interrupt and when that occurs, wait again. That allowed recreate freezing problem in matter of seconds, instead of yours that what it was previously.

Also, i was corrected by devs - there are changes for PPC metarouter too as result of this, so it would be useful if PPC was tried out too on more configuration but my test setups.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 23, 2013 4:38 pm

i would say that all of the credit goes to majbthrd as he created image that what it does is wait for interrupt and when that occurs, wait again. That allowed recreate freezing problem in matter of seconds, instead of yours that what it was previously.
Very true. I actually meant that credit is split between majbthrd (for the image) and MetaROUTER developers (for knowing what to do with the image). :)

I am curious: did it turn out to be a simple problem to fix, after you had majbthrd's image to test with? Are you allowed to talk about what the fix ended up being?
Also, i was corrected by devs - there are changes for PPC metarouter too as result of this, so it would be useful if PPC was tried out too on more configuration but my test setups.
Good to know! I will break out the RB1100s again.

-- Nathan
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 24, 2013 3:13 pm

That issue occurs in my case also timezone, hostname configurations lose after non-proper reboot. So I added custom script into rc.local:
That's a slightly different issue than the one I'm currently fighting, I think. I know others have seen files that were not properly closed (by, say, Asterisk) become lost if the application is still running and the MetaROUTER instance isn't cleanly shut down. That's what is causing your http.conf file to disappear.

What I'm talking about is files becoming temporarily inaccessible or even temporarily hidden. When this happens, the files miraculously reappear if I either reboot the MetaROUTER, or flush the inode cache ('sync; echo 2 > /proc/sys/vm/drop_caches').

In addition, I have also noticed that after extended/prolonged filesystem activity (which also gives the Linux VM subsystem quite a workout), some sort of memory corruption is occurring that is non-reversible once it happens; the only recourse at that point is to reboot the MetaROUTER. This can happen if I copy a file to the MetaROUTER that is larger than the RAM I have allocated to it, or if I read in a file that is larger than the allocated RAM, or 9 times out of 10 on an RB2011, when I flush the pagecache ('sync; echo 1 > /proc/sys/vm/drop_caches'). After doing that, if I try to execute any program in the MetaROUTER, the program will either hang the MetaROUTER, or will immediately quit with "Segmentation fault".

It seemed to be a memory-constraint-related issue at first, and I sought ways of either disabling or limiting both the page and inode caches in the Linux VM subsystem, but apparently as of 2.6.x, this is not possible. It is possible to tune the 'swappiness' (how much weight the VM subsystem gives to clean pages in the RAM cache vs. running code in RAM), but not to instruct it to limit the page cache to only a percentage of available physical RAM. Linux VM is made to consume all possible RAM and then only drop clean pages if RAM is needed by something else. In theory, this should not impact the user negatively, and should even improve overall performance. I was concerned that the fabled Linux "OOM killer" was incorrectly kicking in rather than having clean pages dumped from the cache after available RAM was filled.

So I have tried several things to see if any of them affect the problem:
  • I tried eliminating the 'tmpfs' RAM-disk partition (at /tmp), just in case it was causing issues and holding onto RAM that it shouldn't be.
  • I tried making a small swapfile* and setting '/proc/sys/vm/swappiness' to 0, just in case a bug in the VM subsystem was causing it to behave incorrectly when no swapfile was present.
  • I tried going back to the older MikroTik MetaROUTER patch for OpenWRT (1.1, which only supports MIPSBE) and the older kernel that it supports (2.6.27) just in case there was a bug in either the newer patch (1.2) or the newer kernel (2.6.31).
  • I took the time to create a working 2.6.35 kernel that boots in a MetaROUTER using patches for 2.6.35 that I got from MikroTik a while back.
No change: the system continued to have both randomly "disappearing files" and memory corruption that coincided with filesystem activity (even if it was just filesystem reads, not writes).

Interestingly, it happens within a few minutes of running my guest on top of an RB2011L-IN, but it is very difficult to make a MetaROUTER guest unstable in this way on a RB450G! (...although it does happen, just very infrequently.) I even gave the RB450G guest the same amount of RAM (less, even) as I had configured on the RB2011L guest, and it didn't flinch. It takes a long time before the same kind of memory corruption surfaces on the 450G. The only way I can reliably reproduce the problem on a 450G is when I SCP a 200+ megabyte file over to the guest. Most of the time, after that copy is finished, it starts acting the same way that the 2011L does after I flush the page cache on it.

At this point, I have to believe that this is a problem in the MetaROUTER implementation itself somehow. I will see about putting together a test suite that I can submit as a ticket to MikroTik support. It's going to be tricky because the problem doesn't always manifest itself the same exact way every time...

-- Nathan

* making a swapfile on top of MetaFS is not easy, because MetaFS just maps files in your MetaROUTER to files in the RouterOS host's filesystem, and on MIPSBE, the host filesystem is YAFFS, which can't be used to hold a swapfile because you can't allocate a large block of contiguous free space ('swapon' actually told me "swapfile has holes" when I tried). So I had to fake it out by building a kernel with VFAT support, creating a file large enough to hold the swapfile, formatting it as VFAT, mounting the file via loopback, and then creating the swapfile on THAT. It's not worth it, and the only reason I went to the trouble was to test a theory. There is no good reason to use swap space inside of a MetaROUTER guest; "don't try this at home."
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 24, 2013 5:08 pm

on RB2011RM with 32MB assigned to guest (openwrt) copied over 74MB file. encountered no problems. What i am doing wrong?

Also, if you run RouterOS as guest, does copying of large file causes you any issues?
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Apr 24, 2013 6:46 pm

That's a slightly different issue than the one I'm currently fighting, I think. I know others have seen files that were not properly closed (by, say, Asterisk) become lost if the application is still running and the MetaROUTER instance isn't cleanly shut down. That's what is causing your http.conf file to disappear.

What I'm talking about is files becoming temporarily inaccessible or even temporarily hidden. When this happens, the files miraculously reappear if I either reboot the MetaROUTER, or flush the inode cache ('sync; echo 2 > /proc/sys/vm/drop_caches').
Ohh this issue is too complicate to understand for me :-). My issue was simple than this issue.
Sorry that asking off topic question, I want to learn to compile own asterisk image by myself. How did you make your PPC, MIPSE asterisk images? Did you posted step-by-step guide anywhere?
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 25, 2013 12:42 am

Sorry that asking off topic question, I want to learn to compile own asterisk image by myself. How did you make your PPC, MIPSE asterisk images? Did you posted step-by-step guide anywhere?
I did post a step-by-step of how I built my original OpenWRT "Kamikaze" 8.09.2 image for MetaROUTER, here: http://forum.mikrotik.com/viewtopic.php ... 81#p311681. I suggest that any questions related to this subject be redirected to that thread, or perhaps that a new thread be started.

-- Nathan
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 25, 2013 6:34 am

on RB2011RM with 32MB assigned to guest (openwrt) copied over 74MB file. encountered no problems. What i am doing wrong?
I don't know, but I managed to come up with a way to reproduce the bug 100% of the time (for me). Try following these steps and see if you get the same results:
  1. Fresh Netinstall of 5.25rc1 to RB2011L. No default config.
  2. Copy the MikroTik OpenWRT image (http://www.mikrotik.com/download/metaro ... rootfs.tgz) to the router.
  3. Import it and give it 32MB of RAM.
  4. After it starts up, open the console, and press Enter to get a shell prompt.
  5. Verify you can perform simple tasks without any problems (run 'top', get a listing of the root directory, etc.):
    root@OpenWrt:/# ls /
    bin      etc      lib      overlay  rom      sbin     tmp      var
    dev      kernel   mnt      proc     root     sys      usr      www
  6. Create a large (16MB), empty file in the root directory:
    root@OpenWrt:/# dd if=/dev/zero of=/testfile bs=16M count=1
    1+0 records in
    1+0 records out
  7. Now try to do some of those same simple tasks again:
    root@OpenWrt:/# ls /
    Segmentation fault
    root@OpenWrt:/# top
    Segmentation fault
    root@OpenWrt:/# ifconfig
    Segmentation fault
    root@OpenWrt:/# reboot
    Segmentation fault
As you can see, after exercising the filesystem (and the page cache) with a large write operation, the system becomes unstable. Any EXISTING processes are still running and continue to run, but you cannot start any new ones.
Also, if you run RouterOS as guest, does copying of large file causes you any issues?
I don't know; good question. I have not tried it. The same problem might exist with RouterOS guests, but you might not be able to see them because my understanding is that RouterOS largely consists of a single, monolithic process (Nova?). Since any existing processes that are already running seem to be unaffected, I'm not sure whether you would notice any problems or not. Hopefully, though, my steps above with OpenWRT makes it possible for you and the developers to reproduce the problem, and thus to fix it.

Thanks,

-- Nathan
 
User avatar
payday
Member Candidate
Member Candidate
Posts: 233
Joined: Thu Aug 16, 2012 11:05 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Apr 25, 2013 11:38 pm

I have tested my OpenWRT instance and everything works fine:
BusyBox v1.19.4 (2012-11-09 20:36:06 CET) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 ATTITUDE ADJUSTMENT (Bleeding Edge, r31411)
 -----------------------------------------------------
  * 1/4 oz Vodka      Pour all ingredients into mixing
  * 1/4 oz Gin        tin with ice, strain into glass.
  * 1/4 oz Amaretto
  * 1/4 oz Triple sec
  * 1/4 oz Peach schnapps
  * 1/4 oz Sour mix
  * 1 splash Cranberry juice
 -----------------------------------------------------

/> ls
bin      etc      logs     pckg     root     store    usr      www
cm       kernel   mnt      proc     run      sys      var
dev      lib      overlay  rom      sbin     tmp      vstore

/> dd if=/dev/zero of=/testfile bs=16M count=1
1+0 records in
1+0 records out

/> ls /
bin       etc       logs      pckg      root      store     tmp       vstore
cm        kernel    mnt       proc      run       sys       usr       www
dev       lib       overlay   rom       sbin      testfile  var
I have intentionally included OpenWRT banner to show you the version of OpenWRT that I am using. Obviously top and ifconfig also run normal. I am using v6rc14 and it is really stable. Try upgrading your firmware and try different release of OpenWRT.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Apr 26, 2013 2:41 am

I have tested my OpenWRT instance and everything works fine: [...] I have intentionally included OpenWRT banner to show you the version of OpenWRT that I am using. Obviously top and ifconfig also run normal. I am using v6rc14 and it is really stable. Try upgrading your firmware and try different release of OpenWRT.
Hey, thanks for testing. As you can see from previous posts of mine, I have tried a bajillion different OpenWRT images and Linux kernel versions and they all behave the same way, which is why I was convinced it was a RouterOS host bug/problem. However, I would be interested in trying out your image. Can you post it somewhere for me to download? Also, what kernel version are you running ('uname -a') and what version of the MikroTik MetaROUTER Linux kernel patches did you use to build your kernel?

UPDATE: I just tried my tests on 6.0rc14 with all of the same OpenWRT images, and I don't have the problem! In fact, the guest kernel's pagecache/VM subsystem seems to behave completely differently when the guest is run on 6.0rc14, even though the guest is the exact same image and kernel as I was running under 5.25rc1. So definitely something changed in MetaROUTER itself in 6.0. (If I copy a large file over, it doesn't seem to consume the entire remaining bits of the guest's RAM allotment, for example. I can also flush the clean pages from the pagecache without causing instability in the OpenWRT guest now as well.)

The problem is that I cannot, in good conscience, install BETA software on my customer's devices. 6.0 still has its share of other problems, and I'm not just using the routers to run MetaROUTER on...they do a bunch of other non-MetaROUTER-related tasks as well. So I really need for this to work under 5.x if at all possible. Also, it looks like 6.0 with the same set of default NPK packages enabled uses up a lot more RAM; I didn't have enough free RAM on the RB2011L with the default packages on 6.0 to give my guest 32MB. On 5.x, I have 32MB and then some on top of that available with default packages.

-- Nathan
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Apr 29, 2013 1:21 pm

Just a quick update on the ROS MR front.
I didn't have a single freeze or other issue since I installed 5.25rc1 on my RB450G and RB2011.
Although I didn't conduct any further tests, this feels pretty stable as for now. I will try to do some more elaborate tests if my time allows.
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 30, 2013 9:17 am

thank you all for update. If you see issues, please write here or to support. We are checking out NathanA claim about files going missing till MR guest reboot.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 30, 2013 11:59 am

We are checking out NathanA claim about files going missing till MR guest reboot.
Thank you. I am not sure, but I suspect that the missing files problem and the "segmentation fault" problem are both related to each other. As I documented before, I can reproduce the "segmentation fault" problem with 5.25rc1 on a RB2011L using MikroTik's own OpenWRT image with the steps I provided. Have you been able to reproduce it on your end?

I haven't come up with a reliable way of reproducing the missing files problem, but I can make it happen more often than not using an OpenWRT image I made with Asterisk on it, on an RB2011. After Asterisk automatically starts on bootup and consumes a good amount of available RAM, it is very likely to happen. If you want a copy of that image, I would be happy to provide it to you.

-- Nathan
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 30, 2013 3:34 pm

it would be nice to get link or a file with asterisk image.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Tue Apr 30, 2013 4:14 pm

it would be nice to get link or a file with asterisk image.
You can download the current version of my MIPSBE image here: http://www.nconx.com/~nathan/ast-owrt-m ... ips_b4.tgz. I give it 32MB of RAM when I import it.

Many times, shortly after bootup, on my RB2011L, 'ls -la /usr/share' will show nothing. Then if I 'sync; echo 2 > /proc/sys/vm/drop_caches', usually after that 'ls -la /usr/share' will show the subdirectories.

I will continue to work on finding a way to consistently reproduce the problem, in case you cannot replicate my issue. (So far, though, the instructions I wrote out earlier [here] reproduce "segmentation fault" for me on RB2011L every time.)

Thanks again,

-- Nathan
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri May 03, 2013 9:43 am

Sorry that asking off topic question, I want to learn to compile own asterisk image by myself. How did you make your PPC, MIPSE asterisk images? Did you posted step-by-step guide anywhere?
I did post a step-by-step of how I built my original OpenWRT "Kamikaze" 8.09.2 image for MetaROUTER, here: http://forum.mikrotik.com/viewtopic.php ... 81#p311681. I suggest that any questions related to this subject be redirected to that thread, or perhaps that a new thread be started.

-- Nathan
Thanks Nathan, Is there any possible way just change asterisk default configuration files of the image, without rebuilding it? I tried to use 7-ZIP. It allow me to change files of the image without extracting it. But when I upgraded asterisk configuration files of the image, image become unable to import into metarouter. Metarouter says "Archive is too short"

Second thin is, yesterday I had power issue and router rebooted. But /etc/asterisk/users.conf file has broken. Just became blank file. All other files "/etc/asterisk/extensions.conf, /etc/asterisk/extconfig.conf" are OK. How can I automate process "backup users.conf file every day and replace original users.conf by this backup file after every reboot occurs"
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri May 03, 2013 10:35 am

is there any possible way just change asterisk default configuration files of the image, without rebuilding it? [...]
This is the wrong thread for this discussion. I posted my response to the other thread.

-- Nathan
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu May 16, 2013 7:15 am

Hi MT Support,
Sometimes metarouter disappears after non proper shutdown. Just no metarouter. Metarouter interface is still there but assigned to unknown Virtual machine. What it can be?
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu May 16, 2013 9:57 am

New issue: presence of a MetaROUTER on 5.25 can cause host in certain cases to take much longer to complete tasks than it normally would.

Example: I added a new MetaROUTER to a 450G that a customer had. Instantly, CPU on 450G shot up to 100% and stayed there until I disabled the MetaROUTER. Profiler showed "console" process taking up the majority of the CPU time (70%+) when this was happening.

I traced the issue to a Scheduler event that was kicking off every 10 seconds, and performing a search of the routing table (example: '/ip route find dst-address~"192.168.1.126/32"'). Routing table was roughly 700 entries long ('/ip route print count-only'). While MetaROUTER was running and the CPU was at 100%, I disabled the Scheduler event and killed all active jobs ('/system script job') that it spawned, and CPU went back to normal.

I then tried executing the same '/ip route find' search from a terminal window, both with and without a running MetaROUTER. Without any MetaROUTER running, the routing table search took 1 second (maybe 2, at the most). With a MetaROUTER running, routing table search took almost 15 seconds, and "console" process was maxing out the CPU the entire time.

-- Nathan
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu May 16, 2013 2:04 pm

Nathan, just to be sure, you are talking about a OpenWRT MetaROUTER or a ROS MetaROUTER?
I'll have to try and reproduce this with a ROS MetaROUTER.
 
ayufan
Member
Member
Posts: 331
Joined: Sun Jun 03, 2007 9:35 pm
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu May 16, 2013 2:06 pm

New issue: presence of a MetaROUTER on 5.25 can cause host in certain cases to take much longer to complete tasks than it normally would.

Example: I added a new MetaROUTER to a 450G that a customer had. Instantly, CPU on 450G shot up to 100% and stayed there until I disabled the MetaROUTER. Profiler showed "console" process taking up the majority of the CPU time (70%+) when this was happening.
It's not something new. Use of 1 MetaROUTER roughly decreases performance of host by 50%. Use of 2 even more. Despite the fact that MetaROUTER is idle and not doing anything at all, just idling ;)

Kamil
hAP AC, TP-Link Archer C7 v2, RB951G, RB450G, RPI2, RPI zero
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri May 17, 2013 8:21 am

New issue: presence of a MetaROUTER on 5.25 can cause host in certain cases to take much longer to complete tasks than it normally would.

Example: I added a new MetaROUTER to a 450G that a customer had. Instantly, CPU on 450G shot up to 100% and stayed there until I disabled the MetaROUTER. Profiler showed "console" process taking up the majority of the CPU time (70%+) when this was happening.
It's not something new. Use of 1 MetaROUTER roughly decreases performance of host by 50%. Use of 2 even more. Despite the fact that MetaROUTER is idle and not doing anything at all, just idling ;)

Kamil
I can't acknowledge this, ROS MetaROUTER instances don't seem to consume a significant amount auf CPU cycles when idle. Of course this will change if the instances get real load.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 22, 2013 1:45 am

It's not something new. Use of 1 MetaROUTER roughly decreases performance of host by 50%. Use of 2 even more. Despite the fact that MetaROUTER is idle and not doing anything at all, just idling ;)
We are not talking about 50%, we are talking about a factor of 10x or more. BIG DIFFERENCE.

If what you say were true, then a task which would normally take about 1 second without a MetaROUTER guest running would take roughly 2 seconds with a MetaROUTER guest. I'm seeing 10-15 seconds!

-- Nathan
 
ayufan
Member
Member
Posts: 331
Joined: Sun Jun 03, 2007 9:35 pm
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 22, 2013 12:44 pm

We are not talking about 50%, we are talking about a factor of 10x or more. BIG DIFFERENCE.

If what you say were true, then a task which would normally take about 1 second without a MetaROUTER guest running would take roughly 2 seconds with a MetaROUTER guest. I'm seeing 10-15 seconds!

-- Nathan
I didn't test performance of guests. I see how works the hypervisor (base RouterOS). It's twice slower than usual EVEN if guest MetaROUTER is idle.
hAP AC, TP-Link Archer C7 v2, RB951G, RB450G, RPI2, RPI zero
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 22, 2013 1:12 pm

I didn't test performance of guests. I see how works the hypervisor (base RouterOS). It's twice slower than usual EVEN if guest MetaROUTER is idle.
I am not talking about guests either. I am saying that the HOST is TEN times slower when a guest is running (idling), not TWO times slower. The task that normally takes 1 second on the HOST will take 10-15 seconds on the HOST when a guest is running and idling.

-- Nathan
 
timberwolf
Member Candidate
Member Candidate
Topic Author
Posts: 274
Joined: Mon Apr 25, 2011 12:08 pm
Location: Germany

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 22, 2013 1:20 pm

We are not talking about 50%, we are talking about a factor of 10x or more. BIG DIFFERENCE.

If what you say were true, then a task which would normally take about 1 second without a MetaROUTER guest running would take roughly 2 seconds with a MetaROUTER guest. I'm seeing 10-15 seconds!

-- Nathan
I didn't test performance of guests. I see how works the hypervisor (base RouterOS). It's twice slower than usual EVEN if guest MetaROUTER is idle.
The hypervisor hands out shares of CPU time every 3,33ms if I'm not mistaken, the guests can and do yield back if there is nothing to do.
So what fact you claim to know exactly makes you think, there is a penalty of 50%?
 
ayufan
Member
Member
Posts: 331
Joined: Sun Jun 03, 2007 9:35 pm
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 22, 2013 2:44 pm

The hypervisor hands out shares of CPU time every 3,33ms if I'm not mistaken, the guests can and do yield back if there is nothing to do.
So what fact you claim to know exactly makes you think, there is a penalty of 50%?
Ok. I will later today or tomorrow put some tests. So it happens that I have spare RB450G which can handle 7 MetaROUTERs.
hAP AC, TP-Link Archer C7 v2, RB951G, RB450G, RPI2, RPI zero
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon May 27, 2013 3:55 pm

please note that all RouterOS instances are guests, only one of them have exclusive permission to access actual hardware of the RouterBOARD. Anything else is same for all of them.

Thit is posted here in the same place where it was told about that 1/300 second switch time between guests.

link to actual posts:
http://forum.mikrotik.com/viewtopic.php ... 54#p332654
http://forum.mikrotik.com/viewtopic.php ... 79#p332679
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 29, 2013 7:21 am

please note that all RouterOS instances are guests, only one of them have exclusive permission to access actual hardware of the RouterBOARD. Anything else is same for all of them.
janisk,

Thanks for the reminder of these details; I remember reading your posts after you made them, but had forgotten about these details in the context of the latest discussion.

I have to wonder if some of the latest changes meant to fix the IRQ issue(s) affected the timeslicing that the hypervisor engages in. I should be able to load an older version (say, 5.14 for MIPSBE or something around that time) and test to see if the same 10x performance hit happens to those older versions as well. If it does, then I guess that this huge performance hit just comes with the MetaROUTER territory on MIPSBE, although anything that MikroTik can do to try to improve that would be much appreciated. If older versions perform much better, though, then it's something that the developers should be aware of. Hopefully it is not a side-effect of recent MetaROUTER changes that are impossible to work around if we want the system to be stable.

-- Nathan
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed May 29, 2013 4:15 pm

description there is what it should have. Variance from this is considered a bug or a problem that have to be resolved.

Also note that (as far as i understand) context switching takes place as usually to swap around and commands go through the hypervisor. Yes it uses cashing to make it faster, but it adds extra load.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu May 30, 2013 6:57 am

Okay, I have tested 5.16 vs. 5.25 on an RB450G. They both behave the same. Not sure if it is a bug or not. Here is how to reproduce:
  1. Netinstall 5.25, blank config/no defaults
  2. Add ~1000 or so bogus routes to the default routing table:
    [admin@MikroTik] > :for x from 0 to 3 do={:for y from 0 to 255 do={/ip route add dst-address=("192.168." . $x . "." . $y) gateway=127.0.0.1}}
    [admin@MikroTik] > /ip route print count-only
    1024
  3. Try to traverse the routing table, and time how long it takes:
    [admin@MikroTik] > /ip route find dst-address~"1.2.3.4/32"
    ...with no MetaROUTER running, this will finish in about 1 second.
  4. Now add a new MetaROUTER. RouterOS or imported does not matter; RAM size does not matter:
    [admin@MikroTik] > /metarouter add disabled=no
  5. Traverse the routing table again while this MetaROUTER is running and doing nothing (0-1% CPU), and time how long it takes:
    [admin@MikroTik] > /ip route find dst-address~"1.2.3.4/32"
    ...with 1 MetaROUTER running, this will finish in about 15 seconds!!!
If you have "/tool profiler" running, you will see the "console" process taking up all CPU time on the host during the routing table traversal.

-- Nathan
 
chojrak11
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Apr 05, 2009 10:37 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Jun 19, 2013 2:33 am

[admin@MikroTik] > /ip route find dst-address~"1.2.3.4/32"
...with 1 MetaROUTER running, this will finish in about 15 seconds!!![/*][/list]

If you have "/tool profiler" running, you will see the "console" process taking up all CPU time on the host during the routing table traversal.

-- Nathan
I can confirm this exact scenario and observations. On one of my 493 running RouterOS 6.1 it takes about 45 seconds, not 15 seconds, to complete the command above, while the MetaRouter is running (and I waited until that MetaRouter completely started after creation prior to issuing this command in "host"). I repeated the command after shutting down the MetaRouter and it still took more than 30 seconds to complete. After removing the MetaRouter, things went back to normal (1-2 secs).
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Jun 19, 2013 9:37 am

it seems that this is it. I can confirm that there is a slowdown. As host and guests both are same turf - being guests, it does not matter where you run this script. However adding extra guests does not increase the run-time.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Nov 07, 2014 4:39 pm

Sorry to drag this thread out of retirement, but has anybody been experiencing random reboots of RB450G with recent versions of RouterOS (6.20, for example) when using MetaROUTER?

I just opened up a ticket (#2014110766000602) with MikroTik, so hopefully I will hear back from them next week, but last week I updated an RB450G from 6.11 that was running a MetaROUTER with no problems to 6.20. I did this primarily to get the latest CAPsMAN before implementing CAPsMAN on this client's network (13 APs, all RB751Ui), and now, after the upgrade, the 450G reboots randomly every few hours (although sometimes it happens only after a few minutes). ARGH.

This time, it is not a watchdog timer going off and rebooting the router, it is "kernel failure". At this point, I don't know for sure that the MetaROUTER is even involved -- it could be CAPsMAN, or some other part of 6.20 that is buggy, and not MetaROUTER! -- but if anybody else out there has upgraded a 450G that they are using MetaROUTER on and didn't do anything except upgrade it (configuration is exactly the same), and yours is also now rebooting with "kernel failure", it would be interesting to compare notes. At least I would know what the culprit is...

Thanks,

-- Nathan
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Nov 17, 2014 1:03 pm

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.
Hey janisk,

Did anything ever change with PPC MetaROUTER, as far as stability goes, since you said this a year ago? Did the devs ever find and fix the problems that made PowerPC MR crash? I tried to avoid using PowerPC for MetaROUTER because of this, but now I am testing it out again on RB1100 boards with RouterOS 6.20 and it looks like I am still able to crash it. Results are not 100% conclusive yet, though, so I am going to continue to run tests.

Here is the crazy part: I applied the same basic unsupported workaround that I came up with for RB1100AHx2 (using uniprocessor kernel to enable MetaROUTER) to a RB850Gx2, and not only does it work, but so far, MetaROUTER on a RB850Gx2, which is unsupported, seems to be more stable than MetaROUTER on a RB1100, which is supported! So it is possible that there is something specific to RB1100 that needs to be looked into. If I can manage to get the 850Gx2 to crash, though, then I guess we can conclude that it is a general issue with MR.

-- Nathan
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Nov 19, 2014 12:24 pm

ppc currnetly has no changes. So it is still crashable. As it is completely different from MIPS implementation so they are looked at separately.
Hey janisk,

Did anything ever change with PPC MetaROUTER, as far as stability goes, since you said this a year ago? [...] I am testing it out again on RB1100 boards with RouterOS 6.20 and it looks like I am still able to crash it. Results are not 100% conclusive yet, though, so I am going to continue to run tests.
I can confirm that I can still crash and reboot RB1000 and RB1100 boards (I tried both) when transmitting network traffic to a MetaROUTER VM running on a RB1000-series board.

At the same time, my "hacked" RB850Gx2 runs MetaROUTERs like a champ, and I cannot crash it. It just goes and goes.

Kill me now.

-- Nathan
 
flashberry
just joined
Posts: 1
Joined: Fri Sep 26, 2014 6:08 pm

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Thu Nov 27, 2014 11:27 am

I have the same unstable behaviour as NathanA.

We have about 30 RB951Ui-2HnD devices, all of them have hostpot functionality and we use metarouter with OpenWRT on every device, to filter content with transparent proxy. We tried router firmwares from 6.7 till 6.22 and different build of OpenWRT.

All devices have almost the same configs (SSIDs and hotspot files are the only different things). Routers had different factory firmwares (from 6.0RC till 6.15) and we have several router with "freeze effect". Freezes take place when Metarouter is started and have some network activity. Both Metarouter and Mikrotik systems aren't responding during this time, and we have about 12 seconds of freeze and 20 seconds of normal work after it. We tried different network configs, different firmwares, openwrt images and have no success.

Usually we have issues with routers came was bought with 6.15. Looking for help, because we couldn't scale without overcomimg this problem.
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Dec 01, 2014 4:25 am

I have the same unstable behaviour as NathanA.
Hmm, actually it doesn't sound exactly the same...you are complaining about frequent freezes on a MIPSBE board, while I was complaining in my most recent messages about PPC boards rebooting themselves while running MetaROUTERs. So very different. Your frequent freezing problem, though, is not unlike some of the problems reported earlier in this thread, most of which have been resolved for most people for many, many versions now...
We have about 30 RB951Ui-2HnD devices, [...]
Now that's interesting. Most people here have been running MetaROUTERs on RB450G boards, primarily because of the generous complement of RAM and NAND flash. But, like I said, the symptoms you are reporting sound very similar to the symptoms that people who used to have problems with MR on 450G boards would complain about. Most of the sporadic performance and lockup problems on the 450G were fixed long ago, but it would not surprise me to learn that other boards still had quirky issues specific to them (for example, it used to be that 450Gs were unstable while other boards such as the 433AH were stable running the same RouterOS version).

However, about a week ago, I played with a MetaROUTER on a RB951Ui, and didn't experience any issues like the ones you brought up.
Usually we have issues with routers came was bought with 6.15. Looking for help, because we couldn't scale without overcomimg this problem.
So are you saying that some 951Ui routers have the problem, and others don't, and it's usually the newer shipments that have problems?

It's possible that there are slight hardware differences/updates being made to newer shipments of the 951 series by MikroTik that maybe could explain the different behavior you are seeing with older vs. newer units. For example, this thread would seem to suggest that 951G boards being shipped today differ in subtle ways from older ones; the same thing could also be true of the 951Ui: http://forum.mikrotik.com/viewtopic.php?f=2&t=86802

-- Nathan
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Mon Dec 01, 2014 4:29 am

I just opened up a ticket (#2014110766000602) with MikroTik, so hopefully I will hear back from them next week, but last week I updated an RB450G from 6.11 that was running a MetaROUTER with no problems to 6.20. I did this primarily to get the latest CAPsMAN before implementing CAPsMAN on this client's network (13 APs, all RB751Ui), and now, after the upgrade, the 450G reboots randomly every few hours (although sometimes it happens only after a few minutes). ARGH.

This time, it is not a watchdog timer going off and rebooting the router, it is "kernel failure". At this point, I don't know for sure that the MetaROUTER is even involved -- it could be CAPsMAN, or some other part of 6.20 that is buggy, and not MetaROUTER!
So just to update everyone, as far as the 450G/MIPSBE problem goes, it turns out that my 6.20 "kernel failure" reboots must have been CAPsMAN related and were in fact fixed in later versions, since the problem has gone away ever since I upgraded to 6.23rc7. It may have been MetaROUTER + CAPsMAN causing this in conjunction with each other, because I also tried running CAPsMAN on another MIPSBE board (RB951Ui) for a while, and it never rebooted, but I wasn't running MetaROUTERs on it at the time either. But what's important is that problem seems to be fixed (*knock on wood*). :) So thank you to MikroTik for that.

PPC issues are still a thing.

-- Nathan
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Wed Dec 10, 2014 8:19 pm

Hi Nathan, How are you?
I`m waiting for a too long MT to implement Metarouter on the tile chipset. Do you any news or guess on that? Imagine metarouters on CCRs, will be very nice.
----------------------------
Want to learn more and more...
 
User avatar
NathanA
Forum Veteran
Forum Veteran
Posts: 801
Joined: Tue Aug 03, 2004 9:01 am

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Fri Dec 12, 2014 6:01 am

I`m waiting for a too long MT to implement Metarouter on the tile chipset. Do you any news or guess on that?
Only MikroTik can know their own plans or progress. I suggest you ask them.
Imagine metarouters on CCRs, will be very nice.
Right now I think it would just be nice if MetaROUTER worked on the platforms it is already advertised to be supported on.

-- Nathan
 
User avatar
otgooneo
Trainer
Trainer
Posts: 570
Joined: Tue Dec 01, 2009 3:24 am
Location: Mongolia
Contact:

Re: MetaROUTER stability issues on certain MIPSBE and PPC bo

Sat Jan 10, 2015 8:31 am

Got reply from MT support. They say
For tileX architecture virtualization features will most likely come in one ROS v7 builds. We should be able to provide more info on that when v7 will go into beta testing stage.
----------------------------
Want to learn more and more...

Who is online

Users browsing this forum: No registered users and 4 guests