RB2011 crached when LTE-modem connected

netmanorg · Tue Jan 06, 2015 5:38 pm

Hello everybody! I have some problem.

Please sorry my bad english.

Metarouter works well. As soon as I add LTE modem, the router reboots cyclically
If append LTE modem, and then start the virtual machine, the router also restarts cyclically.

The logs written:

If i reset configuration problem persists.
Even on the default configuration, even when i try to create a virtual machine, when attached LTE modem router reboots.

[admin@MikroTik] > /system routerboard print
       routerboard: yes
             model: 2011UiAS-2HnD
     serial-number: 46730408XXXX
  current-firmware: 3.19
  upgrade-firmware: 3.19

[admin@MikroTik] > /system resource pr
                   uptime: 14m6s
                  version: 6.24
               build-time: Dec/23/2014 13:38:45
              free-memory: 103.9MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V4.12
                cpu-count: 1
            cpu-frequency: 600MHz
                 cpu-load: 1%
           free-hdd-space: 99.3MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 97
         write-sect-total: 2528
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: RB2011UiAS-2HnD
                 platform: MikroTik

[admin@MikroTik] > /system resource usb pr
 # DEVICE VENDOR                   NAME                  SPEED                 
 0 1:1    Linux 3.3.5 ehci_hcd     RB400 EHCI            480 Mbps              
 1 1:2    GCT SEMICONDUCTOR Inc    Modem Yota            480 Mbps

Tue Jan 06, 2015 5:45 pm

currently there is no fix for that - use a separate board for the lte connection or for the metarouter.

netmanorg · Tue Jan 06, 2015 5:53 pm

As modem affects the start VM?

Thanks.

NathanA · Wed Jan 07, 2015 3:09 am

This is VERY interesting. I have been having TONS of problems with MetaROUTER stability lately, and I am about to lose my mind. I have had my stability problems blamed on other services (CAPsMAN, etc.) but it comes and goes regardless of which services I run, and at this point I am convinced it is a memory/RAM corruption bug, which would explain why it is so random and why the behavior seems to change from version to version. The problem reported here might not be related to mine at all, but let's do some exploring...

currently there is no fix for that - use a separate board for the lte connection or for the metarouter.

So, Uldis, MikroTik knew about this issue before netmanorg posted about this? Or is this new information?

Is it possible that root cause of this reboot being reported by netmanorg is related/shared with my open MetaROUTER tickets?

If problem is easily reproducible, it should be possible to debug and fix, correct?

netmanorg, what version of RouterOS are you running? If you are running 6.11 or older, can you try upgrading to the latest? And if you are running 6.12 or newer, can you try downgrading to 6.11 or older to see if behavior changes?

What model of USB modem are you using?

One thing that would be interesting to have you try is to disable the system watchdog ('/system watchdog set watchdog-timer=no') and see if the problem either changes (system locks up completely instead of reboots) or goes away completely. I have noticed that in recent versions of RouterOS, watchdog-initiated reboots are sometimes being reported as "kernel failure" instead.

-- Nathan

netmanorg · Wed Jan 07, 2015 1:25 pm

Today i use latest version of RouterOS.

i try use all 6.x version RouterOS with or without routerboard firmware upgrade.
I try use other LTE modem with other brand.

I turn off watchdogs.

Problem is exist.

The problem exists if append any other USB device (example USB HUB or USB Disk)

NathanA · Wed Jan 07, 2015 2:18 pm

i try use all 6.x version RouterOS with or without routerboard firmware upgrade. [...] The problem exists if append any other USB device (example USB HUB or USB Disk)

Any USB device kills it? Wow. Should be super-easy to reproduce, then.

-- Nathan

NathanA · Thu Jan 08, 2015 5:00 am

i try use all 6.x version RouterOS with or without routerboard firmware upgrade. [...] The problem exists if append any other USB device (example USB HUB or USB Disk)

Okay, I cannot reproduce this. I tried running 6.24 on a 2011UiAS-2HnD, just like you, and I created a MetaROUTER, and then I tried plugging in a USB disk, a USB serial adapter, and a USB hub, and the router did not reboot or freeze. MetaROUTER kept running, too.

I don't have a USB LTE modem to test with right now. But if you can get it to happen with any USB device, I am not seeing it. Can you give me step-by-step what you are doing to cause the problem, starting from a blank config?

[admin@MikroTik] > /system reset-configuration skip-backup=yes keep-users=no no-defaults=yes
Dangerous! Reset anyway? [y/N]: 
y
system configuration will be reset

Rebooting...
Stopping services...
Restarting system.

RouterBOOT booter 3.18

RouterBoard 2011UiAS-2HnD

CPU frequency: 600 MHz
 Memory speed: 200 MHz
  Memory size: 128 MiB
    NAND size: 128 MiB

Press any key within 2 seconds to enter setup..
loading kernel from nand... OK
setting up elf image... OK
jumping to kernel code
Starting...
Generating SSH DSA key...
Starting services...

MikroTik v6.24
Login: admin 
Password: 

  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.24 (c) 1999-2014       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level

[admin@MikroTik] > /system routerboard print 
       routerboard: yes
             model: 2011UiAS-2HnD
     serial-number: XXXXXXXXXXXX
  current-firmware: 3.18
  upgrade-firmware: 3.19
[admin@MikroTik] > /system resource usb print 
 # DEVICE VENDOR                   NAME                   SPEED                  
 0 1:1    Linux 3.3.5 ehci_hcd     RB400 EHCI             480 Mbps               
 1 1:2    Prolific Technology Inc. USB-Serial Controller  12 Mbps      
[admin@MikroTik] > metarouter 
[admin@MikroTik] /metarouter> print 
Flags: X - disabled 
 #   NAME                   MEMORY-SIZE     DISK-SIZE     USED-DISK STATE        
[admin@MikroTik] /metarouter> add 
[admin@MikroTik] /metarouter> 

[...wait 5 minutes, to let new MetaROUTER boot up, and see if host reboots itself...]

[admin@MikroTik] /metarouter> print 
Flags: X - disabled 
 #   NAME                   MEMORY-SIZE     DISK-SIZE     USED-DISK STATE        
 0   mr1                          24MiB     unlimited         79kiB running      
[admin@MikroTik] /metarouter>

I brought up the console for the MetaROUTER, logged into it, etc. Worked fine. Pulled the USB cable out, plugged it back in, no problems...

-- Nathan

netmanorg · Thu Jan 08, 2015 12:13 pm

Ok.. Step by step.

0) Reset routerboard configuration without default configuration
1) When RB2011 is loaded, connect LTE modem without any settings. Only connect. LTE-interface is created
2) Press Metarouter -> Add (Plus button) -> Press OK

RB 2011 reboots.

OR

0) Reset routerboard configuration without default configuration
1) When RB2011 is loaded, press Metarouter -> Add (Plus button) -> Press OK
2) VM is added successful
3) Connect LTE modem is USB port -> wait 3 sec -> RB2011 reboots

About the USB drive

This problem does not appear with all devices. For example, if a directly connected USB drive, everything works fine. When using a USB hub restarts. If i use another hub, all is well again with disk.

But with LTE modems nothing works. Already tried two different models with or without USB hub.

netmanorg · Thu Jan 08, 2015 12:19 pm

if technical support will want to check out, I will give access winbox (I have a static ip), photo modems, usb hubs, etc.

NathanA · Thu Jan 08, 2015 1:17 pm

This problem does not appear with all devices. For example, if a directly connected USB drive, everything works fine. When using a USB hub restarts. If i use another hub, all is well again with disk.

Okay, earlier I thought you said that *all* USB devices cause the problem. I don't have an LTE modem to test with, so I was testing with other USB devices. I tried one hub, one disk, and one serial adapter, and none of them caused the problem.

Are you saying that it rebooted when you plugged in a certain hub, and then plugged a disk into that hub? Or did it reboot if you plugged in a hub without anything attached to the hub? I thought you meant the second one. I will try to round up a few hubs and disks and see if I can find a combination that causes the reboot.

if technical support will want to check out, I will give access winbox (I have a static ip), photo modems, usb hubs, etc.

I have already tried this with different bugs, and MikroTik support absolutely refuses to do remote debugging of MetaROUTER issues because of trade secret leak concerns (their policy is they don't allow debug builds of RouterOS or RouterBOOT outside of MikroTik corporate labs, and they claim they don't have enough visibility into MetaROUTER issues without the debug builds running on the routers they are testing). This is very frustrating to me, because some of these bugs I still don't know how to reliably reproduce artificially in a lab environment, but they happen frequently in the field on production routers that are under normal loads and subject to real-world usage patterns. I have told MikroTik I would be happy to sign a legally-binding NDA, but they have not responded.

But by all means, please give it a shot yourself and offer remote access to your devices to MikroTik and see what they say. Maybe you will be better received better than I was. If you manage to actually get a positive response from them, please let us know.

-- Nathan

netmanorg · Thu Jan 08, 2015 3:50 pm

I'm talking about the fact that some USB devices are not compatible with certain functions Metarouter. In my case it is a USB hub and LTE modem. I understand that it can be any device that is not compatible for some reason.

After joining this device There are 2 options:

1) If you create a virtual machine that is running, the router will reboot to those long as I do not take away the device.

2) If you append the device, then create a VM, when you click OK, the router will reboot once and no configuration changes are not accepted.

My email support@netlife.pro. If the support wants to know more detail, I will give all available and ready assist

NathanA · Thu Jan 08, 2015 4:03 pm

My email support@netlife.pro. If the support wants to know more detail, I will give all available and ready assist

You need to e-mail them at support@mikrotik.com, and attach supout.rif from the affected router. Point them at this discussion thread, and let us know what kind of response you get.

-- Nathan

Fri Jan 09, 2015 4:16 pm

This is a very specific LTE modem, we tested, and there is nothing we can do right now. Only this specific LTE modem causes this issue. The only option is to use a different LTE modem.

Xcelsium · Sun Mar 08, 2015 5:35 am

Got the same problem, already tested with 2 lte different modems, i think the problem is with the hvc0 console or the method to connect to the LTE modem (i guess both use the same method of connection, serial terminal), the problem appear when the lte modem start the call, (we tested with an slow operator, who gave us a window to test the lte modem before the call occurs), and we think that maybe its a probelm with the console (it cant be used by two differents entity or at least, a lte modem and metarouter), so maybe is there anyway to get an openwrt image without console, i tried to disabled on a custom openwrt compilation by editing the mikrotik patch 2.0 but i got no luck on that, i tried to instal u-boot loader but the image doesnt have mtd partition, it will great to give it try :/ if anyone expert on doing can do the compilation to test it :S it would be great.

BR
Xcelsium

Plumbum · Fri Mar 20, 2015 9:46 am

Have same problem on RB751G-2HnD.
Found solution:

What's new in 6.28rc14 (2015-Mar-17 15:41):

*) made metarouter work on boards with 802.11ac support or usb LTE;

Works for me.

RB2011 crached when LTE-modem connected

RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Re: RB2011 crached when LTE-modem connected

Who is online