Community discussions

MikroTik App
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

3 routerboards bricked this week

Sat Jun 05, 2021 7:48 pm

I'm really pissed. I have RB4011, RB2011 and hAP-ac2.
Both my RB4011 and hAP-ac2 wouldn't update (package downloaded automatically or manually to storage but after reboot nothing happened).
I decided to install the latest RouterOS using Netinstall which effectively bricked my 4011 and hAP-ac2. They are in a boot loop and repeated Netinstall does not change anything.
What the hell is going on with Mikrotik?
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 6:39 pm

I have no experience of this myself. Are you using the storage as something more than software and configuration? e.g logs? database? packet-captures?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 6:49 pm

You do not provide any detail, for example software version before and after, etc.
Your thread is useless.

I can write another thread "no routerboards bricked from 2007"...

I'm really pissed. I have RB4011, RB2011 and hAP-ac2.
Both my RB4011 and hAP-ac2 wouldn't update (package downloaded automatically or manually to storage but after reboot nothing happened).

Because you have disabled security package thinking for you are useless,
but without security package installed and enabled on the latest version, you do not upgrade anything.
Last edited by rextended on Mon Jun 07, 2021 6:55 pm, edited 1 time in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 6:52 pm

I decided to install the latest RouterOS using Netinstall which effectively bricked my 4011 and hAP-ac2. They are in a boot loop and repeated Netinstall does not change anything.
Another useless detail: what netinstall version, what routeros version?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 6:54 pm

What the hell is going on with Mikrotik?
If you are unable to use tools provided from mikrotik, it's your fault.
I'm a professionist and no one device brick on update/upgrade or on netinstall.
I'm not fortunate, I only know what I'm doing.
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 9:01 pm

I wrote another thread with details and error messages but moderator did not approve it. Huh?

I had 6.48 on my RB4011 when it stared to refuse to upgrade any further.
I had 6.47 on my first hAP-ac2 when it stopped updating.
I bought new hAP-ac2 with 6.45.9 stock firmware few days agoa and it does not update! When I check installation at stops at 69% saying the image is damaged. This is unit I just unboxed!
I tried Netinstall with all possible LTS and current versions with both 4011 and hAP-ac2 all of them end in boot loop.
I can check what's going on with the RB4011 because it has serial but not with hAP-ac2. It just blinks and reboots.
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 9:03 pm

Regarding use - I am not using and never was using those devices for anything else but WiFi extenders. They have all ports (WiFi included) in a bridge and that's it.
Firewall is off, routing is off. The only enabled packages are system, security and wireless on all devices.
My logging is configured as default (with most going to RAM and only critical mesages to log).
I never downloaded or stored anything on the NAND flash ever. It just cannot be worn out.
Last edited by slaviusz on Mon Jun 07, 2021 10:50 pm, edited 1 time in total.
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 9:07 pm

Here's the log from RB4011 after Netinstall with 6.48 (and other subversions)

Netinstall finishes with no error, sending reboot and the device never comes alive.
AL31400X-140


RouterBOOT booter 6.48

RB4011iGS+5HacQ2HnD

CPU frequency: 1400 MHz
  Memory size: 1024 MiB
    NAND size: 512 MiB

Press any key within 2 seconds to enter setup..

loading kernel... OK
setting up elf image... OK
jumping to kernel code
opendir: No such file or directory
ERROR: no system package found!
Kernel panic - not syncing: Attempted to kill init!
{bf84df0c} panic+0x84/0x1c0
{bf84df44} do_exit+0x80/0x688
{bf84df84} do_group_exit+0x4c/0xc0
{bf84df9c} __wake_up_parent+0x0/0x18
CPU1: stopping
{bf879f5c} handle_IPI+0xf4/0x1e0
{bf879f7c} gic_handle_irq+0x94/0xa4
{bf879f94} __irq_svc+0x40/0x60
Exception stack(0xbf879f98 to 0xbf879fe0)
9f80:                                                       80d2c890 00000000
9fa0: bf879fe0 00000000 bf878000 804e9a48 804a1a78 804e9cc8 0000406a 412fc0f4
9fc0: 00000000 00000000 00000000 bf879fe0 8000e860 8000e864 60000013 ffffffff
{bf879fdc} default_idle+0x24/0x28
{bf879fe4} cpu_idle+0x50/0x8c
{bf879ff4} 0x486174
CPU3: stopping
{bf891f5c} handle_IPI+0xf4/0x1e0
{bf891f7c} gic_handle_irq+0x94/0xa4
{bf891f94} __irq_svc+0x40/0x60
Exception stack(0xbf891f98 to 0xbf891fe0)
1f80:                                                       80d46890 00000000
1fa0: bf891fe0 00000000 bf890000 804e9a48 804a1a78 804e9cc8 0000406a 412fc0f4
<4>1fc0: 00000000 00000000 00000003 bf891fe0 8000e860 8000e864 60000113 ffffffff
{bf891fdc} default_idle+0x24/0x28
{bf891fe4} cpu_idle+0x50/0x8c
{bf891ff4} 0x486174
CPU2: stopping
{bf883f5c} handle_IPI+0xf4/0x1e0
{bf883f7c} gic_handle_irq+0x94/0xa4
{bf883f94} __irq_svc+0x40/0x60
Exception stack(0xbf883f98 to 0xbf883fe0)
3f80:                                                       80d39890 00000000
3fa0: bf883fe0 00000000 bf882000 804e9a48 804a1a78 804e9cc8 0000406a 412fc0f4
3fc0: 00000000 00000000 00000000 bf883fe0 8000e860 8000e864 60000013 ffffffff
{bf883fdc} default_idle+0x24/0x28
{bf883fe4} cpu_idle+0x50/0x8c
{bf883ff4} 0x486174
Rebooting in 1 seconds..:00000050
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 10:54 pm

Because you have disabled security package thinking for you are useless,
but without security package installed and enabled on the latest version, you do not upgrade anything.
Who says that? There's nothing in RouterOS manual or wiki page about upgrading about that.
That doesn't even explain why full .npk Netinstall reders 3 of my 4 routers once worth 1000€ useless bricks laying on my desk.
I followed instructions provided by Mikrotik to upgrade and then Netinstall flash my routers as several times before which bricked them - why shouldn't I be pissed of?
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: 3 routerboards bricked this week

Mon Jun 07, 2021 11:19 pm

When I check installation at stops at 69% saying the image is damaged.
ERROR: no system package found!
I would suspect the downloaded files. What happened to them? Some virus crippling the content? Network errors? Wrong files? Anti-virus software action (inoculation)? Memory bit error (non-ECC memory)? File has correct MD5 checksum? Also after duplication?


As the on-line update failed as well (direct download) it should be somewhere in the download path. Caching somewhere in that path? Firewall with anti-malware inspection?
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 12:30 am

I just checked all downlaoded files (from all of my attempts from LTS 6.47.10, 6.48 and 6.48.3) they all have correct SHA256 as stated on the downlaod page https://mikrotik.com/download

I fully control the whole network since the ISP fiber to my PC. There's no IDS/IDP, no proxy, no anti-virus (running Linux on this computer). The network is basically composed of a router, a bridged mikrotik and unmanagable gigabit switch. I did not experience any other data corruption copying anything over the network ever since it has been assembled.
I've re-downloaded the files several times in the course of 2 days as I was scratching my head so it would be extremely unlucky bit-flip errors on multiple downloads.

I can try to do Netinstall from Windows PC just to be sure.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 12:46 am

You have the higest amount of failed update of all time,

It's the time to give some question to yourself...

Something on your methods are wrong.
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 2:08 am

You have the higest amount of failed update of all time,

It's the time to give some question to yourself...

Something on your methods are wrong.
Can you be more specific? Is following documentation and vendor manuals a "wrong method"?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 9:57 am

Without be here to "view", I just make one example: problem on computer used to send file or do netinstall
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 11:20 am

So I did NetInstall from another PC running Windows. With 6.48 and 6.48.3 the result is exactly the same for RB4011 and hAP-ac2. They still loop in boot, RB4011 producing error:
opendir: No such file or directory
ERROR: no system package found!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 11:42 am

have you try with netinstall 32 bit 6.47.10 and full package routeros 6.47.10 ?
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 12:00 pm

No, I tested latest stable Netinstall x64 on my Win 10 x64.

How does software compiled for 32 bit register size impact higher correctness of data transfer and success rate of bootp transfer exactly?

I will test 6.47.10 today.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 12:15 pm

You think than change "only" register size change application to work from 32 to 64 bit?...
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 12:20 pm

Netinstall is very fragile process. Often netinstall seemingly does its job (returning to ready in very short time) but actually doing nothing .... proper netinstall process takes some time (IIRC something around 10-30 seconds, depending on device's storage size and platform). So it is really vital to go according to the instructions. There have been reports where even NIC type used in laptop was a show stopper ... at the end the user used an USB3-to-ethernet adapter (while disabling built-in NIC) to get things done ... even though use of USB-to-ethernet adapters is usually not recommended.
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 4:31 pm

The Netinstall process works correctly as observed from the output of the tool as well as the RB4011 serial console.
Remember my routers were working fine *until* I flashed them with Netinstall. Should Netinstall encounter any error it would be presented in the output.

Yes, I do know that compiling the same program (especially as rudimentary as C implementation of bootp) on a 64bit OS does essentially nothing but increases available address space and registers to 64 bits. Bootp source code never used anything bigger than 32 bit integers. We're not talking about changing CFLAGS to -march=amd64 -mtune=zen2
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 6:14 pm

Nono, not misunderstand me, sorry for the bad english,
I want to say than two executebles on windows with 32 or 64 bit take different path inside machine, not from source code perspective.
Sorry, I do not reach to explain better on englisn.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2978
Joined: Mon Apr 08, 2019 1:16 am

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 7:20 pm

The Netinstall process works correctly as observed from the output of the tool as well as the RB4011 serial console.
Remember my routers were working fine *until* I flashed them with Netinstall. Should Netinstall encounter any error it would be presented in the output.
I can't follow this.

correctly ... like this?
jumping to kernel code
opendir: No such file or directory
ERROR: no system package found!
Kernel panic - not syncing: Attempted to kill init!
{bf84df0c} panic+0x84/0x1c0
routers working fine ... but no on-line or manual upgrade possible?

Something going wrong with the PXE boot I presume. (I also read somewhere that PXE-boot is only working on the boot-ethernet port of the router. Ether1?).
In addition to the wiki ... https://techsoftcenter.com/tutorial-how ... etinstall/
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 3 routerboards bricked this week

Tue Jun 08, 2021 7:26 pm

the label on the device says "boot" when it is not ether1,
litke the ether12 is the boot port of CCR1016
ether7 on ccr1009
etc.
 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Wed Jun 09, 2021 8:09 pm

So, I managed to revive my RB4011!

I went to boot menu, reformatted NAND, repartitioned. Then I Netinstall flashed ROS7.1b6 and that resulted in reboot to RouterOS.
I was then able to downgrade to 6.48.3.

Now I have to figure out how to do that with hAP-ac2 as it has no serial (and I'm not sure if it has JTAG)
 
Zid2000
just joined
Posts: 10
Joined: Thu Oct 22, 2020 4:31 pm

Re: 3 routerboards bricked this week

Fri Jun 11, 2021 11:56 am

Let me know if you manage to revive (and how) your hAP-ac2. I have the same problem with wAP ac (it worked flawlessly for a year with regular upgrades on stable channel and then v6.48.1 bricked it). As I didn't have LAN adapter for my Macbook, I left it as such for months until I managed to get one of those USB-C to LAN adapters and tried using reset button to trigger boot from backup boot-loader but it's still stuck in reboot loop. As there is no serial port, I too am stuck in figuring a way to revive it.
 
hecatae
Member Candidate
Member Candidate
Posts: 244
Joined: Thu May 21, 2020 2:34 pm

Re: 3 routerboards bricked this week

Fri Jun 11, 2021 3:44 pm

Is the rb2011 working?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: 3 routerboards bricked this week

Fri Jun 11, 2021 4:00 pm

 
slaviusz
just joined
Topic Author
Posts: 13
Joined: Fri Jun 04, 2021 8:53 pm

Re: 3 routerboards bricked this week

Mon Jun 14, 2021 12:26 am

So I also figured out how to revive both of my hAP-ac2 s!

I booted with the backup loader and flashed ROS7.1b6 with Netinstall. It seems the ROS7 has different disk layout (probably also different filesystem with the new kernel) and this resulted in correct boot! Repeated flashing with ROS 6.x didn't help before. Now I'm with ROS 7 on both of my hAP-ac2 s and plan on downgrading to LTS or latest stable.

Good luck to everyone with similar problem!

Who is online

Users browsing this forum: No registered users and 12 guests