I have upgraded the SwOS of my CRS328-24P-4S+RM from 2.15 to 2.17 and had some compatibility issues with a CRS354-48G-4S+2Q+RM (SFP ports did not connect), so I tried downgrading SwOS back to 2.16 (2.14 and 2.15 were not available, factory only) but the switch didn’t came back to life.
Now the power light, all network ports light and the FAN/PoE fault lights keep on (never turning off) when the switch is on.
At the serial port I just get immediately the attachment’s data after powering on, nothing more, nothing less, every time. Not a single response when I press any key, and keeps like this indefinitely.
Netinstall was a dead end as well, switch not even gets recognized.
I also tried Mikrotik support, but they don’t seem to understand the case, they said it’s unfixable.
IMHO this board needs a serial boot file (that’s what the serial output is asking for) to star with or a NAND reflash to fix it for good.
I have already bought some flash memory reader/writer to try something if needed.
It’s not possible that I should throw away a perfectly good switch because a single firmware upgrade has gone wrong…
I have the exact same error messages in the serial console output after a failed firmware upgrade on the same Mikrotik CRS328-24P-4S+RM after clicking the download+upgrade button in SwOS. I also have not been able to get netinstall to recognize the device to allow new firmware to be installed. If you found/find a solution I would love to hear it.
It looks like the downgrade may have corrupted the NAND or bootloader, which is why the switch shows solid LEDs and doesn’t respond via serial or Netinstall. The serial output suggests it’s looking for a boot image, which likely means it needs either a serial boot file or a full NAND reflash. Since MikroTik support isn’t helping much, your best option now is to use your flash programmer to write a clean NAND dump from another working CRS328-24P-4S+RM. Just make sure to back up the current chip first. You can also try pushing MikroTik support again and mention you have UART access and are ready for manual recovery if they can provide a recovery image.
Well, this happened to me today although without downgrade. A simple SWOS download&upgrade bricked the switch. It was running happily for months at 2.17 and today I dared to upgrade to 2.18. After the restart the switch never comes back, all lights are on and the serial console shows the same output as in the original post. I have contacted my distributor for RMA. I can not advise to update to 2.18.
I’m sorry that I do not have posted any updates here, but we have had to buy a new switch as a replacement, RMA was not available for me.
I also bought a NAND flash reader and did the dump of the current flash state.
It appears to me that the Mikrotik updater has messed up some pointers because there was a SwOS flashed in the wrong place (don’t ask me how I know this, I have compared dozen of images with the broken flash file to find out), but the mikrotik variables are intact, I just need the bootloader part to fix it…
I also tried to compile a uboot to send through serial, but the uboot compiler needs the bootloader file, the one thing that I don’t have.
What I really do not understand is why is it so hard for Mikrotik to provide the serial image or the bootloader for us to correct their mistakes…
My plan (due to when I have some spare time) is getting the entire flash through a modified software in the working new switch, to extract the bootloader and fix the flash of the broken switch.
I should not need to do all this to fix a software issue, come on Mikrotik, help us here!
About this, I really tried several times before posting here in the forum, I did explained a lot and very detailed, they sent me to RMA everytime…
I do not believe that this is a single version update issue, this is a problem they might have with the SwOS updater, the memory pointer could be missing some checks or it can have another memory related problem… Of course, this is just speculation, based on what I saw in my corrupted flash memory.
I’m now staring at a broken switch and an unanswered support ticket after upgrading from 2.17 to 2.18. Did you ever manage to get the uboot necessary to fix this or did you just move on from the issue?
This is my first time dealing with Mikrotik’s tech support and I’m not too impressed. Not that they care, but this experience is making me want to just switch my homelab away from their switches after I took a chance on them.
I’ll definitely be recommending people away from their gear.
Add me to the list of users with a failed upgrade v2.17 to v2.18. Also no useful response to my support ticket yet. Just the usual: try netinstall, try backup bootloader, etc. The switch isn’t responding to anything and has the same serial output as OP.
Hello, I’m yet to extract the bootloader from a working unit. I did not gave up yet. All I need is time to deal with it… I need to create a custom firmware and netboot it into the working unit to be able to extract the entire working flash, then separate the bootloader only so that it can be rewritten to the broken unit using the hardware flash writer and/or a custom uboot. I’ll post the results here when there is any progress…
Mikrotik finally replied to my support ticket without any helpful suggestions or files to attempt to send via UART, instead directing me to RMA (not an option for me sadly). I ended up having to buy another replacement unit in the meantime, with the bricked on sitting off to the side.
If anyone is able to figure out and provide a hacky solution here it would be appreciated!
Good to see others have the same problem (actually not good, but at least I am not the culprit). My switch is still at the distributor for RMA. I had a ticket open at Mikrotik support, with the question what is the safest and best way to upgrade, as I did not want to brick my replacement switch. The answer is "There is no best way how to do it, you can upgrade the device in different ways, but the result would always be the same. All the ways you can upgrade the device are the same."
Well, according to my experience and that of other people in this thread this is not true. What I did to upgrade my replacement switch:
It came with ROS 7.16.1. First I updated it through winbox (system/packages update) to 7.19.4. Then I did the firmware upgrade + reboot. Then from ROS I upgraded SWOS to 2.18 (by /system swos upgrade). Rebooted into ROS again. Then I set system/device-mode/update mode=advanced & set system/device-mode/update routerboard=yes. Then came the actual reboot to SWOS. Then the final upgrade from within SWOS by using download&upgrade.
At least that worked and I have a running replacement.
Well Mikrotik finally got back to me and told me if I'm covered under warranty to submit an RMA with the distributor I bought it with. I assume my warranty is over (it's been over 1 year) but I reached out to my distributor anyway. Pretty horrible service for what is clearly a common problem caused by a flawed Mikrotik update.
And my distributor is telling me I'm out of luck. Probably popped up 1 week after my warranty expired. I've enjoyed Mikrotik gear for many years but this is enough to make me jump to another product. Can't believe how poor Mikrotik's response has been for a problem they caused their users.
Paging @felixka - what are the chances we can whip up some recovery process for the CRS328 similar to what you accomplished for the borked RB5009 units? This model also uses Marvell switch and ARM cpu, after all.
Generally, it should be possible.
But it'll be much more work, as you wouldn't be able build on any work to make the CRS328 run on OpenWrt like I was able to. I just put other people's work together in a smart way. Much of that work hasn't been done for the CRS328.
Your best bet would be getting u-boot working and then using the serial console to transfer a working binary and write it into flash (or desolder the flash and write it in a programmer).
The part that's missing is the "backup bootloader" (aka ARM Trusted Firmware) which technically is the first stage bootloader. Having that in place would allow you to netinstall the switch.
The "universal packages" on RouterBOARD - RouterOS - MikroTik Documentation do contain these binaries for many different platforms and you can technically extract them from there.
However, none of the files there contains any binaries for the firmware type used by the CRS328 (dx3230L), unless they use a different naming scheme there.
I personally don't own a CRS328 and it's impossible for me to develop something here without the hardware in question on hand. But I can probably answer questions if anyone wants to attempt this.
@felixka I do own a CRS328, and I am willing to tinker.
Right now it is my primary switch, however since RouterOS igmp querier is basically broken anyway (won’t run on vlans) I’m willing to swap it out for a Cisco CBS I have here for a bit.
Would it be possible for you to, say, relay the steps to me to extract the necessary bits from my current working CRS328? I’m pretty technical and have raspi units, an SOIC clip, etc at my disposal.
If I netinstalled my switch to latest ROS and sent you a full ROM dump from it, would that help?
Maybe stupid question, but did people having these devices bricked try to get to netinstall with BOTH the "main" and "backup" bootloader (the one or the other is loaded depending on when the reset button is pressed/power is applied)?
Maybe they are both corrupted, but you won't know unless you try them.
Or see if via serial CTRL+E work?
Or this only applies to devices running RouterOS (and not SwOS, bricked or not, on dual OS devices such as the CRS 328)?
@felixka
Having the console/serial port should allow more "tricks" before resorting to de-solder the chip, but is there a way to use it to actually write to it?
Even if it is not available in a download package, the bootloader could be extracted from a dump of a working device (I guess this is something that we can probably find willing members to do, if it is safe, while removing the chip or using a probe won't likely be).
Netbooting OpenWRT seems to be possible even if the device is on v7.x bootloader, though changes are not persistent: https://openwrt.org/toh/mikrotik/common
so, if the netboot can be set via console, likely it can boot something else (but WHAT?)