No space left on device. Now what?

Hi all. My box is a:

  Board type: CCR1009-8G-1S
  Firmware version: 3.27

Since a few days I had to put it offline as it was not working anymore.
Once I connected the serial console, the boot shown the following messages:

RouterBOOT booter 3.27                                                                                                                          
                                                                                                                                            
CCR1009-8G-1S                                                                                                                               
                                                                               
CPU frequency: 1200 MHz                                                        
  Memory size: 1024 MiB                                                        
    NAND size: 128 MiB                                                         
                                                                               
Press any key within 2 seconds to enter setup..                                
                                                                               
loading kernel... OK                                                           
setting up elf image... OK                                                     
jumping to kernel code                                                         
Starting...
/etc/rc.d/rc.sysinit: 60: cannot create /var/run/utmp: No space left on device
System halted.

To me this seems that the underlying Linux system is not bootable any more.
As far as i know I have no configuration and license backup but those stored on the unit itself.
I have tried to “force backup-booter loading” but it didn’t help.
I haven’t done a “format nand” as I don’t have a license backup.

Is there a way to fix this situation?

Many thanks in advance.

Don’t do Format NAND ! Use Netinstall to reinstall RouterOS, it will fix the FS and will not touch the license.

I didn’t do that, as it sounded “scary” :laughing:
What about the configuration file? Will be it kept?
TIA.

Netinstall will repair as much as it can. Depends on what caused this problem. Maybe there is some bigger issue that caused data corruption.

OK. I’ll follow your suggestion and will post the news here.
I am going to use XModem as there’s no IP connectivity any more.
Is the “main package” just enough to bring it back to normal operations (provided that there’s no other major reason for failure)?
Paldies!

I have got this after the lengthy XModem transfer:

press any key to continue...                                                                                                                                |
file transfer ok                                                                                                                                             
invalid upgrade file id

I was using the file “routeros-tile-6.35.2.bin” I downloaded from Mikrotik website.

Now what?

Upgrade packages comes in .npk extension. Check this

Xmodem has no relation to my suggestion. I suggested to use Netinstall. You must start the device in Etherboot mode, not in Xmodem mode. Follow instructions here:

http://wiki.mikrotik.com/wiki/Manual:Netinstall

If I go here, I read “npk” in the link but I get a “bin” file with google chrome.
Anyway, I re-downloaded the file with wget and there’s no difference between the two.
I’ll try a transfer with the new file name.

As I have no IP connection any more to the CCR, I need to use the Xmodem.
Isn’t it working?

Xmodem can’t be used for reinstalling. Netinstall will work. You don’t need IP connection, that is the whole point, the RouterBOOT will run the Etherboot server and you will have IP connection.

The CCR is currently only connected with serial port to a Linux server I have there (I am not working in the site now).
All other ethernet connections have been moved to the spare unit we shipped in the remote site.
And, most important thing, we don’t have any Windows machine there.
Do I need to bring a Windows machine on site and attach it directly to the CCR?

Yes

Just to understand: what’d the XModem upgrade procedure be for?

RouterBOOT (BIOS) upgrade only.

If you are linux savvy, you could do this: http://www.cyberciti.biz/tips/running-x-window-graphical-application-over-ssh-session.html
and then run Netinstall in Wine

In remote sites, besides all application servers and storage we only have a single tiny Linux machine for support we access with SSH and that runs a number of SSH-based scripts and applications in order to monitor and “fix” certain services.
But installing XServer and Wine there would be a problem also because there’s no graphical adapter at all.
Yes, there’s XVNC and the likes … that’s not a viable solution, in my case at least.
I asked site controller to ship the unit back to my office.

In some cases it would be very handy if it was possible to recover a router without having a Windows machine onsite.
Maybe if the router could install from a recovery USB stick with the .npk files and a file with some settings (similar to
what netinstall has), used by the bootloader? Netinstall could be extended with a button to put such a file on a USB stick,
knowledgable users could just write a textfile using a text editor.
This would be similar to what many modern motherboards can do: update their BIOS using a USB stick with just a
file on a FAT filesystem, instead of the oldfashioned way where one would have to prepare a bootable DOS diskette
that has some updater .exe on it.

We already have a (powerful) serial console.
We already have XModem via console in the “BIOS”, even if uploading 16+ MB via 115.2K console takes some time.
If that is not possible there could be a way to restart the unit in “single user mode”, setup an ethernet port, download an image via TFTP etc. etc.
This is just my idea, but I have no deep knowledge on how the CCR “BIOS” actually works.
My religion says “thou shall connect the console cabels to some Linux boxe”. So I did.
But I just knew that the current console features are uneffective in my case.

Anyway, effective emergency operations should not rely on specific software and specific OS. Just protocols, like RS232, TFTP, BOOTP etc.

Ok but that is basically what netinstall is doing.