Out of disk space - upgrading hAP Lite from v6.35 to v6.35.2

Erayd · Mon May 09, 2016 2:09 am

What's the recommended process for upgrading a hAP Lite from v6.35 to v6.35.2?

Attempting to upgrade it via System>>Packages>>Check for Updates>>Download & Install results in the following error:

ERROR: not enough disk space, 7.7MiB is required and only 7.3MiB is free

I have reset this router to default config, and removed all files, but this did not help. I do not have any non-core packages installed. System>>Resources says that there is 7.4MiB disk space and 8.7MiB RAM available.

Mon May 09, 2016 9:05 am

please post "/system package print" and "/file print" just to be 100% sure you didn't miss something

Netinstall should be able to solve this, but I would like to learn how this situation became possible.

Erayd · Tue May 10, 2016 12:34 am

please post "/system package print" and "/file print" just to be 100% sure you didn't miss something

Too late sorry - my post got stuck in the moderation queue and I gave up waiting for it - ended up doing a netinstall, which worked.

I do know that prior to the upgrade, I cleared out files and checked packages for anything unnecessary. The only file was a 40B JSON file (skin), and the only packages installed were core ones (default packages plus wireless-rep), and all were at v6.35.

Erayd · Tue May 10, 2016 1:02 am

I've tested this with a spare hAP Lite - I can reliably reproduce the problem.

To reproduce:

1. Downgrade to v6.35 either via System>>Packages or netinstall
2. /system reset-configuration skip-backup=yes
3. Install wireless-rep
4. Reboot
5. Attempt to upgrade via /system package update check-for-updates, install

This does call into question the rationale behind using such a small amount of flash - it's cheap, and going with something slightly larger would be a non-issue cost wise. In my personal opinion, it should be large enough to accommodate *all* possible packages, to ensure update reliability. 16MB is really proving to be inadequate here.

I've seen other posts claiming that the update is done entirely in RAM (and that available disk space is therefore irrelevant), but that's clearly not the case for the hAP Lite.

Thu May 12, 2016 8:08 am

http://forum.mikrotik.com/viewtopic.php ... 78#p522278

Erayd · Thu May 12, 2016 10:56 am

Thanks jarda - looks like things are pretty shocking there!!!

Normis - what's the recommended procedure for upgrading one of these remotely? If I'm using the wireless-rep package to connect to it, then I can't remove that package in order to free enough space for the update to work.

Also, I would very much like to know - what actual *testing* did Mikrotik perform regarding the upgrade process on the hAP Lite before this device was released? I cannot believe that this issue was not obvious then.

Thu May 12, 2016 10:59 am

Actually deleting all files and reboot can help a bit too. You can install wireless package together with the system in the first step. Then in second step the rest.

Thu May 12, 2016 11:01 am

I consider putting only 32mb of ram and almost any flash into new devices to be very bad idea. It is useless source of actual and future problems.

Thu May 12, 2016 12:38 pm

rep package is considered extra. either change over to REP and then upgrade, or remove it first.

koolandrew · Thu Apr 05, 2018 9:20 pm

Is there any solution to this. i now have 2% disk space on a hap lite, and the router is becoming unstable, in that it will run at 100% cpu when i try to do a supout.rtf to report it to Mikrotik.

Surely there has to be a solution, to fix this.

I have many of these, and i am afraid they will all follow suit.

I have installed the latest software 6.41.3 but now i doubt i will ever be able to change that without doing a netinstall.

Please provide a solution Mikrotik.

cgaspar · Thu Apr 05, 2018 10:27 pm

Something similar happening to me using hAP Lite as well. I went to System>Packages>Check For Updates then I clicked on Downlad&Install but I've chosen a different firmware version I really wanted, so I cancelled. For some reason It got stuck on the routerboard's files and I had to reset to get back it clean cause I was out of space

Thu Apr 05, 2018 10:35 pm

Remove unwanted packages. Remove all removable files. Put newer single system package only and reboot. Then add rest of required packages and reboot again.

juliokato · Thu Apr 05, 2018 11:49 pm

Workaround: use netinstall...

coylh · Sat Apr 07, 2018 10:24 pm

Or buy products with a respectable amount of flash... 16MB is embarrassing. It's like running your network with floppy disks.

haik01 · Sun Apr 08, 2018 1:10 am

No, worse than floppy disks...... Punch cards.....

Wyz4k · Thu Mar 07, 2019 4:57 pm

Remove unwanted packages. Remove all removable files. Put newer single system package only and reboot. Then add rest of required packages and reboot again.

There isn't enough space to do that. This is a remote router. Netinstall is not an option.

Trying to upgrade from 6.42.3:
I have deleted everything in Files except the skins folder and rebooted.
Can't download 6.42.6 - this has space saving functionality
Can't download 6.42.12.
Can't download 6.43. - this allows you to use ram for files
Can't download 6.43.12. - this is the version I want to get to

Supout.rif sent to support.

Thu Mar 07, 2019 5:37 pm

I wrote nothing about netinstall. If it is remote installation, add eventually other one or two packages that are critically necessary to log in aside the single system package during the first installation process step.
If you want to be sure that nothing bad happens, prepare second device and let someone to swap it at place. Then you can do whatever with the first one and use elsewhere afterwards.

Ansy · Sat Mar 16, 2019 8:41 pm

May be don't include modules disabled by default in RouterOS package for smips?
advanced-tools, hotspot, mpls, routing, security -- that modules can be uploaded separately.

Wyz4k · Sun Mar 17, 2019 9:35 am

May be don't include modules disabled by default in RouterOS package for smips?
advanced-tools, hotspot, mpls, routing, security -- that modules can be uploaded separately.

That's how I managed to resolve the problem. By only downloading the absolute minimum packages required for the next version and rebooting. You only need system and security for most purposes. Wireless too for non-ethernet connected devices.

mkx · Sun Mar 17, 2019 5:33 pm

May be don't include modules disabled by default in RouterOS package for smips?
advanced-tools, hotspot, mpls, routing, security -- that modules can be uploaded separately.
That's how I managed to resolve the problem. By only downloading the absolute minimum packages required for the next version and rebooting. You only need system and security for most purposes. Wireless too for non-ethernet connected devices.

In recent versions security package depends on dhcp package

Even if DHCP package is installed but disabled, upgrade won't proceed ... so DHCP package has to be installed/enabled before performing upgrade.

Wyz4k · Mon Mar 18, 2019 2:57 am

In recent versions security package depends on dhcp package Even if DHCP package is installed but disabled, upgrade won't proceed ... so DHCP package has to be installed/enabled before performing upgrade.

Meh. It would be nice if they ship the devices with the individual packages installed by default instead of the bundled packages. That way you can "uninstall" what you want and just reboot to free up space.

silversword · Tue Oct 06, 2020 7:39 am

There are too many packages installed by default to be able to be in-place upgraded. Here's what's there now:

Download SMIPS extra Packages zip file. Extract these:

Drag and drop the files to Files

Reboot

Install extra packages if you need them. See descriptions from here:
https://wiki.mikrotik.com/wiki/Manual:System/Packages

mongobongo · Mon Oct 12, 2020 11:33 am

And when we have a network of hundreds of HAP lite ??

Mikrotik should provide a trouble free solution to this situation ...

Mon Oct 12, 2020 11:36 am

Do your hundreds of devices have custom packages installed? Normally there shouldn't be any custom packages and no problem upgrading.

silversword · Mon Oct 12, 2020 7:33 pm

My instructions were based on hap lite factory defaults. There is not enough RAM to upgrade by default methods all factory configured packages. If you have hundreds on hap lites deployed, automate my instructions or hire someone that can do it for you.

I stopped buying hap lites because the hardware doesn't have enough ram. IMHO mikrotik shouldn't have factory images with so many packages installed on these low end devices

Darky · Mon Oct 12, 2020 9:09 pm

Will need to do a NetInstall otherwise :(

Tue Oct 13, 2020 7:59 am

My question was, does anyone have issues using "check for updates" with a device with factory installed version?
Because in the examples I either see custom package sets, or 6.48 beta versions.

pe1chl · Tue Oct 13, 2020 12:12 pm

My question was, does anyone have issues using "check for updates" with a device with factory installed version?
Because in the examples I either see custom package sets, or 6.48 beta versions.

I had this problem on my hAP mini (MUM freebie) but I solved it by one time doing an upgrade using only the separate packages I require instead of the combined package, and from then it upgrades using the one-click method without problem.
There is no wildly unusual config on it: 3 SSIDs each connecting to a VLAN on the ethernet connected to my router. It serves as an AP in the bedroom.
No routing, no NAT, almost no firewall.

I should have tried doing a reboot and then immediately an upgrade, the buildup of RAM usage (and fragmentation) during extended usage could have been just the trigger that pushed it over the limit. Note that the upgrade procedure is VERY CLOSE to the limit of RAM. That also explains why omitting a couple of small packages I do not use solved it for now.

mongobongo · Tue Oct 13, 2020 12:25 pm

Do your hundreds of devices have custom packages installed? Normally there shouldn't be any custom packages and no problem upgrading.

Hi, fortunately I dont have 100s of device I have 10s of devices.

All device have default packages.

This issue didnt just start happening out of the blue.

Currently the best solution (without having to create scripts etc) is to login and drag only needed packages onto the device and reboot .......

mongobongo · Tue Oct 13, 2020 12:27 pm

My question was, does anyone have issues using "check for updates" with a device with factory installed version?
Because in the examples I either see custom package sets, or 6.48 beta versions.

Yes,

from offset factory defaults.

Devices have been upgraded for consecutive number of years.

Last few years this problem is getting worse and worse ..

pe1chl · Tue Oct 13, 2020 6:24 pm

mongobongo: Did you ever try to reboot the device just before trying the update? That may solve it for now.

The reason I mention this is that MikroTik claim in many threads that they "always test an upgrade with a device running with default settings" and they claim it works.
But we as the users see it fail. So there must be something different between their testlab and "the field". Apart from maybe some complex configurations, another difference likely is that they just netinstall their device with a test setup, and immediately try to upgrade. That has more chance of succeeding than trying to update a device that had been running for a couple of weeks and has used up or -fragmented RAM.

silversword · Wed Oct 14, 2020 7:15 am

My question was, does anyone have issues using "check for updates" with a device with factory installed version?

Yes, the first screenshot on my 10/6/2020 post was a default packages preinstalled on hap lite devices. No extra packages installed, had only ever done a simple NAT quickset config and check for update on that Mikrotik device and starting ~6months ago check for updates has been failing due to low disk space when trying to update.

mkx · Wed Oct 14, 2020 8:28 am

I fully agree that on a typical SOHO device a few packages have no place to be installed as factory default. These include: mpls, routing and hotspot, they are clearly advanced stuff that vast majority of users will never need. In addition, these are quite probably not necessary for router to be initially set up and connected to internet, at which point they can be manually uploaded to the router. Many users won't need a few additional packages (such as ppp and advanced-tools), but those might be needed for successful initial device setup, hence they should stay in the default installation list of packages.

pe1chl · Wed Oct 14, 2020 10:28 am

As written in other threads, it was not good to make the "combined package" and thus make it impossible to remove packages and gain space.
Indeed as you point out, there are packages on this class of devices that have absolutely no business being there. But they do take up space.
In my opinion the package combining should never have happened, and instead there should have been a convenient package selection feature that retrieves the wanted packages from the download server without the hassle of downloading a bundle, unzipping, and uploading to the router. Just a list showing all available packages and click-to-install-this. Then it would not be such a hurdle to decide to remove something, it can always be re-installed.

But apparently the move is now further towards a "one package does it all". In v7 there is just the "routeros" package that includes everything that was in the combined package, and more (e.g. the ntp package is now included as well). So the option of omitting what you don't need is going away.
I understand that it is extra work to maintain exact dependency information and that some features introduce dependencies that are not obvious to a new user (e.g. some VPN types requiring DHCP). But then maybe there should be a "RouterOS v7 light" that just does not include all functionality, or else these "mini" devices should be abandoned.

aesmith · Wed Oct 14, 2020 12:59 pm

I've got two devices with 16meg flash. Looking at "Files" both report around 12.6 used out of 16, but the actual files listed only amount to less than 100k. How do you see what is actually occupying flash?

pe1chl · Wed Oct 14, 2020 1:17 pm

I've got two devices with 16meg flash. Looking at "Files" both report around 12.6 used out of 16, but the actual files listed only amount to less than 100k. How do you see what is actually occupying flash?

This is not about 16M flash. That is not the problem (yet). You cannot see the RouterOS files that occupy the flash.
The issue is about devices with only 32M RAM (and 16M flash). These are the ones with the upgrading difficulties.

mongobongo · Tue Oct 20, 2020 9:44 am

mongobongo: Did you ever try to reboot the device just before trying the update? That may solve it for now.

The reason I mention this is that MikroTik claim in many threads that they "always test an upgrade with a device running with default settings" and they claim it works.
But we as the users see it fail. So there must be something different between their testlab and "the field". Apart from maybe some complex configurations, another difference likely is that they just netinstall their device with a test setup, and immediately try to upgrade. That has more chance of succeeding than trying to update a device that had been running for a couple of weeks and has used up or -fragmented RAM.

Yes of course,

the "golden rule" in most things IT related, if it does not work, restart it and try again

:P

flameproof · Thu Oct 22, 2020 12:45 am

I'm in the category of "have 1000s of devices". To be precise, close to 15.000 hAP Lites in our network. We are trying to upgrade them to 6.47.4, to no avail. No custom packages, nothing out of the ordinary.

We get told the CPEs have 7.3MB left and 7.4MB are needed. We need a way to package single .npk files into an upgrade package with just what we need. Would we ever consider MPLS on a hAP Lite? Not in a million years... so why not give us the ability to remove it from the upgrade package?

I have just tried lowering DNS cache from the default to 512kb, and free memory has actually decreased to below 7MB (was suggested in another thread). Rebooting does nothing to help either. Have disabled MPLS package, no noticeable difference.

We have a mix of 6.43.12 up to 6.46.1, due to the way we provision them with a custom-made app. We need 6.47.4 on our CPEs (actually, anything higher than 6.46.3 as that's when the hostpot login redirect got fixed...).

flameproof · Thu Oct 22, 2020 12:55 am

Taking the mick now... can't make its mind up as to how much free is free...

hAP.png

flameproof · Thu Oct 22, 2020 1:31 am

Last one and I'll shut up until we have other views. I have tried to upgrade by downloading separate package files to a test CPE. This was fine until they would not all fit either, and had to do one set, reboot, then another set, reboot... with the added risk that a CPE could lose power at any time, breaking the process and leaving it in a corrupted state, as only some packages (if any got downloaded) would install.

pe1chl · Thu Oct 22, 2020 11:04 am

We get told the CPEs have 7.3MB left and 7.4MB are needed. We need a way to package single .npk files into an upgrade package with just what we need. Would we ever consider MPLS on a hAP Lite? Not in a million years... so why not give us the ability to remove it from the upgrade package?

In 6.47 that capability is actually still there!
First make sure that in your currently running combined package you have set all packages you do not need (like mpls) to disabled, and reboot.
Now on a PC download the all_packages.zip file and extract it. Take ONLY packages you STILL HAVE ENABLED now, and upload them to the device.
Then issue a reboot command on the device.
It will now upgrade the device and at the same time convert it to "separate packages" setup. So the next time you upgrade from the device itself, it will only download those packages and this will give some headroom (enough for now).
Of course when you have many devices you can script the above procedure.

flameproof · Thu Oct 22, 2020 2:51 pm

Thanks for your input pe1chl - however the main issue with it is, what happens if some packages don't make it fully, or at all, and the CPE loses power? We don't have control over, or warning from, Kenya's power grid, which is extremely temperamental, so power outages mid-upgrade are definitely an issue. The standard full-package upgrade is quite safe in this respect, as a broken download will not require a customer site visit to replace/re-provision.

In any case, Mikrotik have listened!

What's new in 6.47.6 (2020-Oct-21 10:41):

*) cap - fixed L2MTU path discovery;
*) crs3xx - fixed hardware offloaded LACP bonding on Ethernet interfaces for CRS354 devices;
.........
*) radius - added "Service-Type" attribute to Access-Request for IPv4 and IPv6 DHCP servers;
*) smips - reduced RouterOS main package size;
*) switch - fixed Ethernet padding for small packets;
.........

They have dropped from 7810370 bytes of 6.47.4 to 7351618 bytes in 6.47.6 - nice, thank you!

flameproof · Fri Oct 23, 2020 1:29 pm

I can confirm this has worked with ~500 of our deployed hAP Lites, which were previously failing to upgrade. Good news!

Who is online