echo: system,error,critical could not save configuration changes, not enough storage space available.

pts0 · Wed Jan 29, 2025 8:52 pm

Hi, after installin 7.17 my disk got space 0.

How to clean it up ? What i have done wrong ?

I have on this switch CAPsMAN (old and new), is this the issue, should i move somewhere else ?

admin@switch012] > /system/resource print 
                   uptime: 39m18s             
                  version: 7.17 (stable)      
               build-time: 2025-01-16 08:19:28
         factory-software: 6.44.6             
              free-memory: 439.8MiB           
             total-memory: 512.0MiB           
                      cpu: ARM                
                cpu-count: 2                  
            cpu-frequency: 800MHz             
                 cpu-load: 5%                 
           free-hdd-space: 0                  
          total-hdd-space: 16.0MiB            
  write-sect-since-reboot: 127                
         write-sect-total: 116823244          
        architecture-name: arm                
               board-name: CRS326-24G-2S+     
                 platform: MikroTik

Thanks

mkx · Wed Jan 29, 2025 9:11 pm

Your switch is one of devices with too little flash space ... and since you need optional package wireless to run old CAPsMAN, you'll have to consider moving legacy CAPsMAN elsewhere (and uninstall wireless package from switch). You can set up legacy CAPsMAN on one of devices which are currrenty CAPs. There's no benefit in running both CAPsMANs on same device from interworking point of view (old and new don't cooperate and don't share config).

pts0 · Wed Jan 29, 2025 10:09 pm

ok, is strange, because same config on other CRS326-24G-2S+ with same config still have space left.
Is disk a special part of memory, or could be possible to shrint memory and expand disk ?

CAPsMAN is absolutly indifferent where is installed ... traffic is not routing there.
Let's move it to RB5009UPr+S+ or L009UiGS-2HaxD ... or maybe directly on a cAP ax.
There is a way to export the config of CAPsMAN ? ok

/caps-man/ export

work for legacy ... how is for wifi ?

sindy · Wed Jan 29, 2025 10:54 pm

Export the configuration to a file (indeed use /export show-sensitive, not /system backup save)), and separately export the certificates if you use them, including private keys. Then netinstall the device and recreate the configuration from the exports.

I have a support ticket open on "something" gradually occupying the disk space already in 7.16.2; the space is periodically released in bulk (128 Mbytes at a time), but a reboot can break this, so the disk space gets completely lost if you reboot the device multiple times at a "wrong" time. Watching the [/system resource get free-hdd-space] value is the way to monitor the changes - I use a scheduled script to monitor the occupied space and log the changes or send Telegram messages.

So far I was unable to make a device without any sensitive information exhibit this behavior so I am unable to provide the much needed supout.rif file to the support. So I'd be grateful for anyone whose device behaves this way and either does not serve as a VPN server or client at all, or it is easy to change the passwords after generating the supout.rif, to provide their supout.rif taken in the middle of the "breathing cycle" to Mikrotik support. The ticket number to refer to is SUP-176219.

pts0 · Thu Jan 30, 2025 1:26 am

Ok, i will do it ... at the moment the switch is working but fully broken, no reboot possible, unistall of package is not possible anymore.
I m too lazy (and not on site) to netinstall it... will solve the problem but

-- so now more the capsman and then provide ticket. I m lucky this switch ist not heavely used ... but the CRS354-48G-4S+2Q+ suffer of similar issue

sindy · Thu Jan 30, 2025 5:59 am

If there is no sensitive information right now and /system/sup-output succeeds (says created: 100% after a while), you can send that one - it is created on the RAM disk, not on the flash that is full, unless you explicitly specify the name and make it start with flash/.

As for exporting the wifi CAPsMAN configuration, use /interface/wifi/export, as the configuration trees for the local wifi interfaces and for capsman-controlled ones have been integrated and the channel, security, datapath, configuration, and provisioning branches are at the same level like the capsman one.

pts0 · Thu Jan 30, 2025 5:58 pm

Submitted SUP-177913

sindy · Fri Jan 31, 2025 9:02 pm

@mkx, since you are already watching this topic anyway, I react to your statement from the 7.17.(1) topic here.

My use case for my hAP ac2 doesn't require any wireless driver and it's not available for experimenting.

viewtopic.php?p=1122624#p1122624 explains why I have quoted that "ran out of flash space" part from your post in the original topic. As in "the wifi-qcom-ac driver may have nothing to do with that".

mkx · Fri Jan 31, 2025 9:37 pm

As in "the wifi-qcom-ac driver may have nothing to do with that".

In my particular case the reason was obvious: with advent of 7.13 I felt adventurous and went ahead with replacing wireless with wifi-qcom-ac. After installation of base ROS and wifi-qcom-ac package only some 300kB of flash remained free. After I added normal router's config (a handful of firewall rules), it ran just fine and wireless worked like a charm. But then I filled IPv4 and IPv6 address lists with blocks of addresses for two countries. And it was still working fine. But when I wanted to update addresses in the lists, device ran out of flash space. Then I did netinstall (because not even removing entries from interface lists was possible), I re-did the setup (not redtoting from backup for obvious reasons) ... and ran into flash problems again. Since physical location of the device (room in underground, placed inside telco rack with it's metallic mesh doors closed most of time) didn't really warrant running wireless interface, I decided to uninstall wifi-qcom-ac driver altogether and that gave me healthy 2.8MB of free space before filling up the address lists. Device now happily serves as router and CAPsMAN (for two very decent APs: one Audience and one wAP ax).

sindy · Fri Jan 31, 2025 9:47 pm

that gave me healthy 2.8MB of free space before filling up the address lists

ok, and that amount of free space does not "autonomously" change, i.e. remains the same unless you cnahge something in the configuration? If so, let me call you "mkx the lucky"

I'm pulling my hair over what is the reason why some devices suffer from the issue and some don't.

optio · Fri Jan 31, 2025 10:06 pm

It grows, but not much in my case running ROS 7.16.2. I have several schedulers that changes configuration, one for eg. performs changes by adapting 25 Queue tree rules several times at day. I didn't monitor how much space is reduced by day, but when creating backups occasionally they are getting a bit larger, less than 1KB per backup so I'm not worried much, still having currently only 420.0KiB I'm not to happy with this either.
Probably this disk space growths depends which configuration is changed and how often.

sindy · Fri Jan 31, 2025 10:23 pm

Probably this disk space growths depends which configuration is changed and how often.

Nope. On mine, the free disk space shrinks by 4096 bytes every 5 to 10 minutes currently, and every 128 kBytes it gets freed in bulk and the cycle repeats, and neither me nor any script touch the configuration during that time. I.e. the ghost is fully autonomous. Two devices under my responsibility behave like this, three other ones don't. All five of them hAP ac² running 7.16.2.

optio · Fri Jan 31, 2025 10:35 pm

Does ROS maybe uses swap file for VM on flash and it's changing depending on RAM usage?

My device is with similar specs (Chateau LTE12) but it has more RAM - 256MB. Also I'm seeing dynamic some small disk usage changes, but much smaller.

disk_usage.png

It is visible that is not straight line, could be just some small temp files used by system processes.

PS.
Actually it's straight, maybe I seeing optical illusions because of dots

disk_usage_line.png

I rebooted router yesterday so I don't have much graph history, but I'm sure from the past when I was observing this graph that disk size varies, but not much in my case.

mkx · Fri Jan 31, 2025 10:49 pm

that gave me healthy 2.8MB of free space before filling up the address lists
ok, and that amount of free space does not "autonomously" change, i.e. remains the same unless you cnahge something in the configuration?

The free space remained constant for some 4 months while running 7.16 (without any reboots or some such) ... after upgrade to 7.17 it did shrink a bit. 7.16, however, did use a bit more RAM (not a problem on my 256MB unit, but could be a problem on 128MB units) and quite more CPU ... both went back to pre 7.16 levels with 7.17. I've yet to see if resource consumption will be stable with 7.17 though.

sindy · Fri Jan 31, 2025 11:02 pm

Does ROS maybe uses swap file for VM on flash and it's changing depending on RAM usage?

By chance, one of the affected hAP ac² has 256 MB RAM too, so that is not a remedy. And no containers run on any of them, so it's not them swapping to the flash - plus, if the RouterOS itself needed to swap the RAM on disk, why would the free space get reclaimed every now and then. Also, the 5 to 10 minutes pace is a property of the current reincarnation of the ghost, it used to be slower before the last reboot.

optio · Fri Jan 31, 2025 11:06 pm

Yes that's true about swap and disk space reclaim, as I mentioned, disk space varies in small amounts, up and down, could be temp files or swap if used.
It seems you have some different issue.

StarBrilliant · Tue Feb 04, 2025 3:51 am

After updating to v7.17.1, my hAP ac^2 started to have the same issue.

echo: system,error,critical could not save configuration changes, not enough storage space available.

My package list:

routeros - 7.17.1 - 11.4 MiB
wifi-qcom-ac - 7.17.1 - 2680.1 KiB

My file list:

flash/pub - empty directory
flash/skins - empty directory

/system/resource/print shows:

free-hdd-space: 0

/system/reboot shows:

action timed out - try again, if error continues contact MikroTik support and send a supout file (13)

I submitted a ticket: SUP-178039
MikroTik support told me to do a netinstall with providing any explanation.
I haven’t wiped this router yet, because it’s my canary device. I need to make sure the update is safe for my other routers.

Tue Feb 04, 2025 8:39 am

And yet ... it might be the only way out for your 16Mb flash space device.

What is happening behind the scenes: some left-over bits and pieces from previous versions are remaining, eating away precious storage space.
And that's what blocks you from performing upgrade. Some devices may pass now but may run in trouble later on. Nobody knows how to get a view on that internal memory usage.
Netinstall is the only way for now to completely clean the internals.

export config using show-sensitive
Move that export away from device and make sure the export is complete
netinstall device, empty config (don't forget that wifi-qcom-ac package, it can be installed in one sweep together with ROS using netinstall)
import config again block by block

PauliusCere · Sat Feb 08, 2025 5:40 pm

Hello,

Just wondering, does anyone solved this problem without Netinstall? Would be simple, but error occurred on remotely managed site, many kilometers away. Even /system reset-configuration doen't work.

Thanks,
Paulius

mkx · Sat Feb 08, 2025 6:25 pm

For me, last time this happened on my hAP ac2 (yesterday

), it wouldn't even reboot properly (by executing reboot command) or shutdown.

You can believe me that I'm following these kinds of discussions for quite some time. This problem is plaguing my device ever since I installed v7, so it's been a few years now. And I've never ran into solution other than netinstall.

On the bright side: device will run more or less normally for quite some time unless it crashes due to unrelated reasons (e.g. memory leak).

And I believe that it woukd boot just fine as well, my guess is that this kind of flash starvation happens when ROS tries to save update to configuration database and it's creating a (full) copy of file before deleting the old one. Which means that probably the old config database is in good shape and shoukd be fine for boot time.
But this is more of an "educated gut feeling", I may be wrong about it and device will explode into neighbour's face.

sindy · Sat Feb 08, 2025 6:46 pm

My gut says the same, I have even experienced it practically while still in ROS 6 - I haven't noticed those complaints but after a reboot, everything was running fine except that I have lost a few ppp secrets I have added over last several weeks. Since I am using the command line almost exclusively, I figure the complaints were simply not output to the console in 6.49.13.

I'm waiting for Mikrotik to respond to my ticket again after I have updated it with a bunch of supout.rif taken after each bite on a machine where the ghost started "eating" and returning the disk space, hoping that they will identify the process that is responsible for that and eventually not only fix the root cause but also add a cleanup .npk like they did back in the days (but that one was removing one particular file so even if it still worked, it would not resolve the current issue), so that devices could be fixed without physically touching them.

mark99i · Thu Feb 27, 2025 3:54 am

I think I've figured out why.

since 7.16-7.17, there seems to be some kind of problem that causes the size of the configuration to grow when changing.
my script periodically updates (but does not add) 25 entries in the /ip/dns/static section. each update spends 4K of free memory
I'm restoring my configuration a month before the problems (netinstall + backup restore), I have about 300KB free with the installed qc-ac, but then they leak
Technical support is not responding to my requests.

I will have to roll back the version to 7.15.3

viewtopic.php?p=1112613#p1112613
viewtopic.php?p=1112617#p1112617

export config using show-sensitive
Move that export away from device and make sure the export is complete
netinstall device, empty config (don't forget that wifi-qcom-ac package, it can be installed in one sweep together with ROS using netinstall)
import config again block by block

It doesn't make any sense. until the problem is fixed, this will return the device to a working state here and now, but after a while it will happen again.

it makes no sense to argue that 16M is not enough: the configuration takes ~ 150 KB, and 300-600KB is free on the device with wifi-qcom-ac installed. What does little mean? It can accommodate 3 more such configurations, which is not so small.
Or do you mean too little for mistakes to go unnoticed? - Yes, but it will take time and devices with a lot of memory will suffer, it's just a matter of time.

ros44 · Fri Feb 28, 2025 9:41 am

I am freaking out. I just found this thread because I have a Chateau 5G (the discontinued model) with 16 MB of storage. The device is a two-hour plane ride away from me and won’t reboot, showing the same error:

system,error,critical could not save configuration changes, not enough storage space available.

mikolajpl · Mon Mar 03, 2025 12:11 pm

I have created an incident at Mikrotik SUP-180945 regarding storage place.

Below the answer:

From: Serhii T. (Jira) [mailto:support@mikrotik.com]
Sent: Monday, March 3, 2025 2:17 PM
To: <removed>
Subject: MikroTik support #[SUP-180945]: Unable to upgrade to RBD52G-5HacD2HnD

—-—-—-—
Please REPLY ABOVE THIS LINE ^ (for faster response, use our support portal).
Hello,
If the device has 0KB of free space during changes, the settings will not be saved, which is the expected behavior.
In this case, you can only reinstall the system on the device using Netinstall.
As the software evolves and consumes more disk space, it continues to fit within the internal storage capacity of the router.
We continuously improve and optimize RouterOS and its features to support our devices.
Consider reviewing your configuration to reduce space usage or upgrading to a more powerful device.
Best regards,
Serhii T.

Your eyes don't fool You !

Advice from Mikrotik is to replace device or go for netinstall. Ridiculous !!! I have some of the devices in remote locations of my customers. Shall I spend time and money to travel from one to another ?
Is this Mikrotik quality ?

Splash · Mon Mar 10, 2025 7:25 pm

I have the same problem with a WAPAC where the disk space slowly disappears.. I removed a backup, no change to the reboot/configuration save issue. I then ran a supout and it wrote 350KB to the disk.... So this is a OS calibration issue with usage, or there is a partition within the OS itself which has run out of disk space.

32 entries in the log
no files store on disk

routeros V7.18.1
wifi-qcom-ac

I am unable to reboot remotely either.

infabo · Mon Mar 10, 2025 7:39 pm

Maybe of help: viewtopic.php?p=1132293#p1132144

Victoare · Tue Mar 11, 2025 2:36 pm

I will have to roll back the version to 7.15.3

I have a similar script (turning dhcp leases to dns static entries).
I postponed the update quite a while and updated from 7.12.1 to 7.18.1 and now facing this 0 hdd error.
Does your rollback solved this issue? Did you tested on any newer version?

Still frustrating that nothing can tell what occupies the hdd space, so we could figure it out what can causes the loss.

Chupaka · Wed Mar 12, 2025 9:34 pm

Netinstalling 7.16.2 (using netinstall from 7.18) to hAP ac^2 gives ~660 KiB of free space. Netinstalling 7.18 leaves only ~100 KiB which drops down to 0 after some time

Chupaka · Wed Mar 12, 2025 11:01 pm

Oh... My... Gosh...

$ cat routeros-7.18.2-arm/squashfs-root/nova/lib/console/logo.txt

  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik routerOS V2.4 (c) 1999-2001       http://mikrotik.com/

The 7.18.2 still contains a 24 years old file!..

mark99i · Thu Mar 20, 2025 3:55 am

I will have to roll back the version to 7.15.3
I have a similar script (turning dhcp leases to dns static entries).
I postponed the update quite a while and updated from 7.12.1 to 7.18.1 and now facing this 0 hdd error.
Does your rollback solved this issue? Did you tested on any newer version?

Still frustrating that nothing can tell what occupies the hdd space, so we could figure it out what can causes the loss.

yes, rolling back to 7.15.3 solves the problem. if you don't need the features of later versions, you can stay on 7.15.3 with the wave2 package installed. in my experience with several dozen different mikrotik devices, this is a fairly stable release.

karlgermany · Sun Mar 30, 2025 7:32 pm

I run into the same issue which the system can not save the config and also refuse to reboot as the flag can not be set.

But I realized there must be an internal issue as I can still create backups even free disk shows 0

[admin@KK_Home_Router_4] > system/resource/print
uptime: 2h32m22s
version: 7.18.2 (stable)
build-time: 2025-03-11 11:59:04
factory-software: 6.41
free-memory: 429.6MiB
total-memory: 512.0MiB
cpu: ARM
cpu-count: 2
cpu-frequency: 800MHz
cpu-load: 1%
free-hdd-space: 0
total-hdd-space: 16.0MiB
write-sect-since-reboot: 3519
write-sect-total: 5432795
architecture-name: arm
board-name: CRS326-24G-2S+
platform: MikroTik
[admin@KK_Home_Router_4] > /system backup save name=pre-netinstall
Saving system configuration
Configuration backup saved
[admin@KK_Home_Router_4] > system/resource/print
uptime: 2h33m40s
version: 7.18.2 (stable)
build-time: 2025-03-11 11:59:04
factory-software: 6.41
free-memory: 429.5MiB
total-memory: 512.0MiB
cpu: ARM
cpu-count: 2
cpu-frequency: 800MHz
cpu-load: 0%
free-hdd-space: 0
total-hdd-space: 16.0MiB
write-sect-since-reboot: 3519
write-sect-total: 5432795
architecture-name: arm
board-name: CRS326-24G-2S+
platform: MikroTik
[admin@KK_Home_Router_4] > /file/print
# NAME TYPE SIZE LAST-MODIFIED
0 pre-netinstall.backup backup 286.8KiB 2025-03-30 18:18:31
1 router4.backup backup 286.8KiB 2025-03-30 18:16:45
2 router4.rsc script 51.9KiB 2025-03-30 18:16:45
3 flash disk 2025-03-30 15:24:17
4 flash/pub directory 2019-04-22 17:48:40
5 flash/skins

so file 0,1,2 could be create with 0 free-hdd-space:.

I had an issue with a packet sniffer pcap file which got to big and I guess it kinda destroyed the filesystem.

For me that's a software bug and not just an issue of free disk space

mkx · Sun Mar 30, 2025 9:05 pm

[admin@KK_Home_Router_4] > /file/print
# NAME TYPE SIZE LAST-MODIFIED
0 pre-netinstall.backup backup 286.8KiB 2025-03-30 18:18:31
1 router4.backup backup 286.8KiB 2025-03-30 18:16:45
2 router4.rsc script 51.9KiB 2025-03-30 18:16:45
3 flash disk 2025-03-30 15:24:17
4 flash/pub directory 2019-04-22 17:48:40
5 flash/skins

Your CRS is one of those "16MB flash, 64MB+ RAM" which have root of file storage on RAM disk ... and remainder of flash is mounted under /flash/ . Backups (and exports) are by default created on root of file storage, on your device that means on RAM disk. Your device has no shortage of RAM, so writing to RAM disk is perfectly possible.

So your post doesn't prove anything.

Victoare · Mon Mar 31, 2025 2:29 am

yes, rolling back to 7.15.3 solves the problem. if you don't need the features of later versions, you can stay on 7.15.3 with the wave2 package installed. in my experience with several dozen different mikrotik devices, this is a fairly stable release.

I tested this rollback and it seems much better, but it still has some "HDD" eating issues. I made a couple changes in my scripts and they not touching the config file as much as they did before with the static DNS entries (so a bit ruined the test for the rollback, but this is my main router at home and need to be working).

Started monitoring the free space using https://github.com/akpw/mktxp-stack and it shows slow but steady decline:

On the left side there is a 24 hour graph showing the garbage collections.
On he right side there s a daily min/max graph showing that the max is getting lower (around 4.5kb) each day.

It would be so nice to be able to check what occupies this space, but if I see it right, there is no such tool.

karlgermany · Mon Mar 31, 2025 12:11 pm

[admin@KK_Home_Router_4] > /file/print
# NAME TYPE SIZE LAST-MODIFIED
0 pre-netinstall.backup backup 286.8KiB 2025-03-30 18:18:31
1 router4.backup backup 286.8KiB 2025-03-30 18:16:45
2 router4.rsc script 51.9KiB 2025-03-30 18:16:45
3 flash disk 2025-03-30 15:24:17
4 flash/pub directory 2019-04-22 17:48:40
5 flash/skins

Your CRS is one of those "16MB flash, 64MB+ RAM" which have root of file storage on RAM disk ... and remainder of flash is mounted under /flash/ . Backups (and exports) are by default created on root of file storage, on your device that means on RAM disk. Your device has no shortage of RAM, so writing to RAM disk is perfectly possible.

So your post doesn't prove anything.

If it would be saved on RAM the files would be gone when the system reboots. But they survived "hard" power off (removing power cable as I can not shutdown/reboot the system anymore)

mkx · Mon Mar 31, 2025 6:56 pm

If it would be saved on RAM the files would be gone when the system reboots. But they survived "hard" power off (removing power cable as I can not shutdown/reboot the system anymore)

I have hard time believing the above sentence.

R1CH · Mon Mar 31, 2025 8:46 pm

Bricked one of my hAP AC2s due to this bug when trying to import a DoH root cert, apparently that pushed it over the edge and they cannot be deleted. And the shitty configuration import after netinstalling constantly errors and has to be handheld all the way due to conflicts with default configurations, wifi1 and wifi2 swapped radios for some reason, my wireguard keys were somehow lost, a terrible experience.

karlgermany · Wed Apr 02, 2025 11:27 am

If it would be saved on RAM the files would be gone when the system reboots. But they survived "hard" power off (removing power cable as I can not shutdown/reboot the system anymore)

I have hard time believing the above sentence.

Would be nice if you would explain this. I am sure there are no secret batteries in the device to keep the data in the RAM when power is gone.
Power Off -> all RAM is gone.

We have a SW issue for sure here.

mkx · Wed Apr 02, 2025 1:17 pm

We have a SW issue for sure here.

As I explained, and output of your files confirms it, that the remaining (which is 0) of flash is mounted under flash/ ... and if you create any file to root (i.e. not under flash/), that file is in RAM disk. And RAM disk does get wiped at reboot (or power off).

And the fact is that 16MB flash is very tight for recent ROS versions. It's possible that those 650kB worth of files can fit on flash (but not on root) after you install such device ... but it is well possible that flash gets depleted right because of creating backups and exports.

So I have hard time believing you that those files survived power down of device. And based on your forum metadata (date of joining, number of posts) I have my doubts about your ROS experiences (but I may be wrong here).

karlgermany · Wed Apr 02, 2025 5:41 pm

We have a SW issue for sure here.

As I explained, and output of your files confirms it, that the remaining (which is 0) of flash is mounted under flash/ ... and if you create any file to root (i.e. not under flash/), that file is in RAM disk. And RAM disk does get wiped at reboot (or power off).

And the fact is that 16MB flash is very tight for recent ROS versions. It's possible that those 650kB worth of files can fit on flash (but not on root) after you install such device ... but it is well possible that flash gets depleted right because of creating backups and exports.

So I have hard time believing you that those files survived power down of device. And based on your forum metadata (date of joining, number of posts) I have my doubts about your ROS experiences (but I may be wrong here).

The backup files were still present after the device was power cycled, so they must be stored on flash.

However, I received an explanation of what went wrong with the flash drive, and it seems that only Netinstall can properly clean it. The issue started after I ran a packet capture (pcap) without setting a size limit, which caused the flash storage to fill up. I would expect a professional system to clean up the filesystem on boot to prevent issues like the one we’re experiencing. The downtime required for Netinstall is significantly higher than just performing a reboot.

Regarding my background: I'm rarely online here, and until now, aside from some configuration issues, my MikroTik routers have worked fine in my network. The one I'm having problems with was purchased seven years ago and has done its job reliably. At work, we use enterprise-grade equipment like Cisco, Juniper, and others. I’ve been qualified to work with such systems for over 20 years.

A problem like the one I encountered here wouldn't be an issue on the systems we use professionally.

Sorry to say this, but your responses were not helpful. Personally, I would have preferred no reply over the kind of answer I received.

mkx · Wed Apr 02, 2025 6:24 pm

Sorry to say this, but your responses were not helpful. Personally, I would have preferred no reply over the kind of answer I received.

My redponse to @OP was to netinstall device. You later chimed in with claim that there's bug in flash handling in ROS (etc.) about which I expressed my doubts. Otherwise you (at least implicitly) referenced to @OP's post (by posting in this thread).
Now you're telling that MT support told you to netinstall device. So how is my suggestion to @OP different ftom what MT support told you?

And I still have my doubts about your explanation. Your extensive experience with equipment by other vendors don't count much when dealing with ROS-specific problems.

And, for the record, I completely agree that ROS should be much more robust when it comes to handling out-of-storage situations, the way things are currently is completely unacceptable.

infabo · Wed Apr 02, 2025 6:40 pm

mkx answer was perfectly accurate and fine for what he explained. You can't make him responsible for not having superpowers inspecting your CRS from just little to no info. How could one expect a packet sniffer pcap file eating up internal flash which is not visible when executing /disk/print? This is a bug for sure and should not happen. I hope you have reported this to Mikrotik support so we can all benefit from a bugfix.

Ca6ko · Thu Apr 03, 2025 9:11 am

If a remote device cannot be rebooted due to lack of memory, it is sometimes possible to do a reboot if you enable it on a schedule

Chupaka · Sat Apr 05, 2025 10:07 am

What's new in 7.19beta8 (2025-Apr-04 13:24):

*) system - fixed "/system reboot" when the system disk is completely full;

Well, at least now we can initiate NetInstall procedure remotely

infabo · Sat Apr 05, 2025 10:21 am

But only if you enabled routerboard device-mode feature flag before to change boot method.

At least downgrade should be possible now.

mkx · Sat Apr 05, 2025 10:56 am

However only if one first installs latest beta. So it doesn't help devices, running older ROS versions, already in the "doomed" state.

infabo · Sat Apr 05, 2025 3:26 pm

But once 7.19 stable released and someone encounters the 0 space issue after upgrading from e.g. 7.16 (this version consumed remarkable less flash), you now can downgrade instead of netinstall your device.

karlgermany · Sun Apr 06, 2025 1:02 pm

Just for your information:

I was able to get the box out of the network.

Before using Netinstall I did a reset of the config and voila:

[admin@MikroTik] /system/resource> print
uptime: 1m52s
version: 7.18.2 (stable)
build-time: 2025-03-11 11:59:04
factory-software: 6.41
free-memory: 455.6MiB
total-memory: 512.0MiB
cpu: ARM
cpu-count: 2
cpu-frequency: 800MHz
cpu-load: 15%
free-hdd-space: 584.0KiB
total-hdd-space: 16.0MiB
write-sect-since-reboot: 2680
write-sect-total: 2680
architecture-name: arm
board-name: CRS326-24G-2S+
platform: MikroTik

Then I restored the .backup file and still there is memory available

[admin@KK_Home_Router_4] /system/resource> print
uptime: 1m41s
version: 7.18.2 (stable)
build-time: 2025-03-11 11:59:04
factory-software: 6.41
free-memory: 447.4MiB
total-memory: 512.0MiB
cpu: ARM
cpu-count: 2
cpu-frequency: 800MHz
cpu-load: 1%
free-hdd-space: 160.0KiB
total-hdd-space: 16.0MiB
write-sect-since-reboot: 575
write-sect-total: 2055
architecture-name: arm
board-name: CRS326-24G-2S+
platform: MikroTik
[admin@KK_Home_Router_4] /system/resource>

Conclusion: Netinstall is not needed and for me it's a BUG in the software. Seems when it runs once against 0 byte free it never shows released space. This explains why I was till able to write configs to the flash ( NOT memory ). It seems, just a thought of me, that when the free space counter is not changed when space is available all internal functions just check the free size counter and then complains with system,error,critical could not save configuration changes, not enough storage space available.

xt22 · Sun Apr 27, 2025 12:46 am

I confirm reset works on my chateaus 5G and 5G R16, but I still have to be at the device and press it, since the device gets to a bricked (although working

) state - I can do absolutely nothing with it remotely, cannot reset it, reboot, shutdown, change anything, remove scripts, certificates, uninstall packages.. nothing.

I'm really "happy" for buying like a dozen $485 Mikrotik devices, when the crap has less flash memory than any aliexpress $2 device, stupid sw bugs so I have to travel every month to the devices to netinstall it and Mikrotik denying the problem and advising netinstall as solution :-/

I'll try the 7.16.2 and see how it works, I downgraded some of my routers to it too, since Mikrotik screwed ovpn

xt22 · Sun Apr 27, 2025 10:20 pm

damn, so it is way worse - it bricks my RBcAPs too, I have hundreds of these :-/ Fortunately most of them are max. 7.16.2 or 7.17, but those are dumb APs, no routing, no firewall filtering etc, just bridged wifi + eth, and still got bricked with 7.18.2 - this is bad.. I have some of these running for more than 1000 days without a restart, running great before 7.18.2

s650mb · Sun May 25, 2025 11:35 am

We found a simple solution for those who have several packages installed.
After installing 7.19, a memory shortage error appeared on two hap ac 2.
Removing the wifi-qcom-ac package.
We are rebooting.
There is a lot of free space

Install the wifi-qcom-ac package.
We do several reboots.
There is some free space, 90kib on one, 60kib on the other.

s650mb · Wed May 28, 2025 6:01 pm

I was wrong in the previous message.
The solution helps for a couple of hours and then there is a lack of space again.
On hapac 2, I had to remove the wifi-qcom-ac package, and with it, the free space is running out.
The wireless package was installed instead.

optio · Wed May 28, 2025 8:41 pm

Never go under 100k on flash, maybe even more is needed for temp files (depends what is used on ROS) and how often config is changed (config db can get fragmented).

xt22 · Sun Jun 08, 2025 9:10 pm

I was wrong in the previous message.
The solution helps for a couple of hours and then there is a lack of space again.
On hapac 2, I had to remove the wifi-qcom-ac package, and with it, the free space is running out.
The wireless package was installed instead.

You can downgrade to 7.16.2 with wifi-qcom-ac and you'll have around 600kB and working device
Or to 7.17-7.17.2, you'll have something over 200 kB, but it will probably still not eat all the free space and work (I now have some cAPs running for 60 days and still around 200 kB of space with 7.17.1 or 7.17.2).

I have tested this on cAPs and Chateaus, hAP should be the same, not sure about the switches - I don't use capsman or wireless on those

lurker888 · Sun Jun 08, 2025 9:36 pm

Just another data point: on a hap ac2 with wifi-qcom-ac I have more than 300k free after netinstall. After about 3 months, with numerous config changes I have approx. 290k free. (And I even do logging to disk, but limited to 2x100 lines)

So the situation is very far from ideal, but to me at least seems to be manageable with careful configuration and no in-place updates...

EDIT: 7.18.2

xt22 · Sun Jun 08, 2025 11:54 pm

it is important to write your ROS version in this issue.

I have checked one of my cAPs running probably something like 6.49.16 and in November updated to 7.16.2 - it has 560Kb of free space and seems pretty solid. With graphs and some logs saved to disk.

There are two problems in this issue:
1, We know Mikrotik made some changes to qifi-qcom-ac in 7.17, and 7.19. The update in 7.17 takes something around 300KB - so who had 500-600KB now has 200KB of space - which is not good when the device has less memory than any chinese aliexpress gadget for $0.50
2, something eats the free space. Probably not related to 1, since the same would happen, it would just take 3x the time - but 1 definitely makes it worse. It seems to me it started in 7.18.

You can't see what is taking the space, everything is empty. Mikrotik support doesn't help, they tell you to review your config or buy more powerful AX device - ignoring the fact, that with the same config, 7.16.2 devices run fine.

It seems 7.19.1 is slightly better in this, at least on my testing chateau, but it is probably too soon to make some conclusions, we'll see.

btw - mikrotiks used to run for decades with 300KB

lurker888 · Mon Jun 09, 2025 12:29 am

Sorry, meant to include version. It's 7.18.2.

7.19.1 adds very little (4k or so) to the wifi package, but the base + wifi-ac together has grown by almost 100k.

CGGXANNX · Mon Jun 09, 2025 1:02 am

Most of the grow between 7.16.x and 7.19.x that you observed are from the main RouterOS package. wifi-qcom-ac itself only increased 8KB in size (compressed size of the SquashFS archive that should be stored as it on the internal storage)

ros-arm-grow.png

New functionalities of course bring more compiled code but also make support data like the .mem files for console and WinBox larger (I think they contain the parameter information, including names and help texts).

The biggest grow of the main package is between 7.16.x and 7.17.x. Also between 7.16.x and 7.17 WebFig has been completely redesigned with new HTML/JS/CSS/SVG as well as embedded Web Fonts (38KB of that

) that increased the compressed size by 75KB for WebFig alone.

Even the built-in CA certificates that came with 7.19 require about additional 20KB (compressed size) that 7.18.x didn't. That hasn't taken into account the code/support data for the implementation of the feature.

xt22 · Mon Jun 09, 2025 2:43 am

I'm not blaming the wifi-qcom-ac package - it makes things a lot worse, since it reduces the free space to one third in 7.17, but the unanswered question is - what is eating the space? Mikrotiks support is worthless, I don't know what is so big problem on simulating the behavior, looking to the file system and saying - bug in vpn logs / memory leak in certificate management etc.

The problem with wifi-qcom-ac package and it's size is that it makes it almost unusable on 16MB devices

infabo · Mon Jun 09, 2025 8:54 am

routeros package size is increasing. At a point even routeros+wireless is going to run into trouble.

Mon Jun 09, 2025 10:43 am

Checked yesterday
Map running 7.18.1 incl wireless still has over 3.3Mb free.

mkx · Mon Jun 09, 2025 12:02 pm

mAP is MIPSBE ... and routeros packages for architectures anything but ARM/ARM64 are quite a bit smaller. Probably also due to smaller number of supported device models (which may require different drivers). Or perhaps due to smaller executables for those platforms ... I'd expect packages for 64-bit platforms slightly larger than packages for 32-bit platforms ... but the difference might be too small to notice due to compression.

xt22 · Mon Jun 09, 2025 1:11 pm

routeros package size is increasing. At a point even routeros+wireless is going to run into trouble.

there is no doubt my toilet paper holder has more storage space than Mikrotik's $485 router and it will cause problems in the future, not sure why, when 1Gb nand chip cost like $0.70 even for end consumer.. but Mikrotik will have to deal with that, probably by making smaller images, tailored for a smaller groups of devices - i.e. separating the qca9984 devices etc.

:mkx plus you can't use any wifi packages except the wireless one on mipsbe, another +800Kb

Santi70 · Mon Jun 09, 2025 1:19 pm

now I found this thread, I am in the same case, going from version 7.19 to version 7.19.1 I was at zero free space. I did a netinstall and it recovered 128 kiB, I thought case solved, but after a few days the errors came back and looking at zero kiB again and I have nothing in flash memory.

use@hap_ac2] > file/print
 # NAME                                 TYPE             SIZE LAST-MODIFIED       
 0 flash                                disk                  1970-01-01 02:00:11 
 1 flash/skins                          directory             1970-01-01 02:00:11

[user@hap_ac2] > system/resource/print 
                   uptime: 1w22h57m11s        
                  version: 7.19.1 (stable)    
               build-time: 2025-05-23 14:27:17
         factory-software: 6.45.9             
              free-memory: 36.3MiB            
             total-memory: 128.0MiB           
                      cpu: ARM                
                cpu-count: 4                  
            cpu-frequency: 448MHz             
                 cpu-load: 2%                 
           free-hdd-space: 0                  
          total-hdd-space: 16.0MiB            
  write-sect-since-reboot: 15423              
         write-sect-total: 46886650           
        architecture-name: arm                
               board-name: hAP ac^2           
                 platform: MikroTik

xt22 · Mon Jun 09, 2025 1:43 pm

netinstall to 7.16.2, with qcom-ac you should have like 500-600 Kb of space.

run console clear-history after you'll finish setting it. Disable store-on-disk if you have graphing set and set small disk logs.
And check it from time to time

optio · Mon Jun 09, 2025 7:13 pm

With containers:

> /system/package/print  
Columns: NAME, VERSION, BUILD-TIME, SIZE
# NAME          VERSION  BUILD-TIME           SIZE     
0 wifi-qcom-ac  7.16.2   2024-11-26 12:09:40  2676.1KiB
1 routeros      7.16.2   2024-11-26 12:09:40  11.1MiB  
2 container     7.16.2   2024-11-26 12:09:40  96.1KiB 

> /system/resource/print 
                   uptime: 22h49m1s
                  version: 7.16.2 (stable)
               build-time: 2024-11-26 12:09:40
         factory-software: 6.99
              free-memory: 63.1MiB
             total-memory: 256.0MiB
                      cpu: ARM
                cpu-count: 4
            cpu-frequency: 672MHz
                 cpu-load: 3%
           free-hdd-space: 272.0KiB
          total-hdd-space: 16.0MiB
  write-sect-since-reboot: 1392
         write-sect-total: 1561657
        architecture-name: arm
               board-name: D53G-5HacD2HnD
                 platform: MikroTik

free-hdd-space with my full config after Netinstall was around 420KiB, after a while with some config changes in meantime dropped to above value and it is stable now with slight variations by some bytes up/down probably due temp files. Currently there is no way to go above ROS 7.16.2 with my config.

infabo · Mon Jun 09, 2025 10:55 pm

Checked yesterday
Map running 7.18.1 incl wireless still has over 3.3Mb free.

I was talking specifically about the wifi-qcom-ac package. This package only works on ARM devices. So I meant that using routeros together with wifi-qcom-ac no longer fits on many 16MB ARM devices, and now we have to use routeros with the older wireless package instead.

But as routeros package keeps getting bigger, Mikrotik will probably have to stop supporting most ARM wireless devices that only have 16MB of storage. Or they need to find another solution.

Santi70 · Tue Jun 10, 2025 2:16 pm

a pity, because the hAP_ac2 still has plenty of power, at least for a router at home or as an AP, and of course the new wireless package is much more efficient than the previous one.
I hope they do something, at least with this model so that it does not lose the updates.

optio · Tue Jun 10, 2025 5:55 pm

a pity, because the hAP_ac2 still has plenty of power, at least for a router at home or as an AP

Same for Chateau LTE12 (old) which is with similar specs (double RAM).

xt22 · Wed Jun 11, 2025 10:14 pm

same for Chateau 5G R16, still sold and offered by Mikrotik for $485 on their web https://mikrotik.com/product/chateau5gr16 or 2025 product catalog https://mt.lv/product_catalog. 2025 router for $485, not capable of WPA3, while Mikrotik says you have to run the wifi-qcom-ac package and clarifies, quote "Chateau (all variants od D53)" supported for the wifi-qcom-ac package. Without any notice that you need to netinstall the device every two weeks because of full disk