Very high sector writes

The times when this used to be enough to feel reasonably safe are unfortunately gone. I don’t say it is the case here, but cross-platform malware does exist, which squats on a PC in your LAN to which it gets via connections permitted by the firewall of the router (such as browsing on infected web sites), and from there it starts malicious activity towards other machines on the LAN and also towards the router, as even security-aware users usually only do what you’ve described above, i.e. prevent attacks from WAN side but keep all managament access open for any source in LAN.

Netinstall will not help you.
All versions above 6.34.6 have nand refresh.

What’s new in 6.35 (2016-Apr-14 12:55):
*) nand - implemented once a week nand refresh to improve stored data integrity (will increase sector writes);

20000 sector writes in 36 hours give something like 93000 sector writes per week. What’s the NAND Flash size in your device?

I have netinstalled 6.45.1 and the issue persists. But the writes are not done once a week, as in the “refresh” update, they are done every couple of minutes. After 1 hour there is already more than 300 writes. So the netinstall procedure hasn’t fixed it. I noticed that even though leases are set to never write to disk, any time a lease is granted, renewed or removed, write count raises by 16. No matter what interval I use, be it never, 1 hour, 5 hours, 1 day, etc, any lease change causes sector writes immediately.
I have already contacted support, they are investigating, so I will keep this thread updated with any info that might help.

128.0 MiB, RB2011UAS-2HnD

My device is old :slight_smile:

did you solve the problem? What have you done for this?

do you enable the vpn service?
or you use vpn?
i find vpn will cause sector writting…

I have similar behaviour with my CAP ac since last couple releases (6.48.x). Used only as ap station with 3-5 clients, no DHCP, no graphing, no logging to disk.

It looks like SNTP client is causing writes to flash - at the moment installed 6.48.2, SNTP client on - 6-8K wrtes per 24h, SNTP client off - 1-1.5K writes per 24h.
I have 3 CAP ac (different configurations), but the ones with h/w revision v2 are not making high sector writes with SNTP on/off, only older model without revision has high sector writes.

From 4 July 2019 to now no one has guessed…

For now only this comes to my mind:

  1. Last logged in on users are stored on NAND, cannot be disabled,
    Just opening winbox or terminal
    For example, just this are 2 writes

  2. RouterBOARD do not have battery for RTC :((((
    SNTP and NTP store time on NAND, it’s happen often.
    It’s for set time at bootup as closest right time.
    it cannot be disabled, also if SNTP and NTP is disabled

  3. On DEFAULT, critical logs, like failed login access, are stored on NAND till 100 events, the newer clear the older, etc.
    These logs are stored on NAND and displayed on terminal on open and removed just after displayed on terminal
    for disble the DEFAULT:

/system logging action set echo remember=no
  1. Terminal history:
    Everytime you write something on terminal, that go to “history” of commands.
    Is persistent and keeped on NAND after reboots
    it cannot be disabled
    hint: for clear history on terminal
/console clear-history
  1. DHCP Leases are stored on default every 5 minutes on “disk”
    For change the update interval:
/ip dhcp-server config set interim-update=00:05:00

For disable saving

/ip dhcp-server config set store-leases-disk=never

Sector write history count?
Yes… it’s writed somewhere… ;))

EDIT: added point 5)

This might be a silly idea, but what if there simply isn’t enough power to commit that much data? Or just an update bug.

I can definitely confirm that my high sector writes occur when SNTP client enabled. With SNTP client disabled I have 600-800 writes per 24h now and nothing else is chagned in configuration to get writes dropped from 8K to 800 per day.

Whether time adjustments are causing other modules to create more NAND writes (kind of chain reaction) or SNTP client itself stores a lot of info on disk - that is question to MikroTik.

If this indeed has anything to do with SNTP client, then it’s NTP client from stand-alone ntp package guilty as well.

If I’m not mistaken, separate NTP package should be installed additionally (I did that recently to get NTP server running on my hex S), SNTP client is part of default packaging.

What I’m saying is that I also see enormous number of sector writes, but my devices all have the separate ntp package installed. AFAIK separate ntp package provides different ntp client than system package. So if it’s ntp client that causes high sector writes, it’s ntp client from separate package doing it as well.

http://forum.mikrotik.com/t/very-high-sector-writes/131330/29

Added point 5)

Hi. Interesting topic.

What number is considered to be high sector writes within a 24 hour period? 1000, 5000?

My hAP ac2 recorded 1.5M sector writes since boot … which was 90 days ago, so it’s averaging more than 15k sector writes per day.

I have 1 cap ac running for 2 years 24x7, it collected 600K (~800 per 24h). After update to 6.48.x this year it got another 90K in 2 weeks, so I started worrying and testing different settings on/off.

Stats for last 7 days with SNTP off (including other 3 MTs at home):
2021-05-08 10_01_18-MikroTik Resources _ Splunk 8.1.3 — Mozilla Firefox.png
PS I wouldn’t be worrying much if I don’t have HP server (gen 8) which NAND flash (soldered) died after 5 years, which made internal SD card slot unworkable + iLO became extremely slow and only way to resolve problem - replace motherboard. Don’t want to replace router in next 3 years because somthing started using NAND not for purpose (I hardly can imagine what should be written to disk when time is adjusted, my very wild theory - it tries to amend everything which is already timestamped and stored on disk with new timings).

My hAP ac2 collected 5.5M sector writes so far, bad blocks is still at 0.0%. This device is my main home router.
My RB951G collected 96k sector writes in 12 weeks, 16.3M in total, bad blocks are at 0.5% since long time ago (pretty sure predates the massive sector write feature). This device is used as switch/AP (stand-alone).
My other RB951G collected 95k writes in 12 weeks, 21.4M in total, bad blocks are at 0.1% (also since long time ago). This one is used as CAPsMAN-controlled AP and some time ago I used it to try to play with ROS v7beta (I partitioned NAND for that).
All devices currently run ROS 6.47.9.

I’m comforting myself with the fact that these devices cost quite less than a HPE ProLiant main board. The RB951Gs have done their tasks “magna cum laude” and I’ll reture tgem light-heartedly if their NAND happen to die. I won’t be so hapoy if RBD52G dies on me (it’s not done with its duties not yet).

But in any case MT should either reduce the number of writes (if it really increased from what it used to be) or explain the situation clearly if actual sector writes did not increase, only counting started to show real numbers. I seem to remember a vague note by one of MT staffers indicating that actual sector writes did not actually increase when counter started to gallop but rather that they fixed counting …

One simple consideration, from 2007 now are 14 years than I work as ISP.
For now, no one single device depleted his memory write times.
Considering I do not use anymore RB1xx, RB5xx, RB4xx, and RB7xx, but I still have many RB1100 working as user-manager from 2010, and all still perfectly working after 11 years of read and write.