Page 1 of 1

Very high sector writes

Posted: Thu Jul 04, 2019 2:48 pm
by arielgrin
Hi All: I'm running RouterOS 6.43.16 (long-term) on RB751g-2HnD and I'm seeing more than 7000 sector writes per day, I've opened webfig and checked the System->Resources menu and I can see how it writes 32 sectors every 6 minutes, approximately. Not only that, it almost reached 400000 sector writes after only 30 days last month, which I think it might be excessive, as nothing is supposed to be writing to disk at all. Just in case, I reset the router to default configuration and disabled DHCP writes to disk, there are no graphs running and logging is to memory only, but anyway the 32 writes every 6 minutes continue to happen, with over 7000 write after just 24 hours. This is my configuration in case anyone sees what could be causing this, unless it might be a bug.
/interface bridge
add admin-mac=D4:CA:6D:51:FB:51 auto-mac=no comment=defconf name=bridge
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX distance=indoors frequency=auto ht-supported-mcs=\
    mcs-0,mcs-1,mcs-2,mcs-3,mcs-4,mcs-5,mcs-6,mcs-7,mcs-8,mcs-9,mcs-10,mcs-11,mcs-12,mcs-13,mcs-14,mcs-15 mode=ap-bridge ssid=MikroTik-51FB55 wireless-protocol=802.11
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=192.168.8.21-192.168.8.100
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge lease-time=1d name=defconf
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wlan1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.8.1/24 comment=defconf interface=ether2 network=192.168.8.0
/ip cloud
set update-time=no
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server config
set store-leases-disk=never
/ip dhcp-server lease
add address=192.168.8.4 client-id=1:40:8d:5c:96:c1:24 mac-address=40:8D:5C:96:C1:24 server=defconf
/ip dhcp-server network
add address=192.168.8.0/24 comment=defconf gateway=192.168.8.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.8.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/system clock
set time-zone-name=America/Argentina/Salta
/system lcd
set contrast=0 enabled=no port=parallel type=24x4
/system lcd page
set time disabled=yes display-time=5s
set resources disabled=yes display-time=5s
set uptime disabled=yes display-time=5s
set packets disabled=yes display-time=5s
set bits disabled=yes display-time=5s
set version disabled=yes display-time=5s
set identity disabled=yes display-time=5s
set bridge disabled=yes display-time=5s
set wlan1 disabled=yes display-time=5s
set ether1 disabled=yes display-time=5s
set ether2 disabled=yes display-time=5s
set ether3 disabled=yes display-time=5s
set ether4 disabled=yes display-time=5s
set ether5 disabled=yes display-time=5s
/system logging
add topics=debug
/system package update
set channel=long-term
/system watchdog
set watchdog-timer=no
/tool graphing
set store-every=24hours
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool user-manager database
set db-path=user-manager

Re: Very high sector write

Posted: Thu Jul 04, 2019 2:54 pm
by mkx
My guess: user manager ... try to attach an USB flash disk and configure user manager to store its database to USB stick.

Re: Very high sector write

Posted: Thu Jul 04, 2019 9:21 pm
by arielgrin
Thanks @mkx for your reply.

I have disabled user-manager package, as I don't need it. I rebooted the router and will check again after a couple of hours to see if the writes have stopped.

Do you happen to see anything else that could be a cause for writes? User manager has been always enabled, but never used, and this write frequency just increased in the last couple of months, it was not like this before.

Re: Very high sector writes

Posted: Thu Jul 04, 2019 10:41 pm
by arielgrin
Indeed it seems it is not user-manager, or at least not only that. The router has been running for less than an hour and there are already 650 writes logged.

Re: Very high sector writes

Posted: Thu Jul 04, 2019 10:42 pm
by Pea
Many DHCP leases? Try:
/ip dhcp-server config set store-leases-disk=never
Edit: I see you have this already, so it must be something else...

Re: Very high sector writes

Posted: Thu Jul 04, 2019 10:45 pm
by mkx
I wonder if this setting

/ip dhcp-server config
set store-leases-disk=never

really disables writing to flash. Try to set it to something like 6h and see if it makes any difference.

Re: Very high sector writes

Posted: Fri Jul 05, 2019 1:13 am
by arielgrin
I will try changing the setting. I thought that store-leases-disk=never was the way to go to avoid leases being written to disk, but I will try other setting. In any case, there are only 5 leases and the lease time is 1 day, so the refresh frequency is really low, once every 12 hours for just 5 leases. The router has been running for 4 hours and it already has more than 2000 sector writes.

Re: Very high sector writes

Posted: Fri Jul 05, 2019 9:16 am
by mkx
I thought that store-leases-disk=never was the way to go to avoid leases being written to disk

It should ... but there's always room for some bugs :wink:

Re: Very high sector writes

Posted: Fri Jul 05, 2019 1:20 pm
by arielgrin
Well I changed the setting to store the leases every 24 hours, but the writes continue to raise. The router has been running for 12 hours and there are more than 4000 writes already. I don't know what else to check or do, in less than 2 months there have been more than 1 million writes. I think that is definitely excessive.

Re: Very high sector writes

Posted: Fri Jul 05, 2019 10:09 pm
by arielgrin
Router has been running for 36 hours and there are more than 20000 sector writes. I don't know what else to do. Should I contact support? I don't have a support contract or anything like that.

Re: Very high sector writes

Posted: Fri Jul 05, 2019 10:28 pm
by sindy
As you have already tried a reset to default configuration, I would try to export (not backup) the configuration into a file, download the file to PC, netinstall the router, and check again with default configuration. If the ghost writes stop appearing, I'd re-import the configuration, otherwise I'd use some other RouterOS release (e.g. 6.43.15, as 6.43.16 only fixes some 60 GHz related issue).

support@mikrotik.com does accept information about well-documented real bugs even without a support contract, but they cannot deal with "how do I change the default IP address" class of questions, that's why they suggest this forum as the primary support channel and refer to the support from your reseller.

Re: Very high sector writes

Posted: Sat Jul 06, 2019 3:25 am
by 2frogs
/system logging
add topics=debug
Have tried disabling this?

Re: Very high sector writes

Posted: Sat Jul 06, 2019 6:52 pm
by mn62
All versions of RouterOS above 6.34.6 in the old line of devices from MikroTik cause an increased amount of write to flash memory.

Re: Very high sector writes

Posted: Sun Jul 07, 2019 4:25 pm
by arielgrin
That setting is no longer active, the router has been reset to factory defaults, but the writes keep on raising no matter what.

Re: Very high sector writes

Posted: Sun Jul 07, 2019 9:10 pm
by sid5632
Netinstall it then. I expect it's been compromised.

Re: Very high sector writes

Posted: Mon Jul 08, 2019 3:37 pm
by arielgrin
Will try netinstall and report. Just curious, compromised how? Do you mean hacked? Or something else?

Re: Very high sector writes

Posted: Mon Jul 08, 2019 7:08 pm
by sid5632
Yes, hacked.

Re: Very high sector writes

Posted: Wed Jul 10, 2019 10:49 pm
by arielgrin
I don't see how could it get hacked. There are no open ports nor forwarded ports on the wan interface. None of the admin tools, like webfig or ssh are open to the exterior. The only one using the router is me.

Re: Very high sector writes

Posted: Wed Jul 10, 2019 11:06 pm
by sid5632
It was only a guess based on almost zero information.
You will only find out if it's cured by Netinstalling it.

Re: Very high sector writes

Posted: Wed Jul 10, 2019 11:18 pm
by 2frogs
Most likely a partially failed update or some corruption in OS.

Re: Very high sector writes

Posted: Wed Jul 10, 2019 11:31 pm
by sindy
There are no open ports nor forwarded ports on the wan interface. None of the admin tools, like webfig or ssh are open to the exterior. The only one using the router is me.
The times when this used to be enough to feel reasonably safe are unfortunately gone. I don't say it is the case here, but cross-platform malware does exist, which squats on a PC in your LAN to which it gets via connections permitted by the firewall of the router (such as browsing on infected web sites), and from there it starts malicious activity towards other machines on the LAN and also towards the router, as even security-aware users usually only do what you've described above, i.e. prevent attacks from WAN side but keep all managament access open for any source in LAN.

Re: Very high sector writes

Posted: Thu Jul 11, 2019 3:54 pm
by mn62
Netinstall will not help you.
All versions above 6.34.6 have nand refresh.

What's new in 6.35 (2016-Apr-14 12:55):
*) nand - implemented once a week nand refresh to improve stored data integrity (will increase sector writes);

Re: Very high sector writes

Posted: Thu Jul 11, 2019 4:36 pm
by sindy
*) nand - implemented once a week nand refresh to improve stored data integrity (will increase sector writes);
20000 sector writes in 36 hours give something like 93000 sector writes per week. What's the NAND Flash size in your device?

Re: Very high sector writes

Posted: Thu Jul 11, 2019 6:33 pm
by arielgrin
I have netinstalled 6.45.1 and the issue persists. But the writes are not done once a week, as in the "refresh" update, they are done every couple of minutes. After 1 hour there is already more than 300 writes. So the netinstall procedure hasn't fixed it. I noticed that even though leases are set to never write to disk, any time a lease is granted, renewed or removed, write count raises by 16. No matter what interval I use, be it never, 1 hour, 5 hours, 1 day, etc, any lease change causes sector writes immediately.
I have already contacted support, they are investigating, so I will keep this thread updated with any info that might help.

Re: Very high sector writes

Posted: Fri Jul 12, 2019 10:54 pm
by mn62
What's the NAND Flash size in your device?
128.0 MiB, RB2011UAS-2HnD

My device is old :-)