Community discussions

 
locodog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 63
Joined: Sun Apr 12, 2015 4:00 am

The Dude 6.40.8 - db failure: database disk image is malformed

Sun Sep 02, 2018 2:25 pm

This morning I was greeted with this error message and my Dude is not running. Database was around 30MB.
I've found on Wiki manual for db repair, but there is a warning: Works only on db versions up to v6.37 software builds.
So since i have 6.40.8, repair from wiki won't work for me.
I did try vacuum-db and it does the same as restarting Mikrotik, Dude starts working for minute or so and error appears again.
Any help on how to proceed?
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 20, 2018 10:28 am

Delete the dude storage and import older dude backup.
 
sid5632
Member
Member
Posts: 336
Joined: Fri Feb 17, 2017 6:05 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 20, 2018 11:44 am

Have you got enough storage space for the database?
When this happens to me, it usually means I've forgotten to increase the disk space on the CHR VM.
 
locodog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 63
Joined: Sun Apr 12, 2015 4:00 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 20, 2018 1:22 pm

I imported older backup of dude but I'm concerned that this will happen again.

I have enough storage available.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 20, 2018 3:00 pm

Yes. It will happen again. Sooner or later it will for sure.
No. Mikrotik does not do anything in order to correct it. Schedule often backups and restore it when necessary. There is no other solution for now.
 
locodog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 63
Joined: Sun Apr 12, 2015 4:00 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 20, 2018 3:42 pm

Do you (or anyone else) know what is causing it?
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Fri Sep 21, 2018 10:45 am

None knows. Even mikrotik didn't say anything more than that they noted that for the further correction while they resurrect the dude development in the future. The dude is on hold again, like usually.
 
eddieb
Member Candidate
Member Candidate
Posts: 126
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Fri Sep 21, 2018 11:07 am

FWIW

I updated Dude with every release, running 6.43.2 now.
Dude runs on a CHR vm over here.
Running for more than a year without problems.
Creating daily backups but never needed to restore
Running 6.45.1 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Fri Sep 21, 2018 1:54 pm

Lucky you. May happen that problems are linked with usb / mSD storage type on routerboards while on CHR this does not apply.
 
locodog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 63
Joined: Sun Apr 12, 2015 4:00 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Mon Sep 24, 2018 8:06 pm

My Dude dB also runs from sdcard. I'm considering to swap micro SD card for USB to sata connector and connect small ssd for database.
I've tried connecting hard disk over USB-to-SATA connector, hard disk is detected, I got it formatted but when I try to write data to it Mikrotik restarts, I'm guessing it's because it doesn't have enough power to run hdd over USB, so I may try to find powered USB to sata adapter or maybe ssd because I'm hoping router will have enough power to run it.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Mon Sep 24, 2018 10:28 pm

My rb750gr3 started to report disk i/o errors on microSD card. I pulled it out, formatted in pc, tested few times with no errors, put back to rb750gr3, reformat to ext3, let the dude run on it and guess what? Again dtb corruption reported, followed by i/o error again. Made tests once more - no problems at all. I moved the dude back to chr but ir has its own problems too (switching off the cpu so the hypervisor decides to stop the virtual machine). At least it means that the rb750gr3 is not able to run dude on microSD card reliably.

 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Tue Sep 25, 2018 12:19 pm

Dude in CHR 6.43.2 does the same too. Mikrotik was officially informed.

There is opened ticket for this problem [Ticket#2017090222000354] by me and as you can see, it celebrated 1st year anniversary few days ago. Stll no positive answer, no hope...
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Tue Sep 25, 2018 12:29 pm

vmware workstation 12. Maybe it is not linked to storage or to architecture, it may be common dude server application programming error which is there since the era of dude 4b3, maybe longer, according to the old forum topics.

Last error was overcome by stopping and starting dude again. Question how long it will last.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 27, 2018 12:17 pm

My latest observations leads me to the actual conclusion that in case of this error it is enough just to restart the dude. As soon as possible in order to gat the smallest gap in recorded data. Finally I have created and regularly scheduled this script that checks the status "running" an in any other case it logs the situation and restarts the dude. It also maintains the counter of restarts, last date and time of restart and the last unwanted dude status. There is no need to make any exclusion, even when the database backup is in progress.

Fell free to use/modify according to your needs and in case of modification / improvement, share here for others.

For ease of use just paste it into the terminal:
/system scheduler
add interval=5m name=DudeCheck on-event=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
/system script
add dont-require-permissions=no name=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":global DudeA [/ dude get status];\r\
    \n:global DudeW running;\r\
    \n:global DudeLR;\r\
    \n:global DudeR;\r\
    \n:global DudeT;\r\
    \n\r\
    \n:if (\$DudeA != \$DudeW)  do={\r\
    \n:log info \"Dude status: \$dDudeA.\";\r\
    \n:set DudeT [/system clock get date];\r\
    \n:set DudeT (\$DudeT . \" \".  [/system clock get time] );\r\
    \n:set DudeLR [/ dude get status];\r\
    \n:set DudeR (\$DudeR + 1);\r\
    \n:log info \"Dude is not running, Make automatic restart.\";\r\
    \n/dude set enabled=no;\r\
    \n:delay 5s;\r\
    \n/dude set enabled=yes\r\
    \n}"
 
locodog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 63
Joined: Sun Apr 12, 2015 4:00 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 27, 2018 2:39 pm

My latest observations leads me to the actual conclusion that in case of this error it is enough just to restart the dude. As soon as possible in order to gat the smallest gap in recorded data. Finally I have created and regularly scheduled this script that checks the status "running" an in any other case it logs the situation and restarts the dude. It also maintains the counter of restarts, last date and time of restart and the last unwanted dude status. There is no need to make any exclusion, even when the database backup is in progress.

Fell free to use/modify according to your needs and in case of modification / improvement, share here for others.

For ease of use just paste it into the terminal:
/system scheduler
add interval=5m name=DudeCheck on-event=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
/system script
add dont-require-permissions=no name=DudeCheck policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":global DudeA [/ dude get status];\r\
    \n:global DudeW running;\r\
    \n:global DudeLR;\r\
    \n:global DudeR;\r\
    \n:global DudeT;\r\
    \n\r\
    \n:if (\$DudeA != \$DudeW)  do={\r\
    \n:log info \"Dude status: \$dDudeA.\";\r\
    \n:set DudeT [/system clock get date];\r\
    \n:set DudeT (\$DudeT . \" \".  [/system clock get time] );\r\
    \n:set DudeLR [/ dude get status];\r\
    \n:set DudeR (\$DudeR + 1);\r\
    \n:log info \"Dude is not running, Make automatic restart.\";\r\
    \n/dude set enabled=no;\r\
    \n:delay 5s;\r\
    \n/dude set enabled=yes\r\
    \n}"
Are you talking about my problem with disk image malformed? I've tried restarting dude and it would start working for minute or two and after that same error happens.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Sep 27, 2018 3:08 pm

That was similar in my case or rb750gr3 /microSD. Now when I gave up running the dude on it and moved it to CHR, the simple restart solves the situation for days without any other noticeable side effects. So far... I suggest you to follow. The client behavior is also much smoother with server in CHR than in rb750gr3. My DB file size is 300MB, backup tgz file has 115MB.
 
locodog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 63
Joined: Sun Apr 12, 2015 4:00 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Oct 03, 2018 9:46 pm

I've found SSD and connected it over USB-to-SATA connector and for now dude is working over it. I will leave it for the night and see how it behaves.
 
billjellis
newbie
Posts: 34
Joined: Mon Dec 15, 2014 11:04 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Mon Oct 15, 2018 10:54 pm

Running 6.42.7 and this started (Oct 14 2018) dude,critical db failure: database disk image is malformed.

It is one hosted on a 750G r3 Hex with a 30gb sd

Going to move to a VM

Thanks for suggestions.
Bill
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Oct 18, 2018 11:14 am

I have not seen this problem in CHR during last two weeks. Really strange.
 
grappy77
just joined
Posts: 1
Joined: Sat Nov 03, 2018 6:26 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Sat Nov 03, 2018 6:46 am

Hello,

Start mongod with --repair . To repair the data files, start the mongod instance with the --repair option. If the mongod instance uses MMAPv1 Storage Engine, you can include the --repairpath option to specify a temporary directory used during repair. For details, see --repairpath .
Thanks...
Last edited by grappy77 on Sat Nov 10, 2018 8:44 am, edited 1 time in total.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Sat Nov 03, 2018 5:37 pm

Recent Dude is the same regardless the subversion number changes. The latest "older" version could be 4.0b3.

I suspect the time changes during the device startup to be the reason for this error. It is something that does not happen in virtual environment as the running vm has still more or less correct time. I have no evidence of this, it is just an idea or feeling.
 
dmitryfromchel
just joined
Posts: 12
Joined: Wed Jan 22, 2014 6:33 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jun 26, 2019 11:13 am

This article helps me. Thanks author! CHR 6.43.4, 240 objects in DUDE.
http://www.abit.ie/how_to_solve_dude_is ... _database/
 
cdemers
Member Candidate
Member Candidate
Posts: 176
Joined: Sun Feb 26, 2006 3:32 pm
Location: Canada
Contact:

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Thu Jun 27, 2019 12:35 pm

If you are on Linux you can also use these instructions:

https://r.tapatalk.com/shareLink?url=ht ... are_type=t


Sent from my SM-A520W using Tapatalk

 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Tue Jul 02, 2019 11:42 pm

This article helps me. Thanks author! CHR 6.43.4, 240 objects in DUDE.
http://www.abit.ie/how_to_solve_dude_is ... _database/
Thanks for hint, I will need to try this procedure also. Is there any long-term experience with this "solution"? Did it help for good or the error comes back after a while again?
 
dmitryfromchel
just joined
Posts: 12
Joined: Wed Jan 22, 2014 6:33 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 03, 2019 7:18 am

Thanks for hint, I will need to try this procedure also. Is there any long-term experience with this "solution"? Did it help for good or the error comes back after a while again?
Today is 7days16hours online without crashes. In this time was added and deleted some objects and links - dude works stable.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 03, 2019 8:35 am

Not so long, though. That was the period the dude worked without an interrution just after switching it off and on until it has started to work just for 1 or two minutes only.

I was thinking of manual reconstruction in new installation, but this looks like way lower amount of workload. The other thing is that the database started to grow recently (during last half of year or so), I suspect it does not delete the old raw measurements
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 03, 2019 9:06 am

OK, looks I have some problems. When I do vacuum command, I get error message and subsequent dump creates zero sized file.
sqlite> vacuum;
Error: UNIQUE constraint failed: objs.id
sqlite> .output dude.sql
sqlite> .dump
sqlite>
Edit: looks like I moved forward, dump successfully ended, but reading the dump file into new database takes ages. There si no progress counter but roughly it does 10% each hour (1GB original database size, 0,5GB SQL dump file size).
 
dmitryfromchel
just joined
Posts: 12
Joined: Wed Jan 22, 2014 6:33 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 03, 2019 1:40 pm

dump file into new database takes ages...
Yes it goes very low speed, and I have some errors during this operation too. I have start dump in the end of day. Next morning new db was created - I don't know how much it longs. db before "repair" 50Mb, after - 33 mb (i3-3.3Ghz, SSD). Errors was in some strings in db, I try to review this strings in .sql file and I think it is something old imported/migrated data, maybe from 4.0bX versions. Maybe really "true repair" is create new dude map?
P.S. All of this like a magic, i think.
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 03, 2019 8:01 pm

Finally the dude works again. Just for one hour so far, but even this is much longer than usual two minutes before...

The dump import took around 10 hours. It was running on i7/SSD, not taking more than 2% of single cpu core. Really "effective" :-). Final db file is 0,25GB comparing to 1GB of original. The graphs are displaying much faster and the client seems not to be locking itself like during last several years.

Hope it will keep working for some time now.

Mikrotik should include the sqlite3 core and provide the db corrective application or involve the corrective function in the dude package so everyone should be able to run the corrective procedure by single command from CLI. At least. Despite they resigned to develop the dude furthermore.
 
bda
Member Candidate
Member Candidate
Posts: 120
Joined: Fri Sep 03, 2010 11:07 am
Location: Russia,Moscow

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 10, 2019 11:19 am

Finally the dude works again. Just for one hour so far, but even this is much longer than usual two minutes before...

The dump import took around 10 hours. It was running on i7/SSD, not taking more than 2% of single cpu core. Really "effective" :-). Final db file is 0,25GB comparing to 1GB of original. The graphs are displaying much faster and the client seems not to be locking itself like during last several years.

Hope it will keep working for some time now.

Mikrotik should include the sqlite3 core and provide the db corrective application or involve the corrective function in the dude package so everyone should be able to run the corrective procedure by single command from CLI. At least. Despite they resigned to develop the dude furthermore.
Does this method applicable for >6.41 dude CHR-installations ?
God bless UNIX!
 
dmitryfromchel
just joined
Posts: 12
Joined: Wed Jan 22, 2014 6:33 am

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 10, 2019 12:52 pm

Does this method applicable for >6.41 dude CHR-installations ?
viewtopic.php?f=8&t=138716&p=737790#p736644
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 10, 2019 4:32 pm

Yes. The dude was not subject to change for few years and it seems mikrotik is not going to change anything on it in near future.
 
bda
Member Candidate
Member Candidate
Posts: 120
Joined: Fri Sep 03, 2010 11:07 am
Location: Russia,Moscow

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 10, 2019 8:18 pm

Yes. The dude was not subject to change for few years and it seems mikrotik is not going to change anything on it in near future.
Thanks, I will try.
One more question. Do I need transfer all dude files or only the .db file?
God bless UNIX!
 
jarda
Forum Guru
Forum Guru
Posts: 7601
Joined: Mon Oct 22, 2012 4:46 pm

Re: The Dude 6.40.8 - db failure: database disk image is malformed

Wed Jul 10, 2019 10:30 pm

Only db file.

Who is online

Users browsing this forum: No registered users and 3 guests