Containers won't start after power loss

dcavni · Mon Nov 06, 2023 2:37 pm

I'm having some problems with Adguard container (there is also udpxy container but for that one it's not a problem if it won't start until i reboot manualy).

Without Adguard DNS isn't working, so practicly nothing is working until i reboot.

We are having quite some power losses this year because of the bad weather and what is happening is that, when power is restored apparently containers won't start with 5009. I'm guessing it's because modem isn't online before 5009 so i'm searching for a way to delay start of the containers for 5 minutes.

If i reboot 5009 manualy after everything else is already running then containers start without problem, but it has to be rebooted. If i just try to start container from Winbox without rebooting, nothing happens, no error in log, nothing.

Any ideas?

zmolnar · Thu Sep 12, 2024 10:00 am

Hi!

I'm facing the exact same problem on my RB5009 running v7.5.13.

Whenever a reboot occurs after a power outage, the AdGuard container does not start. If I try to start it manually, nothing happens, nothing is logged, and the container remains stopped. (I don't know if it's AdGuard-specific or not because I don't have other containers.)

Did you find a solution?

Thu Sep 12, 2024 1:30 pm

Both: are you sure after reboot the disk where the container is stored is still on the place where it needs to be ?
(usually usb1)

Some USB3 keys can after reboot only be accessed using USB2-protocol (a USB reset or delay when starting up the USB-interface solves it).
If they start up as USB2, disk is labeled different (usually usb2) but for your container config it's a completely different location.
So they don't start.

MT support is aware about the issue but no ETA yet for a solution.
Working assumption: don't auto-start containers, use a script after reset to do USB-reset and then start containers from within this script.

Amm0 · Fri Sep 13, 2024 12:10 am

Yeah... And the "Start at boot" is kinda of "one-and-done" situation. If the disk ain't mounted, for any reason, container won't start. It's not just limited USB, but stuff like ROSE+NFS for containers you'll also see this sometimes (or at least I have, occasionally).

A schedule script is like better anyway, and AFAIK it's harmless to call "start" on a already-started container (so likely can skip a check to see if running if desired):

 /container start [find tag~"adguard"]

Cloudisparu · Fri Sep 13, 2024 2:59 pm

+1
Facing the same problem on my 5009 too.
Use a scheduler to start container manually after reboot.

zmolnar · Sun Sep 15, 2024 9:29 am

Hi,

thanks for your reply. Based on your suggestions, I created a script to start adguard, but it does not solve the problem. (This is my very first ROS script I wrote so I'm happy to get feedback.)

:local containers [/container find where tag~"adguard"]

:foreach adguard in $containers do={
    :local rootDir [/container get $adguard root-dir]
    :log error ("root-dir: " . $rootDir)

    :local rootDisk [:pick $rootDir 0 [:find $rootDir "/"]]
    :log error ("root-disk: " . $rootDisk)

    :local mounted false
    :local triesLeft 15

    :while ((false = $mounted) && (0 < $triesLeft)) do={
        :if (0 < [:len [/file find where name=$rootDisk]]) do={
            :delay 5s
            :set mounted true
        } else={
            :set triesLeft ($triesLeft - 1)
            :log error ("adguard root disk not mounted, wait for 2 seconds")
            :delay 2s
        }
    }

    :if (true = $mounted) do={
        :log error "root disk found for adguard"
        /container start $adguard
    } else={
        :log error "failed to find root disk for adguard"
    }

    :if ("running" = [/container get $adguard status]) do={
        :log info "adguard container started successfully"
    } else={
        :log error ("failed to start adguard container" . $error)
        # TODO: set alternative DNS server
    }
}

:log error "script run"

It can be seen in the logs that the root disk is not set up properly for the first check but after that, it gets mounted. The container start command is executed but adguard remains stopped:
(I have already tried to add a 5 seconds delay between container start and status check, but that didn't help either.)

07:53:35 script,error root-dir: usb2/docker/adguard-home
07:53:35 script,error root-disk: usb2
07:53:35 script,error adguard root disk not mounted, wait for 2 seconds
07:53:42 script,error root disk found for adguard
07:53:42 script,error failed to start adguard container
07:53:42 script,error script run

Do you have any ideas what to synchronize the start of the container to in order to make it work?

navigator · Sun Sep 15, 2024 10:07 am

It's the exact same problem on my RB5009 (after reboot or power loss).
I've sheduled script to reset USB-power and delayed start of Adguard.

{
:local diskl [/disk get usb1 label]
:put $diskl
:local diskc "usbflash"
if ($diskl = $diskc) do= {
put "Adguard is running..."
} else={
:delay 10s;
:local contag "adguard"
:local currstat [/container get [find where tag~$contag] status]
:put $currstat
:local runstat "running"
 if ($currstat != $runstat) do= {
/sys routerboard usb power-reset duration=5s
:log warning "USB power reset"
 :delay 10s;
/container start  [find where tag~$contag]
 :delay 10s;
/ip dns set servers=172.18.0.2
  }
 }
}

jaclaz · Sun Sep 15, 2024 11:49 am

You seem to be affected by the USB device detection bug holvoeth mentioned.

It seems to be affecting some devices (USB sticks from a given brand, namely Sandisk) more than other ones, there are a couple workarounds that you could try integrating in your script, either reset the USB bus or rename the slot, see:
viewtopic.php?t=196829
viewtopic.php?t=203217

zmolnar · Sun Sep 15, 2024 3:22 pm

Yes, that was the solution, because I'm using a SanDisk UltraFit 64GB pendrive. I updated the script with USB reset and now it works for me:

# do a USB power reset to fix device detection issue
:system routerboard usb power-reset duration=1s
:delay 3s

:local containers [/container find where tag~"adguard"]

:foreach adguard in $containers do={
    :local rootDir [/container get $adguard root-dir]
    :local rootDisk [:pick $rootDir 0 [:find $rootDir "/"]]

    :local mounted false
    :local triesLeft 15

    :while ((false = $mounted) && (0 < $triesLeft)) do={
        :if (0 < [:len [/file find where name=$rootDisk]]) do={
            :set mounted true
            :delay 2s
        } else={
            :set triesLeft ($triesLeft - 1)
            :log info ($rootDisk . " is not mounted, wait for 2 seconds")
            :delay 2s
        }
    }

    :if (true = $mounted) do={
        :log info "start adguard container"
        /container start $adguard
        :delay 5s
    } else={
        :log error "failed to find root disk for adguard"
    }

    :if ("running" = [/container get $adguard status]) do={
        :log info "adguard container started successfully"
    } else={
        :log error ("failed to start adguard container")
        # TODO: set alternative DNS server
    }
}

Thanks for your help!

Containers won't start after power loss

Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Re: Containers won't start after power loss

Who is online