We have 100’s of R33 boards out in the field that all have LTE interfaces. Majority of them work with no issues but occasionally we have issue with the LTE not starting / connecting properly following a reboot which they all do every day.
I was wondering if there is a way to delay the LTE interface from starting after a reboot. My thinking being give the board to do its thing on reboot and then 30 sec’s after start the LTE interface to see if that helps at all.
Yes there is way to do that with startup script but maybe you first should check if latest version solves this problem (if those boards are not there yet).
I’ve seen something being mentioned in some of the latest release notes related to startup of LTE interface after boot, that’s why I mention it.
Best/safest way:
check how it is done in default config script (terminal: /system def print file=whatever, move file to PC and check contents).
Take over that approach.
I usually use a slightly different approach, I monitor traffic flow via the LTE interface and power-cycle it (they are actually USB ones) if there is none, granting it a recovery period of some minutes after each power cycle. The pros are that this way can handle also weird hangs during operation possibly caused by the mobile network (which may not be of great importance in your case as you reboot nightly), the cons are that the script is a bit more complicated.
Thanks @sindy, I have been speaking to my tech team and apparently what happens for us more often that not following a router reboot is the LTE interface doesn’t actually load, as in doesn’t exist. We then have to wait for our daily reboot to fix it… to combat this i have just written this script which we will run 5 mins after any reboot and every hour just in case
:log info "Checking if LTE1 interface exists..."
:if ([/interface find name="lte1"] = "") do={
:log error "LTE1 interface does not exist. Rebooting router..."
/system reboot
} else={
:log info "LTE1 interface exists."
}
Have you tested that a reboot of the router is indeed necessary to recover from the “LTE not seen” state, i.e. whether power cycling the USB is not a sufficient way to make the LTE modem get recognized?
I did think about doing that but considering all these units are remote, majority are installed at height and the fact that if the LTE doesn’t work we have no remote connectivity and a loss of data capture the result being a day of engineering time to send someone out to fix it plus travel expenses i opted for a belt and braces approach and decided a full router reboot would be a more robust solution.
It is frustrating that its “some” devices that have this issue. I’d admit that not all the routers are the exact same FW as others but most are and the issue happens on devices that match other that are 100% fine. All RB boards are the same as we only use M33’s but over the years LTE cards have changed and we have a mix of LTE, 4G and LTE6 cards with the majority being LTE, all of which seem to run different FW versions. I’ll admit we don’t often run the very latest FW version as we don’t like to be the ones that test in the field, all our boards are on release 6 as we tested 7 and had issues but i do acknowledge that 6.49.17 does mention it has an LTE update that is of interest.
One issue ( more of a concern really ) we have is updating the LTE FW remotely, if that goes wrong we are stuffed and it becomes an expensive fix which we’d prefer to avoid so its rarely done.
Not really a concern.. if it goes into a loop it will be because the LTE card isn’t working and if that’s the case the unit is no good to us and its a swap out anyway
That where the ping watchdog isn’t a bad option… as that will just cause a boot loop, but at least it retry. Since it’s a built-in feature, I think it’s a better “last chance” failsafe since there can always be some script issue (bug/broking changes) that breaks scripting-based recovery. The ping watchdog is pretty simple.
I don’t see why BOTH approaches cannot be tested.
In the script, if LTE cannot be found, try resetting USB.
If after - say - 120 seconds the LTE still cannot be found, then, and only then, reboot the router.
If It works, It works, and if It doesn’t It goes through the full reboot.
120 seconds Is only a rough value, a more sensibile one can be determined experimentally, likely a smaller amount might be enough.
Whhy is there such a reluctance to reboot the router? How many thing should to try before you do something you know works. The only reason we are having to do this is because the manufacturer of the board, modem and developer of the software cant not make it’s own equipment work together properly!
Extra checks and executions just confuse matters however if you feel that strongly about it please feel to share a script that will do all those functions, i’m sure someone on the forum will find it useful at some point. Personally i have commercial decision to consider so i’ll stick with a reboot thanks.
On my side, it is not a reluctance to reboot the router, it is an experience that a software-initiated reboot of the router was in too many cases not sufficient to resolve an issue with an LTE modem that itself needed a power cycle.
Adding:
Important to distinguish perhaps since otherwise we get lost in semantics: software reboot does not necessarily result in power cycle of USB devices.
USB power reset does make sure that modem is power cycled.
During SW reboot it might be it stays powered on (one should monitor USB lines to see if it does or does not get a power cycle but probably not given Sindy’s observations).
Perfect, these are the answers that are really helpful, thank you.
Its a very valid point and is noted. My scripting knowledge is basic so i had to ask ChatGPT for help on this one and it came up with this. I’ll test it on our dev platform but any pointers re the script would be useful.
# Script to check if lte1 exists, reset USB, and reboot if necessary
:local interfaceName "lte1"
:local usbPortNumber "1" # Change to your USB port number if necessary
# Check if lte1 interface exists
:if ([/interface find name=$interfaceName] = "") do={
:log warning "$interfaceName not found. Resetting USB port $usbPortNumber..."
# Reset USB port
/system routerboard usb power-reset duration=3
# Wait for 60 seconds
:delay 60s
# Check again if lte1 interface exists
:if ([/interface find name=$interfaceName] = "") do={
:log warning "$interfaceName still not found. Rebooting the router..."
# Reboot the router
/system reboot
} else={
:log info "$interfaceName detected after USB reset. No further action required."
}
} else={
:log info "$interfaceName is present. No action required."
}