Part of our monitoring is to log and alert of OSPF state changes. At the moment only the transition ‘to DOWN’ is logged as a ‘route, ospf, info’ message but anything else i.e. “state changed from Loading to Full” is in the ‘route, ospf, debug’ logging category
This means I can’t generate messages showing an OSPF neighbor relationship has come up, only when the neighbor goes down (unless I want to send all OSPF debug messages to the SysLog server… no way)
MikroTik devs: would you mind moving these messages from ‘debug’ to ‘info’ so I can log and alert when the neighbor comes back up? Otherwise we only get 1 alert saying “Down” and we have to log in to check if its still down or if it was just a brief interruption
Since MikroTik still hasn’t implemented ‘state changed from Down to Up’ i’ve written a script to simulate it in the meantime. It isn’t perfect but it does the job. It relies on running as a script so it doesn’t notify you immediately
And messages show up under the ‘script,info’ category not ‘route,ospf,info’ so if you are doing remote syslog monitoring you need to send that
The script is set to look at log files going back 1 minute and stores any ospf, info messages. It then enumerates through and compares the active OSPF adjacencies, if it see’s that there was a Down message but that Router-ID is currently active, it’s assumed the link is back up and sends a log message “OSPFv2 neighbor [ID]: state change from Down to Up on [interface]”
I suggest to schedule this to run once per minute. If you want to run it less frequently then adjust the time on line 3 of the script to look further back through the log messages
:global OSPFNeighborList
:if ($OSPFNeighborList != [/routing ospf neighbor find]) do={
:local OSPFDownMessages [/log find where topics=route,ospf,info time>=([/system clock get time]-00:01:00)]
:foreach i in=[$OSPFDownMessages] do={
:local m [/log get $i message]
:local t [:pick $m ([find $m "r"]+2) [find $m ":"]]
:local r [/routing ospf neighbor find where router-id=$t]
:local r [:pick $r 0]
:local ri [/routing ospf neighbor get $r interface]
:if ($r != "") do={:log info "OSPFv2 neighbor $t: state change from Down to Up on $ri"};
};
};
:global OSPFNeighborList [/routing ospf neighbor find];
I’ve had to include some extra checks for duplicate router ID’s (i.e. backup paths) as that was causing an issue. The script will fire off a ‘Down to Up’ message if at least 1 path to a router is still active. This is not strictly correct because an already active link has not transitioned from the Down to Up state
i.e. you have ether1 ↔ ether1 and ether2 ↔ ether2 on RouterA and RouterB respectively
If the link on ether2 went down, technically you should only get 1 message ‘Up to Down’
However this script compares Router-ID’s and see’s that infact there is a path still to the ID of RouterB so it will say ‘Down to Up’
‘Down to Up’ is also not the actual state, it’s just assumed. Technically a router could be stuck in the init/exstart/2way state and it will show ‘Up’, it’s not looking at the actual state
This could be scripted out but I don’t want to go any further with the script making it significantly more complicated. I’d rather MikroTik just implement this properly in the first place…
In addition the script mentions the interface, Again I wish this was included by default in the log message. Because with multiple/redundant paths between routers I like to know which interface went down, not just the adjacency. It’s nice to know if the low bandwidth backup link goes down and it doesn’t affect anybody, or the high bandwidth primary one goes and there’s likely to be congestion and latency increases
I’ve had to disable this script across out entire network. There’s a bug somewhere and I can’t find out what it is. For the most part this script works just fine, however occasionally for some reason the script just keeps on firing and reporting an ‘up’ status, even though no change in neighbors at all
I’ve gone over my script (and i’d appreciate if someone else could) and I really can’t find an issue at all, hence I think its a bug with the mikrotik neighbor output. Maybe its throwing an error/invalid entries and not updating the global value hence it always appears different, refiring the ‘up’ notification
I just don’t know how to work around this. So unfortunately i’ve had to just outright disable this for now
To MikroTik: PLEASE PLEASE PLEASE PLEASE PLEASE PLEASE PLEASE PLEASE just FIX the log reporting! its a really, really simple change to simply add ‘up’ notifications into the logging! you already have ‘down’ notifications in ospf,info so why the heck do you have not have ‘up’ notifications in there as well? they are in debug, which is really stupid because it just floods the log file and is totally impractical to receive FULL ospf debug notifications every 1s (the hello interval we use) on every single router in our network. This is so stupid! this is a 5 minute task for a junior engineer to go and move the notification from debug to info (warning would be even better). Please just do it!!!
Bumping this up. MikroTik please implement this in the next firmware update. It should be an incredibly simple and easy thing to do, the messages are already there, just literally take the ‘up’ message (and all other key state changes) and assign it into the ‘ospf, info’ category. Very simple, 30 minutes if that
I’ve gone over my script (and i’d appreciate if someone else could) and I really can’t find an issue at all, hence I think its a bug with the mikrotik neighbor output. Maybe its throwing an error/invalid entries and not updating the global value hence it always appears different, refiring the ‘up’ notification
I have modified your script to work with a Telegram bot and have noticed similar behavior. I will either get no alerts at all, or I get the the repeated notifications as you described. I have been looking into the neighbor output as I feel like this may be where this matter is stemming from too.
In a Winbox session under System > Scripts > Environment, I notice that the value will not populate at times and also seems to vary quite often on other routers. I am continuing to work on this as we feel that the OSPF state changes are needed alerts.
P.S. Thanks for putting in the work to get the initial script together and starting this thread!
No updates but I’m appalled mikrotik still has done nothing about this. Its a 5 minute task on a Friday afternoon job, literally just move the classification of specific OSPF messages into the correct category, done
I guess it’s a chicken and egg scenario, which comes first?
No one seems to care because no one uses the log messages because they don’t exist in a relevant place…
however occasionally for some reason the script just keeps on firing and reporting an ‘up’ status, even though no change in neighbors at all
diagnosis,
if the neighbors they were really in down state without any notice - then your part of the script is in correct behavior.
do last time stamp check.
let us say we do tail -n 10 /var/log/messages|grep -i something… without any validation then that last time stamp will be taken by your part of the script hence the script was in correct behavior once again.
work around:
if last time stamp < (current time stamp - 1 minute) then exit. # this makes sure no multiple logging in 1 minute
else last time stamp= get message. fi
that “up” part …
depends on your syslog backend - was it MySQL? if yes, then you can just modify/make your “up” status to be included in the column.
tail -n 10 /var/log/messages| grep “up”|sed \debug\info\ >> to sql functions. #or just something like that
and don’t forget ntp time synchronization across the whole network - so the script function properly.
