No syslog on critical alerts

Hi

I have setup all my router to send their logs to Syslog server (Splunk, see my signature).
When I take down a Mikrotik VM router without shuting it down, I do see this when I log inn using WinBox:

dec/13/2019 18:22:12 system,error,critical router was rebooted without proper shutdown
dec/27/2019 14:38:10 system,error,critical router was rebooted without proper shutdown
feb/22/2020 00:28:48 system,error,critical router was rebooted without proper shutdown
mar/07/2020 22:43:42 system,error,critical router was rebooted without proper shutdown
mar/22/2020 17:01:45 system,error,critical router was rebooted without proper shutdown
mar/26/2020 08:15:06 system,error,critical router was rebooted without proper shutdown
apr/07/2020 19:49:52 system,error,critical router was rebooted without proper shutdown
apr/13/2020 07:47:03 system,error,critical router was rebooted without proper shutdown

But for some reason I do not see this i my Syslog. No trace of this message there.
Are these message logged when router going down, or when router boots first time after coming back up?
Are they logged before Syslog starts on the Router?

My logging setup:

Flags: * - DEFAULT
Columns: TOPICS, ACTION, PREFIX
  #     TOPICS    ACTION     PREFIX  
  0  *  info      memory             
  1  *  error     memory             
  2  *  warning   memory             
  3  *  critical  echo               
  4     dhcp      logserver  MikroTik
  5     !debug    logserver  MikroTik
  6     hotspot   logserver  MikroTik

The “!debug” does send all message that not are debug to Syslog, that would be all Critical/Emergency/Alert/Warning/Informational/Notification messages.

My wild guess is that this message is generated so early after the reboot that the network connectivity has not established yet by that time. And since Mikrotik only supports syslog via UDP, there is no second chance for the message.

Since Critical messages are not configured to be sent to your Log server " 3 * critical echo ", you won’t see them there…
Since i ve used a log server messages like “router was rebooted without proper shutdown” appear just fine…

The “!debug” does send all message that not are debug to Syslog, that would be all Critical/Emergency/Alert/Warning/Informational/Notification messages.

Apparently it does not send all messages…
Just try to set critical messages to be loged…

What does the message look like in your syslog server? Is it being parsed correctly for your syslog server to see the critical message correctly?

As Zacharias noted, set each severity separately to go to your syslog server and see if that helps. Thats how I have mine set.

@kd2pm2 They do not come to external Syslog server

@Zacharias
All my message goes to the Syslog server, not just the critical reboot once.
I did try to add critical message to Sylog config, but did not help.
Since I have worked allot with logging I do know that !debug should get all message except debug message from all message logged.
If I do like all message, also debug, I do use !ups (since I do not have UPS)

@sindy
I think you are correct, and this is some that MT should have a look at. If there are new message after reboot and before network up, buffer them and send them out when network start working.

For me these message are important to get to my log so that I could be aware of that some has gone wrong.

Tested on
6.45.8
6.46.beta54
7.0 beta 5

There is only one better way than an e-mail to support@mikrotik.com with a description of the issue - the same e-mail but coming from a Mikrotik distributor, not just and advanced user. At least that’s what Mikrotik keeps declaring - it’s the distributors who actually delivers the customer’s money to Mikrotik, so their requirements get the highest priority.

As for me, adding syslog over TCP, and queueing all messages, would make more sense than buffering just the initial ones until the network comes up after boot, as when something bad is going on in the network, many syslog messages get lost if delivered using UDP.

I have sent several email, and got good response and fixes for the problem I have sent.
So I will send an email :slight_smile:

Here I do agree as well. TCP should be added.

My experience is similar, yet it always depends on how serious the particular issue appears to the guys in Riga.

I think this thinking is correct. And it extends to logging to disks. I seem to miss a few early log messages from disk log.
It seems to me that logging functions, although configured with multiple logging destinations, send messages only to those destinations available at invocation time, no message caching (until logging destination becomes available) is avsilable.
When I think about it, it kinda makes sense … how’s logging subsystem supposed to know whether a particular logging destination is gettin available at all.
But it is annoying, I agree …

On the RB4011 (and also on the 2011), I can say that I would see email errors in the logs stating that it could not send the email (which was a critical alarm). And that makes sense since the WAN port was not yet up so it only makes sense the email could not make it out.

As for syslog…Since its UDP, there is no way for the router to know that the packet never made it to the external syslog server. So it will just keep spitting out logs and eventually when the connection is made on the network, then the syslog server will start reporting, so yes…it makes sense there as well as to why you wont get the initial critical alarms (if there are any).

I can see that there is a problem handling this, but Cisco has solved this. Here is an example message that I get from a switch:

<189> Hemit WS-C3650-48PD NE-02037: 000038: *Apr 14 07:22:09: %SYS-5-RESTART: System restarted --#012Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.5, RELEASE SOFTWARE (fc3)#012Technical Support: http://www.cisco.com/techsupport#012Copyright (c) 1986-2018 by Cisco Systems, Inc.#012Compiled Mon 10-Dec-18 11:34 by mcpre

This was collected in our Splunk from syslog.
I have a view in Splunk that shows when a switch restart and why. (loss of power or reload by user)

Here is a work-around.

Its a script that you can schedule to run every 5 min. (I will add it to my Splunk script)
When script does run it looks if there was a reboot message within the last 5 min, it yes, send a new one.


:local loglistN [:toarray [/log find  time>([/system clock get time] - 5m) message~"rebooted without proper shutdown"]]
:foreach i in=$loglistN do={
	:log error "router was restarted without proper shutdown"
}

@Jotne did you contact support ? Maybe they can implement a TCP session in system logging in a future version…

I did, and here is the reply :slight_smile:

Asked about the missing log and the TCP logging. So, it may help if other request it as well.

Hello,
We will see if that could be implemented if there will be more requests from users. Meanwhile please create custom system to send your logs.
https://wiki.mikrotik.com/wiki/Manual:System/Log
Best regards,

Ok then maybe we will see it in the future…