Community discussions

MikroTik App
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Log Filter

Wed May 24, 2023 1:13 am

Good evening everyone, I took this script from Github and I adapted it to my needs.
It works perfectly, but every day at 00:00 and 02:00, I get the latest notification of the day.

For example, if the latest notification is sent to me at 20:32, every day at 00:00 and at 02:00 it is sent to me again, I don't understand in which part of the script there are problems.


#----------------------------------------
#inserire qui il botID
:local bot "123456789:abcdefghilmnopqrstuvz"
#----------------------------------------
#inserire qui il chatID1 (es: "123456789")
:local ChatID1 "123456789"
#----------------------------------------
#inserire si o no se si vuole utilizzare il chatID2 (es. "si")
:local useChatID2 "si"
#----------------------------------------
#inserire qui il chatID2 (es: "123456789")
:local ChatID2 "123456789"
#----------------------------------------
#inserire si o no se si vuole utilizzare il chatID3 (es: "no")
:local useChatID3 "no"
#----------------------------------------
#inserire qui il chatID3 (es: "123456789")
:local ChatID3 "123456789"
#----------------------------------------
#inserire nome azienda
:local azienda "VidaSPA"
#----------------------------------------
#PAROLE CHIAVE DA CERCARE
#inserire qui la parola chiave 1 
:local key1 "failure"
#----------------------------------------
#inserire qui la parola chiave 2 
:local key2 "loop"
#----------------------------------------
#inserire qui la parola chiave 3 
:local key3 "excessive"
#----------------------------------------
#inserire qui la parola chiave 4 
:local key4 "Netwatch_"
#----------------------------------------
#inserire qui la parola chiave 5 
:local key5 "pppoe-out_EOLO: connected"
#----------------------------------------
#inserire qui la parola chiave 6 
:local key6 "pppoe-out_EOLO: disconnected"
#----------------------------------------
#inserire qui la parola chiave 7 
:local key7 "STATO PEER EDOK="
#----------------------------------------
#inserire qui la parola chiave 8 
:local key8 "DuckDNS: Current IP"
#----------------------------------------
#inserire qui la parola chiave 9 
:local key9 "non usato"
#----------------------------------------
#inserire qui ogni qunto tempo eseguire lo script
:local myRunTime 60s
#----------------------------------------
#inserire qui uno slogan pubblicitario
:local slogan "Sistema di alert by foisfabio.it"



#----------NO MODIFY THIS SECTION----------

#inserire il nome identificativo della macchina
:local myRouterName ([/system identity get name]) 
#----------------------------------------

#----------------------------------------
:local myserver "([/system identity get name])"

:local startBuf [:toarray [/log find message~$key1 || message~$key2 || message~$key3 || message~$key4 || message~$key5 || message~$key6 || message~$key7 || message~$key8 || message~$key9 ]]
#----------------------------------------
:local scheduleName "LogFilter2.0"
:if ([:len [/system scheduler find name="$scheduleName"]] = 0) do={
  /log error "[LogFilter2.0] Alert : lo Scheduler non esiste. Creo lo scheduler"

  /system scheduler add name=$scheduleName interval=$myRunTime start-date=Jan/01/1970 start-time=startup on-event="system script run LogFilter2"

  /log warning "[LogFilter2.0] Alert : Scheduler creato ."
}
#----------------------------------------

# get last time
:local lastTime [/system scheduler get [find name="$scheduleName"] comment]
# for checking time of each log entry
:local currentTime
# log message
:local message
 
# final output
:local output
:local myMac
:local myHost
:local ptr1
:local ptr2
:local emoji "\E2\9C\85"

:local keepOutput false
# if lastTime is empty, set keepOutput to true DISATIVVATO PER PROVA (disativando non funge)
:if ([:len $lastTime] = 0) do={
:set keepOutput true
}


:local counter 0
# loop through all log entries that have been found
:foreach i in=$startBuf do={
 
# loop through all removeThese array items
  :local keepLog true
  :foreach j in=$removeThese do={
#   if this log entry contains any of them, it will be ignored
    :if ([/log get $i message] ~ "$j") do={
      :set keepLog false
    }
  }
  :if ($keepLog = true) do={
   
   :set message [/log get $i message]

#   LOG DATE
#   depending on log date/time, the format may be different. 3 known formats
#   format of jan/01/2002 00:00:00 which shows up at unknown date/time. Using as default
    :set currentTime [ /log get $i time ]
#   format of 00:00:00 which shows up on current day's logs
   :if ([:len $currentTime] = 8 ) do={
     :set currentTime ([:pick [/system clock get date] 0 11]." ".$currentTime)
    } else={
#     format of jan/01 00:00:00 which shows up on previous day's logs
     :if ([:len $currentTime] = 15 ) do={
        :set currentTime ([:pick $currentTime 0 6]."/".[:pick [/system clock get date] 7 11]." ".[:pick $currentTime 7 15])
      }
   }


#   if keepOutput is true, add this log entry to output
   :if ($keepOutput = true) do={
   
      :if ( [:find $message " assigned"] > 0 ) do={
	    :set ptr1 ([:find $message "to"] +3)
	    :set ptr2 [:len $message]
	    :set myMac [:pick $message $ptr1 $ptr2 ]
	    :set myHost [/ip dhcp-server lease get [find where mac-address=$myMac] host-name]
	    :set message ($message." name=".$myHost)
     }
	

      :set output ($output."Data e Ora: ".$currentTime. " \E2\9A\A0 \E2\9A\A0 \E2\9A\A0 \E2\9A\A0--> ".$message."\r\n")
 
  }

    :if ($currentTime = $lastTime) do={
     :set keepOutput true
     :set output ""
   }
  }
  :if ($counter = ([:len $startBuf]-1)) do={
   :if ($keepOutput = false) do={    
     :if ([:len $message] > 0) do={
        :set output ($output."Data e Ora: ".$currentTime. " \E2\9A\A0 \E2\9A\A0 \E2\9A\A0 \E2\9A\A0--> ".$message."\r\n")
      }
    }
  }
  :set counter ($counter + 1)
}


if ([:len $output] > 0) do={

	/system scheduler set [find name="$scheduleName"] comment=$currentTime

	:set output ("Azienda: ".$azienda."\r\n"."Dispositivo: ".$myRouterName."\r\n".$output."\r\n".$slogan)
	 
	:do { 
		:local requestUrl "https://api.telegram.org/bot$bot/sendMessage";
		:local httpData "{\"chat_id\": \"$ChatID1\", \"text\": \"$output\"}";
		/tool fetch url=$requestUrl http-data=$httpData http-header-field=content-type:application/json http-method=post  keep-result=no
	} on-error={
		# log errori http post (disabilitato)
		#:log info "finished with error";
	}
:delay 3s
:if ($useChatID2="si") do={
:local requestUrl "https://api.telegram.org/bot$bot/sendMessage";
		:local httpData "{\"chat_id\": \"$ChatID2\", \"text\": \"$output\"}";
		/tool fetch url=$requestUrl http-data=$httpData http-header-field=content-type:application/json http-method=post  keep-result=no
	} else={
		# log errori http post (disabilitato)
		:log warning "[LogFilter2.0] ChatID2 non utilizzato";
	}
:delay 3s
:if ($useChatID3="si") do={
:local requestUrl "https://api.telegram.org/bot$bot/sendMessage";
		:local httpData "{\"chat_id\": \"$ChatID3\", \"text\": \"$output\"}";
		/tool fetch url=$requestUrl http-data=$httpData http-header-field=content-type:application/json http-method=post  keep-result=no
	} else={
		# log errori http post (disabilitato)
		#:log warning "[LogFilter2.0] ChatID3 non utilizzato";
	}
}
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Wed May 24, 2023 2:00 pm

Bel posto il Lago di Garda.
Il lupo perde il pelo ma non il vizio, ancora con questo metodo dei commenit che usa la NAND invece di usare le variabili globali....
viewtopic.php?p=1003884#p1003398
[…]
:local lastTime [/system scheduler get [find name="$scheduleName"] comment]
[…]
	/system scheduler set [find name="$scheduleName"] comment=$currentTime
[…]
Perhaps it could depend on the fact that at 2 am, Italian time, the format of the logs changes from xx:xx:xx to xxx/xx/xxxx xx:xx:xx and since the comment only says xx:xx:xx the script send again the logs another time?....
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Wed May 24, 2023 2:37 pm

It works perfectly, but every day at 00:00 and 02:00, I get the latest notification of the day.
My script is the same base as yours, but adapted to my needs,
viewtopic.php?t=151953#p1003030

I have the same problem you mentioned: It repeats events of the previous day, same time but with day and month of the current day.



..
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Wed May 24, 2023 3:48 pm

Perhaps it could depend on the fact that at 2 am, Italian time, the format of the logs changes from xx:xx:xx to xxx/xx/xxxx xx:xx:xx and since the comment only says xx:xx:xx the script send again the logs another time?....
???

When I have time I check that, but probably timezone is involved... 2H is actual timezione offset on italy & co.
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Re: Log Filter

Wed May 24, 2023 4:03 pm

ipsec script was copied from this one i found on github. so by copying I also copied the errors. I'll try to change this. more than anything I don't understand how to solve the continuous sending of messages at 00:00 and 02:00
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Re: Log Filter

Wed May 24, 2023 4:04 pm

could this be solved by uncommenting it and using the global variable?
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Wed May 24, 2023 5:24 pm

ipsec script was copied from this one i found on github. so by copying I also copied the errors. I'll try to change this. more than anything I don't understand how to solve the continuous sending of messages at 00:00 and 02:00
My time zone is the same as Italy (from Spain) and the times previously handled by the script are modified thanks to the "anydate2isodate" function (you have it in my previous post) courtesy of @rextended which converts "any" time to full format.

hh:mm:ss -> YYYY-MM-DD hh:mm:ss
MMM/DD hh:mm:ss -> YYYY-MM-DD hh:mm:ss
MMM/DD/YYYY hh:mm:ss -> YYYY-MM-DD hh:mm:ss
...
etc.

I also don't understand the problem of repeating events before 00:00 and 02:00, the rest of the day works OK.
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Wed May 24, 2023 5:27 pm

Hello again,

I have tested this code (original script) without any modification (only Telegram token and chat_id) and made a simulation by setting the router time to 23:58:00. When reaching 00:00:00:00 no event from the previous day has been repeated, it has worked as expected.

@abbio90, try with this code to see if it works well for you.
:local myserver ("\E2\84\B9"." "."MikroTik"." ".[/system identity get name])
:local scheduleName "LogFilter"
:local bot          "XXXXXXXXXXXXXXXX:XXXXXXXXXXX-XXXXXXXXXXXXXXXXXXXXXX"
:local ChatID       "XXXXXXXXX"
:local startBuf     [:toarray [/log find message~"login failure" || message~"logged in" || message~"rebooted" || message~"loop" || message~"fcs" || message~"excessive" || message~"system"]]
:local removeThese  [:toarray ""]

:if ([:len [/system scheduler find name="$scheduleName"]] = 0) do={
    /log warning "[LogFilter] Alert : Schedule does not exist. Creating schedule ...."
    /system scheduler add name=$scheduleName interval=60s start-date=may/24/2023 start-time=12:00:00 on-event=LogFilter
    /log warning "[LogFilter] Alert : Schedule created ."
}

:local lastTime     [/system scheduler get [find name="$scheduleName"] comment]
:local currentTime  ""
:local message      ""
:local output       ""

:local keepOutput false
:if ([:len $lastTime] = 0) do={:set keepOutput true}

:local counter 0

:foreach i in=$startBuf do={

    :local keepLog true
    :foreach j in=$removeThese do={
        :if ([/log get $i message] ~ "$j") do={:set keepLog false}
    }

    :if ($keepLog = true) do={
        :set message [/log get $i message]
        :set currentTime [ /log get $i time ]

        :if ([:len $currentTime] = 8 ) do={
            :set currentTime ([:pick [/system clock get date] 0 11]." ".$currentTime)
        } else={
            :if ([:len $currentTime] = 15 ) do={
                :set currentTime ([:pick $currentTime 0 6]."/".[:pick [/system clock get date] 7 11]." ".[:pick $currentTime 7 15])
            }
        }

        :if ($keepOutput = true) do={:set output ($output.$currentTime." ".$message." %0A%0A ")}
        :if ($currentTime = $lastTime) do={:set keepOutput true; :set output ""}
    }                                                                                                                            

    :if ($counter = ([:len $startBuf])-1) do={
        :if ($keepOutput = false) do={
            :if ([:len $message] > 0) do={:set output ($output.$currentTime." ".$message)}
        }
    }

    :set counter ($counter + 1)
}

if ([:len $output] > 0) do={
    /system scheduler set [find name="$scheduleName"] comment=$currentTime
    /tool fetch url="https://api.telegram.org/bot$bot/sendMessage\?chat_id=$ChatID&text=$myserver%0A%0A$output" keep-result=no
}
Maybe @rextended is right and the problem is going to be the dates, I can't find another explanation.

BR.
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Thu May 25, 2023 10:22 am

Well, the test performed tonight with the original script has been failed, some events of the previous day have been repeated tonight at 00:00 and 02:00.

I think the problem then cannot be the date. I don't know where it fails.
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Re: Log Filter

Thu May 25, 2023 5:39 pm

yes, I uncommented it and save to a global variable, but the result is the same. In your opinion, changing the rules of the date and updated to 7.10 with the new ISO format of the date, could it be solved?
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Thu May 25, 2023 6:06 pm

yes, I uncommented it and save to a global variable, but the result is the same. In your opinion, changing the rules of the date and updated to 7.10 with the new ISO format of the date, could it be solved?
On a Miktotik with v7.9.1 I tried tonight to leave the date format of the original script: may/25/2023 00:00:00:00 and the events repeated again at 00:00 and
02:00. Next night I will try on a Mikrotik, original script with v7.10beta+ to see the results.

I'm no expert, but I think you can only make progress by trying different options.
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Fri May 26, 2023 1:02 am

Nothing, the same problem persists.
:(
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Fri May 26, 2023 1:09 am

try to set the device to GMT +00:00...... (and disable auto-detect)
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Re: Log Filter

Fri May 26, 2023 1:14 am

Are you saying it could be a time zone related system? What if you try to insert a row where at 00:00 and 02:00 set the keepOutput variable to false?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Fri May 26, 2023 10:46 am

From post #4:
[…] probably timezone is involved... 2H is actual timezione offset on italy & co.
I still do not have time to test it.
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Fri May 26, 2023 8:43 pm

try to set the device to GMT +00:00...... (and disable auto-detect)
Ok, I'll do the test tonight.

..
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Fri May 26, 2023 8:45 pm

try to set the device to GMT +00:00...... (and disable auto-detect)
Ok, I'll do the test tonight.

..
Thanks
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Sat May 27, 2023 11:16 am

try to set the device to GMT +00:00...... (and disable auto-detect)
Good morning, I have already performed the tests with the following conditions:

a) Configured System > Clock
- Time: Local time
- Date: Current date
- Time Zone Autodetected: Unchecked
- Time Zone Name: GMT+0

b) System > NTP Server: Unchecked

c) System > NTP Client: Unchecked

d) IP > Cloud
- DDNS Enabled: Checked
- Update Time: Checked

Result: Some LogFilter events at 00:00 and 02:00 were not repeated.

The recommendation of @rextended is good (thanks) but I don't know if the Time Zone configuration could be done in the script itself instead of configuring it in System > Clock so as not to affect the rest of Mikrotik services.

BR.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sat May 27, 2023 12:16 pm

for sure is possible avoid duplication using log index instead of last date.... the original script is not mine and I do not like it... ;)

select * from log where id > last id ;)
 
S8T8
newbie
Posts: 27
Joined: Thu Sep 15, 2022 7:15 pm

Re: Log Filter

Sat May 27, 2023 12:32 pm

Hello @diamuxin, I was reading your previous post and noticed that the script is looking for the "last log" for the date that was written on the scheduled comment.
Druvis from MikroTik wrote a script that may help you to fix the problem and you could try to your integrate: https://github.com/Druvis-Timma/Mikroti ... coding.rsc
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sat May 27, 2023 12:52 pm

Hello @diamuxin, I was reading your previous post and noticed that the script is looking for the "last log" for the date that was written on the scheduled comment.
Druvis from MikroTik wrote a script that may help you to fix the problem and you could try to your integrate: https://github.com/Druvis-Timma/Mikroti ... coding.rsc
Completely useless in the context of this topic.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sat May 27, 2023 1:07 pm

one telegram message for each new log entry code

:global anydate2isodate do={
    :local dtime [:tostr $1]
    /system clock
    :local cyear [get date] ; :if ($cyear ~ "....-..-..") do={:set cyear [:pick $cyear 0 4]} else={:set cyear [:pick $cyear 7 11]}
    :if (([:len $dtime] = 10) or ([:len $dtime] = 11)) do={:set dtime "$dtime 00:00:00"}
    :if ([:len $dtime] = 15) do={:set dtime "$[:pick $dtime 0 6]/$cyear $[:pick $dtime 7 15]"}
    :if ([:len $dtime] = 14) do={:set dtime "$cyear-$[:pick $dtime 0 5] $[:pick $dtime 6 14]"}
    :if ([:len $dtime] =  8) do={:set dtime "$[get date] $dtime"}
    :if ([:tostr $1] = "") do={:set dtime ("$[get date] $[get time]")}
    :local vdoff [:toarray "0,4,5,7,8,10,11,19"]
    :local MM    [:pick $dtime ($vdoff->2) ($vdoff->3)]
    :if ($dtime ~ ".../../....") do={
        :set vdoff [:toarray "7,11,1,3,4,6,12,20"]
        :set MM    ([:find "xxanebarprayunulugepctovecANEBARPRAYUNULUGEPCTOVEC" [:pick $dtime ($vdoff->2) ($vdoff->3)] -1] / 2)
        :if ($MM>12) do={:set MM ($MM - 12)} ; :if ($MM<10) do={:set MM "0$MM"}
    }
    :return "$[:pick $dtime ($vdoff->0) ($vdoff->1)]-$MM-$[:pick $dtime ($vdoff->4) ($vdoff->5)] $[:pick $dtime ($vdoff->6) ($vdoff->7)]"
}

:global UTF8toURLencode do={
    :local ascii "\00\01\02\03\04\05\06\07\08\09\0A\0B\0C\0D\0E\0F\
                  \10\11\12\13\14\15\16\17\18\19\1A\1B\1C\1D\1E\1F\
                  \20\21\22\23\24\25\26\27\28\29\2A\2B\2C\2D\2E\2F\
                  \30\31\32\33\34\35\36\37\38\39\3A\3B\3C\3D\3E\3F\
                  \40\41\42\43\44\45\46\47\48\49\4A\4B\4C\4D\4E\4F\
                  \50\51\52\53\54\55\56\57\58\59\5A\5B\5C\5D\5E\5F\
                  \60\61\62\63\64\65\66\67\68\69\6A\6B\6C\6D\6E\6F\
                  \70\71\72\73\74\75\76\77\78\79\7A\7B\7C\7D\7E\7F\
                  \80\81\82\83\84\85\86\87\88\89\8A\8B\8C\8D\8E\8F\
                  \90\91\92\93\94\95\96\97\98\99\9A\9B\9C\9D\9E\9F\
                  \A0\A1\A2\A3\A4\A5\A6\A7\A8\A9\AA\AB\AC\AD\AE\AF\
                  \B0\B1\B2\B3\B4\B5\B6\B7\B8\B9\BA\BB\BC\BD\BE\BF\
                  \C0\C1\C2\C3\C4\C5\C6\C7\C8\C9\CA\CB\CC\CD\CE\CF\
                  \D0\D1\D2\D3\D4\D5\D6\D7\D8\D9\DA\DB\DC\DD\DE\DF\
                  \E0\E1\E2\E3\E4\E5\E6\E7\E8\E9\EA\EB\EC\ED\EE\EF\
                  \F0\F1\F2\F3\F4\F5\F6\F7\F8\F9\FA\FB\FC\FD\FE\FF"
    :local UTF8toURLe {"00";"01";"02";"03";"04";"05";"06";"07";"08";"09";"0A";"0B";"0C";"0D";"0E";"0F";
                       "10";"11";"12";"13";"14";"15";"16";"17";"18";"19";"1A";"1B";"1C";"1D";"1E";"1F";
                       "+";"21";"22";"23";"24";"25";"26";"27";"28";"29";"2A";"2B";"2C";"-";".";"2F";
                       "0";"1";"2";"3";"4";"5";"6";"7";"8";"9";"3A";"3B";"3C";"3D";"3E";"3F";
                       "40";"A";"B";"C";"D";"E";"F";"G";"H";"I";"J";"K";"L";"M";"N";"O";
                       "P";"Q";"R";"S";"T";"U";"V";"W";"X";"Y";"Z";"5B";"5C";"5D";"5E";"_";
                       "60";"a";"b";"c";"d";"e";"f";"g";"h";"i";"j";"k";"l";"m";"n";"o";
                       "p";"q";"r";"s";"t";"u";"v";"w";"x";"y";"z";"7B";"7C";"7D";"~";"7F";
                       "80";"81";"82";"83";"84";"85";"86";"87";"88";"89";"8A";"8B";"8C";"8D";"8E";"8F";
                       "90";"91";"92";"93";"94";"95";"96";"97";"98";"99";"9A";"9B";"9C";"9D";"9E";"9F";
                       "A0";"A1";"A2";"A3";"A4";"A5";"A6";"A7";"A8";"A9";"AA";"AB";"AC";"AD";"AE";"AF";
                       "B0";"B1";"B2";"B3";"B4";"B5";"B6";"B7";"B8";"B9";"BA";"BB";"BC";"BD";"BE";"BF";
                       "C0";"C1";"C2";"C3";"C4";"C5";"C6";"C7";"C8";"C9";"CA";"CB";"CC";"CD";"CE";"CF";
                       "D0";"D1";"D2";"D3";"D4";"D5";"D6";"D7";"D8";"D9";"DA";"DB";"DC";"DD";"DE";"DF";
                       "E0";"E1";"E2";"E3";"E4";"E5";"E6";"E7";"E8";"E9";"EA";"EB";"EC";"ED";"EE";"EF";
                       "F0";"F1";"F2";"F3";"F4";"F5";"F6";"F7";"F8";"F9";"FA";"FB";"FC";"FD";"FE";"FF"
                      }
    :local string $1
    :if (([:typeof $string] != "str") or ($string = "")) do={ :return "" }
    :local lenstr [:len $string]
    :local constr ""
    :for pos from=0 to=($lenstr - 1) do={
        :local urle ($UTF8toURLe->[:find $ascii [:pick $string $pos ($pos + 1)] -1])
        :local sym $urle
        :if ([:len $urle] = 2) do={:set sym "%$[:pick $urle 0 2]" }
        :set constr "$constr$sym"
    }
    :return $constr
}

:global lastLog
:if ([:typeof $lastLog] != "num") do={:set lastLog 0}

{
:local tgBot    "XXXXXXXXXXXXXXXX:XXXXXXXXXXX-XXXXXXXXXXXXXXXXXXXXXX"
:local tgChatID "XXXXXXXXX"
:local tgPrefix "\E2\84\B9 MikroTik $[/system identity get name] $[/system resource get board-name]"
:local wtopics  "critical|error|warning"
:local utopics  "ipsec"
:local mkwd     "login failure|logged in|loop|down|fcs|excessive|system|rebooted|ipsec|ike2"
:local ukwd     "unwanted|example|phase1 negotiation failed"

:local id2num do={:return [:tonum "0x$[:pick $1 1 [:len $1]]"]}
:local tgmessg ""
:local emotico ""
:local temp    ""

/log
:foreach item in=[find where (((([$id2num $".id"] > $lastLog) and (buffer=memory)) and (!([:tostr $topics]~"($utopics)"))) \
                             and \
                             ((([:tostr $topics]~"($wtopics)") or (message~"($mkwd)")) and (!(message~"($ukwd)"))))] do={
    :set lastLog [$id2num $item]
    :set emotico ""
    :local temp [:tostr [get $item topics]]
    :if ($temp~"critical") do={:set emotico "\E2\98\A0\20"}
    :if ($temp~"error")    do={:set emotico "$emotico\E2\9D\8C\20"}
    :if ($temp~"warning")  do={:set emotico "$emotico\E2\9A\A0\20"}
    :set tgmessg [$UTF8toURLencode ("$tgPrefix\r\n$[$anydate2isodate [get $item time]] $emotico$[get $item message]")]
    /tool fetch url="https://api.telegram.org/bot$tgBot/sendMessage\3Fchat_id=$tgChatID&text=$tgmessg" keep-result=no
}

}
EDIT 1,2,3 and 4: bugfix
EDIT 5: Separate the two version of the script on 2 posts.
EDIT 6: https://forum.mikrotik.com/viewtopic.php?p=1004648#p1004648
EDIT 7: Exclude on default ipsec error messages phase1 negotiation failed, added the possibility to exclude topics regardless are critical or error or warning
(thanks to @S8T8 for the idea)
EDIT 8: Misspell error
Last edited by rextended on Tue May 30, 2023 8:25 pm, edited 8 times in total.
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Sat May 27, 2023 2:39 pm

Impressive work Rex, thank you very much, really.

I'm away with the family right now, I'll try it as soon as I can.

BR.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sat May 27, 2023 2:41 pm

one unique telegram message with all new log entry code

:global anydate2isodate do={
    :local dtime [:tostr $1]
    /system clock
    :local cyear [get date] ; :if ($cyear ~ "....-..-..") do={:set cyear [:pick $cyear 0 4]} else={:set cyear [:pick $cyear 7 11]}
    :if (([:len $dtime] = 10) or ([:len $dtime] = 11)) do={:set dtime "$dtime 00:00:00"}
    :if ([:len $dtime] = 15) do={:set dtime "$[:pick $dtime 0 6]/$cyear $[:pick $dtime 7 15]"}
    :if ([:len $dtime] = 14) do={:set dtime "$cyear-$[:pick $dtime 0 5] $[:pick $dtime 6 14]"}
    :if ([:len $dtime] =  8) do={:set dtime "$[get date] $dtime"}
    :if ([:tostr $1] = "") do={:set dtime ("$[get date] $[get time]")}
    :local vdoff [:toarray "0,4,5,7,8,10,11,19"]
    :local MM    [:pick $dtime ($vdoff->2) ($vdoff->3)]
    :if ($dtime ~ ".../../....") do={
        :set vdoff [:toarray "7,11,1,3,4,6,12,20"]
        :set MM    ([:find "xxanebarprayunulugepctovecANEBARPRAYUNULUGEPCTOVEC" [:pick $dtime ($vdoff->2) ($vdoff->3)] -1] / 2)
        :if ($MM>12) do={:set MM ($MM - 12)} ; :if ($MM<10) do={:set MM "0$MM"}
    }
    :return "$[:pick $dtime ($vdoff->0) ($vdoff->1)]-$MM-$[:pick $dtime ($vdoff->4) ($vdoff->5)] $[:pick $dtime ($vdoff->6) ($vdoff->7)]"
}

:global UTF8toURLencode do={
    :local ascii "\00\01\02\03\04\05\06\07\08\09\0A\0B\0C\0D\0E\0F\
                  \10\11\12\13\14\15\16\17\18\19\1A\1B\1C\1D\1E\1F\
                  \20\21\22\23\24\25\26\27\28\29\2A\2B\2C\2D\2E\2F\
                  \30\31\32\33\34\35\36\37\38\39\3A\3B\3C\3D\3E\3F\
                  \40\41\42\43\44\45\46\47\48\49\4A\4B\4C\4D\4E\4F\
                  \50\51\52\53\54\55\56\57\58\59\5A\5B\5C\5D\5E\5F\
                  \60\61\62\63\64\65\66\67\68\69\6A\6B\6C\6D\6E\6F\
                  \70\71\72\73\74\75\76\77\78\79\7A\7B\7C\7D\7E\7F\
                  \80\81\82\83\84\85\86\87\88\89\8A\8B\8C\8D\8E\8F\
                  \90\91\92\93\94\95\96\97\98\99\9A\9B\9C\9D\9E\9F\
                  \A0\A1\A2\A3\A4\A5\A6\A7\A8\A9\AA\AB\AC\AD\AE\AF\
                  \B0\B1\B2\B3\B4\B5\B6\B7\B8\B9\BA\BB\BC\BD\BE\BF\
                  \C0\C1\C2\C3\C4\C5\C6\C7\C8\C9\CA\CB\CC\CD\CE\CF\
                  \D0\D1\D2\D3\D4\D5\D6\D7\D8\D9\DA\DB\DC\DD\DE\DF\
                  \E0\E1\E2\E3\E4\E5\E6\E7\E8\E9\EA\EB\EC\ED\EE\EF\
                  \F0\F1\F2\F3\F4\F5\F6\F7\F8\F9\FA\FB\FC\FD\FE\FF"
    :local UTF8toURLe {"00";"01";"02";"03";"04";"05";"06";"07";"08";"09";"0A";"0B";"0C";"0D";"0E";"0F";
                       "10";"11";"12";"13";"14";"15";"16";"17";"18";"19";"1A";"1B";"1C";"1D";"1E";"1F";
                       "+";"21";"22";"23";"24";"25";"26";"27";"28";"29";"2A";"2B";"2C";"-";".";"2F";
                       "0";"1";"2";"3";"4";"5";"6";"7";"8";"9";"3A";"3B";"3C";"3D";"3E";"3F";
                       "40";"A";"B";"C";"D";"E";"F";"G";"H";"I";"J";"K";"L";"M";"N";"O";
                       "P";"Q";"R";"S";"T";"U";"V";"W";"X";"Y";"Z";"5B";"5C";"5D";"5E";"_";
                       "60";"a";"b";"c";"d";"e";"f";"g";"h";"i";"j";"k";"l";"m";"n";"o";
                       "p";"q";"r";"s";"t";"u";"v";"w";"x";"y";"z";"7B";"7C";"7D";"~";"7F";
                       "80";"81";"82";"83";"84";"85";"86";"87";"88";"89";"8A";"8B";"8C";"8D";"8E";"8F";
                       "90";"91";"92";"93";"94";"95";"96";"97";"98";"99";"9A";"9B";"9C";"9D";"9E";"9F";
                       "A0";"A1";"A2";"A3";"A4";"A5";"A6";"A7";"A8";"A9";"AA";"AB";"AC";"AD";"AE";"AF";
                       "B0";"B1";"B2";"B3";"B4";"B5";"B6";"B7";"B8";"B9";"BA";"BB";"BC";"BD";"BE";"BF";
                       "C0";"C1";"C2";"C3";"C4";"C5";"C6";"C7";"C8";"C9";"CA";"CB";"CC";"CD";"CE";"CF";
                       "D0";"D1";"D2";"D3";"D4";"D5";"D6";"D7";"D8";"D9";"DA";"DB";"DC";"DD";"DE";"DF";
                       "E0";"E1";"E2";"E3";"E4";"E5";"E6";"E7";"E8";"E9";"EA";"EB";"EC";"ED";"EE";"EF";
                       "F0";"F1";"F2";"F3";"F4";"F5";"F6";"F7";"F8";"F9";"FA";"FB";"FC";"FD";"FE";"FF"
                      }
    :local string $1
    :if (([:typeof $string] != "str") or ($string = "")) do={ :return "" }
    :local lenstr [:len $string]
    :local constr ""
    :for pos from=0 to=($lenstr - 1) do={
        :local urle ($UTF8toURLe->[:find $ascii [:pick $string $pos ($pos + 1)] -1])
        :local sym $urle
        :if ([:len $urle] = 2) do={:set sym "%$[:pick $urle 0 2]" }
        :set constr "$constr$sym"
    }
    :return $constr
}

:global lastLog
:if ([:typeof $lastLog] != "num") do={:set lastLog 0}

{
:local tgBot    "XXXXXXXXXXXXXXXX:XXXXXXXXXXX-XXXXXXXXXXXXXXXXXXXXXX"
:local tgChatID "XXXXXXXXX"
:local tgPrefix "\E2\84\B9 MikroTik $[/system identity get name] $[/system resource get board-name]"
:local wtopics  "critical|error|warning"
:local utopics  "ipsec"
:local mkwd     "login failure|logged in|loop|down|fcs|excessive|system|rebooted|ipsec|ike2"
:local ukwd     "unwanted|example|phase1 negotiation failed"

:local id2num do={:return [:tonum "0x$[:pick $1 1 [:len $1]]"]}
:local tgmessg ""
:local emotico ""
:local temp    ""

/log
:foreach item in=[find where (((([$id2num $".id"] > $lastLog) and (buffer=memory)) and (!([:tostr $topics]~"($utopics)"))) \
                             and \
                             ((([:tostr $topics]~"($wtopics)") or (message~"($mkwd)")) and (!(message~"($ukwd)"))))] do={
    :set lastLog [$id2num $item]
    :set emotico ""
    :local temp [:tostr [get $item topics]]
    :if ($temp~"critical") do={:set emotico "\E2\98\A0\20"}
    :if ($temp~"error")    do={:set emotico "$emotico\E2\9D\8C\20"}
    :if ($temp~"warning")  do={:set emotico "$emotico\E2\9A\A0\20"}
    :set tgmessg "$tgmessg$[$anydate2isodate [get $item time]] $emotico$[get $item message]\r\n"
}

:if ($tgmessg != "") do={
    :set tgmessg [$UTF8toURLencode ("$tgPrefix\r\n$tgmessg")]
    /tool fetch url="https://api.telegram.org/bot$tgBot/sendMessage\3Fchat_id=$tgChatID&text=$tgmessg" keep-result=no
}

}
EDIT 1: https://forum.mikrotik.com/viewtopic.php?p=1004648#p1004648
EDIT 2: Exclude on default ipsec error messages phase1 negotiation failed, added the possibility to exclude topics regardless are critical or error or warning
(thanks to @S8T8 for the idea)
EDIT 3: Misspell fixed
Last edited by rextended on Tue May 30, 2023 8:22 pm, edited 3 times in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sat May 27, 2023 2:43 pm

Impressive work Rex, thank you very much, really.
Thanks, I hope is error free. I do not have Telegram, but the URL "text" is correctly generated.
If you ignore the date conversion and the correct urlencode, are less lines of code for do the same thing... ;)
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Sun May 28, 2023 2:28 am

Impressive work Rex, thank you very much, really.
Thanks, I hope is error free. I do not have Telegram, but the URL "text" is correctly generated.
If you ignore the date conversion and the correct urlencode, are less lines of code for do the same thing... ;)
I confirm that your script works perfect, it is much better to work on the last Id.
I love how summarized the code is. Thanks again.
 
S8T8
newbie
Posts: 27
Joined: Thu Sep 15, 2022 7:15 pm

Re: Log Filter

Sun May 28, 2023 4:04 am

@rextended testing right now, works also for me!
This should be a pinned topic...
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sun May 28, 2023 1:39 pm

Since I never seen at same time critical that is also error or warning on same log, no matter, but I want fix code for this.
from
:if ($temp~"critical") do={:set emotico "\E2\98\A0\20"}
:if ($temp~"error") do={:set emotico "\E2\9D\8C\20"}
:if ($temp~"warning") do={:set emotico "\E2\9A\A0\20"}
to
:if ($temp~"critical") do={:set emotico "\E2\98\A0\20"}
:if ($temp~"error") do={:set emotico "$emotico\E2\9D\8C\20"}
:if ($temp~"warning") do={:set emotico "$emotico\E2\9A\A0\20"}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sun May 28, 2023 1:44 pm

@rextended testing right now, works also for me!
This should be a pinned topic...
Thanks, added to my "Fragments of Snippets":
viewtopic.php?p=1004649#p1004649
 
S8T8
newbie
Posts: 27
Joined: Thu Sep 15, 2022 7:15 pm

Re: Log Filter

Sun May 28, 2023 7:13 pm

Also if I don't have any knowledge in scripts I applied few edits;
- added a dash " - " before a new line of log sent on the text
Example:
- 2023-05-28 12:20:46 ether3 link down
- 2023-05-28 12:21:03 ether3 link up (speed 100M, full duplex)

- changed: and !(message~" -> and !(topic~"
I fell that is easier for me to exclude topics instead of content of the log.

- it's possibile (and easy) to have just the time instead of the full date in the text? (Only if is provided by the script, don't want to waste your time)
Example:
- 12:20:46 ether3 link down

Have a nice Sunday!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Sun May 28, 2023 7:43 pm

- changed: and !(message~" -> and !(topic~"
I fell that is easier for me to exclude topics instead of content of the log.
Nice Idea, I consider that on next version.
But doing that in your way, you can accidentally avoid the reporting of critical / error logs if are critical/error at the same time of your unwanted topic...

- it's possibile (and easy) to have just the time instead of the full date in the text? (Only if is provided by the script, don't want to waste your time)
Example:
- 12:20:46 ether3 link down
Yes, replace the line
:return "$[:pick $dtime ($vdoff->0) ($vdoff->1)]-$MM-$[:pick $dtime ($vdoff->4) ($vdoff->5)] $[:pick $dtime ($vdoff->6) ($vdoff->7)]"
with what you want, like
:return "$[:pick $dtime ($vdoff->6) ($vdoff->7)]"
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Re: Log Filter

Mon May 29, 2023 8:57 am

sorry guys but i've been busy. I'll try as soon as I have time. but this script left in chat last, send any event log?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Mon May 29, 2023 10:45 am

sorry guys but i've been busy. I'll try as soon as I have time. but this script left in chat last, send any event log?
Both on post #22 and post #24, choose your preferred version
 
abbio90
Member Candidate
Member Candidate
Topic Author
Posts: 129
Joined: Fri Aug 27, 2021 9:16 pm
Contact:

Re: Log Filter

Tue May 30, 2023 7:40 pm

I tried the script and I adapted it to my needs, and I would say that it works very well.. even if some error alerts are annoying and not very sensible like the ipsec phase 1 which fails when the router fails over, it doesn't make sense. .how could I do to create a list of phrases that should not be communicated on telegram? this script I understand will also work from version 7.10 with the ISO adaptations of the date and time. Right?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Tue May 30, 2023 7:46 pm

will also work from version 7.10 with the ISO adaptations of the date and time. Right?
Yes


Both scripts are updated for:
EDIT x: Exclude on default ipsec error messages phase1 negotiation failed, added the possibility to exclude topics regardless are critical or error or warning
(thanks to @S8T8 for the idea)
 
User avatar
diamuxin
Member Candidate
Member Candidate
Posts: 209
Joined: Thu Sep 09, 2021 5:46 pm

Re: Log Filter

Tue May 30, 2023 9:04 pm

Both scripts are updated for:
EDIT x: Exclude on default ipsec error messages phase1 negotiation failed, added the possibility to exclude topics regardless are critical or error or warning
(thanks to @S8T8 for the idea)
Updated! thx

..
 
S8T8
newbie
Posts: 27
Joined: Thu Sep 15, 2022 7:15 pm

Re: Log Filter

Wed May 31, 2023 1:25 am

EDIT 2: Exclude on default ipsec error messages phase1 negotiation failed, added the possibility to exclude topics regardless are critical or error or warning
EDIT 3: Misspell fixed
Nice update, this was exactly what I meant!
You mind to spend few words on the last Edit?
Hope to have other suggestion to share.
THANKS!!!

FYI, I'm using it as Scheduler running every 10min.
As I'm not a fan of Telegram I'll check if possible an easy integration with other solutions like Elements (Matrix).
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 10797
Joined: Tue Feb 25, 2014 12:49 pm
Location: Inside the Cat's Eye
Contact:

Re: Log Filter

Wed May 31, 2023 1:29 am

You mind to spend few words on the last Edit?
Really? :lol:

Exclude on defoult ipsec ==>> Exclude on default ipsec

Who is online

Users browsing this forum: DyadyaGenya and 8 guests