Page 1 of 1

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:00 pm
by Hunty
I've attached two screenshot

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:03 pm
by Hunty
please note that I've inserted "Mikrotik" under System/Logging

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:13 pm
by Jotne
All apps needs to be in
$SPLUNK_HOME/etc/apps
So on windows you should have:
C:\Program Files\Splunk\etc\apps\MikroTik

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:14 pm
by Hunty
All apps needs to be in
$SPLUNK_HOME/etc/apps
So on windows you should have:
C:\Program Files\Splunk\etc\apps\MikroTik
Yes

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:16 pm
by Jotne
Uninstall everything.
Install follow post #1 step by step.

You should also see
C:\Program Files\Splunk\etc\apps\MikroTik\default
C:\Program Files\Splunk\etc\apps\MikroTik\metadata
etc
Not
C:\Program Files\Splunk\etc\apps\MikroTik\MikroTik\default
If that does not work, I will try to do an install my self from the #1 post and test it.

PS no need to quote post above you.

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:23 pm
by Hunty
are you sure?
I've attached a screenshot of the content of the folder

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:43 pm
by Jotne
You have restarted Splunk after app install?
All looks correct.

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:50 pm
by Hunty
Yes several times

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 8:55 pm
by Jotne
You have installed Splunk free as in post #1, or do you use Splunk before to some else?
I have seen problem with installed version that using other index.

From the picture above, it does not seem that splunk does the filed extraction.

If you like, I can try teamviewer to see what is wrong.
Not able to post a private message to you, so post an email so I can get in touch with you.

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Nov 15, 2018 9:08 pm
by Hunty
thanks for your help, but I'll try tomorrow with the linux VM but I've to solve first why the 514 port is not available even if I followed your guide to install the app with a non root user

Re: Using Splunk to analyse MikroTik logs 2.4 (Graphing everything)

Posted: Tue Nov 20, 2018 12:20 pm
by Jotne
2.4 Released

Nearly all code are rewritten to get better speed and make it cleaner.
Dark Theme makes a big visual change.

# v2.4 (20.11.2018)
# Updated "MikroTik Hotspot login/logout information" to show IP
# Fixed when inn interface= unknown
# Updated view 2.4 to handel more hits
# Updated "MikroTik DNS" to not view revers lookup "site!=*.in-addr.arpa"
# Rewritten "Microtik Traffic" Error in all calculation
# Fixed data rounding and fixed typo
# Fixed formating in "MikroTik Remote Connection"
# Set permission view the view to show in app only
# Added System Changes as a new default menu
# Fixed missing host in "MikroTik Uptime"
# Added Host to "MikroTik Traffic"
# Added view "MikroTik Wifi strength"
# Added view "MikroTik System Changes"
# Dark theme needs >=7.2
# Removed global time (use default time)
# Removed searchWhenChanged="true" (default)
# Cleaned code
# Fixed error in "13. OSCam config changes"
# Added Sprakline to "MikroTik Device List"
2.4 Device list.jpg
.
2.4 System Changes.jpg
.
2.4 Traffic.jpg

Re: Using Splunk to analyse MikroTik logs 2.4 (Graphing everything)

Posted: Sat Nov 24, 2018 12:06 am
by jareckib12
Hi,
First - thx for update.
Second - in MikroTik DNS request view, client filtering does not work. When selecting any item in addition to "any" does not show any results.

Jarecki

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sun Nov 25, 2018 10:40 am
by Jotne
Good catch,

I have updated to 2.5

# 2.5 (25.11.2018)
# Change all "if" test to use "coalesce"
# Fixed error in "MikroTik DNS request"
# Moved more to base search
# Removed some code not needed in "MikroTik Web Proxy"
# Fixed error with src_port in dest_ip dropdown in "MikroTik Firewall Rules"

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sun Nov 25, 2018 9:26 pm
by sherics
Hello,

I have installed it completely and except the Traffic, everything work.

In the traffic I see just few MBs, even if I download 500MB or 1GB, it does not shows up there, just few % of the downloaded amount.

I do not have a public IP on my internal network, the public IP is on the WAN port, ether1, as a standard home router, other clients are on WiFi on first VLAN and 2 computers on second VLAN.

Do you have an idea what is wrong?

Thank you.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Nov 26, 2018 6:08 pm
by Jotne
I did download an 1GB file from here: http://www.ovh.net/files/
And it showed up correctly.

Do you have Fasttrack on?
If so try to disable it, it may be that packed are not accounted when Fasttrack is on.
https://www.youtube.com/watch?v=6LaqhDm6PHI

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Nov 26, 2018 8:45 pm
by MSandoval
Hello Jotne and the whole community.
First I want to tell you, good job, really good jobs, and thanks for sharing with us Jotne.

Secondly I have a question, in version 2.4 I see in the record that wrote "List of devices" this function indicates that it already supports multi-router log ?, in such case as it is identified in each module to which router belongs each record?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Nov 26, 2018 9:32 pm
by Jotne
You are correct, it does support as many routers as you like to add.
Going away from SNMP to Syslog only was driven by the simpler way to do thing.
With SNMP, you need to set up the monitor system to request SNMP from the device.
This is ok for singel router ans small system.
But if you like to monitor a router across public internet, you end up in a security risk by open for SNMP.

Whit using script and Syslog this is a one way communication. All data are sent from the device to the monitor system.
No need to open ports. Same script for all routers. No need to configure any configuration on the monitoring system for each router.

I have four routers/host that sends log to my sentral log server.

On every view you can select host to view only that host.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Nov 26, 2018 9:41 pm
by MSandoval
Great, you're right, forget that each module has a drop-down menu Hosts. I'm going to try it and anything I write. Thanks again.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Nov 26, 2018 10:59 pm
by sherics
I did download an 1GB file from here: http://www.ovh.net/files/
And it showed up correctly.

Do you have Fasttrack on?
If so try to disable it, it may be that packed are not accounted when Fasttrack is on.
https://www.youtube.com/watch?v=6LaqhDm6PHI
Well, I forgot about fastrack... without fastrack it works now, but unfortunately without fastrack my router is on 95-99% CPU while I download/upload anything; and the speed is lowered for 300mbit/s... With fastrack enable, the cpu is approx on 70% on full gigabit connection, about 90MB/s real speed. Well, after 4 years, I think, I need to purchase a more powerul and new router :)

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Nov 26, 2018 11:22 pm
by Jotne
It may also be that you could configure your router to use hardware offloading. Depending on type and software version.
But old boxes do have less power so upgrade may be the only option.

Its a good point to now that traffic monitoring does not work when fast track is enabled, so I will mention that in the first post.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Fri Nov 30, 2018 3:45 pm
by MSandoval
Hello everyone, I have a problem with module MikroTik_Traffic section Public IP. when reviewing this, I found a small error when declaring the variable host, in this case that variable is capitalized Host, it does that the section does not work, changing this I achieved that it works correctly.
<title>Public IP</title>
        <search base="base_search">
          <query>
            search
              Host="$Host$"         >>>   change with host="$Host$"
            | eval ip_in=if("$direction$"=="in",src_address,dst_address)
            | eval ip_out=if("$direction$"!="in",src_address,dst_address)
Image

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Fri Nov 30, 2018 5:28 pm
by Jotne
Thanks for the feedback :)

It will be fixed in 2.6. For others you can edit det file and correct the typo.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Fri Dec 07, 2018 4:18 am
by ariefwido
Hello there,

This is my first time using splunk and I have no result on dashboard anyway also I did every step on the post #1, any idea why this happen?
The logs already show up on the splunk but the MikroTik app dashboard have no result at all.

Thanks and appreciate your help.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Fri Dec 07, 2018 11:51 pm
by Jotne
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug
I would guess you have typed wrong prefix. Any other word than MikroTik would brake the index of the data.
Make sure its 100% equal with capital M and K

Cut and Past is the best option to get it correct.

Do a search like this in Splunk, change to your MikroTik Routers IP, what is the output?
index=* host=192.168.88.1 | rex "^\S+\s(?<prefix>\S+)\s" | stats count by prefix

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 1:23 am
by ariefwido
Hello,

I did copy and paste that command on cli.

The result prefix search on attachment
Search MT.JPG
And then I found something that on the search section if I remove module=xxx then I got the result on the dashboard.
For the example on the device list dashboard I use this
No Module.JPG
instead of your originally script
With Module.JPG
I think that module=xx didn't work on my splunk search. Any idea?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 9:38 am
by Jotne
Strange.

Can you post output of sourcetype=mikrotik script=sysinfo
Make sure you have Smart Mode selected (see circle on picture)
Click the arrow to expand one post so I see the extraction. >
.
test_output.jpg

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 10:33 am
by ariefwido
Hi Jotne,

Here is the output and just different from yours.
test_output_1.JPG

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 12:26 pm
by Jotne
I see two strange things.
1. It seems that Splunk does not handle the date/time correctly since its shown within your event.
2.I do not see the information from router that shows where it comes from and type (ipsec/DNS/DHCP) (debug packets)

Is this a clean Splunk installation, followed the steps above?

You are running on a 951G a common box, I have a 941 and 750Gr3 and some other.
Your RouterOS software 6.43.4 is the same as I do run, so should be ok

Can you post the last lines of the output on the Router of /log print and /log print detail
Just cut and paste the line, so I do see how it looks like.

On mine
11:21:32 script,info script=pool pool=default-dhcp used=1 total=245
and
time=11:21:32 topics=script,info
message="script=pool pool=default-dhcp used=1 total=245"
I do miss the stuff in bold from your logg message and would like to see how it looks like on the router to compare what Splunk sees.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 12:52 pm
by ariefwido
Hi Jotne,

Yes, this is fresh install splunk and I did several time remove my VM and install again to make sure that.

Here is the output
/log print

17:47:19 firewall,info FW_INTERNAL forward: in:PJX out:BRX-LAN, proto TCP (ACK), 10.99.100.102:7332->10.121.61.108:52380, len 40

/log print detail

time=17:48:19 topics=firewall,info message="FW_INTERNAL forward: in:PJX out:BRX-LAN, proto TCP (ACK), 10.99.100.102:7332->10.121.61.108:52380, len 40"

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 1:09 pm
by Jotne
This looks correct, so it have to be some wrong with Splunk implementation since message looks different there.
Several other has used this, so should not be an big error in the code.

If you tyoe index=* in splunk, do you see any message that have the module tag coming from the router?

Like this
firewall,info
PS If you set time to: real time 1-minute window you should see data live as they arrive.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 1:28 pm
by ariefwido
Unfortunately I didn't see that message on my splunk,
test_output_2.JPG
Any idea what is happening on my splunk?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 2:10 pm
by Jotne
You have some strange in your message that I have not see with other: RTZPKN02

Can you post this? /system logging export

How did you install the files in Splunk?

Why do you get Des 9 in your log, I am still at Des 8?
Your logs has two different time stamp.
See if all clock is equal everywhere. Router, Computer ++

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 2:23 pm
by ariefwido
Sorry the Dec 9 is from date server, I already change the NTP :D

Here is the output
 /system logging export
# dec/08/2018 19:21:37 by RouterOS 6.43.4
# software id = 29W1-FTPT
#
# model = 951G-2HnD
# serial number = 642E05A9020A
/system logging action
add name=syslog remote=10.99.100.77 remote-port=7514 src-address=10.122.82.200 \
    target=remote
add bsd-syslog=yes name=logserver remote=10.100.10.105 src-address=\
    10.122.82.200 target=remote
/system logging
add action=syslog disabled=yes topics=info,error,interface,warning
add action=logserver prefix=MikroTik topics=dhcp
add action=logserver prefix=MikroTik topics=!debug

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 2:44 pm
by Jotne
Seems that you do not have a clean install.
You are logging to several system at the same time.
It should work.

Try this: Remove all logg line and add this:
/system logging action
add name=myserver remote=10.100.10.105 target=remote
/system logging
add action=myserver prefix=MikroTik topics=!debug
add action=myserver prefix=MikroTik topics=dhcp
Is this your Splunk server? 10.100.10.105
If not, do you relay your message (rslyslog or other server)?
Do you send your log message passing trough several routers?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 3:45 pm
by ariefwido
Seems that you do not have a clean install.
You are logging to several system at the same time.
It should work.

Try this: Remove all logg line and add this:
/system logging action
add name=myserver remote=10.100.10.105 target=remote
/system logging
add action=myserver prefix=MikroTik topics=!debug
add action=myserver prefix=MikroTik topics=dhcp
Is this your Splunk server? 10.100.10.105
If not, do you relay your message (rslyslog or other server)?
Do you send your log message passing trough several routers?
Ok I will reinstall my splunk VM again and change all log line and I will tell you the result

And yes my log message passing through several routers.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 3:46 pm
by Jotne
But is this your Splunk server? 10.100.10.105
Or do you send data to an rsyslog or other syslog server, that then sends it to your Splunk server?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Sat Dec 08, 2018 5:30 pm
by ariefwido
Hi Jotne,

It seems I found the problem, the problem is marking the BSD Syslog on log remote action.
test_output_3.JPG
Finally the result is come.

Thanks and very appreciate your help.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Tue Dec 11, 2018 1:53 pm
by WeWiNet
Hi Jotne,

Wanted to say thank you, very nice job.
Also to highlight that this tutorial works perfect on MacOS 10.14.
I just followed your tutorial and installed it with the Splunk Enterprise version
and all is working perfect (Ok I had to restart my machine once as splunk did not launch first time correctly).

I now try to make sense out of all that data and nice graphs ... :-)

PS: How can you know how much data you log per day (which is the limitation of the free version)?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Tue Dec 11, 2018 2:52 pm
by Jotne
Thanks.

You find license information her:

Settings->Licensing
There you see this for free version
Licensed daily volume 500 MB

Select:
Usage-Report->Previous 30 days

Here you will see how much of the license you use each day, last 30 days.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Tue Jan 01, 2019 11:47 pm
by zandhaas
Thank You for this post and all the work to get all the information in Graphs.

Only I had a hard time to get all the information in Splunk.
After three hours of trying a lot of different things I finaly discoverd that I missed the last "}" in the Router script.

Perhaps you can change the post where the script is too make the beginning and end of the script more clear.

Regards Peter

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Wed Jan 02, 2019 5:50 pm
by Jotne
Thanks for the feedback.
Added some space in the script to make it better to see start end.

Next time you can click Select ALL, behind the Code: at the top of the script and you get all that is needed.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Thu Jan 03, 2019 10:37 am
by Hunty
Thank You for this post and all the work to get all the information in Graphs.

Only I had a hard time to get all the information in Splunk.
After three hours of trying a lot of different things I finaly discoverd that I missed the last "}" in the Router script.

Perhaps you can change the post where the script is too make the beginning and end of the script more clear.

Regards Peter
I had the same problem,
now everything works

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Thu Jan 03, 2019 11:06 am
by Hunty
I'm seeing two problems:
The script reports a cpu higher than usual, it detects the cpu loads when the scripts is running, so instead of reading a normal 10% load, it reads a load near to 100%

The second is the Disk graph.
I've attached two screenshot

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Thu Jan 03, 2019 11:21 am
by Hunty
I've modified the script in order to read the cpu load at the beginning, now the readings are correct
# This script is used to send data to Splunk using syslog.
#===================================

# Collect system resource
# ----------------------------------
:local cpuload ([/system resource get cpu-load]);
:local freemem ([/system resource get free-memory]/1000000);
:local totmem ([/system resource get total-memory]/1000000);
:local freehddspace ([/system resource get free-hdd-space]/1000000);
:local totalhddspace ([/system resource get total-hdd-space]/1000000);
:local up ([/system resource get uptime]);
:log info message="script=resource free_memory=$freemem MB total_memory=$totmem MB free_hdd_space=$freehddspace MB total_hdd_space=$totalhddspace MB cpu_load=$cpuload uptime=$up";



# Collect accounting traffic
# ----------------------------------
# Take a snapshoot
if ([/ip accounting get enabled]=yes) do={
/ip accounting snapshot take
# Send data to loggin server
foreach logline in=[/ip accounting snapshot find] do={:log info message="$[/ip accounting snapshot print as-value from=$logline]"}};

# Finding dynmaic lines used in uPnP
# ----------------------------------
:foreach logline in=[/ip firewall nat find dynamic=yes] do={:log info message="$[/ip firewall nat print as-value from=$logline]"};

# Collect system information
# ----------------------------------
:local version ([/system resource get version]);
:local board ([/system resource get board-name]);
:local model ([/system routerboard get model]);
:local serial ([/system routerboard get serial-number]);
:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial";

# Collect system health
# ----------------------------------
:local voltage ([/system health get voltage]/10);
:local temperature ([/system health get temperature]);
:log info message="script=health voltage=$voltage V temperature=$temperature C";

# Sends wireless client data to log server
# ----------------------------------
:foreach logline in=[/interface wireless registration-table find] do={:log info message="$[/interface wireless registration-table print  as-value from=$logline]"};

# Collect DHCP Pool information
# ----------------------------------
/ip pool {
   :local poolname
   :local pooladdresses
   :local poolused
   :local minaddress
   :local maxaddress
   :local findindex
   :local tmpint
   :local maxindex


 #  :put ("IP Pool Statistics")
 #  :put ("------------------")

# Iterate through IP Pools
   :foreach p in=[find] do={

      :set poolname [get $p name]
      :set pooladdresses 0
      :set poolused 0


#   Iterate through current pool's IP ranges
      :foreach r in=[:toarray [get $p range]] do={

#      Get min and max addresses
         :set findindex [:find [:tostr $r] "-"]
         :if ([:len $findindex] > 0) do={
            :set minaddress [:pick [:tostr $r] 0 $findindex]
            :set maxaddress [:pick [:tostr $r] ($findindex + 1) [:len [:tostr $r]]]
         } else={
            :set minaddress [:tostr $r]
            :set maxaddress [:tostr $r]
         }

#       Convert to array of octets (replace '.' with ',')
         :for x from=0 to=([:len [:tostr $minaddress]] - 1) do={
            :if ([:pick [:tostr $minaddress] $x ($x + 1)] = ".") do={
               :set minaddress ([:pick [:tostr $minaddress] 0 $x] . "," . \
                                       [:pick [:tostr $minaddress] ($x + 1) [:len [:tostr $minaddress]]]) }
         }
         :for x from=0 to=([:len [:tostr $maxaddress]] - 1) do={
            :if ([:pick [:tostr $maxaddress] $x ($x + 1)] = ".") do={
               :set maxaddress ([:pick [:tostr $maxaddress] 0 $x] . "," . \
                                       [:pick [:tostr $maxaddress] ($x + 1) [:len [:tostr $maxaddress]]]) }
         }

#      Calculate available addresses for current range
         :if ([:len [:toarray $minaddress]] = [:len [:toarray $maxaddress]]) do={
            :set maxindex ([:len [:toarray $minaddress]] - 1)
            :for x from=$maxindex to=0 step=-1 do={
#             Calculate 256^($maxindex - $x)
               :set tmpint 1
               :if (($maxindex - $x) > 0) do={
                  :for y from=1 to=($maxindex - $x) do={ :set tmpint (256 * $tmpint) }
               }
               :set tmpint ($tmpint * ([:tonum [:pick [:toarray $maxaddress] $x]] - \
                                                    [:tonum [:pick [:toarray $minaddress] $x]]) )
               :set pooladdresses ($pooladdresses + $tmpint)
#         for x
            }

#      if len array $minaddress = $maxaddress
         }

#      Add current range to total pool's available addresses
         :set pooladdresses ($pooladdresses + 1)

#   foreach r
      }

          :set poolused [:len [used find pool=[:tostr $poolname]]]
#   Send data
    #      :log info message=("pool=" . $poolname  . " used=" . $poolused . " total=" . $pooladdresses)
          :log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")

# foreach p
   }
# /ip pool
}

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Thu Jan 03, 2019 3:10 pm
by Jotne
Good idea moving the cpu reading to the top. I have updated fist post view new version.
PS mine du not give much difference in CPU when script is running. Maybe you device is some under-powered or you have som wrong in your configuration (fasttrack or hw acceleration missing)

I see that MB is wrongly reported due to dividing on 1000000 and not 1048576 (1024*1024). Corrected in the script.
Since graph is in percentage it should not make any change to the view.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Fri Jan 18, 2019 3:08 am
by fengyuclub
Good idea moving the cpu reading to the top. I have updated fist post view new version.
PS mine du not give much difference in CPU when script is running. Maybe you device is some under-powered or you have som wrong in your configuration (fasttrack or hw acceleration missing)

I see that MB is wrongly reported due to dividing on 1000000 and not 1048576 (1024*1024). Corrected in the script.
Since graph is in percentage it should not make any change to the view.
No matter how it is set, my splunk can't get my ccr1016 data. Splunk is a virtual machine ubuntu server in the LAN, ccr on what you said, every 5 minutes running script can see a lot of log information generated, but still no data into splunk, there is nothing I did not do Is it? Please forgive my english, from google translation.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Fri Jan 18, 2019 8:29 am
by Jotne
No need to quote message above you, only part of it when needed. Always use Post Reply button under the post.

Are you 100% sure you have tagged the packet with MikroTik? There are no firewall?
Try in the search page and search for a star last 15 min. Do you get any?
*

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Jan 21, 2019 9:50 am
by Hunty
After three hours of trying a lot of different things I finaly discoverd that I missed the last "}" in the Router script.
Check this!

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Jan 21, 2019 3:30 pm
by zandhaas
I'm Using Splunk for a couple of weeks now.

In the Firewall Rule section I see beside the attacks from the large big spooky internet also local adresses appear as a result of the "FW_Drop_All_From_Wan" rules.
and that are mainly request with dest_port 53 (DNS).

Is it possible to filter out the local addresses?

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Jan 21, 2019 6:38 pm
by Jotne
You should not see local address in the outside inn block rule.

Can you post an example like this:
2019-01-21 17:25:25	FW_Drop_all_from_WAN	input	ether1-Wan	(unknown 0)	00:05:00:01:00:01	TCP	104.131.145.9	45167	92.31.200.211	2082	San Francisco	United States
And yes, you can get rid of the message in two ways.
1. Add a rule on the fw above the outside in block rule that block the specific ip/port of your choice.
2. Modify Splunk to exclude the ip/port you like.

First is the simplest solution.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Jan 21, 2019 7:07 pm
by zandhaas
Below an example of the local 192.168.0.1 address
_time	rule	chain	in_if	out_if	src_mac	protocol	src_ip	src_port	dest_ip	dest_port	City	Country
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	42597	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	57660	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	56630	192.168.0.1	53	Unknown	
And I wil try the solution with an extra Drop rule in the firewall.

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Jan 21, 2019 7:34 pm
by Jotne
It looks like your router tries to resolve DNS on it self and get blocked.
From router console try this.
:put [/resolve mikrotik.com]
You should get an IP as result, like 159.148.147.196

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Posted: Mon Jan 21, 2019 8:20 pm
by zandhaas
I get the same IP as a result biut perhaps My PI-Hole implementation has something to do with it.

I'm using PI-Hole as an "Ad blocker for my Internal network"
And for this I'm using DHCP option 6 to force all internal clients to go to the PI-Hole server for the DNS resolving.


By the way I changed the "Drop all from not coming from LAN" rule.
I replaced the "In Interface list" from !LAN to "In Interface" Ether1-WAN.

This seemed to have resolved my issue.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jan 22, 2019 8:43 am
by Jotne
2.6 released

# 2.6 (22.01.2019)
# Added information about fast track in "traffic monitor"
# Fixed typo in Traffic view. Added fast track info
# Changed to checkbox in "DNS Request"
# Added better sparkline "in Device List"
# Added identity to "Device List"
# Updated script to get identity
# Removed parentheses from services from "MikroTik uPnP"
# Added ip to client drop-down list to "MikroTik uPnP"
# Added more disk info to "MikroTik Resources"
# Changed to last 12 hour instead of 4 in "MikroTik DNS Live usage"
# Changed to sort by count in "Sort by count"
# Added timeline dashboard to "DNS Request"
# Fixed public IP speed by reducing lookup in "Traffic"

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jan 22, 2019 11:05 am
by zandhaas
I see some strange things happen.

I have added three devices to the Splunk Mikrotik environment.

1. RB750Gr3 as a router. (sending over UDP 514)
2. HAPac2 configured as a switch (Accesspoint) (sending over UDP 515)
3. Mikrotik CHR as Dude server. (sending over UDP 516)

Everything seems to log all information to splunk but after somtime the data of the HAPac2 is not examind any more by Splunk.
After restarting the splunk server Everything is OK again for a short time.
The Router and the DUDE server have no issues.

When i check the Splunkd.log file I see a lot "Failed to parse timestamp" messages for the HAPac2 syslog.
01-22-2019 09:46:45.504 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:515|host=192.168.0.8|syslog|
What can be wrong?

This morning I updated to version 2.6.
But I had this problem before. So it is not version related.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jan 22, 2019 11:40 am
by Jotne
It may be that the props filter only look at UDP:514 and syslog.
When data comming in on UDP:515 it will not see that its MikroTik data.

You can fix this by edit etc/apps/MikroTik/default/props.conf and add
[source::udp:515]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik

[source::udp:516]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik
But my questioon to you is, why use more than on UDP?
I do see noe good reason to use on port for each device. Send all to UDP/514

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jan 22, 2019 1:10 pm
by zandhaas
I started with using port 514 for all 3 mikrotik devices.
At that moment I had the same problem. No data in visible in Splunk.
After that I changed to port 515 and restarted splunk. And yes I saw data in Splunk. but some time later Splunk stopped showing data in the graphs.
Then I restarted splunk again and yes Splunk is showing data for an hour or so.
The Router and the Dude device are showing Up as expected.

See the picture below:
2019-01-22 11_52_06-MikroTik Wifi strength _ Splunk 7.png
At the moment I changed all 3 devices back to UDP port 514. With the same result as before.

I still see below messages in the splunkd.log file saying it suppresses messages:
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|\n                                1295 similar messages suppressed.  First occurred at: Tue Jan 22 11:57:40 2019
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.345 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.348 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.356 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jan 22, 2019 1:19 pm
by Jotne
Do examine time on all your devices. It must be in sync.
Do use NTP on all devices to make sure time is ok.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jan 22, 2019 1:52 pm
by zandhaas
The router is used as the timeserver for my local environment.
the HAPac2, the Dude server and the Splunk server synchronize time with the router and all have the same time and date.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Jan 23, 2019 9:57 am
by Jotne
It may have something to do that you have used different UDP ports. I may not recognize the message correctly.
You may try to start over and follow the example step by step.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Jan 23, 2019 7:10 pm
by zandhaas
I made some progress.

After an other look at the messages in the splunkd.log file
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
I focused on the MAX_TIMESTAMP_LOOKAHEAD option. according to the default props.conf file this option default to 32 for syslog events.
Looking at the mikrotik log events the consists of 19 characters (excluding the mili seconds).

To change the default 32 to 19 I added the MAX_TIMESTAMP_LOOKAHEAD option to the "/opt/splunk/etc/apps/MikroTik/default/props.conf " file and restarted Splunk.
[syslog]
TRANSFORMS-force_mikrotik = force_mikrotik
MAX_TIMESTAMP_LOOKAHEAD = 19
After this change I do not see the above message in the Splunkd.log file anymore. And more important, the Hapac2 is logging events for more as 3 hours now. This is already an hour longer as before (max 2 hours).

I will keep an eye on the Mikrotik splunk environment to see if everything keeps running.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Jan 23, 2019 9:11 pm
by Jotne
Interesting. Have not used much time in my splunkd.log, but have the same problem as you,
But only in one of 4 routers. Other are ok.

Tried bot 19 and 23 but still get samme message.
01-23-2019 20:08:40.136 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event (Wed Jan 23 20:08:39 2019). Context: source=udp:514|host=193.1.1.100|syslog|

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Jan 23, 2019 10:41 pm
by zandhaas
At my site it was 1 out of 3 that failed and I was missing information for that router.
Are you also missing data?
After the change my failing router is still visible in Splunk so for mee it seems the solution.
But I did not check the log files that come from the routers. Do you now were I can find them?
Perhaps it has something to do with too many events during a short time period.

We need to debug this.

Regards Peter

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Jan 24, 2019 8:12 am
by Jotne
I do get event from all routers. To see if you get from one specific router use search and type host=1.2.3.4 (change to your IP)

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Fri Jan 25, 2019 3:38 pm
by Egert143
Hello

Could i get instructions how to create splunk source type manualy ? I have splunk light (paid) and it doesent support apps (as far as i know).

Current problem is that source and dest addres fields are merged with port numbers.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Fri Jan 25, 2019 10:07 pm
by Jotne
I have no idea on how to use Splunk Light.
In normal Splunk, source type based on the source it comes from udp:514

props.conf
[source::udp:514]
TRANSFORMS-force_mikrotik = force_mikrotik
transforms.conf
[force_mikrotik]
DEST_KEY =  MetaData:Sourcetype
REGEX =  \sMikroTik:\s
FORMAT =  sourcetype::mikrotik

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sat Jan 26, 2019 10:07 pm
by Egert143
And how would i turn 123.123.123.123:1234->12.34.45.67:80 to Source Address = 123.123.123.123 Source Port = 1234 Dest Address = 12.34.45.67 Dest Port 80 So they would be searchable ?

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sat Jan 26, 2019 11:02 pm
by Jotne
The traffic solution are based on that you have private ip inside your net and public on the outside.
Private IPv4 addresses
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16


But if you like to log other IP and know what is inside/outside, you have to modify the Splunk files.

Edit:
MikroTik Traffic
Replace all
 | search (ip_in="10.0.0.0/8" OR ip_in="172.16.0.0/12" OR ip_in="192.168.0.0/16")
with
 | search ip_in="12.34.45.0/8"
That if you like 12.34.45.0/8 to be your inside net.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Fri Feb 01, 2019 8:41 am
by JieYu2001
Where can I find the link to download MikroTik2.6 spl? Thanks.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sat Feb 02, 2019 9:47 pm
by zandhaas
In the first post of this topic :)
Or the below link
download/file.php?id=35231

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 04, 2019 9:40 am
by JieYu2001
Thanks zandhaas - I got it downloaded.

I also installed everything per our topic owner Jotne's procedure but cannot get the data flow from MikroTik to Splunk, after verifying port 514 is open. Upon diving into some details, I suspect it's due to the lack of SSL of my MikronTik (192.168.88.1 shows "Not secure") - anyone know if this is the root cause? If yes what is the easiest way to enable SSL under RouterOS v6.40.8 and Win10? Appreciate any tips there.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Feb 05, 2019 10:55 am
by Jotne
There are no certificates involved in the transaction. All data are sent using UDP/514 Syslog (not encrypted).

In Splunk search, type only a * and do a search for the last 24 hour. Do you see any data at all?
Make sure you follow all steps in the first post 1 by 1.
Do you have any deviation? Using a clean Splunk install? Windows firewall opened if you run on Windows?

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Feb 06, 2019 7:16 am
by JieYu2001
Thanks Jotne. Now I see the data (events) through the Splunk search, though MikroTik2.6 app still not sees the data yet and I am still debugging.

BTW the Splunk observed event entry looks like - do you see any anomaly there?

2/5/19
9:09:54.000 PM
Feb 5 21:09:54 router.lan Feb 5 21:09:54 MikroTik MikroTik: Router = 192.168.88.1
host = router.lan
source = udp:514
sourcetype = mikrotik

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Feb 06, 2019 8:22 am
by Jotne
Can you post some example line from search in Splunk that shows what you got in the log from using * search?

Do you have tagget all packet with MikroTik? This will fail Mikrotik since its not the same name.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Feb 07, 2019 6:19 am
by JieYu2001
Hi Jonte, here're three snapshots

1. Splunk Event entry sample from the MikroTik UDP feed - great if you can help review the "Host", "Source", "Sourcetype" field to see if they are right for the MikroTik2.6 App
Splunk Event Entry from UDP and MikroTik.png
2. Splunk UDP input setting
Splunk UDP Input Setting.png
3. MikroTik2.6 App snapshot (system change search, with no data found while the Splunk search gives items like above)
Splunk MikroTik 2.6 App Lauch Snapshot.png
Thanks

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Feb 07, 2019 8:27 am
by Jotne
1. Is this the only type of event you see?

Here are some example on how they should look like: (various modules)
firewall,info MikroTik: NAT_Web_server dstnat: in:ether1-Wan out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 91.12.58.49:49145->92.220.200.251:80, len 60
dhcp,debug,packet MikroTik:     Parameter-List = Subnet-Mask,Router,Domain-Server,Domain-Name,NETBIOS-Name-Server,Static-Route
dns,packet MikroTik: --- sending reply to 10.10.10.244:53720:
script,info MikroTik: script=health voltage=24 V temperature=42 
wireless,info MikroTik: 04:62:73:xx:xx:21@wlan1 established connection on 2437000, SSID GjestenettHMN
ipsec MikroTik: invalied encryption algorithm=6.
interface,info MikroTik: ether1 link up (speed 100M, full duplex)
Have you followed tutorial in post#1?
Do you use Splunk for other stuff?

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Feb 07, 2019 10:26 am
by JieYu2001
Hi I followed your first post but skipped 2c~2e (FW/NAT/Traffic logging since not sure about the detailed steps). I did have Home Monitor app before that affected the MikroTik data inputs, and I have it removed so the data inputs seems right (though not complete if without 2c~2e). The question I have is that, even with incomplete but valid data (say only DHCP request part), should MikroTik2.6 App see them and populate some view right? But now it seems the app does not pick up anything and I am not sure if the app has access to the log. Thanks.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Feb 07, 2019 11:52 am
by Jotne
You should get DHCP and other stuff from the router if you skipped 2c-2e.
Thats why I asked about how the log lines looks like.
You could use a search for host=192.168.88.1 and post some line.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 11, 2019 9:17 am
by JieYu2001
Thanks again Jotne. Here's a screenshot. Seems the Splunk events have the right contents, but the format is different from yours.
Splunk MikroTik 2.6 Event Snapshots.png
Basically, before the identifier "MikroTik", there are timestamps and another "MikroTik", but without the log field name like "dns,packet" as in your snapshots.
I copied the MikroTik scripts exactly, so do you think I missed something on the Splunk side? My Splunk version is 7.2.3.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 11, 2019 9:44 am
by zandhaas
Are you sure your "router script" is complete?

I had problems getting my data visible in splunk to.
It turned out that I missed the last "}" in the Router script.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 11, 2019 10:04 am
by JieYu2001
Hi Jotne ~ some progress - for some reason, the "Module" field picks up part of the timestamp (the Month) since their is no syslog field name for some reason (the event item format difference I mentioned). After tweaking the Volt/Temperature code (removing the module key from the search), I was able to get that view right. Encouraged and will see how to get the module field right in the first place - help appreciated.
Splunk MikroTik 2.6 Volt_n_Temp.png

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 11, 2019 12:04 pm
by Larsa
Since I'm not a Splunk expert I wonder if anyone has some bright ideas how to optimize Splunk / Mongodb?

We have about 15.5 million entries and the reports are getting really slow to produce. In a regular SQL database you can run a "Query Execution Plan" and then add indexes to columns that performs table scans. Is there an equivalent way in Splunk or any other way to optimize the environment? We're running Splunk with 12 cores, 20 Gb ram and SSD which ought to be sufficient.

Any suggestions are welcome!

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 11, 2019 6:41 pm
by Jotne
@Larsa
Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.

I do recommend that you start a thread about your problem over here:
https://answers.splunk.com/index.html

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Feb 11, 2019 6:51 pm
by Jotne
@JieYu2001

There are some wrong with extraction of the data in the Splunk or the format that your MT Router sends it.
In List view in Splunk your should not see time and date in the Event space, only in Time column.
In your view, I do not see it only one time extra, but two times in front of the data. This breaks all view.
You get it to work since you adjusted to view to accept your wrong data.
source=udp:514 and sourcetype=mikrotik looks correct.

I would recommend you to start over.
Clean Install of Splunk, remove all connection to Splunk in your router.

@zandhaas
You do not need the script to get data inn to splun, so it could also be removed to rule out problems.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Feb 12, 2019 9:26 am
by JieYu2001
Thanks Jotne - the issue is resolved. In the MK Logging setting, I checked "BSD Syslog" which caused issue (still don't know why since that is the correct syslog protocol supported in Splunk). Uncheck it and things look fine now.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Feb 12, 2019 3:08 pm
by Jotne
There was nothing in the first post telling you to select it so not sure why you did it.
Will update post #1 to say not to select it.
Good you find out what was wrong :)

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Feb 12, 2019 9:07 pm
by Larsa
Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.I do recommend that you start a thread about your problem over here: https://answers.splunk.com/index.html

Thanks for the suggestion, I'll report back if I find out an appropriate solution!

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Fri Feb 15, 2019 5:41 pm
by oaas
Great work!

Had some issues with parsing messages from one cAP ac where the messages suddenly dropped due to "Failed to parse timestamp" warning messages.

Seems it got solved by adding
TIME_FORMAT = %b/%d/%Y %H:%M:%S
to the props.conf file.

Please consider adding this to future releases.

/Thanks

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sun Feb 17, 2019 11:28 am
by frankcale
Hi, Can u pls help with displaying Vlan info

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sun Feb 17, 2019 11:59 am
by Jotne
Not sure what you asks for.
A list of Vlan on the router?
Traffic going trough Vlan?

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sun Feb 17, 2019 4:59 pm
by frankcale
Hi, Can u include vlan traffic monitoring and if possible protocols like youtube, torrent, updates, etc

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sun Feb 17, 2019 7:30 pm
by Jotne
Protocol are complicated to monitor due to https, near to impossible.
Vlan can be monitored used SNMP or you can use script and syslog to send data.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Mar 05, 2019 12:19 pm
by Jotne
Updated 1a to mention that you need an account at splunk.com to download software.
Account is free to create.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Mar 06, 2019 6:18 am
by ithelp
Hi, thanks for this magnificent explanation.
Can you give me on how to see the PPP and PPPOE information from the log?
I've already configure it on the rules tab, but nothing shows on any dashboard.
Thanks,

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Wed Mar 06, 2019 8:09 am
by Jotne
I do not have PPP nor PPPOE so I can not easily make log for it.

But if you could post 3-4 pages of logs that involves PPP and PPPOE output I could have look at it.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Mar 07, 2019 12:32 pm
by cavin12
thanks for such a clear presentation for the newbies to understand, appreciate the efforts.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sat Mar 16, 2019 8:07 pm
by neutronlaser
Price is ridiculous.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sat Mar 16, 2019 9:40 pm
by Jotne
500MB/day for free is ridiculous much to pay.

But I do agree that if you pay retail price for Splunk and need eks 500GB/day, price is high.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Apr 29, 2019 9:25 am
by Halfeez92
Hi how can I remove the MikroTik device list in the splunk dashboard view? I have multiple same devices showing up because I forgot to disable NAT and enable routing. Now it have 2 same devices with different IP

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Apr 29, 2019 1:14 pm
by Jotne
I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Apr 29, 2019 7:08 pm
by Halfeez92
I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.
Ok thanks for the help. Already delete the duplicate device.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Mon Jun 10, 2019 5:49 pm
by Jotne
Updated section 2c regarding Log prefix.

NB Do not use more than 20 charters, or else it start to clip other part of the log
firewall,info MikroTik: 123456789012345678901234567890 : in:ether1-Wan ...
firewall,info MikroTik: 1234567890123456789012345 forwa: in:ether1-Wan ...
firewall,info MikroTik: 12345678901234567890123 forward: in:ether1-Wan...
firewall,info MikroTik: 12345678901234567890 forward: in:ether1-Wan ...
As you see here the chain word forward is eat'n up by the prefix.
MT is this a bug???
If not, set a warning in the gui :)

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Jun 13, 2019 1:54 pm
by Jotne
Updated section 2f)

Script updated to collect and show how many dynamic/static address lists entry there are.
Eks output
script,info MikroTik: script=address_lists list=rdp_stage2 dynamic=24 static=0
script,info MikroTik: script=address_lists list=rdp_stage1 dynamic=28 static=0
script,info MikroTik: script=address_lists list=ftp_stage2 dynamic=1 static=0
script,info MikroTik: script=address_lists list=ftp_stage1 dynamic=1 static=0
script,info MikroTik: script=address_lists list=black_list_rdp dynamic=42 static=0
script,info MikroTik: script=address_lists list=black_list_ftp dynamic=1 static=0
script,info MikroTik: script=address_lists list=Whitelist_IP dynamic=3 static=2
script,info MikroTik: script=address_lists list=Router dynamic=0 static=1
script,info MikroTik: script=address_lists list=IPSEC dynamic=1 static=0
script,info MikroTik: script=address_lists list=FW_Block_user_try_unkown_port dynamic=1089 static=0
script,info MikroTik: script=address_lists list=Clients dynamic=0 static=2
script,info MikroTik: script=address_lists list=Blocked dynamic=1 static=7
This will later be used in its own graph to see variation in the lists.

PS only one IP en the ssh black list black_list_ssh is due to that I do not use default port.

You can update script only and wait for new Mikrotik Splunk app to be updated later.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Jun 20, 2019 9:59 am
by zandhaas
Hello Jotne,

I want to upgrade my Splunk version 7.2 environment tot Splunk 7.3

Is the mikrotik app compatible with Splunk 7.3?

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Thu Jun 20, 2019 1:43 pm
by Jotne
Yes, I do try to not use anything special in the APP so it should be compatible with all new version.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Fri Jun 21, 2019 9:29 pm
by Jotne
Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Sun Jun 23, 2019 2:59 am
by pidde
Hi!

Must say you did a great work with this app!
Is it possible to add option82 to dhcpserver part?
And is it also possible decode the option82 from hex?

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jun 25, 2019 10:34 am
by zandhaas
Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
When I look at the current script under 2f I only see the "# Collect DHCP Pool information" part.

It seems the rest of the script is missing.

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Tue Jun 25, 2019 1:09 pm
by Jotne
You are 100% correct. Copy past error.

Fixed.

PS It's getting closer to the release of v 2.7 of Splunk for MikroTik

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Posted: Fri Jun 28, 2019 2:10 pm
by Jotne
Script to get information on the router is upgraded to 2.6 section 2f

Simpler DHCP calculation.
Fixed comment so it start on the beginning of the line.
Fixed Script names

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 01, 2019 1:15 pm
by Jotne
Upgraded to 2.7

There are a lot of new changes to the app as listed below, so its a larger upgrade.
Simplest way to upgrade, if you have not made changes your self, remove (uninstall) previous version, install new version.
Please report any problems back to this thread, and I will try to fixed.

PS If you do upgrade, you also need to upgrade script in section 2f (fist post) on all router you like to get data from.
Just cut/past the script over the old one.

PS2 File is found under section 1g first post

Request to changes are also welcome :)

What new:
# 2.7 (01.07.2019)
# New view added "Address Lists Counters"
# Changes most view to use "Base Search"
# Changed "MikroTik DHCP request" to use stats and fixed host flaw
# Changed "MikroTik System Changes" to use 30 day and 4 hour span and maxspan in transaction
# Removed changes to "DHCP leases" in "MikroTik System Changes"
# Added search in dropdown for "MikroTik DNS Live usage"
# Added Time picker for "MikroTik Device List"
# Speeded up "MikroTik Remote Connection"
# Fixed wrong timestamp of packets logged
# Changed "MikroTik DHCP request" to use stats and fixed host flaw and maxspan in trnsaction
# Added search in dropdown for "MikroTik DNS Live usage" and added IP to client and change sorting
# Fixed "MikroTik DNS request" to use correct dropdown lists
# Fixed "MikroTik Firewall Rules" to use better searh, removed base level, added counters, long prefix
# Rewritten "MikroTik Live attack" to speed up and added more dropdown
# Fixed "MikroTik Resources" to give correct host number
# Changed "MikroTik System Changes" to use 30 day and 4 hour span, removed DHCP info
# Fixed "MikroTik Traffic" to use script= and some clean up
# Fixed "MikroTik uPnP" script name, added ip to dropdown
# Added to ">MikroTik Uptime" dropdown menu
# Fixed "MikroTik Volt/Temperature" sorting
# Fixed "MikroTik VPN Connection" faster search
# Fixed "MikroTik Web Proxy" sorting and some code clean up
# Changed "MikroTik Wifi strength" to use script tag and some clean up
# Added "dashboard.css" to set menu color global
# Fixed "props.conf" to better handel wrong prefixed and some other changes

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Wed Jul 03, 2019 5:36 am
by fengyuclub
I have been paying attention to this post, very powerful chart, but the cumbersome construction and the lack of relevant knowledge have been unsuccessful. I can only temporarily use the mrtg icon inside routeros to temporarily cope with it. I hope the poster can write the deployment manual from the perspective of the technology-poor. .

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Wed Jul 03, 2019 3:17 pm
by Jotne
Its written so that a user with some knowlege should be able to set it up.
You can start by telling me what your problem is, and we may be able to help you out.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 06, 2019 6:50 am
by fengyuclub
Reinstalled splunk on ubuntu18.04, is a virtual machine under esxi, the deployment is very simple and normal, according to the steps of the top post, but the splunk dashboard can not see the task data incoming. Very strange, what else do I need to pay attention to? Please forgive my English using Google Translate, I am from China
1.png

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 06, 2019 10:43 am
by Jotne
After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik 
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 08, 2019 12:57 pm
by fengyuclub
After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
According to what you said carefully, but still can not receive the data, I introduced the cdb1016 log file db format, can be displayed to splunk, indicating that splunk no problem, is the data input problem, I see ros is the log The output is udp514 port, but I only see tcp listening port settings in splunk's receiving settings. Is this the reason?
1.png
2.png
3.jpg

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 08, 2019 1:36 pm
by Jotne
It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Jul 09, 2019 5:51 am
by fengyuclub
There is no local listening udp514, now there is data in, but click on the meter in the Mikrotik2.7 dashboard, most of them do not have any charts, how to add or customize the dashboard you need here, for example, I want The wan's real-time or past and downstream traffic in a certain period of time, as well as the system temperature, the number of online hosts, and so on. How to do it?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Jul 09, 2019 8:29 am
by Jotne
514 UDP do need to be active
Do you run it on Linux?

If so, as Root, type:
netstat -opan | grep 514
You should see one line like this:
udp        0      0 0.0.0.0:514             0.0.0.0:*                           23557/splunkd        off (0.00/0/0)
if not UDP/514 is not running.

One the mikrotik, post the output of:
/system logging export
You should see some like:
# jul/09/2019 07:26:37 by RouterOS 6.43.16
# software id = E4B6-94N8
#
# model = RouterBOARD 750G r3
# serial number = 6F3806E0A160
/system logging action
set 3 remote=ip_your_syslog_server
/system logging
set 0 disabled=yes
add action=remote prefix=MikroTik topics=dhcp
add action=remote prefix=MikroTik topics=hotspot
add action=remote prefix=MikroTik topics=!debug
There should be IP for your server, and prefix for all action with MikroTik. If one letter is wrong in the prefix, it will fail. See capital M and T in the MikroTik.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Jul 11, 2019 1:03 pm
by fengyuclub
It’s true that I set it wrong, Mikrotik changed to MikroTik, and it should be fine, then I will report it.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Jul 11, 2019 1:32 pm
by haaroons
Hello Jotne,
I am new to this forum.

I have install MikroTik logs 2.7.

MikroTik DNS Live usage and MikroTik DNS Live request is not working. if i do search eventtype=dns_query No item found

Do advice how to fix this.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Jul 11, 2019 11:43 pm
by Jotne
DNS information are coming from standard logs on the router.

What do you get if you go to search window and search with the following line:
sourcetype=mikrotik earliest=-24h latest=now() | stats count by module
I do get some like this:
module		count
dhcp		12764
dns		324512
firewall	1349
ipsec		7
script		91182
upnp		308

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 12, 2019 6:02 am
by fengyuclub
The data is coming, some of the tables are already filled, some still have no data, such as dns, it doesn't matter, I want to know how to monitor the flow table of an interface (wan), just like mirkrotik's built-in mrtg chart, every 5 minutes, 30 minutes and so on. . . As shown
1.png

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 12, 2019 8:05 am
by Jotne
That is why I need the output of the above command.
Some data are coming from the logg.
Some are comming from scripting

Log:
-------
dhcp,dhcp_static,dns,firewall,ipsec,upnp

script:
-------
IPSEC_failed,address_list,healt,pool,resource,sysinfo,traffic,uncounted,upnp

So I guess you have some log problems. Read section 2b carefully.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 12, 2019 11:42 am
by fengyuclub
Splunk is too powerful. If I have multiple ccr1016, how can I transfer data to the splunk server, how do I distinguish syslogs from different mikrotik routers?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 12, 2019 1:29 pm
by Jotne
All the view for MikroTik in Splunk has a host drop down. So if you have more than one router, just select the host you like to monitor.
There is one possible problem, if you have many routers with same IP that sends log to same Splunk.
That could be solved using unique ID for each router and some small change to the code.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 15, 2019 6:24 am
by fengyuclub
How can I write the interface tx-bits-per-second parameter to the log and then plot it in splunk.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 15, 2019 8:06 am
by Jotne
What command do you use on the router to see this data?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 15, 2019 10:07 am
by fengyuclub
What command do you use on the router to see this data?
interface monitor-traffic ether1

Search forums see scripts with such calls
  "/interface monitor-traffic ether1 once do={
:put ($"tx-bits-per-second"/1000 /1000 )
}"

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Jul 16, 2019 8:12 pm
by Jotne
It can be done.
I do use IP accounting to see the traffic going trough the router.
This way are more generic and does work without any modification.
If you monitor one and one interface, this has to be adopted for each setup.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 5:03 am
by fengyuclub
{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX (($speedRX/1000)/1000);
:set $mbpsTX (($speedTX/1000)/1000);
:put "$iname RX:$mbpsRX Mbps TX:$mbpsTX Mbps";
}
}
I found the script for this post available, but after running it is all interfaces, I don't want all interfaces, only a few interfaces are needed, for example, I only need ether1, ether2, how to modify the script, and how can I get it? Let him display in the log, I use the splunk search call, and display it as 14.5Mbps instead of 14528. I hope to get everyone's help.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 9:29 am
by Jotne
Info
It seems that data you get from monitor are just moment blink of data going through the interface. So it will fly up and down for every time you run it. If it would be like cisco, average last 5 min, it would be perfect to rune every 5 min. Not sure if it are useful at as is.


If you have not renamed interface
:foreach interface in=[/interface find] do={
To
:foreach interface in=[/interface find where (name~"^ether1\$" || name~"^ether2\$") ] do={
or use regex
:foreach interface in=[/interface find where name~"^ether[12]\$" ] do={
Anchor ^ \$ are used to distinguish ether1 from ether11 etc.

Edit
You can use ID instead of name, so you can change from:
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
to
:set $monitor [/interface monitor-traffic $interface as-value once]
PS2, no need to declare variables, use them directly
do not divide data by 1000 two times, let splunk do that, so you do not loose any resolution
use equal sign for splunk to read data directly
you do not need semicolon behind each line ;

So final script could be some like this
:foreach interface in=[/interface find where name~"^ether[12]\$"] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 11:47 am
by fengyuclub
Your script, the regular expression method, no success without any output, it doesn't matter, my code is as follows
I want to know how to search for rich charts, and there are 14.8Mbps and 14833 display problems. This is not important. The important thing is how splunk draws charts.

mycode
{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "WAN-ether2 down RX=$mbpsRX Kbps";
:log info "WAN-ether2 up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"adsl-tx") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "adsl-tx down RX=$mbpsRX Kbps";
:log info "adsl-tx up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"bonding1") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "bonding1 down RX=$mbpsRX Kbps";
:log info "bonding1 up   TX=$mbpsTX Kbps"
}
}
After the schedule is displayed as follows
1.png

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 12:26 pm
by fengyuclub
Tested on other ccr1016 your script is successful, it should be the problem of the interface name, but it is important to draw the splunk graphics, I hope you can add to the new version.
3.png

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 1:38 pm
by Jotne
When you have multiple interface, use only one section, no a section for every interface

change
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
to
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
Test code that should output data to screen:
{
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:put "script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
}

PS, when testing cut and past on the cli, you need to wrape all script in brackets {} !!!

PS how often would you like to run the script? every 5 min. Do you know if monitor could show average 5 min data?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 1:58 pm
by Jotne
Try this

Add this to the Data_to_Splunk_using_Syslog script
# Get interface data (test)
# ----------------------------------
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1")(name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
Then in Splunk do this search for the last 4 hour.
sourcetype=mikrotik script=monitor| timechart avg(RX) as RX avg(TX) as TX by interface limit=10
May take some time to nice graphs.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 7:06 pm
by zandhaas
Nice,
I have added the additional script entries and changed the inteface names to the names I use.
But............
The sourcetype entry in the search entry schould be "sourcetype=MikroTik" 8) 8)

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 7:32 pm
by Jotne
In Splunk, search ignore case :)

Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Jul 19, 2019 9:06 pm
by zandhaas

Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.
The current "Mikrotik Traffic" overview is indeed a nice oveview.
But apart from knowing who is generating the traffic I am very interested in the amount of traffic that floats over each individual interface. And especially the WAN interface(s) and ISL interfaces. And when you see a bottleneck on one of your interfaces you can drill down to your traffic overview to identify the source of all that traffic.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 20, 2019 5:46 am
by fengyuclub
Jotne,Great, I did it according to your script, and the beautiful chart shows normal. I tried to add scripts to my multiple ccr and routerboards, so my interface has a lot of duplicate names, such as bonding1 and bridge1, how can I distinguish between them, or change the name for each interface.
4.png

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 20, 2019 5:59 am
by fengyuclub
Understand, add host=x.x.x.x in front of the search statement you gave to open my ccr and rb.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 20, 2019 7:10 am
by fengyuclub
Host=x.x.x.x Although this option is available, some devices have an internet connection that is a dynamic ip obtained by adsl dialing. So before the log warning, add an identity=xxxxx to distinguish the mikrotik device. After testing, it is feasible and runs very well.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 22, 2019 10:24 am
by Jotne
@ fengyuclub
Nice to see you are getting it to work.

@ All
Section 2c) Logging prefix has been updated with sample on how to name to logs.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Jul 22, 2019 3:44 pm
by Jotne
Script in section 2f) updated to 2.9

It now support to get interface counters and you can also set modules true/false if you do not like to monitor one section.
If you do not have wifi/dhcp, you can just set them to false.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Wed Jul 24, 2019 8:23 am
by Jotne
Script in section 2f) updated to 3.0

Do now get CDP neighbors

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Jul 25, 2019 12:46 pm
by fengyuclub
Splunk is really powerful, I see splunk have a lot of apps to install, in our China use wechat (similar to facebook, telegram) this social software, I saw this social software related app, WeChat Alert App for Splunk, I installed this App, sending test messages from wechat is successful, but I don't know much about splun's alert settings, set it many times, only a single success, can you help me?
5.png

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Jul 25, 2019 1:44 pm
by Jotne
No need for extra app to send message. Sending email using a gmail account is easy and works well.

But there is a big issue.

If you have a free Splunk license, you do loose a lot of thing.
* Monitor and Alerting (needed for sending alerts)
* 500MB pr day maximum
* Cluster
* Universal Forwarder
* HA
* Distributed Search
* Perfomance Acceleration
* Access controll (only on user)
* LDAP
+++

This is why I have not included any Alerting in the project.

There is a workaround. You can setup an batch job that runs search from command line and do stuff from it. (I have not tested it)

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 27, 2019 4:56 am
by fengyuclub
I have Splunk Enterprise license, gmail alert can't be real-time, mobile mail client can't update mail in real time, there is a delay of about 10 minutes, so I choose wechat alert.I received some wechat alert, but some of them use the search to save as an alert, I can't receive a wechat alert, I don't know where the problem is.
6.jpg
7.jpg

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Jul 27, 2019 8:30 am
by Jotne
Splunk do handle real time alerts (or close to)
https://docs.splunk.com/Documentation/S ... TimeAlerts
It should not depend of type of action you are using, starting a program, sending sms, email, wechat etc. Alerts should go out.
But you should not use to many alerts, since it will use more CPU to handle them.

Not sure what your problem is.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Fri Aug 02, 2019 10:20 am
by Jotne
Updated script to 3.1

Fixed CDP, since some devices sends long version with new lines breaking up the log lines. (Cisco)

PS still have problem that line is cut in Splunk. Not sure if its MT not sending whole line, or Splunk that cuts the lines.
I do only get 278 characters.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Aug 06, 2019 3:34 pm
by antispam
Using 2.7, it's mentioned that the "defconf: drop all not coming from LAN" rule should have the prefix 'FI_D_port-test'. When I set that, the Live Attack dashboard doesn't populate as it appears from the source in the dashboard it is searching for 'FW_Drop_all_from_WAN'. When changing this to 'FI_D_port-test' in the Live Attack dashboard source, it works. Is the FW_Drop_all_from_WAN still required?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Aug 06, 2019 4:24 pm
by Jotne
The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Wed Aug 07, 2019 9:39 am
by fengyuclub
Recently, one of my ccr is a bit problematic. I can only recover from the time when the device is powered off and restarted. The top of the log in winbox can see red like "system, error System rebooted because of kernel failure" or "Out of memory condition was detected", but I can't see it in splunk search. These log messages, how can I output all the log information to splunk for easy query.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Wed Aug 07, 2019 1:12 pm
by Jotne
A search like this should give all message:
sourcetype=mikrotik module=system
IF not try this:
sourcetype=mikrotik
Or at last just this
*

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 10, 2019 6:04 am
by antispam
The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik
Thanks for the prompt reply, that was exactly what I did to get it fixed - keep up the great work!

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 10, 2019 2:28 pm
by stuartkoh
It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.

One thing to be careful of if you're setting this up in an existing Splunk environment - unless you're really familiar with how things are setup, don't enable Splunk's UDP/514 input without first checking that syslog isn't already being received by something like syslog-ng or rsyslog. You could wind up with data loss or have events put into the wrong index or sourcetype.

It's also not best practices to run Splunk as root. For home use I guess you can get away with it, but for any production Splunk environment you will want to have Splunk running as a restricted user (user = splunk and group = splunk is commonly used).

When you install Splunk, you can set it to autostart on boot and also set the user if you want.
[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

For receiving syslog - I'm more familiar with syslog-ng, but I also see rsyslog being used successfully. Either one of these will work well for you.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 10, 2019 5:32 pm
by Jotne
This is already mention in section 1b)

If you install Ubuntu, (i think from 16.x), rsyslog is installed as default. But its not listening on port 514/UPD as default and you need to edit the config and restart syslog to get it running. So it should normally not be any conflict.

But in production environment I do also recommend running Splunk as a non root user, then use rsyslog to listen on 514/UDP. Then make Splunk index rsyslogs config.

If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 10, 2019 8:31 pm
by stuartkoh

If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.
I think that syslog-ng has an option that can be used to do this.
keep-timestamp()
Description: Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
https://www.syslog-ng.com/technical-doc ... -timestamp

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 10, 2019 8:50 pm
by stuartkoh
I also wanted to note that I'm not advocating that anyone switch from rsyslog or whatever they're currently using to syslog-ng unless they have good reason to do so.

I don't even really have an opinion on how they compare. I've been working with syslog-ng a bit so that's what I'm familiar with. I'm not trying to start a flame war over which to use. :-)

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Aug 15, 2019 1:39 am
by Spotegg
Hello! I have installed Splunk with Mikrotik module. Thanks, it's great!
Is there a way to organize monitoring of Internet connection on the router. For example, there is an Internet channel on ether1, and you need to somehow download data to Splunk about when the Internet crashed on the router. Maybe there is already such a script?

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Thu Aug 15, 2019 4:10 pm
by ferdytao

Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Posted: Fri Aug 16, 2019 12:21 pm
by ferdytao

Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?

Resolved! :D

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 17, 2019 5:35 pm
by ferdytao
I'm having a strange problem both on my synology with docker and my windows pc also, I configured everything as described, I got many logs from my router but after a while it stops reading while counters are still increasing.
Checking via tcpdump, logs are arriving to the server but is like they are not processed.
Immagine.jpg
Someone could help me? Maybe I got something wrong, I cannot image the server is flooded by a single router.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 17, 2019 5:53 pm
by Jotne
See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 17, 2019 7:19 pm
by ferdytao
See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing
Immagine.jpg

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 17, 2019 10:03 pm
by Jotne
What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 17, 2019 10:34 pm
by ferdytao
What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.
Yes the script is installed, I had the 30 min windows search and no data are showed even when the script starts. The stranger thing is that I have the same problem running Splunk on two different system
Immagine.jpg

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sun Aug 18, 2019 8:39 pm
by Jotne
Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sun Aug 18, 2019 11:12 pm
by ferdytao
Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?
Thanks for all your help and your time, it's very strange as you said... I opened syslog on port 5014 udp, i also checked the file props.conf on MikroTik app match the port.
I also tried as you said with echo "hello" and it's working.

I think there is some problem with timestamp because then it stops collect i got some error in internal index:
08-18-2019 19:09:43.054 +0200 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Sun Aug 18 01:28:00 2019). Context: source=udp:5014|host=192.168.1.1|syslog|
I checked around on google but I didn't find a solution

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Aug 19, 2019 8:21 am
by Jotne
You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Mon Aug 19, 2019 9:30 am
by ferdytao
You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.
It's a real riddle! :D
The server is running as a root on my last test with ubuntu but nothing is changed. Now I just changed the port from 5014 to default 514 (I used 5014 because on my synology is already kept) and seems to be working. I will keep it running to see what happen!

Thanks for all your support and time spent! ;)

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Aug 20, 2019 12:19 am
by ferdytao
Finally it's working!
Just uncommented the option: MAX_TIMESTAMP_LOOKAHEAD = 23 "inside /opt/splunk/etc/apps/MikroTik/default/props.conf" :D

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Tue Aug 20, 2019 8:27 am
by Jotne
For me it then sounds like you have a time problem. Its important that all clock is synced by using NTP.
Look at time on your router and on Splunk server. It should be within the same second.

Or date stamp is in the wrong format. Not sure how you can get this, since MT sends it in correct format.
You sends UDP directly to Splunk, not using rsyslog or syslog-ng etc?
Something adding data to your UDP packets?

From the manual
MAX_TIMESTAMP_LOOKAHEAD = <integer>
* Specifies how far (in characters) into an event Splunk should look for a timestamp.
* This constraint to timestamp extraction is applied from the point of the TIME_PREFIX-set location.
* For example, if TIME_PREFIX positions a location 11 characters into the event, and MAX_TIMESTAMP_LOOKAHEAD is set to 10, timestamp extraction will be constrained to characters 11 through 20.
So this sets where to look for the time stamp.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 24, 2019 5:00 pm
by stuartkoh
After I posted this reply I realized that I goofed.

I didn't read the post I replied to very well. I looked at the event you posted and just looked at the timestamp at the beginning of it and completely missed that it was a Splunk internal log.

So you can probably ignore what I had posted shown below about how to handle the timestamp in your logs. It does show how to setup timestamp detection, but for the wrong logs. :-)

What I originally posted is below:

Finally it's working!
Just uncommented the option: MAX_TIMESTAMP_LOOKAHEAD = 23 "inside /opt/splunk/etc/apps/MikroTik/default/props.conf" :D

I think you could probably change it to MAX_TIMESTAMP_LOOKAHEAD = 29

The timestamp in your events is "08-18-2019 19:09:43.054 +0200" and Splunk needs to know that all of that is part of the timestamp in order to parse things properly. If you use 23 for the lookahead, Splunk may not catch the timezone.

So maybe:

MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_PREFIX = ^
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%3N %z

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 24, 2019 5:05 pm
by stuartkoh
BTW, Packt is giving away a free copy of a decent book on Splunk today. https://www.packtpub.com/free-learning

Implementing Splunk 7 - Third Edition

James D. Miller

Mar 2018

576 pages

What will you learn

Enrich machine-generated data and transform it into useful, meaningful insights
Perform search operations and configurations, build dashboards, and manage logs
Extend Splunk services with scripts and advanced configurations to process optimal results

You have to sign up for an account with them, but I've had one for several years and they haven't been obnoxious about sending me too much e-mail or anything. The offer expires later today (in about 10 hours from when I'm posting this) so I hope people will see this in time to take advantage of this.

My only connection with Packt is as a customer. I've bought some of their books and gotten some good free ones from them too.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 24, 2019 5:24 pm
by stuartkoh
See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing

Sometimes Splunk will mistakenly identify the timestamp of an event as coming from the future because it misinterprets the timezone (or the sending device has the incorrect time set and actually is telling Splunk the events are from the future.

So it can be useful to run a search with "All time" selected in the time picker. If there are events "from the future" they will show up this way. You can also run an "All time (real time)" search and see events as they come in and see what the timestamps are. Just be careful with real-time searches on Splunk because they use up a lot of resources. So I only use them when I really need to, and I stop them as soon as I have what I want.

(There are some times when you need to have a real-time search running all the time for very time-critical alerting, but it's normally better for performance to change a real-time search into a scheduled one running every 5 minutes or whatever. Each real-time search basically ties up a CPU core, so if you have several real-time searches running at once you can see poor overall performance on your search heads.)

I don't know if you're getting events "from the future" or not, but it might be worth checking.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 24, 2019 5:29 pm
by stuartkoh
For me it then sounds like you have a time problem. Its important that all clock is synced by using NTP.
Look at time on your router and on Splunk server. It should be within the same second.

Yes! What Jotne said! :-)

It is extremely important to make sure the clocks on your devices and your Splunk servers are all in sync. Having time properly synced is very important for logging in general, but especially so for things like Splunk where you'll want to correlate events from different devices.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 24, 2019 9:28 pm
by Jotne
I think you could probably change it to MAX_TIMESTAMP_LOOKAHEAD = 29

The timestamp in your events is "08-18-2019 19:09:43.054 +0200" and Splunk needs to know that all of that is part of the timestamp in order to parse things properly. If you use 23 for the lookahead, Splunk may not catch the timezone.

So maybe:

MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_PREFIX = ^
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%3N %z
If you see props.conf in the MikroTik app, the MAX_TIMESTAMP_LOOKAHEAD = 23 line is not active due to # in front of it
So you should only change this if it does not work.
Still not sure why you need to sett it. Maybe you have other settings that influence on how Splunk handle MikroTik data.

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sun Aug 25, 2019 11:00 am
by ferdytao
Thanks everyone for the answers, It's seems to be working since 7 days now so I prefer to not changing anything! :D

I just uncommented the line MAX_TIMESTAMP_LOOKAHEAD = 23 and the time zone is also working properly, to be sure I also set the NTP to all devices and nothing else.

If I will have some time to spend, I will try to manage another install of splunk to try the LOOKAHEAD of 29

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Thu Aug 29, 2019 4:52 pm
by jacauc
I created a dropdown to select rules on the Live Attack View. All my mikrotik prefixes start with either Allow or Drop - Such as DropPings, AllowDNS etc.
Now I can quickly see the entire overview of my rules.

The updated dashboard code is below:
<form theme="dark">
  <label>MikroTik Live attack</label>
  <description>Shows form where ip comes from that tries to access your router on ports that are not opened using rule "FW_Drop_all_from_WAN"</description>
  <!--Version 
  2.4.0
  2.5.1 Change to "coalesce"
  2.7.1 Change to base searc
  2.7.2 Revritten to speed up and added more dropdown
  ######################################################
  #
  # Mikrotik Add-On for Splunk
  #
  # Copyright (C) 2019 Jotne
  # All Rights Reserved
  #
  # v2.7
  #
  ######################################################
  -->
  <search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule="*"
      | table _time host dest_port src_ip protocol rule
      | iplocation src_ip
      | replace "" with "Unknown" in City,Country
      | fields - Region
      | search 
          host="$Host$"
          protocol="$protocol$"
          dest_port="$port$"
          Country="$Country$"
          rule="$Rule$"
    </query>
  </search>
  <fieldset submitButton="false" autoRun="true">
    <input type="time">
      <label>Time span</label>
      <default>
        <earliest>rt-5m</earliest>
        <latest>rt</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Host</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="protocol">
      <label>Protocol</label>
      <search base="base_search">
        <query>
          | eval data=protocol
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="port">
      <label>Port</label>
      <search base="base_search">
        <query>
          | eval data=dest_port
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Country">
      <label>Country</label>
      <search base="base_search">
        <query>
          | eval data=Country
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Rule">
      <label>Rule</label>
      <search base="base_search">
        <query>
          | eval data=rule
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <choice value="Allow*">AnyAllow</choice>
      <choice value="Drop*">AnyDrop</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Drop*</default>
      <initialValue>Drop*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <map>
        <search base="base_search">
          <query>
            | eval info=src_ip."-".City."-".Country."-".protocol."/".dest_port
            | geostats globallimit=0 count by info
          </query>
        </search>
        <option name="height">600</option>
        <option name="refresh.display">progressbar</option>
      </map>
    </panel>
  </row>
</form>
2019-08-29_15-50-50.jpg

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 31, 2019 10:45 am
by Jotne
Interesting.

I may implement this in ver 2.8 :)

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 31, 2019 11:01 am
by jacauc
Interesting.

I may implement this in ver 2.8 :)
Please do!
Do you have a beta of 2.8 yet, or is it imminent for release?

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Posted: Sat Aug 31, 2019 10:29 pm
by Jotne
I am always working with some changes, and release new version when I have spare time to do it :)

Here is the working list for 2.8
# 2.8 (xx.xx.2019)
# Added interface changes
# Updated script to 2.7, added uncounted traffic and fastrack test, fixed when missing temperature
# Updated script to 2.8, fixed where system healt does not show anything (x86)
# Updated script to 2.9, get interface counters and you can also set modules true/false
# Updated script to 3.0, get CDP neigbhours
# Added new view "MikroTik Neighbor"
# Added uncounted packages to "MikroTik Traffic"
# Added board_name to "MikroTik Device List"
# Added Mikrotik interface Traffic"
# Fixed l2tp user extraction
# Fixed VPN when user info is missing
# Added access rule dropdown to "Mikrotik Live attack"