Using Splunk to monitor and graph various data from our MikroTik Routers is a nice and free way to help you showing what is going on in your network.
Splunk is free to use for logging up to 500MB pr day.
You can request a 10GB/day developer license here: https://dev.splunk.com/enterprise/dev_license/
NB logging large amount of Accouning, DNS or firewall rules quickly eats up license, so I do recommend to turn off Accouning/DNS logging to start with.
Splunk can be used to monitor multiple devices. No ports needs to be opened (like with SNMP monitoring). All data are sent from the device to the Splunk monitor (using sctipt and syslog). Devices could be all around the world.
PS:
Traffic monitoring does not work correctly while fast track is enabled (and its removed in v7.x of RouterOS. Changed to Kid control). Turn fast track off and you may loose throughput, so its something you should consider when using this type of monitoring. How to disable fast track: https://www.youtube.com/watch?v=6LaqhDm6PHI
latest changes
# 3.6 (05.05.2022)
# NB Delete old app (copy custom made config) before install v3.6
# Change data to store in Mikrotik index, instead of default index
# Change how rsyslog handles data. Did fail if there was more than one type of input
# Updeted script in "MikroTik DHCP to Static"
# Uses new Index, important to look at macros.conf and set correct index.
# Added colors to "MikroTik Admin user login"
Installation
On your PC
1) Works on Windows and Linux, but use Linux (clearly the best choice and also used in all post here)
-----------------------------------
1a) Download and install Splunk (Windows or Linux(Ubuntu recommended))
PS you need an account to download. It's free to create.
https://www.splunk.com/en_us/download/s ... prise.html
PS you need to create an account to download the file. Free to download and use (up to 500MB/day)
PS remember to set timezone on Windows/Linux, or else logging time will be wrong.
1b) PS: To install Splunk as a non root user, recommended. (needs an external syslog reciver)
Splunk setup:
viewtopic.php?t=179960#p888802
rsyslog setup
viewtopic.php?t=179960#p888803
Splunk can run as root user, but not recommended.
1c) Change to free license group. Very important to do before 30 day of use. !!!!!!!!!!!!!!!!!!!!
Web gui:
1d) Settings->licensing->Change license group->Free licnse->Save
1e) Open Windows Firewall for UDP on Windows (On linux its not blocked)
Web gui:
Start->type "adv"->Select:Widows Firewall with Advanced Security->Sect Inbound rules->Right Click "Inbound Rules">New Rule-Port-Next->UDP->Specific local ports->514->Next->Next->Next->Name "syslog"
1f) Allow UDP 514 (syslog), only if you run Splunk as root and not using rsyslog server. (rsyslog can be used when splunk runs as root or as non-root user.)
If running Splunk as non ROOT user or like to use external syslog reciever, see 1b for non-root)
Web gui:
Setting->Datainputs->Add new (behind the UDP)->Port 514->Next->Sourcetype type syslog and select syslog->Next-Submit
1g) Download the Splunk spl file: 1h) Extract the spl file
From Start page in Splunk, click the gear behind Apps or
from top meny click Apps->Manage Apps
Then select Install app from file and select the spl file
1i) A restart of Splunk may be needed.
Web gui:
Settings->Server controls->Restart Splunk
1j) Upgrade form previous version.
Some time files are renamed, so if you have not change any original files, just delete the MikroTik folder.
No logged data will be deleted.
If you have custom dashboards, menus, saved search (reports) etc, you need to merge the configuration files.
They are normal stored in "local" folder.
2) On Your MikroTik Router
-----------------------------
Before you setup logging, you should make an unique identifier of your route. Important if you have more than one router to monitor.
Code: Select all
/system identity set name=Router-London-222a) Syslog
You need to make your Router able to send Syslog messages.
Web gui:
System->Logging->Action->Add New->Name (your server name)->Type:Remote->Remote Address:ip your syslog->Ok
Cli
Code: Select all
/system logging action add name=logserver target=remote remote=192.168.1.50 remote-port=5142b) Then select what modules to log.
I do suggest that you send all DHCP logs including debug and all other logs that are not debug.
It is very important to name the prefix like this "MikroTik" and not "mikrotik" or some other.
Splunk uses the MikroTik prefix to find out what type of syslog data that is coming to it.
Uppercase T and uppercase M, rest are lowercase
Web gui:
Code: Select all
System->Logging->Rules->Add new->Topics:dhcp->Prefix:MikroTik->action:your syslog server->Ok
System->Logging->Rules->Add new->Topics:!debug,!packet,!snmp->Prefix:MikroTik->action:your syslog server->OkCode: Select all
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=hotspotCode: Select all
/system logging set [find topics="info"] topics=info,!firewall2c) Select what rules to log
NB Do not use more than 20 charters, or else it start to clip other part of the log!!!!!!!!!!!
To log the Firewall and Nat rules, you need to turn on logging and add Log Prefix (under action).
Do not log more than needed. Logging rules like defconf: accept established,related rules will flod your log,
Below is a sample on how to name the log rules. You do not need to follow this rule, but it makes it more uniform.
Code: Select all
Rule name logging
==================
Format:
x_y_z
x=<where and direction>
y=<what to do>
z=<name/info>
Example
-------
Filter Rule Forard allow HTTP
FF_A_Http
Filter Route Input Drop ICMP
FI_D_Icmp
Nat HTTP
ND_DE_Http
Mangle Mark HTTP packets
MF_MP_Http
Filter Rule
------------------
x=
FF Filter Forward
FI Filter Input
FO Filter Output
FX Filter Custom list
y=
A Accept
AD Add to dst address list
AS Add to src address list
D Dropp
F Fast track
J Jump
L Log
P Passthrough
RJ Reject
RT Return
T Tarpit
Nat Rule
------------------
x=
ND Dest nat
NS Source nat
y=
A Accept
AD Add to dst address list
AS Add to src address list
DE Dst-nat
J Jump
L Log
M Masquerade
N Netmap
P Passthrough
RE Redirect
RT Return
SA same
S Src-nat
Raw
------------------
x=
RP Filter Raw Prerouting
RO Filter Raw Output
y=
A Accept
AD Add to dst address list
AS Add to src address list
F Fast track
D Dropp
J Jump
L Log
N No track
P Passthrough
RT Return
Mangle
------------------
x=
MF Mangle Forward
MI Mangle Input
MP Mangle Postrouing
MR Mangle Prerouting
y=
A Accept
AD Add to address list
AS Add to dst address list
CD Change DSCP
CM Change MSS
CT Change TTL
CL Clear DF
F Fast track
J Jump
L Log
MC Marc connection
MP Mark packets
MR Mark routing
P Passthrough
RT Return
RO Route
S Set proirity
SP Sniff PC
ST Sniff TZSP
SI Strip IPv4 options2d) You should at least log this rule "defconf: drop all not coming from LAN" with this prefix: FI_D_port-test
Web gui:
IP->Firewall->selec:defconf: drop all not coming from LAN->Log:v->Log Prefix:FI_D_port-test
This will populate the MikroTik Live attack view.
2e) Accounting (new version in 3.5)
To get accounting data, you need to turn on Kid Control on the MikroTik router. (MikroTik Traffic dashboard)
Cli:
Code: Select all
/ip kid-control
add fri=0s-1d mon=0s-1d name=Monitor sat=0s-1d sun=0s-1d thu=0s-1d tue=0s-1d wed=0s-1d
2f) Main Collector Script
To get all the other data like Traffic accounting, uPnP, System health, System resources and DHCP pool information you need this script on the MikroTik. Create this script with name Data_to_Splunk_using_Syslog and cut and past code using gui.
In the top of the script, you can set a module to true/false. If you do not use wifi, set :local Wireless false
Code: Select all
# Collect information from Mikrotik RouterOS
# Jotne 2021
:log info message="script=version ver=4.8"
# ----------------------------------
# What data to collect. Set to false to skip the section
# ----------------------------------
:local SystemResource true
:local SystemInformation true
:local SystemHealth true
:local TrafficData true
:local AccuntData true
:local uPnP true
:local Wireless true
:local AddressLists true
:local DHCP true
:local Neighbor true
:local InterfaceData true
:local CmdHistory true
:local CAPsMANN false
# Collect system resource
# ----------------------------------
:if ($SystemResource) do={
/system resource
:local cpuload [get cpu-load]
:local freemem ([get free-memory]/1048576)
:local totmem ([get total-memory]/1048576)
:local freehddspace ([get free-hdd-space]/1048576)
:local totalhddspace ([get total-hdd-space]/1048576)
:local up [get uptime]
:local sector [get write-sect-total]
:log info message="script=resource free_memory=$freemem MB total_memory=$totmem MB free_hdd_space=$freehddspace MB total_hdd_space=$totalhddspace MB cpu_load=$cpuload uptime=$up write-sect-total=$sector"
}
# Make some part only run every hours
# ----------------------------------
:global Hour
:local run false
:local hour [:pick [/system clock get time] 0 2]
:if ($Hour != $hour) do={
:global Hour $hour
:set run true
}
# Get NTP status
# ----------------------------------
:local ntpstatus ""
:if ([:len [/system package find where !disabled and name=ntp]] > 0 or [:tonum [:pick [/system resource get version] 0 1]] > 6) do={
:set ntpstatus [/system ntp client get status]
} else={
:if ([:typeof [/system ntp client get last-update-from]] = "nil") do={
:set ntpstatus "using-local-clock"
} else={
:set ntpstatus "synchronized"
}
}
:log info message="script=ntp status=$ntpstatus"
# Get interface traffic data for all interface
# ----------------------------------
:if ($TrafficData) do={
:foreach id in=[/interface find] do={
:local output "$[/interface print stats as-value where .id=$id]"
:set ( "$output"->"script" ) "if_traffic"
:log info message="$output"
}
}
# Get traffic data v2 (Kid Control)
# ----------------------------------
:if ($AccuntData) do={
:foreach logline in=[/ip kid-control device find] do={
:local output "$[/ip kid-control device get $logline]"
:set ( "$output"->"script" ) "kids"
:log info message="$output"
}
}
# Finding dynmaic lines used in uPnP
# ----------------------------------
:if ($uPnP) do={
:foreach logline in=[/ip firewall nat find where dynamic=yes and comment~"^upnp "] do={
:local output "$[/ip firewall nat print as-value from=$logline]"
:set ( "$output"->"script" ) "upnp"
:log info message="$output"
}
}
# Collect system information
# ----------------------------------
:local model na
:local serial na
:local ffirmware na
:local cfirmware na
:local ufirmware na
:if ($SystemInformation and $run) do={
:local version ([/system resource get version])
:local board ([/system resource get board-name])
:if ($board!="CHR") do={
/system routerboard
:set model ([get model])
:set serial ([get serial-number])
:set ffirmware ([get factory-firmware])
:set cfirmware ([get current-firmware])
:set ufirmware ([get upgrade-firmware])
}
:local identity ([/system identity get name])
:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial identity=\"$identity\" factory-firmware=\"$ffirmware\" current-firmware=\"$cfirmware\" upgrade-firmware=\"$ufirmware\""
}
# Collect system health
# ----------------------------------
:if ($SystemHealth) do={
:do {
# New version
:foreach id in=[/system health find] do={
:local health "$[/system health get $id]"
:set ( "$health"->"script" ) "health"
:log info message="$health"
}
} on-error={
# Old version
:if (!([/system health get]~"(state=disabled|^\$)")) do={
:local health "$[/system health get]"
:set ( "$health"->"script" ) "health"
:log info message="$health"
}
}
}
# Sends wireless client data to log server
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wlan]]>0) do={
/interface wireless registration-table
:foreach i in=[find] do={
:log info message=".id=$i;ap=$([get $i ap]);interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=$([get $i signal-strength]);tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
}
}
# Count IP in address-lists
#----------------------------------
:if ($AddressLists) do={
:local array [ :toarray "" ]
:local addrcntdyn [:toarray ""]
:local addrcntstat [:toarray ""]
:local test
:foreach id in=[/ip firewall address-list find] do={
:local rec [/ip firewall address-list get $id]
:local listname ($rec->"list")
:local listdynamic ($rec->"dynamic")
:if (!($array ~ $listname)) do={ :set array ($array , $listname) }
:if ($listdynamic = true) do={
:set ($addrcntdyn->$listname) ($addrcntdyn->$listname+1)
} else={
:set ($addrcntstat->$listname) ($addrcntstat->$listname+1)}
}
:foreach k in=$array do={
:log info message=("script=address_lists list=$k dynamic=".(($addrcntdyn->$k)+0)." static=".(($addrcntstat->$k)+0))}
}
# Get MNDP (CDP) Neighbors
# ----------------------------------
:if ($Neighbor and $run) do={
:foreach neighborID in=[/ip neighbor find] do={
:local nb [/ip neighbor get $neighborID]
:local id [:pick ("$nb"->".id") 1 99]
:foreach key,value in=$nb do={
:local newline [:find $value "\n"]
:if ([$newline]>0) do={
:set value [:pick $value 0 $newline]
}
:log info message="script=neighbor nid=$id $key=\"$value\""
}
}
}
# Collect DHCP Pool information
# ----------------------------------
:if ($DHCP and $run) do={
/ip pool {
:local poolname
:local pooladdresses
:local poolused
:local minaddress
:local maxaddress
:local findindex
# Iterate through IP Pools
:foreach pool in=[find] do={
:set poolname [get $pool name]
:set pooladdresses 0
:set poolused 0
# Iterate through current pool's IP ranges
:foreach range in=[:toarray [get $pool range]] do={
# Get min and max addresses
:set findindex [:find [:tostr $range] "-"]
:if ([:len $findindex] > 0) do={
:set minaddress [:pick [:tostr $range] 0 $findindex]
:set maxaddress [:pick [:tostr $range] ($findindex + 1) [:len [:tostr $range]]]
} else={
:set minaddress [:tostr $range]
:set maxaddress [:tostr $range]
}
# Calculate number of ip in one range
:set pooladdresses ($maxaddress - $minaddress)
# /foreach range
}
# Test if pools is used in DHCP or VPN and show leases used
:local dname [/ip dhcp-server find where address-pool=$poolname]
:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
:set poolused [:len [used find pool=[:tostr $poolname]]]
} else={
# DHCP server found, count leases
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
:set poolused [:len [/ip dhcp-server lease find where server=$dname]]}
# Send data
:log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")
# /foreach pool
}
# /ip pool
}
}
# Get detailed command history RouterOS >= v7
# ----------------------------------
:if ([:tonum [:pick [/system resource get version] 0 1]] > 6 and $CmdHistory) do={
:global cmd
:local f 0
:foreach i in=[/system history find] do={
:if ($i = $cmd) do={ :set f 1 }
:if ($f != 1) do={
:log info message="StartCMD"
:log info message=[/system history get $i]
:log info message="EndCMD"
}
}
:global cmd [:pick [/system history find] 0]
}
# Test if CAPsMANN is installed, if yes, run capsmann script.
# ----------------------------------
:if ( ([:len [/interface find where type="cap"]] > 0) and $CAPsMANN) do={ /system script run capsman }
# End Script
2g) Then schedule the script to run every 5 minutes:
Code: Select all
/system scheduler
add interval=5m name="Data to Splunk" on-event=Data_to_Splunk_using_Syslog3) Starting up
3a) Startup information
Some parts of the scripts runs only every hour, and some jobs in Splunk runs once a day.
So it will take time before all devices er named correctly. To speed things up some, do this:
After running for some hour and you see that data are coming, in Splunk go to:
Apps->Mikrotik->Reports and run both Device table updater and DHCP table updater by clicking on Open In Search behind the app
4) Debugging
4a) See if any data are comming inn to splunk at all. Do a search in Splunk for:
Code: Select all
index=*Code: Select all
index=* | table _time sourcetype _raw4c). See that _raw does contain only data and not time and other info. Do a search in Splunk for:
Code: Select all
index=* | table _raw
dns MikroTik: done query: #640030 adservice.google.no 216.58.211.2
dhcp,debug,packet MikroTik: Client-Id = 01-6C-3B-6B-88-34-3F
firewall,info MikroTik: FI_D_port-test input: in:ether1 out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 47.118.40.92:52503->92.220.205.91:2376, len 404d). Verify that all files has same user:group (root:root or splunk:splunk if run as non-root user)
4e). Look for error written in file or error in file name. inputs.conf not input.conf
4f). Read trough all steps on how to install if some does not work
4g). License problems.
Not convert license to Free license before 30 days or indexing more than 500MB/day?
The Free license will prevent searching if there are 3 license warnings in a rolling 30 day window. If that happens, Splunk Free continues to index your data but disables search functionality. You will regain search when you are below 3 license violation warnings in a 30 day period. See About license violations.
How to solve this+
1. Convert to free and wait 30 days if you did not convert it.
2. Passing 3 times? Reduce license <500MB and wait 30 days.
3. Reintall Splunk
4. Get a Free 10GB/day developer license ( https://dev.splunk.com/enterprise/dev_license )
4h) Limit inout. Data comes from two sources in Splunk.
1. The log setup. DNS and other stuff. To remove DNS change to
Code: Select all
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp,!dnsChange true to false each block you like to stop.
4i) 1. You see you get Duplicate Values under host. This may be that you did change name on one device ore have multiple host with same name.
Install Lookup Editor (a Splunk addon app) in Splunk if you do not already have done. Got to Apps -> Lookup Editor, select device_kvstore and open it. Remove the duplicate/old items.
4x). Still problems: ask here
but certainly not overly complex.
