Using Splunk to monitor and graph various data from our MikroTik Routers is a nice and free way to help you showing what is going on in your network.
Splunk is free to use for logging up to 500MB pr day.
It can be used to monitor multiple devices. No ports needs to be opened (like with SNMP monitoring). All data are sent from the device to the Splunk monitor. Devices could be all around the world.
PS:
Traffic monitoring does not work correctly while fast track is enabled. Turn it off and you may loose throughput, so its something you should think about when using this type of monitoring. How to disable it: https://www.youtube.com/watch?v=6LaqhDm6PHI
What's new
A lot have changed since previous version. So much that its better to replace all MikroTik files in the Splunk server instead of trying to update files.
Before data was collected in three ways.
1. Syslog
2. SNMP
3. Scripts (remote)
Problem with SNMP is that you have to add duplicate configuration in Syslog to handle each box you monitor.
Problem with script is that you need to setup a job on the Splunk server that SSH inn to each box to get data.
The script tools did not log host IP, so you did not now from what box data was coming from.
SNMP and Scipts has been removed and change inn to local script on the Router that sends outs needed data to the Splunk using Syslog.
This way all data get correctly marked and you know for what Router you gets data from.
latest changes
# 2.8 (07.12.2019)
# Added interface changes
# Updated script to 2.7, added uncounted traffic and fastrack test, fixed when missing temperature
# Updated script to 2.8, fixed where system healt does not show anything (x86)
# Updated script to 2.9, get interface counters and you can also set modules true/false
# Updated script to 3.0, get CDP neigbhours
# Added new view "MikroTik Neighbor"
# Added uncounted packages to "MikroTik Traffic"
# Added board_name to "MikroTik Device List"
# Added Mikrotik interface Traffic"
# Fixed l2tp user extraction
# Fixed VPN when user info is missing
# Added access rule dropdown to "Live attack"
# Added some color and more field to "DHCP Request"
# Fixed Span and removed time scal "System Changes"
# Fixed missing mac in "Admin Conection"
# Added more lines and color in "DHCP request"
# Added output color in "Firewall Rules"
# Added rule menu to "Live attack"
# Added "DATETIME_CONFIG = CURRENT" to props.conf
Installation
1) On your PC (Windows/Linux)
-----------------------------------
1a) Download and install Splunk (Windows or Linux(recommended))
PS you need an account to download. It's free to create.
https://www.splunk.com/en_us/download/s ... prise.html
PS you need to create an account to download the file. Free to download and use (up to 500MB/day)
1b) PS: To install Splunk as a non root user, look here: viewtopic.php?p=677233#p677233
Splunk can fine run as root user, but not recommended.
1c) Change to free license group. Very important to do before 30 day of use. !!!!!!!!!!!!!!!!!!!!
Web gui:
1d) Settings->licensing->Change license group->Free licnse->Save
1e) Open Windows Firewall for UDP on Windows (On linux its not blocked)
Web gui:
Start->type "adv"->Select:Widows Firewall with Advanced Security->Sect Inbound rules->Right Click "Inbound Rules">New Rule-Port-Next->UDP->Specific local ports->514->Next->Next->Next->Name "syslog"
1f) Allow UDP 514 (syslog)
Web gui:
Setting->Datainputs->Add new (behind the UDP)->Port 514->Next->Sourcetype type syslog and select syslog->Next-Submit
1g) Download the Splunk spl file: 1h) Extract the spl file
From Start page in Splunk, click the gear behind Apps or
from top meny click Apps->Manage Apps
Then select Install app from file and select the spl file
1i) A restart of Splunk may be needed.
Web gui:
Settings->Server controls->Restart Splunk
1j) Upgrade form previous version.
Since no files are renamed, just data added, you should be fine by just replacing current files.
You can delete the MikroTik folder if you like. No logged data will be deleted.
If you have custom dashboards, menus, saved search (reports) etc, you need to merge the configuration files.
2) On Your MikroTik Router
-----------------------------
Before you setup logging, you should make an unique identifier of your route. Important if you have more than one router to monitor.
Code: Select all
/system identity set name=Router-London-22You need to make your Router able to send Syslog messages.
Web gui:
System->Logging->Action->Add New->Name (your server name)->Type:Remote->Remote Address:ip your syslog->Ok
Cli
Code: Select all
/system logging action add name=logserver target=remote remote=192.168.1.50 remote-port=5142b) Then select what to log.
I do suggest that you send all DHCP logs including debug and all other logs that are not debug.
It is very important to name the prefix like this "MikroTik" and not "mikrotik" or some other.
Splunk uses the MikroTik prefix to find out what type of syslog data that is coming to it.
Uppercase T and uppercase M, rest are lowercase
Web gui:
System->Logging->Rules->Add new->Topics:dhcp->Prefix:MikroTik->action:your syslog server->Ok
System->Logging->Rules->Add new->Topics:!debug->Prefix:MikroTik->action:your syslog server->Ok
Cli:
Code: Select all
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug
/system logging add action=logserver prefix=MikroTik topics=hotspot2c) Select what to log
NB Do not use more than 20 charters, or else it start to clip other part of the log!!!!!!!!!!!
To log the Firewall and Nat rules, you need to turn on logging and add Log Prefix (under action).
Do not log more than needed. Logging rules like defconf: accept established,related rules will flod your log,
Below is a sample on how to name the log rules. Do not need to follow this rule, but it makes it more uniform.
Code: Select all
Rule name logging
==================
Format:
x_y_z
z=name/info
Example
-------
Filter Rule Forard allow HTTP
FF_A_Http
Filter Route Input Drop ICMP
FI_D_Icmp
Nat HTTP
ND_DE_Http
Mangle Mark HTTP packets
MF_MP_Http
Filter Rule
------------------
x=
FF Filter Forward
FI Filter Input
FO Filter Output
FX Filter Custom list
y=
A Accept
AD Add to dst address list
AS Add to src address list
D Dropp
F Fast track
J Jump
L Log
P Passthrough
RJ Reject
RT Return
T Tarpit
Nat Rule
------------------
x=
ND Dest nat
NS Source nat
y=
A Accept
AD Add to dst address list
AS Add to src address list
DE Dst-nat
J Jump
L Log
M Masquerade
N Netmap
P Passthrough
RE Redirect
RT Return
SA same
S Src-nat
Raw
------------------
x=
RP Filter Raw Prerouting
RO Filter Raw Output
y=
A Accept
AD Add to dst address list
AS Add to src address list
F Fast track
D Dropp
J Jump
L Log
N No track
P Passthrough
RT Return
Mangle
------------------
x=
MF Mangle Forward
MI Mangle Input
MP Mangle Postrouing
MR Mangle Prerouting
y=
A Accept
AD Add to address list
AS Add to dst address list
CD Change DSCP
CM Change MSS
CT Change TTL
CL Clear DF
F Fast track
J Jump
L Log
MC Marc connection
MP Mark packets
MR Mark routing
P Passthrough
RT Return
RO Route
S Set proirity
SP Sniff PC
ST Sniff TZSP
SI Strip IPv4 optionsWeb gui:
IP->Firewall->selec:defconf: drop all not coming from LAN->Log:v->Log Prefix:FI_D_port-test
This will populate the MikroTik Live attack view.
2e) Accounting
To get accounting data, you need to enable it on the MikroTik router. (MikroTik Traffic dashboard)
Web gui:
IP-> Accounting -> Enable Accounting -> mark Threshould:2560 OK
Cli:
Code: Select all
/ip accounting set enabled=yes threshold=2560To get all the other data like Traffic accounting, uPnP, System health, System resources and DHCP pool information you need this script on the MikroTik. It need to be named: Data_to_Splunk_using_Syslog
I did prefer to list the script without all the newline code etc need for cut and past from CLI, så do use WinBox or Webgui to add the script.
In the top of the script, you can set a module to true/false. If you do not use wifi, set :local Wireless false
Code: Select all
# Collect information from Mikrotik RouterOS
# v 3.2 Jotne 2019
# ----------------------------------
# What data to collect. Set to false to skip the section
# ----------------------------------
:local SystemResource true
:local SystemInformation true
:local SystemHealth true
:local TrafficData true
:local uPnP true
:local Wireless true
:local AddressLists true
:local DHCP true
:local Neighbor true
:local InterfaceData true
# Interface to get data from (using regex)
:local IF "ether.*"
# Example
# "ether.*" All ethernet interfaces
# "^ether[1-5]\$" Only ethernet 1 to 5
# ".*" All interfaces (Briges/VLAN/pptp/Ether ++)
# "ether(1|2)\$" interface ethernet 1 and 2 (/$ needed to prevent ether11 etc)
# Collect system resource
# ----------------------------------
if ($SystemResource) do={
:local cpuload ([/system resource get cpu-load])
:local freemem ([/system resource get free-memory]/1048576)
:local totmem ([/system resource get total-memory]/1048576)
:local freehddspace ([/system resource get free-hdd-space]/1048576)
:local totalhddspace ([/system resource get total-hdd-space]/1048576)
:local up ([/system resource get uptime])
:log info message="script=resource free_memory=$freemem MB total_memory=$totmem MB free_hdd_space=$freehddspace MB total_hdd_space=$totalhddspace MB cpu_load=$cpuload uptime=$up"
}
# Get traffic data (accounting data)
# ----------------------------------
if ($TrafficData) do={
# Test if fasttrack is enabled and give warning
:if ([/ip firewall filter find where (action=fasttrack-connection && !disabled)] != "") do={
:log info message=("script=traffic,fasttrack=1")
} else={
:log info message=("script=traffic,fasttrack=0")
}
# Test if accounting is enabled and if yes, get data
if ([/ip accounting get enabled]=yes) do={
/ip accounting snapshot take
# Get uncounted data
/ip accounting uncounted {
:log info message=("script=uncounted,bytes=".[get bytes].",packets=".[get packets])}
# Send data to loggin server
foreach logline in=[/ip accounting snapshot find] do={
:local output "$[/ip accounting snapshot print as-value from=$logline]"
:set ( "$output"->"script" ) "traffic"
:log info message="$output"
}
}
}
# Get interface data
# ----------------------------------
if ($InterfaceData) do={
:foreach interface in=[/interface find where name~"$IF"] do={
:delay 100ms
:local iname [/interface get $interface name]
:local monitor [/interface monitor-traffic $interface as-value once]
:local speedRX ($monitor->"rx-bits-per-second")
:local speedTX ($monitor->"tx-bits-per-second")
:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
}
}
# Finding dynmaic lines used in uPnP
# ----------------------------------
if ($uPnP) do={
:foreach logline in=[/ip firewall nat find dynamic=yes] do={
:local output "$[/ip firewall nat print as-value from=$logline]"
:set ( "$output"->"script" ) "upnp"
:log info message="$output"
}
}
# Collect system information
# ----------------------------------
if ($SystemInformation) do={
:local version ([/system resource get version])
:local board ([/system resource get board-name])
:local model ([/system routerboard get model]);
:local serial ([/system routerboard get serial-number])
:local identity ([/system identity get name])
:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial identity=\"$identity\""
}
# Collect system health
# ----------------------------------
if ($SystemHealth) do={
:if (([/system health get]~"state=disabled" || [/system health get]="")=false) do={
:local voltage ([/system health get voltage]/10)
:local temperature ([/system health get temperature])
:log info message="script=health voltage=$voltage V temperature=$temperature C"
}
}
# Sends wireless client data to log server
# ----------------------------------
if ($Wireless) do={
:do {
:if ([:len [/interface wireless find ]]>0) do={
:foreach logline in=[/interface wireless registration-table find] do={
:local output "$[/interface wireless registration-table print as-value from=$logline]"
:set ( "$output"->"script" ) "wifi"
:log info message="$output"
}
}
} on-error={}
}
# Count IP in address-lists
#----------------------------------
if ($AddressLists) do={
:local array [ :toarray "" ]
:local addrcntdyn [:toarray ""]
:local addrcntstat [:toarray ""]
:local test
:foreach id in=[/ip firewall address-list find] do={
:local rec [/ip firewall address-list get $id]
:local listname ($rec->"list")
:local listdynamic ($rec->"dynamic")
:set ( $array->$listname ) 1
if ($listdynamic = true) do={
:set ($addrcntdyn->$listname) ($addrcntdyn->$listname+1)
} else={
:set ($addrcntstat->$listname) ($addrcntstat->$listname+1)}
}
:foreach k,v in=$array do={
:log info message=("script=address_lists list=$k dynamic=".(($addrcntdyn->$k)+0)." static=".(($addrcntstat->$k)+0))}
}
# Get MNDP (CDP) Neighbors
# ----------------------------------
if ($Neighbor) do={
:foreach neighborID in=[/ip neighbor find] do={
:local nb [/ip neighbor get $neighborID]
:foreach key,value in=$nb do={
:local newline [:find $value "\n"]
:if ([$newline]>0) do={
:set $value [:pick $value 0 $newline]
}
:set ( "$nb"->"$key" ) "\"$value\""
}
:set ( "$nb"->"script" ) "\"neighbor\""
:log info message="$nb"
}
}
# Collect DHCP Pool information
# ----------------------------------
if ($DHCP) do={
/ip pool {
:local poolname
:local pooladdresses
:local poolused
:local minaddress
:local maxaddress
:local findindex
# Iterate through IP Pools
:foreach pool in=[find] do={
:set poolname [get $pool name]
:set pooladdresses 0
:set poolused 0
# Iterate through current pool's IP ranges
:foreach range in=[:toarray [get $pool range]] do={
# Get min and max addresses
:set findindex [:find [:tostr $range] "-"]
:if ([:len $findindex] > 0) do={
:set minaddress [:pick [:tostr $range] 0 $findindex]
:set maxaddress [:pick [:tostr $range] ($findindex + 1) [:len [:tostr $range]]]
} else={
:set minaddress [:tostr $range]
:set maxaddress [:tostr $range]
}
# Calculate number of ip in one range
:set pooladdresses ($maxaddress - $minaddress)
# /foreach range
}
# Test if pools is used in DHCP or VPN and show leases used
:local dname [/ip dhcp-server find where address-pool=$poolname]
:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
:set poolused [:len [used find pool=[:tostr $poolname]]]
} else={
# DHCP server found, count leases
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
:set poolused [:len [/ip dhcp-server lease find where server=$dname]]}
# Send data
:log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")
# /foreach pool
}
# /ip pool
}
}
Code: Select all
/system scheduler
add interval=5m name="Data to Splunk" on-event=Data_to_Splunk_using_SyslogIf you have problems or comments, please feel free to ask
Example screen shots:
Maybe you will find some errors. Please let me know.
