📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Version 4.0 21.03.2024
.
NB, there seems to be an error with script running in routerOS 7.13.3, so upgrade to 7.13.4 or later if you have 7.13.3

4.0 important Information:

  1. All routers needs serial number in their logging tags (section 2a). If not you will not get any dashboard to work. To add serial, run the script in section 2a on all routers. (cut and past to a terminal windows.
  2. If you for some reason has an other system logging action other than logserver, you need to edit the serial update script in 2a to use your action name
    .
    Top_logo.jpg
    Using Splunk to monitor and graph various data from our MikroTik Routers is a nice and free way to help you showing what is going on in your network.
    Splunk is free to use for logging up to 500MB pr day.
    You can request a 10GB/day developer license here: https://dev.splunk.com/enterprise/dev_license/

NB logging large amount of Accouning, DNS or firewall rules quickly eats up license, so I do recommend to turn off Accouning/DNS logging to start with.

Splunk can be used to monitor multiple devices. No ports needs to be opened (like with SNMP monitoring). All data are sent from the device to the Splunk monitor (using sctipt and syslog). Devices could be all around the world.

PS:
Traffic monitoring does not work correctly while fast track is enabled (and its removed in v7.x of RouterOS. Changed to Kid control). Turn fast track off and you may loose throughput, so its something you should consider when using this type of monitoring. How to disable fast track: https://www.youtube.com/watch?v=6LaqhDm6PHI

latest changes

Added to GIT

4.0 (21.03.2024)

Changed to use serial in all dashboard

Changed many regex due to added serial

Changed to use MikroTik index directly without macro

Removed host_name and use identity in all dashboard

Change device_table script to update every hour, not every day

Fixed form version. Should always be 1.1

Added DHCP lookup of client name in mikrotik_accounting_traffic

Fixed romon info extraction. Use host_name in graphs in mikrotik_admin_user_login

Joined multipe IP for the same host, Fixed list for multiple firmware, fixed errors in varios dashboards in mikrotik_device_list

Added Time Span and separated IP address from name with - in mikrotik_dns_request.

Rewritten calcualtion to give correctly bps and now works with multiple hosts, Added graph to show total bytex tx/rx in mikrotik_interface_traffic

Installation
On your PC
1) Works on Windows and Linux, but use Linux (clearly the best choice and also used in all post here)

1a) Download and install Splunk (Windows or Linux(Ubuntu recommended))
PS you need an account to download. It’s free to create.
https://www.splunk.com/en_us/download/splunk-enterprise.html
PS you need to create an account to download the file. Free to download and use (up to 500MB/day)
PS remember to set timezone on Windows/Linux, or else logging time will be wrong.


1b) PS: To install Splunk as a non root user, recommended. (needs an external syslog reciver)
Splunk setup:
https://forum.mikrotik.com/viewtopic.php?t=179960#p888802
rsyslog setup
https://forum.mikrotik.com/viewtopic.php?t=179960#p888803

Splunk can run as root user, but not recommended.

1c) Change to free license group. Very important to do before 30 day of use. !!!
Web gui:
1d) Settings->licensing->Change license group->Free licnse->Save

1e) Open Windows Firewall for UDP on Windows (On linux its not blocked)
Web gui:
Start->type “adv”->Select:Widows Firewall with Advanced Security->Sect Inbound rules->Right Click “Inbound Rules”>New Rule-Port-Next->UDP->Specific local ports->514->Next->Next->Next->Name “syslog”

1f) Allow UDP 514 (syslog), only if you run Splunk as root and not using rsyslog server. (rsyslog can be used when splunk runs as root or as non-root user.)
If running Splunk as non ROOT user or like to use external syslog reciever, see 1b for non-root)
Web gui:
Setting->Datainputs->Add new (behind the UDP)->Port 514->Next->Sourcetype type syslog and select syslog->Next-Submit

1g) Get the Splunk
You can use one of the two option
…1. Download the file from the forum here:
MikroTik4.0.rar (73.2 KB)
.
…2 Get the GIT repository
https://github.com/Jotne/MikroTik

1h)Installing the Splunk
…1 If you get the file form the from the forum, extract the compressed file to get the Mikrotik.spl file and install it using Splunk Gui:.
From Start page in Splunk, click the gear behind Apps or
from top meny click Apps->Manage Apps
Then select Install app from file and select the spl file

…2 Download it from GIT.
If you download it from GIT copy folder with all content to %Splunk%/etc/apps. so it becomes:
%Splunk%/etc/apps/MikroTik/default/…
%Splunk%/etc/apps/MikroTik/metadata/…
%Splunk%/etc/apps/MikroTik/README
…

1i) A restart of Splunk may be needed.
Web gui:
Settings->Server controls->Restart Splunk

1j) Upgrade form previous version.
Some time files are renamed, so if you have not change any original files, just delete the MikroTik folder.
No logged data will be deleted.
If you have custom dashboards, menus, saved search (reports) etc, you need to merge the configuration files.
They are normal stored in “local” folder.


2a) Syslog
You need to make your Router able to send Syslog messages.

Cut and past the following to a terminal window.

Set the log server: (change IP to your log server)

/system logging action add name=logserver target=remote remote=192.168.1.50 remote-port=514

Sets the correct tag MikroTik so data can be picked up. It also set the RouterBoard serial to all packet so multiple routers behind nat can be identified. If its not a routerBard and do not have a serial, a random hex number of 12 digits are set,

{
:local serial na
:foreach id in=[/system/logging/find action prefix~"MikroTik"] do={
/system logging remove numbers=$id
}
:local boardName [/system resource get board-name]

:if ($boardName = "CHR") do={
	:set serial [/system license get system-id]
} else= {
	:local arch [/system resource get architecture-name]

	:if ($arch = "x86_64") do={
		:set serial [/system license get software-id]
	} else={
		:set serial ([/system routerboard get serial-number])
	}
}
# Log hotspot at debug level
/system logging add action=logserver prefix="serial=$serial MikroTik" topics=hotspot

# Do not log debug, packet nor snmp, but all other will be logged.
/system logging add action=logserver prefix="serial=$serial MikroTik" topics=!debug,!packet,!snmp

# Do not log info internal logs, only warning and error logs
/system/logging/disable numbers=[/system logging/find where topics~"info"] 
}

PS Do NOT select BSD Syslog. It will mess up the logging format. BSD adds date and severity, but in the same time remove the prefix that tells what modules that do send logs to syslog.


2b) Select what rules to log
NB Do not use more than 20 charters, or else it start to clip other part of the log!!!
To log the Firewall and Nat rules, you need to turn on logging and add Log Prefix (under action).
Do not log more than needed. Logging rules like defconf: accept established,related rules will flod your log,
Below is a sample on how to name the log rules. You do not need to follow this rule, but it makes it more uniform.

Rule name logging
==================

Format:
x_y_z

x=<where and direction>
y=<what to do>
z=<name/info>

Example
-------
Filter Rule Forard allow HTTP
FF_A_Http

Filter Route Input Drop ICMP
FI_D_Icmp

Nat HTTP
ND_DE_Http

Mangle Mark HTTP packets
MF_MP_Http


Filter Rule
------------------
x=
FF Filter Forward
FI Filter Input
FO Filter Output
FX Filter Custom list

y=
A  Accept
AD Add to dst address list
AS Add to src address list
D  Dropp
F  Fast track
J  Jump
L  Log
P  Passthrough
RJ Reject
RT Return
T  Tarpit

Nat Rule
------------------
x=
ND Dest nat
NS Source nat

y=
A  Accept
AD Add to dst address list
AS Add to src address list
DE Dst-nat
J  Jump
L  Log
M  Masquerade
N  Netmap
P  Passthrough
RE Redirect
RT Return
SA same
S  Src-nat

Raw
------------------
x=
RP Filter Raw Prerouting
RO Filter Raw Output

y=
A  Accept
AD Add to dst address list
AS Add to src address list
F  Fast track
D  Dropp
J  Jump
L  Log
N  No track
P  Passthrough
RT Return

Mangle
------------------
x=
MF Mangle Forward
MI Mangle Input
MP Mangle Postrouing
MR Mangle Prerouting

y=
A  Accept
AD Add to address list
AS Add to dst address list
CD Change DSCP
CM Change MSS
CT Change TTL
CL Clear DF
F  Fast track
J  Jump
L  Log
MC Marc connection
MP Mark packets
MR Mark routing
P  Passthrough
RT Return
RO Route
S  Set proirity
SP Sniff PC
ST Sniff TZSP
SI Strip IPv4 options

2d) You should at least log this rule “defconf: drop all not coming from LAN” with this prefix: FI_D_port-test
Web gui:
IP->Firewall->selec:defconf: drop all not coming from LAN->Log:v->Log Prefix:FI_D_port-test
This will populate the MikroTik Live attack view.


2e) Accounting (new version in 3.5)
To get accounting data, you need to turn on Kid Control on the MikroTik router. (MikroTik Traffic dashboard)
Cli:

/ip kid-control
add fri=0s-1d mon=0s-1d name=Monitor sat=0s-1d sun=0s-1d thu=0s-1d tue=0s-1d wed=0s-1d

2f) Scripts

Main Script:

To get all the other data like Traffic accounting, uPnP, System health, System resources and DHCP pool information you need this script on the MikroTik. Create this script with name Data_to_Splunk_using_Syslog and cut and past code using gui.
In the top of the script, you can set a module to true/false. If you do not use wifi, set :local Wireless false

You can from the main script adjust what it should collect or not collect by change between true and false in the first part of the script,

# Collect information from Mikrotik RouterOS
# Jotne 2025
# Script name=Data_to_Splunk_using_Syslog
:log info message="script=version ver=5.8"
# ----------------------------------

# Auto update syslog server. 5.3-5.4.
# Change <your syslog dns name> to the dns of your syslog server.
# The update is disabled by default.  Remove the # from the two next line to use it.

#:local mySyslog [resolve <your syslog dns name>]
#/system/logging/action/set [find where name="logserver"] remote=$mySyslog


# What data to collect.  Set to false to skip the section 
# ----------------------------------
:local SystemResource true
:local SystemInformation true
:local SystemHealth true
:local TrafficData true
:local AccountData true
:local uPnP true
:local Wireless true
:local AddressLists true
:local DHCP true
:local Neighbor true
:local InterfaceData true
:local CmdHistory true
:local CAPsMANN false

:local Routing true
:local OSPF false
:local BGP false

:local PPP true
:local IPSEC true

# Get RouterOS main version (used to run different script on different version)
:local train [:tonum [:pick [/system resource get version] 0 1]] 

# Collect system resource
# ----------------------------------
:if ($SystemResource) do={
	/system resource
	:local cpuload [get cpu-load]
	:local freemem ([get free-memory]/1048576)
	:local totmem ([get total-memory]/1048576)
	:local freehddspace ([get free-hdd-space]/1048576)
	:local totalhddspace ([get total-hdd-space]/1048576)
	:local up [get uptime]
	:local sector [get write-sect-total]
	:log info message="script=resource free_memory=$freemem MB total_memory=$totmem MB free_hdd_space=$freehddspace MB total_hdd_space=$totalhddspace MB cpu_load=$cpuload uptime=$up write-sect-total=$sector"
}


# Make some part only run every hours
# ----------------------------------
:global Hour
:local run false
:local hour [:pick [/system clock get time] 0 2]
:if ($Hour != $hour) do={
	:global Hour $hour
	:set run true
}


# Get NTP status
# ----------------------------------
:local ntpstatus ""
:if ([:len [/system package find where !disabled and name=ntp]] > 0 or [:tonum [:pick [/system resource get version] 0 1]] > 6) do={
    :set ntpstatus [/system ntp client get status]
} else={
    :if ([:typeof [/system ntp client get last-update-from]] = "nil") do={
        :set ntpstatus "using-local-clock"
    } else={
        :set ntpstatus "synchronized"
    }
}
:log info message="script=ntp status=$ntpstatus" 


# Get interface traffic data for all interface
# ----------------------------------
:if ($TrafficData) do={
	:foreach id in=[/interface find] do={
		:local output "$[/interface print stats as-value where .id=$id]"
		:set ( "$output"->"script" ) "if_traffic"
		:log info message="$output"
	}
}


# Get traffic data v2 (Kid Control)
# ----------------------------------
:if ($AccountData) do={
	:foreach logline in=[/ip kid-control device find] do={
		:local output "$[/ip kid-control device get $logline]"
		:set ( "$output"->"script" ) "kids"
		:log info message="$output"
	}
}


# Finding dynmaic lines used in uPnP
# ----------------------------------
:if ($uPnP) do={
	:foreach logline in=[/ip firewall nat find where dynamic=yes and comment~"^upnp "] do={
		:local output "$[/ip firewall nat print as-value from=$logline]"
		:set ( "$output"->"script" ) "upnp"
		:log info message="$output" 
	}
}


# Collect system information 5.5 added ID for non routerBoard 5.6 Remvoed serial
# ----------------------------------
:local model na
:local ffirmware na
:local cfirmware na
:local ufirmware na
:if ($SystemInformation and $run) do={
	:local version ([/system resource get version])
	:local board ([/system resource get board-name])
	:local identity ([/system identity get name])
	:do {
		:if ($board!="CHR" OR $board!="x86") do={
			/system routerboard
			:set model ([get model])
			:set ffirmware ([get factory-firmware])
			:set cfirmware ([get current-firmware])
			:set ufirmware ([get upgrade-firmware])
		}
	} on-error={}
	:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" identity=\"$identity\" factory-firmware=\"$ffirmware\" current-firmware=\"$cfirmware\" upgrade-firmware=\"$ufirmware\""
}


# Collect system health (health removed from some routers 5.8)
# ----------------------------------
:do {
	:if ($train > 6 and $SystemHealth) do={
		# New version (RouterOS >6)
		:foreach id in=[/system health find] do={
			:local health "$[/system health get $id]"
			:set ( "$health"->"script" ) "health"
			:log info message="$health"
		}
	} else={
		# Old version (RouterOS 6 or older)
		:if (!([/system health get]~"(state=disabled|^\$)")) do={
			:local health "$[/system health get]"
			:set ( "$health"->"script" ) "health"
			:log info message="$health"
		}
	}
} on-error={}



# Sends wireless client data to log server 
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wlan]]>0) do={
	/interface wireless registration-table
	:foreach i in=[find] do={
		:log info message=".id=$i;ap=$([get $i ap]);interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=$([get $i signal-strength]);tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
	}
}


# Count IP in address-lists
#----------------------------------
:if ($AddressLists) do={
	:local array [ :toarray "" ]
	:local addrcntdyn [:toarray ""] 
	:local addrcntstat [:toarray ""] 
	:local test
	:foreach id in=[/ip firewall address-list find] do={
		:local rec [/ip firewall address-list get $id]
		:local listname ($rec->"list")
		:local listdynamic ($rec->"dynamic")
		:if (!($array ~ $listname)) do={ :set array ($array , $listname) }
		:if ($listdynamic = true) do={
			:set ($addrcntdyn->$listname) ($addrcntdyn->$listname+1)
		} else={
			:set ($addrcntstat->$listname) ($addrcntstat->$listname+1)}
	}
	:foreach k in=$array do={
		:log info message=("script=address_lists list=$k dynamic=".(($addrcntdyn->$k)+0)." static=".(($addrcntstat->$k)+0))}
}


# Get MNDP (CDP) Neighbors
# ----------------------------------
:if ($Neighbor and $run) do={
	:foreach neighborID in=[/ip neighbor find] do={
		:local nb [/ip neighbor get $neighborID]
		:local id [:pick ("$nb"->".id") 1 99]
		:foreach key,value in=$nb do={
			:local newline [:find $value "\n"]
			:if ([$newline]>0) do={
				:set value [:pick $value 0 $newline]
			}
			:log info message="script=neighbor nid=$id $key=\"$value\""
		}
	}
}


# Collect DHCP Pool information
# ----------------------------------
:if ($DHCP and $run) do={
	/ip pool {
		:local poolname
		:local pooladdresses
		:local poolused
		:local minaddress
		:local maxaddress
		:local findindex

# Iterate through IP Pools
		:foreach pool in=[find] do={
			:set poolname [get $pool name]
			:set pooladdresses 0
			:set poolused 0

# Iterate through current pool's IP ranges
			:foreach range in=[:toarray [get $pool range]] do={

# Get min and max addresses
				:set findindex [:find [:tostr $range] "-"]
				:if ([:len $findindex] > 0) do={
					:set minaddress [:pick [:tostr $range] 0 $findindex]
					:set maxaddress [:pick [:tostr $range] ($findindex + 1) [:len [:tostr $range]]]
				} else={
					:set minaddress [:tostr $range]
					:set maxaddress [:tostr $range]
				}

# Calculate number of ip in one range
				:set pooladdresses ($maxaddress - $minaddress)

# /foreach range
			}

# Test if pools is used in DHCP or VPN and show leases used
			:local dname [/ip dhcp-server find where address-pool=$poolname]
			:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
				:set poolused [:len [used find pool=[:tostr $poolname]]]
			} else={
# DHCP server found, count leases. 5.7 Fixed where a pool is used in multiple DHCP servers
				:local dname [/ip dhcp-server get [:pick [find where address-pool=$poolname] 0] name]
				:set poolused [:len [/ip dhcp-server lease find where server=$dname]]}

# Send data
			:log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")

# /foreach pool
		}
# /ip pool
	}
}


# Get detailed command history RouterOS >= v7
# ----------------------------------
:if ($train > 6 and $CmdHistory) do={
	:global cmd
	:local f 0
	:foreach i in=[/system history find] do={
		:if ($i = $cmd) do={ :set f 1 }
		:if ($f != 1) do={
			:log info message="StartCMD"
			:log info message=[/system history get $i]
			:log info message="EndCMD"
		}
	}
	:global cmd  [:pick [/system history find] 0]
}


# Test if CAPsMANN is installed and run script 5.5
# ----------------------------------
:if ( ([:len [/interface find where type="cap"]] > 0) and $CAPsMANN) do={ 
	/system script run CAPsMANN
}



# Collect routing information
# ----------------------------------
:if ($Routing) do={
	/ip route
	:foreach id in=[find] do={
		:local route "$[get $id]"
		:set ( "$route"->"script" ) "route"
		:log info message="$route"
	}
}

:if ($OSPF) do={
	/routing ospf neighbor
	:foreach id in=[find] do={
		:local ospf "$[get $id]"
		:set ( "$ospf"->"script" ) "ospf"
		:log info message="$ospf"
	}
}

:if ($BGP) do={
	/routing bgp session
	:foreach id in=[find] do={
		:local bgp "$[get $id]"
		:set ( "$bgp"->"script" ) "bgp"
		:log info message="$bgp"
	}
}


# Collect PPP/IPSEC
# ----------------------------------
:if ($PPP) do={
	/ppp active
	:foreach id in=[find] do={
		:local ppp "$[get $id]"
		:set ( "$ppp"->"script" ) "ppp"
		:log info message="$ppp"
	}
}

:if ($IPSEC) do={
	/ip ipsec active-peers
	:foreach id in=[find] do={
		:local ipsec "$[get $id]"
		:set ( "$ipsec"->"script" ) "ipsec"
		:log info message="$ipsec"
	}
}

# End Script

CAPsMANN scrip. Only needed if you are using CAPsMANN

# CAPsMANN script

:local capsregistered ([/caps-man registration-table print count-only])
 
/caps-man interface
:local name
:local mac
 
# ignore all master interfaces
:foreach i in=[find where master-interface="none"] do={
	:set name [get $i name]
	:set mac [get $i radio-mac]
	:local counter ([/caps-man registration-table print count-only  where interface=$name])
	:log info message="script=caps-man name=$name counter=$counter"
}
:log info message="script=caps-man capsregistered=$capsregistered"
#added to log frequency&chennel
:if ( ([:len [/interface find where type="cap"]] > 0) and $CAPsMANN and $run) do={
:foreach i in=[/caps-man interface find where master-interface="none"] do={
	:local name [/caps-man interface get $i name]
	:local channel [/caps-man interface get $i current-channel]
	:log info "$name: selected channel $channel"
	}
}

2g) Then schedule the script to run every 5 minutes:

/system scheduler
add interval=5m name="Data_to_Splunk" on-event=Data_to_Splunk_using_Syslog

3) Starting up
3a) Startup information
Some parts of the scripts runs only every hour, and some jobs in Splunk runs once a day.
So it will take time before all devices er named correctly. To speed things up some, do this:
After running for some hour and you see that data are coming, in Splunk go to:
Apps->Mikrotik->Reports and run both Device table updater and DHCP table updater by clicking on Open In Search behind the app


4) Debugging
4a) See if any data are comming inn to splunk at all. Do a search in Splunk for:

index=*

4b). Test if data has correct tag “MikroTik” (Capital M & T) Do a search in Splunk for:

index=* | table _time sourcetype _raw

You should see correct time, sourcetype should show “mikrotik” and _raw should show data

4c). See that _raw does contain only data and not time and other info. Do a search in Splunk for:

index=* | table  _raw
dns MikroTik: done query: #640030 adservice.google.no 216.58.211.2
dhcp,debug,packet MikroTik:     Client-Id = 01-6C-3B-6B-88-34-3F
firewall,info MikroTik: FI_D_port-test input: in:ether1 out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 47.118.40.92:52503->92.220.205.91:2376, len 40

If you see date, format of packet from Mikrotik has BDS set or Rsyslog is not setup correctly.

4d). Verify that all files has same user:group (root:root or splunk:splunk if run as non-root user)

4e). Look for error written in file or error in file name. inputs.conf not input.conf

4f). Read trough all steps on how to install if some does not work

4g). License problems.
Not convert license to Free license before 30 days or indexing more than 500MB/day?
The Free license will prevent searching if there are 3 license warnings in a rolling 60 day window. If that happens, Splunk Free continues to index your data but disables search functionality. You will regain search when you are below 3 license violation warnings in a 60 day period. See About license violations.
How to solve this+

  1. Convert to free and wait 30 days if you did not convert it.
  2. Passing 3 times? Reduce license <500MB and wait 60 days.
  3. Reintall Splunk
  4. Get a Free 10GB/day developer license ( https://dev.splunk.com/enterprise/dev_license )

4h) Limit inout. Data comes from two sources in Splunk.

  1. The log setup. DNS and other stuff. To remove DNS change to
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp,!dns
  1. From the logging script: Data_to_Splunk_using_Syslog
    Change true to false each block you like to stop.

4i) 1. You see you get Duplicate Values under host. This may be that you did change name on one device ore have multiple host with same name.
Install Lookup Editor (a Splunk addon app) in Splunk if you do not already have done. Got to Apps → Lookup Editor, select device_kvstore and open it. Remove the duplicate/old items.

4j) Make sure you not have setup BSD syslog. System->Logging->Action->Logserver->Make sure BSD are not selected.

4k) Make sure time are correct on the router. NTP running and connected to a server.

4x). Still problems: ask here :slight_smile:

1Mikrotik Firewall.jpg
2DNS Live view.jpg
3Volt_temperature.jpg
4Resources.jpg

Netwatch

This part shows how to use the MikroTik Netwatch dashboard.
Idea with this part are to monitor one or many IP and get a good logging information of up/down time of a watched IP.

Setup.
Lets say you like to monitor a WireGuard VPN tunnel. There are noe traces of up down status for WireGuard in the RouterOS, so using Netwarch to see if remote IP is up or down is a way to see status of it.

Script
System->Scripts->Add script.
Name: Netwatch
Script:

####################################
# Netwatch script
#
# Used as both up and down script
# Created Jotne 2021 v1.5
#
####################################
:local Host $host
/tool netwatch
:local Status [get [find where host="$Host"] status]
:local Comment [get [find where host="$Host"] comment]
:local Interval [get [find where host="$Host"] interval]
:local Since [get [find where host="$Host"] since]
:log info "script=netwatch watch_host=$Host comment=\"$Comment\" status=$Status interval=$Interval since=\"$Since\""

Tools->Netwatch
Add Host ip. For WireGuard, that would be ip on the other side of the tunnel.
Host: 10.0.0.2
Up: Netwatch
Down: Netwatch
Comment: WG-Tunnel-22 (This name is important to set, since this will identify what this Netwatch do watch.)

/tool netwatch
add comment=WG-Tunnel-22 down-script=Netwatch host=10.0.0.2 up-script=Netwatch

You can ass as many netwatch IP as you like. It will take resource from the router, so do not add to many that test to often.
.
Netwatch.jpg

Placeholder p1

How to install Splunk as a non root user.
Its a security risk to run everything as a root user, so if you can, you should use a dedicated user for your program.

This tutorial will show how to install Splunk as a user with name splunk on your Ubuntu server (may work on other as well)

Download latest Splunk Enterprise to you /tmp folder

Create the splunk user:

sudo useradd -c "splunk user" -m -s /bin/bash -U -d /opt/splunk splunk

Log in a the splunk user:

sudo su - splunk

Download Splunk Linux tgz file to /tmp folder

Extract the Splunk software to /opt folder (name of file will change with new version):

tar xvzf /tmp/splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz -C /opt

Start your Splunk server (accept license agrement and set a password for Spkunk admin user):

~/bin/splunk start

Add a user/password to login to Splunk.

As a root user, make Splunk autostart with user splunk as a startup script:

sudo /opt/splunk/bin/splunk enable boot-start -user splunk

You should now be up and running on port 8000 (can be changed)

http://you-server.ip:8000

Remember to use splunk user whenever you change/add files or do anything else with Splunk from the CLI

sudo su - splunk

PS:
If you run Splunk as a non root user then you can not use UDP/514 as a syslog receiver port in Splunk.
Since all port below 1024 need root permission to work.

Workarounds.

  1. Send syslog to other port above 1023, like 1514 for UDP syslog. (need to change many routers to send to correct port)
  2. Set up a local syslog server like r-syslog and let Splunk read the r-syslog log files.
    http://forum.mikrotik.com/t/tool-using-splunk-to-analyse-mikrotik-logs-3-3-graphing-everything/121810/13

How to install RSyslog for Linux

If you do use Splunk as a non root (recomended) user, you need an external Syslog server.

This is how to set it up using Ubuntu server. Should work on most version.

rsyslog comes default with Ubuntu so no need to install any extra software.


PS do not modify these file to use other location. If you do so you will need to modify udp.conf rsyslog and inputs.conf splunk for every upgrade.

Copy these two files to /etc/rsyslog.d/

udp.conf (sets up rsyslog to accept Sylog on udp/514)

# rsyslog.d/udp.conf
#
# This receives UDP syslog on port 514 and stores it in a reliable format
# in /data/syslog/udp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imudp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="udp_split_filename" type="list") {
  constant(value="/data/syslog/udp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="udp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="udp_split_filename"
  )
}

# setting
input(type="imudp" port="514" ruleset="udp_split")

tcp.conf (sets up rsyslog to accept Syslog on tcp/1514) PS MikroTik only sends UDP syslog, so this part is not needed

# rsyslog.d/tcp.conf
#
# This receives TCP syslog on port 1514 and stores it in a reliable format
# in /data/syslog/tcp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imtcp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="tcp_split_filename" type="list") {
  constant(value="/data/syslog/tcp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="tcp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="tcp_split_filename"
  )
}

# settings
input(type="imtcp" port="1514" ruleset="tcp_split")

Create the following folders

mkdir /data
mkdir /data/syslog
mkdir /data/syslog/tcp
mkdir /data/syslog/udp

Change folder rights to syslog and restart rsyslog

chown -R syslog:syslog /data/syslog
service rsyslog restart

run ss or netstat as root user to see that rsylog is running

ss -pultn | grep syslog
udp   UNCONN 0      0                                0.0.0.0:514        0.0.0.0:*     users:(("rsyslogd",pid=5532,fd=8))
udp   UNCONN 0      0                                   [::]:514           [::]:*     users:(("rsyslogd",pid=5532,fd=9))
tcp   LISTEN 0      25                               0.0.0.0:1514       0.0.0.0:*     users:(("rsyslogd",pid=5532,fd=6))
tcp   LISTEN 0      25                                  [::]:1514          [::]:*     users:(("rsyslogd",pid=5532,fd=7))



netstat -pultn | grep rsyslog
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      1459/rsyslogd
tcp6       0      0 :::1514                 :::*                    LISTEN      1459/rsyslogd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1459/rsyslogd
udp6       0      0 :::514                  :::*                                1459/rsyslogd

To make Splunk read rsyslog data make this file: %SplunkHome%/etc/system/local/inputs.conf

[monitor:///data/syslog/udp/.../*.log]
sourcetype = rsyslog
host_segment=4

[monitor:///data/syslog/tcp/.../*.log]
sourcetype = rsyslog
host_segment=4

Test your server:

echo '<14>sourcehost message text' | nc -v -u -w 0 127.0.0.1 514

This should create a folder /data/syslog/udp/127.0.0.1 with a *.log file

Clean UP
Since rsyslog does not delete anything, you need a script that delete old files

Create file /etc/cron.d/rsyslog_cleanup with:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

UDPLOGDIR=/data/syslog/udp/.
TCPLOGDIR=/data/syslog/tcp/.

# Age are n+1 days
DELETE_AGE=2

# Every hour, as user syslog, clean out ancient log files
00 * * * *  root /usr/bin/find ${UDPLOGDIR} -mtime +${DELETE_AGE} \( -name \*.log -o -name \*.log.gz \) -print -delete 2>&1
00 * * * *  root /usr/bin/find ${TCPLOGDIR} -mtime +${DELETE_AGE} \( -name \*.log -o -name \*.log.gz \) -print -delete 2>&1

Placeholder P2

Placeholder P3

Hi Jotne,
Are you working on / thinking about incorporating Netflow into your app/dashboard design already ? It can be right next to all the existing stuff anyway.
Offcourse there is also the question about the backend ingesting netflow data from Mikrotik device.
Because today I have 2 separate pages with some Netflow pages found on Github, might be interesting to start including it in the general design ?

Yes I have been thinking about netflow, since MT is removing IP accounting.
Will see how much time I find to do some testing.

IP accounting are sent over Syslog, so no need for extra setup anywhere. Scripts take care of sending data.
Netflow on the other hand, need some server to receive data and an extra port (can not be sent over syslog port).
Netflow server also needs to be stup and communicate with Splunk.
Netflow plugins for Splunk also seems to cost money… https://splunkbase.splunk.com/app/489/
So there will be a more complex solution using Netflow.

I’m using the Splunk App for Stream, which is capable of many things including netflow-decoding.
Its free as far as I know, I’ve been using is more then 1 year, actually never failed on me. But I agree installation + adapting config-file was not click-click-click ready :wink: but certainly not overly complex.

https://splunkbase.splunk.com/app/1809/


Targeted full packet capture to NAS for forensic investigation of raw packets. Aggregate data using familiar SPL aggregation methods to reduce the volume of data indexed. Capture Flow-type records, including NetFlow v5, v9, jFlow, and sFlow, and IPFIX, and send Flow Records directly into your Indexers, with optional filtering and aggregation.

That seems to look better.
Could you give me a quick guide on how to get netflow in on 9995 to Splunk, I could update the app to show the data.

Ok,
Let’s start with some github-references where I downloaded some dashboards. You probably will have to pick & match what you want to include in your own dashboard

https://github.com/JohnnyMirza/Splunk_Netflow
https://github.com/danucalovj/Splunk-Netflow-Analyzer
https://github.com/lucas-alados/netflow_sample_dashboards

I followed the guides on Splunkbase itself for installing the Stream_TA package, its not that complex.

https://docs.splunk.com/Documentation/StreamApp/7.3.0/DeployStreamApp/UseStreamtoingestNetflowandIPFIXdata

Contents of my /opt/splunk/etc/apps/Splunk_TA_stream/default/streamfwd.conf
(not too sure why I added the customs with their flow-id, I think it was during the time I wanted IPFIX to work (which is buggy on 6.x ROS) but v5 works fine.
I agree you miss out on some field, but “the basics” are there.

[streamfwd]
port = 8889
ipAddr = 127.0.0.1

netflowReceiver.0.ip = X.X.X.X (=IP of the sending interface on the Mikrotik)
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow


netflowElement.0.enterpriseid = 14988
netflowElement.0.id = 225
netflowElement.0.termid = netflow.postNATSourceIPAddress

netflowElement.1.enterpriseid = 14988
netflowElement.1.id = 226
netflowElement.1.termid = netflow.postNATDestinationIPAddress

netflowElement.2.enterpriseid = 14988
netflowElement.2.id = 227
netflowElement.2.termid = netflow.postNAPTSourceTransportPort

netflowElement.3.enterpriseid = 14988
netflowElement.3.id = 228
netflowElement.3.termid = netflow.postNAPTDestinationTransportPort




And I think that this should be about it. The binary is started also with the rest of Splunk
(I’m running it as root, not the smartest thing to do I guess :wink:
(needed to reboot my NAS on October 12 hence te low uptime)

root 1748 1023 0 Oct12 ? 00:00:00 /bin/sh -c /opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd
root 1749 1748 0 Oct12 ? 01:14:27 /opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd


Hope this helps already a bit…

On the Mikrotik, under IP > Flows I’ve enabled it, selected all possible field.
I have 1 “target” (=IP of the listener Streams/Splunk) and selected “v5”
That’s it…

Hi, I try to upgrade from 3.2 and reading the instructions says :

1j) Upgrade form previous version.
Some time files are renamed, so if you have not change any original files, just delete the MikroTik folder.
No logged data will be deleted.
If you have custom dashboards, menus, saved search (reports) etc, you need to merge the configuration files.
They are normal stored in “local” folder.

HOW do I delete “the MikroTik folder” thru the splunk interface ???
I am running splunk in docker on synology, not that familiar with splunk, it just works here :wink:

It can not be deleted trough Splunk, you need to delete the folder maualy:
~/etc/apps/MikroTik

What if you have many devices sending netflow?
Can you open it so it listen for any IP?

netflowElement.1.enterpriseid = 14988
netflowElement.1.id = 226
netflowElement.1.termid = netflow.postNATDestinationIPAddress

What are these section?

I think you can add multiple , like netflowReceiver.0.ip=X.X.X.X
netflowReceiver.1.ip=Y.Y.Y.Y

As each of these instances has fields for IP, port etc , I think its purpose it to be able to add multiple.
Never tried it :wink: I only have 1 Mikrotik

These sections on
netflowElement.1.id = 226
netflowElement.1.termid = netflow.postNATDestinationIPAddress
etc..

I think I added them while playing to get IPFIX working (IPFIX works with periodic templates of fields being transmitted).
I believe the default “dictionary” did not recognise certain Netflow field, hence the possibility to add new ones.
To be honest I don’t know anymore. They are not present in the “default” files so for sure some custom work.

Perhaps it something to do with this thread http://forum.mikrotik.com/t/netflow-no-longer-showing-natd-destination-address-something-chnaged/90137/1

I did hope for a solution where you have a netflow listener, and that it does not care about where data is coming from.
Would be hard to maintain and setup if you have a lot of routers.

I quickly tested and it seems not to like 0.0.0.0 :wink:
I agree this add some task to the admin, but I’m not even sure this Stream-plugin would “scale” well if you have hundreds of routers sending Netflow data to it.
Then you probably have to run it on a separate box and not on the same “Splunk” host that like I do.
As an “ip accounting” replacement, I would enable netflow only on the Internet-facing devices or some centralized Internet breakout boxes.
Enabling netflow for all internal interfaces on a larger network will generate quite some flows.

I re-tested and 0.0.0.0 seems to work.
Because Splunk is running on my NAS I really need to be patient, it’s quite heavy :wink:

Anyway, I’ve adapted my config to

netflowReceiver.0.ip = 0.0.0.0
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow


And then killed the streamfwd-processes and restarted Splunk. Since then, events seems to arrive just fine.
The “exporter_ip” field indicates which element is sending the Netflow data.