Community discussions

MikroTik App
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) 💾 🛠 💻 📊

Fri Jul 27, 2018 11:23 am

This thread is deprecated use new thread:

viewtopic.php?t=179960



Version 3.3 13.08.2021
Top_logo.jpg
Using Splunk to monitor and graph various data from our MikroTik Routers is a nice and free way to help you showing what is going on in your network.
Splunk is free to use for logging up to 500MB pr day.
NB logging large amount of Accouning, DNS or firewall rules quickly eats up license, so I do recommend to turn off Accouning/DNS logging to start with.

It can be used to monitor multiple devices. No ports needs to be opened (like with SNMP monitoring). All data are sent from the device to the Splunk monitor. Devices could be all around the world.

PS:
Traffic monitoring does not work correctly while fast track is enabled (and its removed in v7.x of RouterOS). Turn it off and you may loose throughput, so its something you should think about when using this type of monitoring. How to disable it: https://www.youtube.com/watch?v=6LaqhDm6PHI


What's new
A lot have changed since previous version. So much that its better to replace all MikroTik files in the Splunk server instead of trying to update files.
Before data was collected in three ways.
1. Syslog
2. SNMP
3. Scripts (remote)
Problem with SNMP is that you have to add duplicate configuration in Syslog to handle each box you monitor.
Problem with script is that you need to setup a job on the Splunk server that SSH inn to each box to get data.
The script tools did not log host IP, so you did not now from what box data was coming from.

SNMP and Scipts has been removed and change inn to local script on the Router that sends outs needed data to the Splunk using Syslog.
This way all data get correctly marked and you know for what Router you gets data from.

latest changes
# 3.3 (13.08.2021)
# New KV store (used to store info about Devices and DHCP info
# Changed from table to fields in base search for all dashboard to speed up searches.
# Added more DNS extraction
# Updated script to 4.4 (Added firmware info)
# Added frimware info "MikroTik Device List"
# Fixed lookup "MikroTik Wifi strength"
# Added new DoH dashboard "MikroTik DoH information"
# Change table format "MikroTik Interface Changes"
# Fixed typo in eventtype "MikroTik VPN Connections"
# Added user local ip "MikroTik VPN Connections"
# Fixed missing search in 2b, duplicate entry in user-name, added 1b login info "MikroTik VPN Connections"
# Fixed to not show false logged in user "MikroTik VPN Connections"
# Fixed html formatting in "Mikrotik DHCP to Static"
# Fixed missing City in "Mikrotik Firewall Rules"
# Moved "Mikrotik uPnP" to Connections menu
# Added table view to "Mikrotik uPnP"

Installation
1) On your PC Works on Windows and Linux, but use Linux (clearly the best choice)
-----------------------------------
1a) Download and install Splunk (Windows or Linux(Ubuntu recommended))
PS you need an account to download. It's free to create.
https://www.splunk.com/en_us/download/s ... prise.html
PS you need to create an account to download the file. Free to download and use (up to 500MB/day)
PS remember to set timezone on Windwows/Linux, or else logging time will be wrong.

1b) PS: To install Splunk as a non root user, recomended. (needs an external syslog reciever)
Splunk setup:
viewtopic.php?p=677233#p677233
rsyslog setup
viewtopic.php?p=677233#p793342

Splunk can fine run as root user, but not recommended.

1c) Change to free license group. Very important to do before 30 day of use. !!!!!!!!!!!!!!!!!!!!
Web gui:
1d) Settings->licensing->Change license group->Free licnse->Save

1e) Open Windows Firewall for UDP on Windows (On linux its not blocked)
Web gui:
Start->type "adv"->Select:Widows Firewall with Advanced Security->Sect Inbound rules->Right Click "Inbound Rules">New Rule-Port-Next->UDP->Specific local ports->514->Next->Next->Next->Name "syslog"

1f) Allow UDP 514 (syslog) (if running Splunk as non ROOT user or like to use external syslog reciever, see 1b for non-root)
Web gui:
Setting->Datainputs->Add new (behind the UDP)->Port 514->Next->Sourcetype type syslog and select syslog->Next-Submit

1g) Download the Splunk spl file:
MikroTik3.3.rar
1h) Extract the spl file
From Start page in Splunk, click the gear behind Apps or
from top meny click Apps->Manage Apps
Then select Install app from file and select the spl file

1i) A restart of Splunk may be needed.
Web gui:
Settings->Server controls->Restart Splunk

1j) Upgrade form previous version.
Since no files are renamed, just data added, you should be fine by just replacing current files.
You can delete the MikroTik folder if you like. No logged data will be deleted.
If you have custom dashboards, menus, saved search (reports) etc, you need to merge the configuration files.

2) On Your MikroTik Router
-----------------------------
Before you setup logging, you should make an unique identifier of your route. Important if you have more than one router to monitor.
/system identity set name=Router-London-22
2a) Syslog
You need to make your Router able to send Syslog messages.
Web gui:
System->Logging->Action->Add New->Name (your server name)->Type:Remote->Remote Address:ip your syslog->Ok
Cli
/system logging action add name=logserver target=remote remote=192.168.1.50 remote-port=514
PS Do NOT select BSD Syslog. It will mess up the logging format.

2b) Then select what modules to log.
I do suggest that you send all DHCP logs including debug and all other logs that are not debug.
It is very important to name the prefix like this "MikroTik" and not "mikrotik" or some other.
Splunk uses the MikroTik prefix to find out what type of syslog data that is coming to it.
Uppercase T and uppercase M, rest are lowercase
Web gui:
System->Logging->Rules->Add new->Topics:dhcp->Prefix:MikroTik->action:your syslog server->Ok
System->Logging->Rules->Add new->Topics:!debug,!packet,!snmp->Prefix:MikroTik->action:your syslog server->Ok
Cli:
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=hotspot
To not fill up internal logs with firwall logs etc, remove firewall from memory logs
/system logging set [find topics="info"] topics=info,!firewall
PS Hotspot is not needed if you do not use it.

2c) Select what rules to log
NB Do not use more than 20 charters, or else it start to clip other part of the log!!!!!!!!!!!
To log the Firewall and Nat rules, you need to turn on logging and add Log Prefix (under action).
Do not log more than needed. Logging rules like defconf: accept established,related rules will flod your log,
Below is a sample on how to name the log rules. You do not need to follow this rule, but it makes it more uniform.
Rule name logging
==================

Format:
x_y_z

z=name/info

Example
-------
Filter Rule Forard allow HTTP
FF_A_Http

Filter Route Input Drop ICMP
FI_D_Icmp

Nat HTTP
ND_DE_Http

Mangle Mark HTTP packets
MF_MP_Http


Filter Rule
------------------
x=
FF Filter Forward
FI Filter Input
FO Filter Output
FX Filter Custom list

y=
A  Accept
AD Add to dst address list
AS Add to src address list
D  Dropp
F  Fast track
J  Jump
L  Log
P  Passthrough
RJ Reject
RT Return
T  Tarpit

Nat Rule
------------------
x=
ND Dest nat
NS Source nat

y=
A  Accept
AD Add to dst address list
AS Add to src address list
DE Dst-nat
J  Jump
L  Log
M  Masquerade
N  Netmap
P  Passthrough
RE Redirect
RT Return
SA same
S  Src-nat

Raw
------------------
x=
RP Filter Raw Prerouting
RO Filter Raw Output

y=
A  Accept
AD Add to dst address list
AS Add to src address list
F  Fast track
D  Dropp
J  Jump
L  Log
N  No track
P  Passthrough
RT Return

Mangle
------------------
x=
MF Mangle Forward
MI Mangle Input
MP Mangle Postrouing
MR Mangle Prerouting

y=
A  Accept
AD Add to address list
AS Add to dst address list
CD Change DSCP
CM Change MSS
CT Change TTL
CL Clear DF
F  Fast track
J  Jump
L  Log
MC Marc connection
MP Mark packets
MR Mark routing
P  Passthrough
RT Return
RO Route
S  Set proirity
SP Sniff PC
ST Sniff TZSP
SI Strip IPv4 options
2d) You should at least log this rule "defconf: drop all not coming from LAN" with this prefix: FI_D_port-test
Web gui:
IP->Firewall->selec:defconf: drop all not coming from LAN->Log:v->Log Prefix:FI_D_port-test
This will populate the MikroTik Live attack view.

2e) Accounting
To get accounting data, you need to enable it on the MikroTik router. (MikroTik Traffic dashboard)
Web gui:
IP-> Accounting -> Enable Accounting -> mark Threshould:2560 OK
Cli:
/ip accounting set enabled=yes threshold=2560
2f) Main Collector Script
To get all the other data like Traffic accounting, uPnP, System health, System resources and DHCP pool information you need this script on the MikroTik. Create this script with name Data_to_Splunk_using_Syslog and cut and past code using gui.
In the top of the script, you can set a module to true/false. If you do not use wifi, set :local Wireless false



# Collect information from Mikrotik RouterOS
# Jotne 2021
:log info message="script=version ver=4.7"
# ----------------------------------


# What data to collect.  Set to false to skip the section 
# ----------------------------------
:local SystemResource true
:local SystemInformation true
:local SystemHealth true
:local TrafficData true
:local AccuntData true
:local uPnP true
:local Wireless true
:local AddressLists true
:local DHCP true
:local Neighbor true
:local InterfaceData true
:local CmdHistory true
:local CAPsMANN false


# Collect system resource
# ----------------------------------
:if ($SystemResource) do={
	/system resource
	:local cpuload [get cpu-load]
	:local freemem ([get free-memory]/1048576)
	:local totmem ([get total-memory]/1048576)
	:local freehddspace ([get free-hdd-space]/1048576)
	:local totalhddspace ([get total-hdd-space]/1048576)
	:local up [get uptime]
	:local sector [get write-sect-total]
	:log info message="script=resource free_memory=$freemem MB total_memory=$totmem MB free_hdd_space=$freehddspace MB total_hdd_space=$totalhddspace MB cpu_load=$cpuload uptime=$up write-sect-total=$sector"
}


# Make some part only run every hours
# ----------------------------------
:global Hour
:local run false
:local hour [:pick [/system clock get time] 0 2]
:if ($Hour != $hour) do={
	:global Hour $hour
	:set run true
}


# Get NTP status
# ----------------------------------
:local ntpstatus ""
:if ([:len [/system package find where !disabled and name=ntp]] > 0 or [:tonum [:pick [/system resource get version] 0 1]] > 6) do={
    :set ntpstatus [/system ntp client get status]
} else={
    :if ([:typeof [/system ntp client get last-update-from]] = "nil") do={
        :set ntpstatus "using-local-clock"
    } else={
        :set ntpstatus "synchronized"
    }
}
:log info message="script=ntp status=$ntpstatus" 


# Get traffic data (accounting data)
# ----------------------------------
:if ([:tonum [:pick [/system resource get version] 0 1]] < 7 and $AccuntData) do={
# Test if fasttrack is enabled and give warning
	:if ([/ip firewall filter find where (action=fasttrack-connection && !disabled)] != "") do={
		:log info message=("script=traffic,fasttrack=1")
	} else={
		:log info message=("script=traffic,fasttrack=0")
	}
# Test if accounting is enabled and if yes, get data
	:if ([/ip accounting get enabled]=yes) do={
		/ip accounting snapshot take
# Send data to loggin server
		:foreach logline in=[/ip accounting snapshot find] do={
			:local output "$[/ip accounting snapshot print as-value from=$logline]"
			:set ( "$output"->"script" ) "traffic"
			:log info message="$output"
		}
	}
}


# Get interface traffic data for all interface
# ----------------------------------
:if ($TrafficData) do={
	:foreach id in=[/interface find] do={
		:local output "$[/interface print stats as-value where .id=$id]"
		:set ( "$output"->"script" ) "if_traffic"
		:log info message="$output"
	}
}


# Finding dynmaic lines used in uPnP
# ----------------------------------
:if ($uPnP) do={
	:foreach logline in=[/ip firewall nat find where dynamic=yes and comment~"^upnp "] do={
		:local output "$[/ip firewall nat print as-value from=$logline]"
		:set ( "$output"->"script" ) "upnp"
		:log info message="$output" 
	}
}


# Collect system information
# ----------------------------------
:local model na
:local serial na
:local ffirmware na
:local cfirmware na
:local ufirmware na
:if ($SystemInformation and $run) do={
	:local version ([/system resource get version])
	:local board ([/system resource get board-name])
	:if ($board!="CHR") do={
		/system routerboard
		:set model ([get model])
		:set serial ([get serial-number])
		:set ffirmware ([get factory-firmware])
		:set cfirmware ([get current-firmware])
		:set ufirmware ([get upgrade-firmware])
	}
	:local identity ([/system identity get name])
	:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial identity=\"$identity\" factory-firmware=\"$ffirmware\" current-firmware=\"$cfirmware\" upgrade-firmware=\"$ufirmware\""
}


# Collect system health
# ----------------------------------
:if ($SystemHealth) do={
	:if (!([/system health get]~"(state=disabled|^\$)")) do={
		:local voltage ([/system health get voltage]/10)
		:local temperature ([/system health get temperature])
		:log info message="script=health voltage=$voltage V temperature=$temperature C"
	}
}


# Sends wireless client data to log server 
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wlan]]>0) do={
	/interface wireless registration-table
	:foreach i in=[find] do={
		:log info message=".id=$i;ap=$([get $i ap]);interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=$([get $i signal-strength]);tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
	}
}


# Count IP in address-lists
#----------------------------------
:if ($AddressLists) do={
	:local array [ :toarray "" ]
	:local addrcntdyn [:toarray ""] 
	:local addrcntstat [:toarray ""] 
	:local test
	:foreach id in=[/ip firewall address-list find] do={
		:local rec [/ip firewall address-list get $id]
		:local listname ($rec->"list")
		:local listdynamic ($rec->"dynamic")
		:if (!($array ~ $listname)) do={ :set array ($array , $listname) }
		:if ($listdynamic = true) do={
			:set ($addrcntdyn->$listname) ($addrcntdyn->$listname+1)
		} else={
			:set ($addrcntstat->$listname) ($addrcntstat->$listname+1)}
	}
	:foreach k in=$array do={
		:log info message=("script=address_lists list=$k dynamic=".(($addrcntdyn->$k)+0)." static=".(($addrcntstat->$k)+0))}
}


# Get MNDP (CDP) Neighbors
# ----------------------------------
:if ($Neighbor and $run) do={
	:foreach neighborID in=[/ip neighbor find] do={
		:local nb [/ip neighbor get $neighborID]
		:local id [:pick ("$nb"->".id") 1 99]
		:foreach key,value in=$nb do={
			:local newline [:find $value "\n"]
			:if ([$newline]>0) do={
				:set value [:pick $value 0 $newline]
			}
			:log info message="script=neighbor nid=$id $key=\"$value\""
		}
	}
}


# Collect DHCP Pool information
# ----------------------------------
:if ($DHCP and $run) do={
	/ip pool {
		:local poolname
		:local pooladdresses
		:local poolused
		:local minaddress
		:local maxaddress
		:local findindex

# Iterate through IP Pools
		:foreach pool in=[find] do={
			:set poolname [get $pool name]
			:set pooladdresses 0
			:set poolused 0

# Iterate through current pool's IP ranges
			:foreach range in=[:toarray [get $pool range]] do={

# Get min and max addresses
				:set findindex [:find [:tostr $range] "-"]
				:if ([:len $findindex] > 0) do={
					:set minaddress [:pick [:tostr $range] 0 $findindex]
					:set maxaddress [:pick [:tostr $range] ($findindex + 1) [:len [:tostr $range]]]
				} else={
					:set minaddress [:tostr $range]
					:set maxaddress [:tostr $range]
				}

# Calculate number of ip in one range
				:set pooladdresses ($maxaddress - $minaddress)

# /foreach range
			}

# Test if pools is used in DHCP or VPN and show leases used
			:local dname [/ip dhcp-server find where address-pool=$poolname]
			:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
				:set poolused [:len [used find pool=[:tostr $poolname]]]
			} else={
# DHCP server found, count leases
				:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
				:set poolused [:len [/ip dhcp-server lease find where server=$dname]]}

# Send data
			:log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")

# /foreach pool
		}
# /ip pool
	}
}


# Get detailed command history RouterOS >= v7
# ----------------------------------
:if ([:tonum [:pick [/system resource get version] 0 1]] > 6 and $CmdHistory) do={
	:global cmd
	:local f 0
	:foreach i in=[/system history find] do={
		:if ($i = $cmd) do={ :set f 1 }
		:if ($f != 1) do={
			:log info message="StartCMD"
			:log info message=[/system history get $i]
			:log info message="EndCMD"
		}
	}
	:global cmd  [:pick [/system history find] 0]
}


# Test if CAPsMANN is installed, if yes, run capsmann script.
# ----------------------------------
:if ( ([:len [/interface find where type="cap"]] > 0) and $CAPsMANN) do={ /system script run capsman }


# End Script

If you do use CAPsMan, create a script with name capsman and add this code.

:local capsregistered ([/caps-man registration-table print count-only])
 
/caps-man interface
:local name
:local mac
 
# ignore all master interfaces
:foreach i in=[find where master-interface="none"] do={
	:set name [get $i name]
	:set mac [get $i radio-mac]
	:local counter ([/caps-man registration-table print count-only  where interface=$name])
	:log info message="script=caps-man name=$name counter=$counter"
}
:log info message="script=caps-man capsregistered=$capsregistered"

2g) Then schedule the script to run every 5 minutes:
/system scheduler
add interval=5m name="Data to Splunk" on-event=Data_to_Splunk_using_Syslog

2h) Debugging
1. See if any data are comming inn to splunk at all.
index=*
2. Test if data has correct tag "MikroTik" (Capital M & T)
index=* | table _time sourcetype _raw
You should see corect time, sourcetype should show "mikrotik" and _raw should show data

3. See that _raw does contain only data and not time and other info
index=* | table  _raw
dns MikroTik: done query: #640030 adservice.google.no 216.58.211.2
dhcp,debug,packet MikroTik:     Client-Id = 01-6C-3B-6B-88-34-3F
firewall,info MikroTik: FI_D_port-test input: in:ether1 out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 47.118.40.92:52503->92.220.205.91:2376, len 40
If you see date, format of packet from Mikrotik has BDS set or Rsyslog is not setup correctly.

4. Read trough all steps on how to install if some does not work
5. Still problems: ask here :)



Example screen shots:

DNS Live view.jpg
Volt_temperature.jpg
Live_attac.jpg
You do not have the required permissions to view the files attached to this post.
Last edited by Jotne on Tue Nov 02, 2021 12:08 pm, edited 154 times in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.0

Fri Jul 27, 2018 11:35 am

More screenshots
DHCP_pool_information.jpg
DHCP_request.jpg
Treaffic.jpg
Firewall_rules.jpg
DNS_request.jpg
You do not have the required permissions to view the files attached to this post.
Last edited by Jotne on Fri Jul 27, 2018 12:26 pm, edited 1 time in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.0

Fri Jul 27, 2018 11:36 am

Screenshot 2
Remote_connection.jpg
upnp.jpg
Uptime.jpg
Wifi_connection.jpg
You do not have the required permissions to view the files attached to this post.
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.0

Fri Jul 27, 2018 3:04 pm

Hi Jotne. Really is a Great Job, thank very much to share this new version. I'll try it and I'll update the post.

:D
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.0

Sat Jul 28, 2018 6:35 pm

Hi Jotne.
Hi. I have three specific issues that happen to me when I mount the new version.
The first is a warning that issues when I start splunk in the props.conf configuration file. Attached image.
Image
http://subirimagen.me/uploads/20180728073918.jpg

The second thing is that the MikroTik DNS Request module did not work for me, it did not show me any information. But I solved it by changing this in the query.
Image
http://subirimagen.me/uploads/20180728101744.jpg
Make those changes and it worked correctly.

The third thing is that the MikroTik Traffic module worked partially because it does not bring me the information of "Percent data pr client". But I solved it by changing this in the query.
Image
http://subirimagen.me/uploads/20180728102956.jpg

Make the following changes and it worked correctly.
Image
http://subirimagen.me/uploads/20180728103102.jpg

Now everything I need is working correctly. Again congratulate you for the great job and especially for sharing it.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.1

Sun Jul 29, 2018 6:13 pm

Thanks MSandoval for testing and feedback.

I have updated to v2.1
It should correct hopefully all the errors in 2.0.
Error was du to some manual change I did make on my installation du to some fixed IP.
This has now been removed, so I do use the same files as I post

# 2.1
# Fixed typo in "MikroTik DNS request"
# Fixed wrong eval in "MikroTik Traffic"
# Removed search used localy in "MikroTik DNS Live usage"
# Removed not needed SED line from props.conf
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.1 (Graphing everything)

Tue Jul 31, 2018 12:36 am

Testing all the modules, I realized that the MikroTik Wifi connection module was not working at all. Since I did not bring data from the Connected section, I only showed data in Disconnected.
Image
http://subirimagen.me/uploads/20180730163143.jpg

What I found is that in the search criterion of eventtypes, it only searches if it connects with signal strength,
Image
http://subirimagen.me/uploads/20180730160920.jpg

To show me more information what I did was create a new type of event that will look for me within the wireless module sent by syslog what will be connected.
Image
http://subirimagen.me/uploads/20180730163417.jpg

Then, edit the criteria in the search performed by the query in the dashboard adding the new criterion of "eventtype = wifi_connected" to each query with the "eventtype = wifi_connected_* "
Image
http://subirimagen.me/uploads/20180730162141.jpg

And with this already if you showed me information in the connected section.
Image
http://subirimagen.me/uploads/20180730163559.jpg
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Tue Jul 31, 2018 8:06 am

Updated files to 2.2

# 2.2
# Removed Host info at the bootom in "MikroTik DNS Live usage"
# Fixed overnship of various dashboard
# Fixed "MikroTik Wifi connection" to show connected if it has no signal strength

@MSandoval
Thanks for the feedback and the effort to fix it :)

I did thought that all connection did show signal strength.
What you have done is partially correct.

In eventtype.conf this:
search = "wireless,info *: connected"
will make it for both:
wireless,info *: connected
wireless,info *: connected, signal strength
So only this is needed (remove signalstrengt)
[wifi_connected]
search = "wireless,info *: connected"
Since evnttype [wifi_connected] does not change, no changes are needed in the dashboard.
 
santong7
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Tue Jun 04, 2013 1:40 pm
Location: Heraklion Crete Greece
Contact:

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Tue Jul 31, 2018 9:28 am

Where is the zip file ?

Do you have any clue why splunk linux stops logging after a while, and with splunk server restart it starts again and then after a while it stops again until the next splunk server restart ?

Same happens with windows splunk version.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Tue Jul 31, 2018 9:37 am

File is there now, se this post why it was missing:
viewtopic.php?f=2&t=137476

Not sure why you have problem with Splunk. I have used it for many years and its for me very stable.
Test a PC with Ubuntu 18.04 and Splunk. Should work very vell together.
Windows should be ok too, but I do recommend Linux (feels much faster om same hw)

You are using latest Splunk, downloaded from Splunk.com?

I will later when I have time post how to install Splunk as a non root user.
 
santong7
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Tue Jun 04, 2013 1:40 pm
Location: Heraklion Crete Greece
Contact:

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Tue Jul 31, 2018 10:56 am

File is there now, se this post why it was missing:
viewtopic.php?f=2&t=137476

Not sure why you have problem with Splunk. I have used it for many years and its for me very stable.
Test a PC with Ubuntu 18.04 and Splunk. Should work very vell together.
Windows should be ok too, but I do recommend Linux (feels much faster om same hw)

You are using latest Splunk, downloaded from Splunk.com?

I will later when I have time post how to install Splunk as a non root user.
I have splunk running on ubuntu server 18.04 with the latest splunk version 7.1.2
When I first run splunk it starts, then it stops, then I restart the server. You can see the empty log on the firewall timeline attached image..
Capture.PNG
I tried also to see if this happens with another syslog server application, like the Kiwi Syslog Server, and I am getting logs all the time without interrupts.

Any clue ?
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Tue Jul 31, 2018 11:23 am

No idea why it stops working.
I have run Splunk for years without it stopping by it self, always using Ubuntu Server.
Maybe there are some other stuff/software on you Ubuntu that kills it.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Tue Jul 31, 2018 1:40 pm

How to install Splunk as a non root user.
Its a security risk to run everything as a root user, so if you can, you should use a dedicated user for your program.

This tutorial will show how to install Splunk as a user with name splunk on your Ubuntu server (may work on other as well)

Download latest Splunk Enterprise to you /tmp folder

Create the splunk user:
sudo useradd -c "splunk user" -m -s /bin/bash -U -d /opt/splunk splunk
Log in a the splunk user:
sudo su - splunk
Download Splunk Linux tgz file to /tmp folder

Extract the Splunk software to /opt folder (name of file will change with new version):
tar xvzf /tmp/splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz -C /opt
Start your Splunk server (accept license agrement and set a password for Spkunk admin user):
~/bin/splunk start
Add a user to login to Splunk.

As a root user, make Splunk autostart with user splunk as a startup script:
sudo /opt/splunk/bin/splunk enable boot-start -user splunk
You should now be up and running. :)

Remember to use splunk user whenever you change/add files or do anything else with Splunk from the CLI
sudo su - splunk
PS:
If you run Splunk as a non root user then you can not use UDP/514 as a syslog receiver port in Splunk.
Since all port below 1024 need root permission to work.

Workarounds.
1. Send syslog to other port above 1023, like 1514 for UDP syslog. (need to change many routers to send to correct port)
2. Set up a local syslog server like r-syslog and let Splunk read the r-syslog log files.
viewtopic.php?p=677233#p793342
Last edited by Jotne on Sun Feb 21, 2021 9:26 am, edited 6 times in total.
 
ismaelac2
just joined
Posts: 1
Joined: Tue Feb 06, 2018 8:45 pm

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Wed Aug 01, 2018 8:01 pm

Good job !! I'll test it, thanks for sharing Jotne !!
 
santong7
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Tue Jun 04, 2013 1:40 pm
Location: Heraklion Crete Greece
Contact:

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Thu Aug 02, 2018 11:34 am

No idea why it stops working.
I have run Splunk for years without it stopping by it self, always using Ubuntu Server.
Maybe there are some other stuff/software on you Ubuntu that kills it.

Finally there is a problem with timestamp, found on splunkd.log
08-01-2018 15:52:07.610 +0300 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Wed Aug 1 02:35:00 2018). Context: source=udp:514|host=192.168.1.1|syslog|

And it stops logging.

This is an example of what mikrotik sends to me, on another syslog server
192.168.1.1 Jul 12 00:17:05 firewall,info DROP INPUT input in:pppoe-WAN out:(unknown 0), proto TCP (SYN), 79.129.108.120:41236->79.129.36.201:7547, len 44


Any work around ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Thu Aug 02, 2018 6:21 pm

You do miss a very important thing.
Where is your MikroTik tag?

A message need to looks like this:
Raw format:
firewall,info MikroTik: NAT_Web_Varg dstnat: in:ether1-Wan out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 195.29.234.174:8505->92.220.197.134:80, len 52
List format:
02/08/2018 17:15:43.000	firewall,info MikroTik: NAT_Web_Varg dstnat: in:ether1-Wan out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 195.29.234.174:8505->92.220.197.134:80, len 52
See my first post, starting from:
Then select what to log.
Make user you add prefix=MikroTik to all syslog rules.
 
Mordor
just joined
Posts: 1
Joined: Mon Jul 30, 2018 11:31 am

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Thu Aug 09, 2018 7:09 am

help me set up a collection of logs with Mikrotik.
I can not understand what to add to the firewall and NAT in Mikrotik, what would the logs in to the syslog go to the Splunk server on CentOS.
if you can, make a little instruction "How to setup collect of logs".
Thanks you.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Thu Aug 09, 2018 2:34 pm

It is clearly written in the first post under Select what to log
1 select what fw or nat rule you like to log.
2 open the fw/nat rule
3 go to action
4 mark log
5 add a text to log prefix for the log, like NAT_RDP
This text could be anything you like, but it god to have a discription tha tell you what the rule does.

PS if you see number increase in webgui you should also get syslog

PS2 do not log everything, start some small and increase with more rules when it works. Logging everything will log allot of data.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Fri Aug 10, 2018 3:32 pm

Upgraded to 2.3

What new:
# v2.3 (10.08.2018)
# Created an Splunk app version

So for user of 2.2, there are no need to upgrade. No new function.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sun Aug 19, 2018 1:23 pm

Here is an example on how Hotspot data looks in Splunk

Splunk-MikroTik-Hotspot.jpg
You do not have the required permissions to view the files attached to this post.
 
helmi1987
just joined
Posts: 2
Joined: Fri Aug 24, 2018 4:28 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sat Aug 25, 2018 9:29 am

Hello

very good integration.
how can i record capsman wifi?

greeting helmi1987
 
Gabana
just joined
Posts: 1
Joined: Sat Sep 08, 2018 8:26 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sat Sep 08, 2018 8:56 pm

hello

thanks for your great job

also does it log "Commands" executed on the device ?
 
Dindihi
newbie
Posts: 27
Joined: Tue Jan 07, 2014 7:12 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sun Sep 09, 2018 12:02 am

Hi,
thanks for this work!
One question, how often do you execute the script with the scheduler?

+1 for capsman
Thanks
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sun Sep 09, 2018 8:26 am

I do schedule the script to run every 5 minutes. (first post updated)

Since I do not have Capsman, I have no possibility to make Dashboard for it.
 
Dindihi
newbie
Posts: 27
Joined: Tue Jan 07, 2014 7:12 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sat Sep 15, 2018 1:19 pm

Hi,
i added some functionalities for Capsman logging.
Maybe this will be helpful for someone.
I'm not a Mikrotik neither Splunk expert !! ;-) Maybe you will find some errors. Please let me know.
But at least this works fine here.

Settings->Sourcetype->Mikrotik->Advanced->Add:
Name: EXTRACT-mikrotik_capsman_state
Value: caps,info.*(?<src_mac>.{2}:.{2}:.{2}:.{2}:.{2}:.{2})@(?<cap_device>.*)\s(?<state>connected|disconnected),\s


Script: Data_to_Splunk_using_Syslog_capsman
:local capsregistered ([/caps-man registration-table print count-only]);

 /caps-man interface
:local name
:local mac


# ignore all master interfaces
:foreach p in=[find where master-interface!="none"] do={
:set name [get $p name]
:set mac [get $p radio-mac]
:local counter ([/caps-man registration-table print count-only  where interface=$name]);
:log info message="script=caps-man name=$name counter=$counter";
}

:log info message="script=caps-man capsregistered=$capsregistered";
Schedule this script every 5 minutes.


Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
Download export_dhcp_splunk.txt to your PC.

Splunk->Settings->Lookups->Lookup table files->New
Destination App: Mikrotik
Destination filename: dhcp_clients
And upload the file export_dhcp_splunk.txt


Splunk->Settings->Lookups->Lookup Definition->New
Destination App: Mikrotik
Type: File
Lookup file: dhcp_clients



And finally the Dashboard (add to Mikrotik app)
<form>
  <label>MikroTik CapsMan</label>
  <fieldset submitButton="false">
    <input type="time" token="global_time">
      <label>Time Span</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="Span" searchWhenChanged="true">
      <label>Time Span</label>
      <choice value="bins=100">Default</choice>
      <choice value="span=1m">1 min</choice>
      <choice value="span=5m">5 min</choice>
      <choice value="span=10m">10 min</choice>
      <choice value="span=20m">20 min</choice>
      <choice value="span=1h">1 hour</choice>
      <choice value="span=2h">2 hour</choice>
      <default>bins=100</default>
    </input>
    <input type="dropdown" token="cap_device">
      <label>CapsMan</label>
      <choice value="*">Any</choice>
      <fieldForLabel>cap_device</fieldForLabel>
      <fieldForValue>cap_device</fieldForValue>
      <search>
        <query>sourcetype=mikrotik
          module=caps
          | top limit=0 cap_device
          | sort cap_device</query>
        <earliest>$global_time.earliest$</earliest>
        <latest>$global_time.latest$</latest>
      </search>
      <default>*</default>
      <prefix>cap_device="</prefix>
      <suffix>"</suffix>
    </input>
    <input type="dropdown" token="srcmac">
      <label>Source Mac</label>
      <choice value="*">Any</choice>
      <default>*</default>
      <fieldForLabel>hostname</fieldForLabel>
      <fieldForValue>src_mac</fieldForValue>
      <search>
        <query>sourcetype=mikrotik
          module=caps
          | top limit=0 src_mac
          | sort src_mac
          |lookup dhcp_clients src_mac OUTPUT hostname</query>
        <earliest>$global_time.earliest$</earliest>
        <latest>$global_time.latest$</latest>
      </search>
      <prefix>src_mac="</prefix>
      <suffix>"</suffix>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Connection messages by cap_device</title>
      <chart>
        <search>
          <query>module=caps 
$cap_device$ $srcmac$
|timechart $Span$  count(_raw) by cap_device</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Devices connection messages by device</title>
      <chart>
        <search>
          <query>sourcetype=mikrotik module=caps state=connected 
$cap_device$ $srcmac$
|lookup dhcp_clients src_mac OUTPUT hostname
|timechart $Span$  count(_raw) by hostname</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Connected devices by cap</title>
      <chart>
        <search>
          <query>sourcetype=mikrotik caps-man |timechart $Span$ values(counter) by name | fillnull  |appendcols  [ search sourcetype=mikrotik caps-man|timechart $Span$ values(capsregistered) as TOTAL |fillnull]</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">connect</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
    <panel>
      <title>Connected devices by cap</title>
      <chart>
        <search>
          <query>sourcetype=mikrotik counter&gt;0  |chart values(counter) by name</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">pie</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
</form>
Edit:
i will probably add some more features in the next week when i find time :-)
You do not have the required permissions to view the files attached to this post.
Last edited by Dindihi on Sat Sep 15, 2018 1:33 pm, edited 2 times in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Sat Sep 15, 2018 1:24 pm

Nice
I will look trough it and add it to the package in the first post.
A problem with Splunk when you add stuff trough GUI, you do not know were it goes.
I may end up in your user, wrong app or correct app :)

And some raw loglines of various types?
 
philamonster
just joined
Posts: 13
Joined: Mon Apr 03, 2017 4:08 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 09, 2018 6:15 pm

Jotne, just wanted to post a note of thanks again for Splunk integration and that I successfully upgraded from 1.1 in place without too many hoops to jump through. Added script and scheduled it on MikroTik device and data was visible in Splunk immediately. I had edited some of the accounting scripts to change time I was collecting in 1.1 due to high CPU generated on Splunk vm w/5 minute interval. New version is much better on resources overall which I am thankful for!
 
maperezdelrio
just joined
Posts: 1
Joined: Fri Oct 12, 2018 8:01 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 16, 2018 3:45 am

Hello
Download your app, for splunk, and I have followed your instructions to install it. However, I notice that the dashboards do not load information, as your screenshots show.
Inside the folders of the application is not found the scripts that would process the logs sent from the device, Mikrotik as healt or resources. According to the splunk documentation this should be located in the Mikrotik / bin folder, is my assessment correct?

Thank you!
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 16, 2018 7:05 pm

In this version there are no script at the Splunk (bin folder). Script is moved to the Mikrotik side that do send out everything using syslog.
In splunk set it to show last 24 hour and add * i search field to see what is going on.
This should get you lots of log line from the MikroTik
 
Dindihi
newbie
Posts: 27
Joined: Tue Jan 07, 2014 7:12 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 16, 2018 7:33 pm

I added index=mikrotik (your index name) to all dashboard searches.
Without this i also had no results
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 16, 2018 9:54 pm

When you run Splunk as a free license, the user should see all index data without need specify it.
Sounds you are either running it in a full licensed version, or are still in the 30 day free trail.
If you are in the trail mode you need to convert it as soon as possible to free version.
If not it will block your search fro on month if it passes 30 days before converting.
See installation instruction on first post.

It seems that you also have made MikroTik logs goes into another Index than default.
Follow the #1 post, it should go to the main index. It will work in other index, but you may need to adjust some like you did.
Last edited by Jotne on Tue Oct 16, 2018 10:00 pm, edited 1 time in total.
 
Dindihi
newbie
Posts: 27
Joined: Tue Jan 07, 2014 7:12 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 16, 2018 10:00 pm

I agree, i have paid license.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Oct 16, 2018 10:12 pm

Then is should be fine to use and as you did write, fix was to use correct index :)

I do have a 500GB/day license at my work, that I do manage, so do know some about who it works in large settings.
 
vitvickis
just joined
Posts: 1
Joined: Wed Oct 10, 2018 4:09 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Oct 18, 2018 3:14 pm

Hi Jotne!
Thank You for this post! Really appreciate your work.
I have some problems : I really can't make to log accounting and for example Resources/Voltage.
What need to be done, to see that in Splunk logs?

For example - DNS,DHCP, Firewall logs are working wihtout any problems.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Wed Nov 07, 2018 11:17 am

Working in 2.4

Here are some teasers.

Dark Theme
Added view to better show system changes
Added view to show wifi client strength for all clients connected
++++
.
MikroTik Wifi strength.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Wed Nov 07, 2018 11:42 am

Hi Jotne!
Thank You for this post! Really appreciate your work.
I have some problems : I really can't make to log accounting and for example Resources/Voltage.
What need to be done, to see that in Splunk logs?

For example - DNS,DHCP, Firewall logs are working wihtout any problems.
It seems that you miss the data coming from the script part.

Try these test to see if you see data in a command:
--------------
Show if ip accounting works, used to get firewall data
{
/ip accounting snapshot take
# Send data to loggin server
foreach logline in=[/ip accounting snapshot find] do={:put message="$[/ip accounting snapshot print as-value from=$logline]"}}
List health info:
{:local voltage ([/system health get voltage]/10);
:local temperature ([/system health get temperature]);
:put message="script=health voltage=$voltage V temperature=$temperature C";}
If cut and past of these commands gives information, you need to look at the script.
Have you created the script?
Does it have correct name?
Does it show a run count behind it greater than 0?
Do you get data out if you run the script manually?
Have you setup the scheduler?
Does it have correct name for the script to run?
Does the scheduler run (show run count)?
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Mon Nov 12, 2018 4:28 pm

I'm not able to see any result into splunk server.
I've followed the guide step by step, I'm seeing that the Mikrotik's script has Run Count = 40 so it is sending to Splunk server, I've added the windows firewall inbound rules, but I'm not able to see any data in splunk server.
Can you please help me?
Thanks

p.s. thanks a lot for your guide and effort, it is the tool I was searching for a long time.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Tue Nov 13, 2018 6:50 pm

In Splunk under:
Settings -> Data Inputs -> UDP
Do you se port 514 like this?
UDP port	Source type	Status	Actions
514	syslog	Enabled | Disable	Clone | Delete
If you do run Splunk on a windows, have you opened Windows firewall for Splunk or UDP:514?
You can try to disable firewall temporary.

Is logging correcty setup?
Can you on your MT post output of this:
 /system logging export
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Wed Nov 14, 2018 3:41 pm

Hi Jotne, thanks for your reply
In Splunk under:
Settings -> Data Inputs -> UDP
Do you se port 514 like this?
UDP port	Source type	Status	Actions
514	syslog	Enabled | Disable	Clone | Delete
Yes
If you do run Splunk on a windows, have you opened Windows firewall for Splunk or UDP:514?
Yes
You can try to disable firewall temporary.
I've tried but without any result
Is logging correcty setup?
Can you on your MT post output of this:
 /system logging export
[admin@MikroTik CRS125] > /system logging export
# nov/14/2018 14:33:07 by RouterOS 6.43.4
# software id =
#
# model = CRS125-24G-1S-2HnD
# serial number =
/system logging action
add name=logserver remote=192.168.88.210 target=remote
/system logging
add action=logserver prefix=MikroTik topics=dhcp
add action=logserver prefix=MikroTik topics=!debug
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Wed Nov 14, 2018 4:47 pm

Can you ping the Splunk server from MT?
Are you running Widows/Linux? I do recommend Linux.
On windows, you can confirm that Splunk is listening on port UDP/514 by running this command:
netstat -toan | find "514"
You should get one line like this:
UDP    0.0.0.0:514            *:*                                    5108
For Linux
netstat -pan | grep 514
udp    62848      0 0.0.0.0:514             0.0.0.0:*                           17949/splunkd
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 11:12 am

Can you ping the Splunk server from MT?
yes
[admin@MikroTik CRS125] > ping 192.168.88.210
  SEQ HOST                                     SIZE TTL TIME  STATUS             
    0 192.168.88.210                             56 128 0ms  
    1 192.168.88.210                             56 128 0ms  
    2 192.168.88.210                             56 128 0ms  
    sent=3 received=3 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms 
Are you running Widows/Linux? I do recommend Linux.
On windows, you can confirm that Splunk is listening on port UDP/514 by running this command:
netstat -toan | find "514"
You should get one line like this:
UDP    0.0.0.0:514            *:*                                    5108
For Linux
netstat -pan | grep 514
udp    62848      0 0.0.0.0:514             0.0.0.0:*                           17949/splunkd
I'm running it on Windows Server 2016
C:\Windows\system32>netstat -toan | find "514"
  UDP    0.0.0.0:514            *:*                                    9784
  UDP    0.0.0.0:58514          *:*                                    3464
  UDP    0.0.0.0:59514          *:*                                    3464
  UDP    [::]:60514             *:*                                    3464
  UDP    [::]:61514             *:*                                    3464
  UDP    [::]:62514             *:*                                    3464
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 1:02 pm

Then the only thing I do not see if the Windows Server block some in the firewall. But as you write, you have opened it.
If you did not try it, try to disable the whole fw for some time.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 2:47 pm

I've tried to disable the entire firewall for 10 minutes and I've executed manually the script at least three times, but no info was present in the dashboards :(
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 6:22 pm

Try this.
On terminal of the Router OS type:
:log info message="mandarin"
In Splunk, set it to last 15 min and do a search like this:
mandarin
You should get at least one line like this (in raw mode)
script,info MikroTik: mandarin
If you get output you have communication.
If it does not show script,info MikroTik: in front of mandarin, you MikroTik app is not correctly installed in Splunk
Post your output.
 
Dindihi
newbie
Posts: 27
Joined: Tue Jan 07, 2014 7:12 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 6:26 pm

Is there a tcpdump or similar on windows?
Maybe check if udp packets are coming from your MT.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 6:44 pm

This is UPD, so tcpdump would not help.
Did the last test not give you anything?
 
Dindihi
newbie
Posts: 27
Joined: Tue Jan 07, 2014 7:12 pm

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 6:47 pm

Sure,
tcpdump also shows udp packets.

[~] # tcpdump udp port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:46:01.876313 IP 192.168.215.4.58292 > NAS.syslog: [|syslog]
17:46:01.876371 IP 192.168.215.4.58292 > NAS.syslog: [|syslog]
17:46:05.144568 IP 192.168.214.117.syslog > NAS.syslog: SYSLOG user.info, length: 66

Edit:
You tried to manually search for events (not the dashboard).
index=YOURINDEX
(on ALL TIME)
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 7:15 pm

Try this.
On terminal of the Router OS type:
:log info message="mandarin"
In Splunk, set it to last 15 min and do a search like this:
mandarin
You should get at least one line like this (in raw mode)
script,info MikroTik: mandarin
If you get output you have communication.
If it does not show script,info MikroTik: in front of mandarin, you MikroTik app is not correctly installed in Splunk
Post your output.
I've found the mandarin entry

In the Search app I'm seeing this:

490.761 event and it is growing!

Why I'm not seeing anything under mikrotik app?
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 7:17 pm

Anyway I've also tried to install Splunk in a UbuntuVM using Virtualbox, I've followed your guide to add the splunk user, I'm trying to configure the UDP 514 input port but I'm having this error:

Parameter name: UDP port 514 is not available.

I prefere to solve the issue under Windows server that is installed bare metal
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 7:49 pm

You do get data inn to Splink

Do a search like this in Splunk last 15 min
*
Post some lines, so that I do see how it looks like.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:00 pm

I've attached two screenshot
You do not have the required permissions to view the files attached to this post.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:03 pm

please note that I've inserted "Mikrotik" under System/Logging
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:13 pm

All apps needs to be in
$SPLUNK_HOME/etc/apps
So on windows you should have:
C:\Program Files\Splunk\etc\apps\MikroTik
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:14 pm

All apps needs to be in
$SPLUNK_HOME/etc/apps
So on windows you should have:
C:\Program Files\Splunk\etc\apps\MikroTik
Yes
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:16 pm

Uninstall everything.
Install follow post #1 step by step.

You should also see
C:\Program Files\Splunk\etc\apps\MikroTik\default
C:\Program Files\Splunk\etc\apps\MikroTik\metadata
etc
Not
C:\Program Files\Splunk\etc\apps\MikroTik\MikroTik\default
If that does not work, I will try to do an install my self from the #1 post and test it.

PS no need to quote post above you.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:23 pm

are you sure?
I've attached a screenshot of the content of the folder
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:43 pm

You have restarted Splunk after app install?
All looks correct.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:50 pm

Yes several times
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:55 pm

You have installed Splunk free as in post #1, or do you use Splunk before to some else?
I have seen problem with installed version that using other index.

From the picture above, it does not seem that splunk does the filed extraction.

If you like, I can try teamviewer to see what is wrong.
Not able to post a private message to you, so post an email so I can get in touch with you.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 9:08 pm

thanks for your help, but I'll try tomorrow with the linux VM but I've to solve first why the 514 port is not available even if I followed your guide to install the app with a non root user
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.4 (Graphing everything)

Tue Nov 20, 2018 12:20 pm

2.4 Released

Nearly all code are rewritten to get better speed and make it cleaner.
Dark Theme makes a big visual change.

# v2.4 (20.11.2018)
# Updated "MikroTik Hotspot login/logout information" to show IP
# Fixed when inn interface= unknown
# Updated view 2.4 to handel more hits
# Updated "MikroTik DNS" to not view revers lookup "site!=*.in-addr.arpa"
# Rewritten "Microtik Traffic" Error in all calculation
# Fixed data rounding and fixed typo
# Fixed formating in "MikroTik Remote Connection"
# Set permission view the view to show in app only
# Added System Changes as a new default menu
# Fixed missing host in "MikroTik Uptime"
# Added Host to "MikroTik Traffic"
# Added view "MikroTik Wifi strength"
# Added view "MikroTik System Changes"
# Dark theme needs >=7.2
# Removed global time (use default time)
# Removed searchWhenChanged="true" (default)
# Cleaned code
# Fixed error in "13. OSCam config changes"
# Added Sprakline to "MikroTik Device List"
2.4 Device list.jpg
.
2.4 System Changes.jpg
.
2.4 Traffic.jpg
You do not have the required permissions to view the files attached to this post.
 
jareckib12
just joined
Posts: 1
Joined: Fri Jan 12, 2018 1:04 am

Re: Using Splunk to analyse MikroTik logs 2.4 (Graphing everything)

Sat Nov 24, 2018 12:06 am

Hi,
First - thx for update.
Second - in MikroTik DNS request view, client filtering does not work. When selecting any item in addition to "any" does not show any results.

Jarecki
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sun Nov 25, 2018 10:40 am

Good catch,

I have updated to 2.5

# 2.5 (25.11.2018)
# Change all "if" test to use "coalesce"
# Fixed error in "MikroTik DNS request"
# Moved more to base search
# Removed some code not needed in "MikroTik Web Proxy"
# Fixed error with src_port in dest_ip dropdown in "MikroTik Firewall Rules"
 
sherics
just joined
Posts: 10
Joined: Sun Nov 25, 2018 10:02 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sun Nov 25, 2018 9:26 pm

Hello,

I have installed it completely and except the Traffic, everything work.

In the traffic I see just few MBs, even if I download 500MB or 1GB, it does not shows up there, just few % of the downloaded amount.

I do not have a public IP on my internal network, the public IP is on the WAN port, ether1, as a standard home router, other clients are on WiFi on first VLAN and 2 computers on second VLAN.

Do you have an idea what is wrong?

Thank you.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 6:08 pm

I did download an 1GB file from here: http://www.ovh.net/files/
And it showed up correctly.

Do you have Fasttrack on?
If so try to disable it, it may be that packed are not accounted when Fasttrack is on.
https://www.youtube.com/watch?v=6LaqhDm6PHI
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 8:45 pm

Hello Jotne and the whole community.
First I want to tell you, good job, really good jobs, and thanks for sharing with us Jotne.

Secondly I have a question, in version 2.4 I see in the record that wrote "List of devices" this function indicates that it already supports multi-router log ?, in such case as it is identified in each module to which router belongs each record?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 9:32 pm

You are correct, it does support as many routers as you like to add.
Going away from SNMP to Syslog only was driven by the simpler way to do thing.
With SNMP, you need to set up the monitor system to request SNMP from the device.
This is ok for singel router ans small system.
But if you like to monitor a router across public internet, you end up in a security risk by open for SNMP.

Whit using script and Syslog this is a one way communication. All data are sent from the device to the monitor system.
No need to open ports. Same script for all routers. No need to configure any configuration on the monitoring system for each router.

I have four routers/host that sends log to my sentral log server.

On every view you can select host to view only that host.
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 9:41 pm

Great, you're right, forget that each module has a drop-down menu Hosts. I'm going to try it and anything I write. Thanks again.
 
sherics
just joined
Posts: 10
Joined: Sun Nov 25, 2018 10:02 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 10:59 pm

I did download an 1GB file from here: http://www.ovh.net/files/
And it showed up correctly.

Do you have Fasttrack on?
If so try to disable it, it may be that packed are not accounted when Fasttrack is on.
https://www.youtube.com/watch?v=6LaqhDm6PHI
Well, I forgot about fastrack... without fastrack it works now, but unfortunately without fastrack my router is on 95-99% CPU while I download/upload anything; and the speed is lowered for 300mbit/s... With fastrack enable, the cpu is approx on 70% on full gigabit connection, about 90MB/s real speed. Well, after 4 years, I think, I need to purchase a more powerul and new router :)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 11:22 pm

It may also be that you could configure your router to use hardware offloading. Depending on type and software version.
But old boxes do have less power so upgrade may be the only option.

Its a good point to now that traffic monitoring does not work when fast track is enabled, so I will mention that in the first post.
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Nov 30, 2018 3:45 pm

Hello everyone, I have a problem with module MikroTik_Traffic section Public IP. when reviewing this, I found a small error when declaring the variable host, in this case that variable is capitalized Host, it does that the section does not work, changing this I achieved that it works correctly.
<title>Public IP</title>
        <search base="base_search">
          <query>
            search
              Host="$Host$"         >>>   change with host="$Host$"
            | eval ip_in=if("$direction$"=="in",src_address,dst_address)
            | eval ip_out=if("$direction$"!="in",src_address,dst_address)
Image
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Nov 30, 2018 5:28 pm

Thanks for the feedback :)

It will be fixed in 2.6. For others you can edit det file and correct the typo.
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Dec 07, 2018 4:18 am

Hello there,

This is my first time using splunk and I have no result on dashboard anyway also I did every step on the post #1, any idea why this happen?
The logs already show up on the splunk but the MikroTik app dashboard have no result at all.

Thanks and appreciate your help.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Dec 07, 2018 11:51 pm

/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug
I would guess you have typed wrong prefix. Any other word than MikroTik would brake the index of the data.
Make sure its 100% equal with capital M and K

Cut and Past is the best option to get it correct.

Do a search like this in Splunk, change to your MikroTik Routers IP, what is the output?
index=* host=192.168.88.1 | rex "^\S+\s(?<prefix>\S+)\s" | stats count by prefix
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 1:23 am

Hello,

I did copy and paste that command on cli.

The result prefix search on attachment
Search MT.JPG
And then I found something that on the search section if I remove module=xxx then I got the result on the dashboard.
For the example on the device list dashboard I use this
No Module.JPG
instead of your originally script
With Module.JPG
I think that module=xx didn't work on my splunk search. Any idea?
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 9:38 am

Strange.

Can you post output of sourcetype=mikrotik script=sysinfo
Make sure you have Smart Mode selected (see circle on picture)
Click the arrow to expand one post so I see the extraction. >
.
test_output.jpg
You do not have the required permissions to view the files attached to this post.
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 10:33 am

Hi Jotne,

Here is the output and just different from yours.
test_output_1.JPG
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 12:26 pm

I see two strange things.
1. It seems that Splunk does not handle the date/time correctly since its shown within your event.
2.I do not see the information from router that shows where it comes from and type (ipsec/DNS/DHCP) (debug packets)

Is this a clean Splunk installation, followed the steps above?

You are running on a 951G a common box, I have a 941 and 750Gr3 and some other.
Your RouterOS software 6.43.4 is the same as I do run, so should be ok

Can you post the last lines of the output on the Router of /log print and /log print detail
Just cut and paste the line, so I do see how it looks like.

On mine
11:21:32 script,info script=pool pool=default-dhcp used=1 total=245
and
time=11:21:32 topics=script,info
message="script=pool pool=default-dhcp used=1 total=245"
I do miss the stuff in bold from your logg message and would like to see how it looks like on the router to compare what Splunk sees.
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 12:52 pm

Hi Jotne,

Yes, this is fresh install splunk and I did several time remove my VM and install again to make sure that.

Here is the output
/log print

17:47:19 firewall,info FW_INTERNAL forward: in:PJX out:BRX-LAN, proto TCP (ACK), 10.99.100.102:7332->10.121.61.108:52380, len 40

/log print detail

time=17:48:19 topics=firewall,info message="FW_INTERNAL forward: in:PJX out:BRX-LAN, proto TCP (ACK), 10.99.100.102:7332->10.121.61.108:52380, len 40"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 1:09 pm

This looks correct, so it have to be some wrong with Splunk implementation since message looks different there.
Several other has used this, so should not be an big error in the code.

If you tyoe index=* in splunk, do you see any message that have the module tag coming from the router?

Like this
firewall,info
PS If you set time to: real time 1-minute window you should see data live as they arrive.
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 1:28 pm

Unfortunately I didn't see that message on my splunk,
test_output_2.JPG
Any idea what is happening on my splunk?
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 2:10 pm

You have some strange in your message that I have not see with other: RTZPKN02

Can you post this? /system logging export

How did you install the files in Splunk?

Why do you get Des 9 in your log, I am still at Des 8?
Your logs has two different time stamp.
See if all clock is equal everywhere. Router, Computer ++
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 2:23 pm

Sorry the Dec 9 is from date server, I already change the NTP :D

Here is the output
 /system logging export
# dec/08/2018 19:21:37 by RouterOS 6.43.4
# software id = 29W1-FTPT
#
# model = 951G-2HnD
# serial number = 642E05A9020A
/system logging action
add name=syslog remote=10.99.100.77 remote-port=7514 src-address=10.122.82.200 \
    target=remote
add bsd-syslog=yes name=logserver remote=10.100.10.105 src-address=\
    10.122.82.200 target=remote
/system logging
add action=syslog disabled=yes topics=info,error,interface,warning
add action=logserver prefix=MikroTik topics=dhcp
add action=logserver prefix=MikroTik topics=!debug
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 2:44 pm

Seems that you do not have a clean install.
You are logging to several system at the same time.
It should work.

Try this: Remove all logg line and add this:
/system logging action
add name=myserver remote=10.100.10.105 target=remote
/system logging
add action=myserver prefix=MikroTik topics=!debug
add action=myserver prefix=MikroTik topics=dhcp
Is this your Splunk server? 10.100.10.105
If not, do you relay your message (rslyslog or other server)?
Do you send your log message passing trough several routers?
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 3:45 pm

Seems that you do not have a clean install.
You are logging to several system at the same time.
It should work.

Try this: Remove all logg line and add this:
/system logging action
add name=myserver remote=10.100.10.105 target=remote
/system logging
add action=myserver prefix=MikroTik topics=!debug
add action=myserver prefix=MikroTik topics=dhcp
Is this your Splunk server? 10.100.10.105
If not, do you relay your message (rslyslog or other server)?
Do you send your log message passing trough several routers?
Ok I will reinstall my splunk VM again and change all log line and I will tell you the result

And yes my log message passing through several routers.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 3:46 pm

But is this your Splunk server? 10.100.10.105
Or do you send data to an rsyslog or other syslog server, that then sends it to your Splunk server?
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 5:30 pm

Hi Jotne,

It seems I found the problem, the problem is marking the BSD Syslog on log remote action.
test_output_3.JPG
Finally the result is come.

Thanks and very appreciate your help.
You do not have the required permissions to view the files attached to this post.
 
WeWiNet
Long time Member
Long time Member
Posts: 591
Joined: Thu Sep 27, 2018 4:11 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Tue Dec 11, 2018 1:53 pm

Hi Jotne,

Wanted to say thank you, very nice job.
Also to highlight that this tutorial works perfect on MacOS 10.14.
I just followed your tutorial and installed it with the Splunk Enterprise version
and all is working perfect (Ok I had to restart my machine once as splunk did not launch first time correctly).

I now try to make sense out of all that data and nice graphs ... :-)

PS: How can you know how much data you log per day (which is the limitation of the free version)?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Tue Dec 11, 2018 2:52 pm

Thanks.

You find license information her:

Settings->Licensing
There you see this for free version
Licensed daily volume 500 MB

Select:
Usage-Report->Previous 30 days

Here you will see how much of the license you use each day, last 30 days.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Tue Jan 01, 2019 11:47 pm

Thank You for this post and all the work to get all the information in Graphs.

Only I had a hard time to get all the information in Splunk.
After three hours of trying a lot of different things I finaly discoverd that I missed the last "}" in the Router script.

Perhaps you can change the post where the script is too make the beginning and end of the script more clear.

Regards Peter
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Wed Jan 02, 2019 5:50 pm

Thanks for the feedback.
Added some space in the script to make it better to see start end.

Next time you can click Select ALL, behind the Code: at the top of the script and you get all that is needed.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Thu Jan 03, 2019 10:37 am

Thank You for this post and all the work to get all the information in Graphs.

Only I had a hard time to get all the information in Splunk.
After three hours of trying a lot of different things I finaly discoverd that I missed the last "}" in the Router script.

Perhaps you can change the post where the script is too make the beginning and end of the script more clear.

Regards Peter
I had the same problem,
now everything works
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Thu Jan 03, 2019 11:06 am

I'm seeing two problems:
The script reports a cpu higher than usual, it detects the cpu loads when the scripts is running, so instead of reading a normal 10% load, it reads a load near to 100%

The second is the Disk graph.
I've attached two screenshot
You do not have the required permissions to view the files attached to this post.
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Thu Jan 03, 2019 11:21 am

I've modified the script in order to read the cpu load at the beginning, now the readings are correct
# This script is used to send data to Splunk using syslog.
#===================================

# Collect system resource
# ----------------------------------
:local cpuload ([/system resource get cpu-load]);
:local freemem ([/system resource get free-memory]/1000000);
:local totmem ([/system resource get total-memory]/1000000);
:local freehddspace ([/system resource get free-hdd-space]/1000000);
:local totalhddspace ([/system resource get total-hdd-space]/1000000);
:local up ([/system resource get uptime]);
:log info message="script=resource free_memory=$freemem MB total_memory=$totmem MB free_hdd_space=$freehddspace MB total_hdd_space=$totalhddspace MB cpu_load=$cpuload uptime=$up";



# Collect accounting traffic
# ----------------------------------
# Take a snapshoot
if ([/ip accounting get enabled]=yes) do={
/ip accounting snapshot take
# Send data to loggin server
foreach logline in=[/ip accounting snapshot find] do={:log info message="$[/ip accounting snapshot print as-value from=$logline]"}};

# Finding dynmaic lines used in uPnP
# ----------------------------------
:foreach logline in=[/ip firewall nat find dynamic=yes] do={:log info message="$[/ip firewall nat print as-value from=$logline]"};

# Collect system information
# ----------------------------------
:local version ([/system resource get version]);
:local board ([/system resource get board-name]);
:local model ([/system routerboard get model]);
:local serial ([/system routerboard get serial-number]);
:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial";

# Collect system health
# ----------------------------------
:local voltage ([/system health get voltage]/10);
:local temperature ([/system health get temperature]);
:log info message="script=health voltage=$voltage V temperature=$temperature C";

# Sends wireless client data to log server
# ----------------------------------
:foreach logline in=[/interface wireless registration-table find] do={:log info message="$[/interface wireless registration-table print  as-value from=$logline]"};

# Collect DHCP Pool information
# ----------------------------------
/ip pool {
   :local poolname
   :local pooladdresses
   :local poolused
   :local minaddress
   :local maxaddress
   :local findindex
   :local tmpint
   :local maxindex


 #  :put ("IP Pool Statistics")
 #  :put ("------------------")

# Iterate through IP Pools
   :foreach p in=[find] do={

      :set poolname [get $p name]
      :set pooladdresses 0
      :set poolused 0


#   Iterate through current pool's IP ranges
      :foreach r in=[:toarray [get $p range]] do={

#      Get min and max addresses
         :set findindex [:find [:tostr $r] "-"]
         :if ([:len $findindex] > 0) do={
            :set minaddress [:pick [:tostr $r] 0 $findindex]
            :set maxaddress [:pick [:tostr $r] ($findindex + 1) [:len [:tostr $r]]]
         } else={
            :set minaddress [:tostr $r]
            :set maxaddress [:tostr $r]
         }

#       Convert to array of octets (replace '.' with ',')
         :for x from=0 to=([:len [:tostr $minaddress]] - 1) do={
            :if ([:pick [:tostr $minaddress] $x ($x + 1)] = ".") do={
               :set minaddress ([:pick [:tostr $minaddress] 0 $x] . "," . \
                                       [:pick [:tostr $minaddress] ($x + 1) [:len [:tostr $minaddress]]]) }
         }
         :for x from=0 to=([:len [:tostr $maxaddress]] - 1) do={
            :if ([:pick [:tostr $maxaddress] $x ($x + 1)] = ".") do={
               :set maxaddress ([:pick [:tostr $maxaddress] 0 $x] . "," . \
                                       [:pick [:tostr $maxaddress] ($x + 1) [:len [:tostr $maxaddress]]]) }
         }

#      Calculate available addresses for current range
         :if ([:len [:toarray $minaddress]] = [:len [:toarray $maxaddress]]) do={
            :set maxindex ([:len [:toarray $minaddress]] - 1)
            :for x from=$maxindex to=0 step=-1 do={
#             Calculate 256^($maxindex - $x)
               :set tmpint 1
               :if (($maxindex - $x) > 0) do={
                  :for y from=1 to=($maxindex - $x) do={ :set tmpint (256 * $tmpint) }
               }
               :set tmpint ($tmpint * ([:tonum [:pick [:toarray $maxaddress] $x]] - \
                                                    [:tonum [:pick [:toarray $minaddress] $x]]) )
               :set pooladdresses ($pooladdresses + $tmpint)
#         for x
            }

#      if len array $minaddress = $maxaddress
         }

#      Add current range to total pool's available addresses
         :set pooladdresses ($pooladdresses + 1)

#   foreach r
      }

          :set poolused [:len [used find pool=[:tostr $poolname]]]
#   Send data
    #      :log info message=("pool=" . $poolname  . " used=" . $poolused . " total=" . $pooladdresses)
          :log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")

# foreach p
   }
# /ip pool
}
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Thu Jan 03, 2019 3:10 pm

Good idea moving the cpu reading to the top. I have updated fist post view new version.
PS mine du not give much difference in CPU when script is running. Maybe you device is some under-powered or you have som wrong in your configuration (fasttrack or hw acceleration missing)

I see that MB is wrongly reported due to dividing on 1000000 and not 1048576 (1024*1024). Corrected in the script.
Since graph is in percentage it should not make any change to the view.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Jan 18, 2019 3:08 am

Good idea moving the cpu reading to the top. I have updated fist post view new version.
PS mine du not give much difference in CPU when script is running. Maybe you device is some under-powered or you have som wrong in your configuration (fasttrack or hw acceleration missing)

I see that MB is wrongly reported due to dividing on 1000000 and not 1048576 (1024*1024). Corrected in the script.
Since graph is in percentage it should not make any change to the view.
No matter how it is set, my splunk can't get my ccr1016 data. Splunk is a virtual machine ubuntu server in the LAN, ccr on what you said, every 5 minutes running script can see a lot of log information generated, but still no data into splunk, there is nothing I did not do Is it? Please forgive my english, from google translation.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Jan 18, 2019 8:29 am

No need to quote message above you, only part of it when needed. Always use Post Reply button under the post.

Are you 100% sure you have tagged the packet with MikroTik? There are no firewall?
Try in the search page and search for a star last 15 min. Do you get any?
*
 
Hunty
just joined
Posts: 20
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 9:50 am

After three hours of trying a lot of different things I finaly discoverd that I missed the last "}" in the Router script.
Check this!
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 3:30 pm

I'm Using Splunk for a couple of weeks now.

In the Firewall Rule section I see beside the attacks from the large big spooky internet also local adresses appear as a result of the "FW_Drop_All_From_Wan" rules.
and that are mainly request with dest_port 53 (DNS).

Is it possible to filter out the local addresses?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 6:38 pm

You should not see local address in the outside inn block rule.

Can you post an example like this:
2019-01-21 17:25:25	FW_Drop_all_from_WAN	input	ether1-Wan	(unknown 0)	00:05:00:01:00:01	TCP	104.131.145.9	45167	92.31.200.211	2082	San Francisco	United States
And yes, you can get rid of the message in two ways.
1. Add a rule on the fw above the outside in block rule that block the specific ip/port of your choice.
2. Modify Splunk to exclude the ip/port you like.

First is the simplest solution.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 7:07 pm

Below an example of the local 192.168.0.1 address
_time	rule	chain	in_if	out_if	src_mac	protocol	src_ip	src_port	dest_ip	dest_port	City	Country
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	42597	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	57660	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	56630	192.168.0.1	53	Unknown	
And I wil try the solution with an extra Drop rule in the firewall.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 7:34 pm

It looks like your router tries to resolve DNS on it self and get blocked.
From router console try this.
:put [/resolve mikrotik.com]
You should get an IP as result, like 159.148.147.196
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 8:20 pm

I get the same IP as a result biut perhaps My PI-Hole implementation has something to do with it.

I'm using PI-Hole as an "Ad blocker for my Internal network"
And for this I'm using DHCP option 6 to force all internal clients to go to the PI-Hole server for the DNS resolving.


By the way I changed the "Drop all from not coming from LAN" rule.
I replaced the "In Interface list" from !LAN to "In Interface" Ether1-WAN.

This seemed to have resolved my issue.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 8:43 am

2.6 released

# 2.6 (22.01.2019)
# Added information about fast track in "traffic monitor"
# Fixed typo in Traffic view. Added fast track info
# Changed to checkbox in "DNS Request"
# Added better sparkline "in Device List"
# Added identity to "Device List"
# Updated script to get identity
# Removed parentheses from services from "MikroTik uPnP"
# Added ip to client drop-down list to "MikroTik uPnP"
# Added more disk info to "MikroTik Resources"
# Changed to last 12 hour instead of 4 in "MikroTik DNS Live usage"
# Changed to sort by count in "Sort by count"
# Added timeline dashboard to "DNS Request"
# Fixed public IP speed by reducing lookup in "Traffic"
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 11:05 am

I see some strange things happen.

I have added three devices to the Splunk Mikrotik environment.

1. RB750Gr3 as a router. (sending over UDP 514)
2. HAPac2 configured as a switch (Accesspoint) (sending over UDP 515)
3. Mikrotik CHR as Dude server. (sending over UDP 516)

Everything seems to log all information to splunk but after somtime the data of the HAPac2 is not examind any more by Splunk.
After restarting the splunk server Everything is OK again for a short time.
The Router and the DUDE server have no issues.

When i check the Splunkd.log file I see a lot "Failed to parse timestamp" messages for the HAPac2 syslog.
01-22-2019 09:46:45.504 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:515|host=192.168.0.8|syslog|
What can be wrong?

This morning I updated to version 2.6.
But I had this problem before. So it is not version related.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 11:40 am

It may be that the props filter only look at UDP:514 and syslog.
When data comming in on UDP:515 it will not see that its MikroTik data.

You can fix this by edit etc/apps/MikroTik/default/props.conf and add
[source::udp:515]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik

[source::udp:516]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik
But my questioon to you is, why use more than on UDP?
I do see noe good reason to use on port for each device. Send all to UDP/514
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 1:10 pm

I started with using port 514 for all 3 mikrotik devices.
At that moment I had the same problem. No data in visible in Splunk.
After that I changed to port 515 and restarted splunk. And yes I saw data in Splunk. but some time later Splunk stopped showing data in the graphs.
Then I restarted splunk again and yes Splunk is showing data for an hour or so.
The Router and the Dude device are showing Up as expected.

See the picture below:
2019-01-22 11_52_06-MikroTik Wifi strength _ Splunk 7.png
At the moment I changed all 3 devices back to UDP port 514. With the same result as before.

I still see below messages in the splunkd.log file saying it suppresses messages:
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|\n                                1295 similar messages suppressed.  First occurred at: Tue Jan 22 11:57:40 2019
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.345 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.348 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.356 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 1:19 pm

Do examine time on all your devices. It must be in sync.
Do use NTP on all devices to make sure time is ok.
Last edited by Jotne on Tue Jan 22, 2019 3:53 pm, edited 1 time in total.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 1:52 pm

The router is used as the timeserver for my local environment.
the HAPac2, the Dude server and the Splunk server synchronize time with the router and all have the same time and date.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 9:57 am

It may have something to do that you have used different UDP ports. I may not recognize the message correctly.
You may try to start over and follow the example step by step.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 7:10 pm

I made some progress.

After an other look at the messages in the splunkd.log file
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
I focused on the MAX_TIMESTAMP_LOOKAHEAD option. according to the default props.conf file this option default to 32 for syslog events.
Looking at the mikrotik log events the consists of 19 characters (excluding the mili seconds).

To change the default 32 to 19 I added the MAX_TIMESTAMP_LOOKAHEAD option to the "/opt/splunk/etc/apps/MikroTik/default/props.conf " file and restarted Splunk.
[syslog]
TRANSFORMS-force_mikrotik = force_mikrotik
MAX_TIMESTAMP_LOOKAHEAD = 19
After this change I do not see the above message in the Splunkd.log file anymore. And more important, the Hapac2 is logging events for more as 3 hours now. This is already an hour longer as before (max 2 hours).

I will keep an eye on the Mikrotik splunk environment to see if everything keeps running.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 9:11 pm

Interesting. Have not used much time in my splunkd.log, but have the same problem as you,
But only in one of 4 routers. Other are ok.

Tried bot 19 and 23 but still get samme message.
01-23-2019 20:08:40.136 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event (Wed Jan 23 20:08:39 2019). Context: source=udp:514|host=193.1.1.100|syslog|
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 10:41 pm

At my site it was 1 out of 3 that failed and I was missing information for that router.
Are you also missing data?
After the change my failing router is still visible in Splunk so for mee it seems the solution.
But I did not check the log files that come from the routers. Do you now were I can find them?
Perhaps it has something to do with too many events during a short time period.

We need to debug this.

Regards Peter
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jan 24, 2019 8:12 am

I do get event from all routers. To see if you get from one specific router use search and type host=1.2.3.4 (change to your IP)
 
Egert143
just joined
Posts: 17
Joined: Tue Apr 24, 2018 4:05 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jan 25, 2019 3:38 pm

Hello

Could i get instructions how to create splunk source type manualy ? I have splunk light (paid) and it doesent support apps (as far as i know).

Current problem is that source and dest addres fields are merged with port numbers.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jan 25, 2019 10:07 pm

I have no idea on how to use Splunk Light.
In normal Splunk, source type based on the source it comes from udp:514

props.conf
[source::udp:514]
TRANSFORMS-force_mikrotik = force_mikrotik
transforms.conf
[force_mikrotik]
DEST_KEY =  MetaData:Sourcetype
REGEX =  \sMikroTik:\s
FORMAT =  sourcetype::mikrotik
 
Egert143
just joined
Posts: 17
Joined: Tue Apr 24, 2018 4:05 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Jan 26, 2019 10:07 pm

And how would i turn 123.123.123.123:1234->12.34.45.67:80 to Source Address = 123.123.123.123 Source Port = 1234 Dest Address = 12.34.45.67 Dest Port 80 So they would be searchable ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Jan 26, 2019 11:02 pm

The traffic solution are based on that you have private ip inside your net and public on the outside.
Private IPv4 addresses
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16


But if you like to log other IP and know what is inside/outside, you have to modify the Splunk files.

Edit:
MikroTik Traffic
Replace all
 | search (ip_in="10.0.0.0/8" OR ip_in="172.16.0.0/12" OR ip_in="192.168.0.0/16")
with
 | search ip_in="12.34.45.0/8"
That if you like 12.34.45.0/8 to be your inside net.
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Feb 01, 2019 8:41 am

Where can I find the link to download MikroTik2.6 spl? Thanks.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Feb 02, 2019 9:47 pm

In the first post of this topic :)
Or the below link
download/file.php?id=35231
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 04, 2019 9:40 am

Thanks zandhaas - I got it downloaded.

I also installed everything per our topic owner Jotne's procedure but cannot get the data flow from MikroTik to Splunk, after verifying port 514 is open. Upon diving into some details, I suspect it's due to the lack of SSL of my MikronTik (192.168.88.1 shows "Not secure") - anyone know if this is the root cause? If yes what is the easiest way to enable SSL under RouterOS v6.40.8 and Win10? Appreciate any tips there.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 05, 2019 10:55 am

There are no certificates involved in the transaction. All data are sent using UDP/514 Syslog (not encrypted).

In Splunk search, type only a * and do a search for the last 24 hour. Do you see any data at all?
Make sure you follow all steps in the first post 1 by 1.
Do you have any deviation? Using a clean Splunk install? Windows firewall opened if you run on Windows?
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Feb 06, 2019 7:16 am

Thanks Jotne. Now I see the data (events) through the Splunk search, though MikroTik2.6 app still not sees the data yet and I am still debugging.

BTW the Splunk observed event entry looks like - do you see any anomaly there?

2/5/19
9:09:54.000 PM
Feb 5 21:09:54 router.lan Feb 5 21:09:54 MikroTik MikroTik: Router = 192.168.88.1
host = router.lan
source = udp:514
sourcetype = mikrotik
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Feb 06, 2019 8:22 am

Can you post some example line from search in Splunk that shows what you got in the log from using * search?

Do you have tagget all packet with MikroTik? This will fail Mikrotik since its not the same name.
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 6:19 am

Hi Jonte, here're three snapshots

1. Splunk Event entry sample from the MikroTik UDP feed - great if you can help review the "Host", "Source", "Sourcetype" field to see if they are right for the MikroTik2.6 App
Splunk Event Entry from UDP and MikroTik.png
2. Splunk UDP input setting
Splunk UDP Input Setting.png
3. MikroTik2.6 App snapshot (system change search, with no data found while the Splunk search gives items like above)
Splunk MikroTik 2.6 App Lauch Snapshot.png
Thanks
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 8:27 am

1. Is this the only type of event you see?

Here are some example on how they should look like: (various modules)
firewall,info MikroTik: NAT_Web_server dstnat: in:ether1-Wan out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 91.12.58.49:49145->92.220.200.251:80, len 60
dhcp,debug,packet MikroTik:     Parameter-List = Subnet-Mask,Router,Domain-Server,Domain-Name,NETBIOS-Name-Server,Static-Route
dns,packet MikroTik: --- sending reply to 10.10.10.244:53720:
script,info MikroTik: script=health voltage=24 V temperature=42 
wireless,info MikroTik: 04:62:73:xx:xx:21@wlan1 established connection on 2437000, SSID GjestenettHMN
ipsec MikroTik: invalied encryption algorithm=6.
interface,info MikroTik: ether1 link up (speed 100M, full duplex)
Have you followed tutorial in post#1?
Do you use Splunk for other stuff?
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 10:26 am

Hi I followed your first post but skipped 2c~2e (FW/NAT/Traffic logging since not sure about the detailed steps). I did have Home Monitor app before that affected the MikroTik data inputs, and I have it removed so the data inputs seems right (though not complete if without 2c~2e). The question I have is that, even with incomplete but valid data (say only DHCP request part), should MikroTik2.6 App see them and populate some view right? But now it seems the app does not pick up anything and I am not sure if the app has access to the log. Thanks.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 11:52 am

You should get DHCP and other stuff from the router if you skipped 2c-2e.
Thats why I asked about how the log lines looks like.
You could use a search for host=192.168.88.1 and post some line.
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 9:17 am

Thanks again Jotne. Here's a screenshot. Seems the Splunk events have the right contents, but the format is different from yours.
Splunk MikroTik 2.6 Event Snapshots.png
Basically, before the identifier "MikroTik", there are timestamps and another "MikroTik", but without the log field name like "dns,packet" as in your snapshots.
I copied the MikroTik scripts exactly, so do you think I missed something on the Splunk side? My Splunk version is 7.2.3.
You do not have the required permissions to view the files attached to this post.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 9:44 am

Are you sure your "router script" is complete?

I had problems getting my data visible in splunk to.
It turned out that I missed the last "}" in the Router script.
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 10:04 am

Hi Jotne ~ some progress - for some reason, the "Module" field picks up part of the timestamp (the Month) since their is no syslog field name for some reason (the event item format difference I mentioned). After tweaking the Volt/Temperature code (removing the module key from the search), I was able to get that view right. Encouraged and will see how to get the module field right in the first place - help appreciated.
Splunk MikroTik 2.6 Volt_n_Temp.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 12:04 pm

Since I'm not a Splunk expert I wonder if anyone has some bright ideas how to optimize Splunk / Mongodb?

We have about 15.5 million entries and the reports are getting really slow to produce. In a regular SQL database you can run a "Query Execution Plan" and then add indexes to columns that performs table scans. Is there an equivalent way in Splunk or any other way to optimize the environment? We're running Splunk with 12 cores, 20 Gb ram and SSD which ought to be sufficient.

Any suggestions are welcome!
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 6:41 pm

@Larsa
Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.

I do recommend that you start a thread about your problem over here:
https://answers.splunk.com/index.html
Last edited by Jotne on Mon Feb 11, 2019 6:54 pm, edited 1 time in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 6:51 pm

@JieYu2001

There are some wrong with extraction of the data in the Splunk or the format that your MT Router sends it.
In List view in Splunk your should not see time and date in the Event space, only in Time column.
In your view, I do not see it only one time extra, but two times in front of the data. This breaks all view.
You get it to work since you adjusted to view to accept your wrong data.
source=udp:514 and sourcetype=mikrotik looks correct.

I would recommend you to start over.
Clean Install of Splunk, remove all connection to Splunk in your router.

@zandhaas
You do not need the script to get data inn to splun, so it could also be removed to rule out problems.
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 12, 2019 9:26 am

Thanks Jotne - the issue is resolved. In the MK Logging setting, I checked "BSD Syslog" which caused issue (still don't know why since that is the correct syslog protocol supported in Splunk). Uncheck it and things look fine now.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 12, 2019 3:08 pm

There was nothing in the first post telling you to select it so not sure why you did it.
Will update post #1 to say not to select it.
Good you find out what was wrong :)
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 12, 2019 9:07 pm

Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.I do recommend that you start a thread about your problem over here: https://answers.splunk.com/index.html

Thanks for the suggestion, I'll report back if I find out an appropriate solution!
 
User avatar
oaas
just joined
Posts: 4
Joined: Sun Feb 10, 2019 7:15 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Feb 15, 2019 5:41 pm

Great work!

Had some issues with parsing messages from one cAP ac where the messages suddenly dropped due to "Failed to parse timestamp" warning messages.

Seems it got solved by adding
TIME_FORMAT = %b/%d/%Y %H:%M:%S
to the props.conf file.

Please consider adding this to future releases.

/Thanks
 
frankcale
just joined
Posts: 4
Joined: Sat Nov 03, 2018 6:39 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 11:28 am

Hi, Can u pls help with displaying Vlan info
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 11:59 am

Not sure what you asks for.
A list of Vlan on the router?
Traffic going trough Vlan?
 
frankcale
just joined
Posts: 4
Joined: Sat Nov 03, 2018 6:39 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 4:59 pm

Hi, Can u include vlan traffic monitoring and if possible protocols like youtube, torrent, updates, etc
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 7:30 pm

Protocol are complicated to monitor due to https, near to impossible.
Vlan can be monitored used SNMP or you can use script and syslog to send data.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Mar 05, 2019 12:19 pm

Updated 1a to mention that you need an account at splunk.com to download software.
Account is free to create.
 
ithelp
just joined
Posts: 2
Joined: Sun Aug 16, 2015 9:41 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Mar 06, 2019 6:18 am

Hi, thanks for this magnificent explanation.
Can you give me on how to see the PPP and PPPOE information from the log?
I've already configure it on the rules tab, but nothing shows on any dashboard.
Thanks,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Mar 06, 2019 8:09 am

I do not have PPP nor PPPOE so I can not easily make log for it.

But if you could post 3-4 pages of logs that involves PPP and PPPOE output I could have look at it.
 
neutronlaser
Member
Member
Posts: 445
Joined: Thu Jan 18, 2018 5:18 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Mar 16, 2019 8:07 pm

Price is ridiculous.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Mar 16, 2019 9:40 pm

500MB/day for free is ridiculous much to pay.

But I do agree that if you pay retail price for Splunk and need eks 500GB/day, price is high.
 
User avatar
Halfeez92
Member Candidate
Member Candidate
Posts: 101
Joined: Tue Oct 30, 2012 12:58 pm
Location: 127.0.0.1
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 9:25 am

Hi how can I remove the MikroTik device list in the splunk dashboard view? I have multiple same devices showing up because I forgot to disable NAT and enable routing. Now it have 2 same devices with different IP
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 1:14 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.
 
User avatar
Halfeez92
Member Candidate
Member Candidate
Posts: 101
Joined: Tue Oct 30, 2012 12:58 pm
Location: 127.0.0.1
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 7:08 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.
Ok thanks for the help. Already delete the duplicate device.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Jun 10, 2019 5:49 pm

Updated section 2c regarding Log prefix.

NB Do not use more than 20 charters, or else it start to clip other part of the log
firewall,info MikroTik: 123456789012345678901234567890 : in:ether1-Wan ...
firewall,info MikroTik: 1234567890123456789012345 forwa: in:ether1-Wan ...
firewall,info MikroTik: 12345678901234567890123 forward: in:ether1-Wan...
firewall,info MikroTik: 12345678901234567890 forward: in:ether1-Wan ...
As you see here the chain word forward is eat'n up by the prefix.
MT is this a bug???
If not, set a warning in the gui :)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 13, 2019 1:54 pm

Updated section 2f)

Script updated to collect and show how many dynamic/static address lists entry there are.
Eks output
script,info MikroTik: script=address_lists list=rdp_stage2 dynamic=24 static=0
script,info MikroTik: script=address_lists list=rdp_stage1 dynamic=28 static=0
script,info MikroTik: script=address_lists list=ftp_stage2 dynamic=1 static=0
script,info MikroTik: script=address_lists list=ftp_stage1 dynamic=1 static=0
script,info MikroTik: script=address_lists list=black_list_rdp dynamic=42 static=0
script,info MikroTik: script=address_lists list=black_list_ftp dynamic=1 static=0
script,info MikroTik: script=address_lists list=Whitelist_IP dynamic=3 static=2
script,info MikroTik: script=address_lists list=Router dynamic=0 static=1
script,info MikroTik: script=address_lists list=IPSEC dynamic=1 static=0
script,info MikroTik: script=address_lists list=FW_Block_user_try_unkown_port dynamic=1089 static=0
script,info MikroTik: script=address_lists list=Clients dynamic=0 static=2
script,info MikroTik: script=address_lists list=Blocked dynamic=1 static=7
This will later be used in its own graph to see variation in the lists.

PS only one IP en the ssh black list black_list_ssh is due to that I do not use default port.

You can update script only and wait for new Mikrotik Splunk app to be updated later.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 20, 2019 9:59 am

Hello Jotne,

I want to upgrade my Splunk version 7.2 environment tot Splunk 7.3

Is the mikrotik app compatible with Splunk 7.3?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 20, 2019 1:43 pm

Yes, I do try to not use anything special in the APP so it should be compatible with all new version.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jun 21, 2019 9:29 pm

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
 
User avatar
pidde
just joined
Posts: 3
Joined: Fri Aug 24, 2012 5:22 pm
Location: Sweden
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Jun 23, 2019 2:59 am

Hi!

Must say you did a great work with this app!
Is it possible to add option82 to dhcpserver part?
And is it also possible decode the option82 from hex?
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jun 25, 2019 10:34 am

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
When I look at the current script under 2f I only see the "# Collect DHCP Pool information" part.

It seems the rest of the script is missing.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jun 25, 2019 1:09 pm

You are 100% correct. Copy past error.

Fixed.

PS It's getting closer to the release of v 2.7 of Splunk for MikroTik
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jun 28, 2019 2:10 pm

Script to get information on the router is upgraded to 2.6 section 2f

Simpler DHCP calculation.
Fixed comment so it start on the beginning of the line.
Fixed Script names
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 01, 2019 1:15 pm

Upgraded to 2.7

There are a lot of new changes to the app as listed below, so its a larger upgrade.
Simplest way to upgrade, if you have not made changes your self, remove (uninstall) previous version, install new version.
Please report any problems back to this thread, and I will try to fixed.

PS If you do upgrade, you also need to upgrade script in section 2f (fist post) on all router you like to get data from.
Just cut/past the script over the old one.

PS2 File is found under section 1g first post

Request to changes are also welcome :)

What new:
# 2.7 (01.07.2019)
# New view added "Address Lists Counters"
# Changes most view to use "Base Search"
# Changed "MikroTik DHCP request" to use stats and fixed host flaw
# Changed "MikroTik System Changes" to use 30 day and 4 hour span and maxspan in transaction
# Removed changes to "DHCP leases" in "MikroTik System Changes"
# Added search in dropdown for "MikroTik DNS Live usage"
# Added Time picker for "MikroTik Device List"
# Speeded up "MikroTik Remote Connection"
# Fixed wrong timestamp of packets logged
# Changed "MikroTik DHCP request" to use stats and fixed host flaw and maxspan in trnsaction
# Added search in dropdown for "MikroTik DNS Live usage" and added IP to client and change sorting
# Fixed "MikroTik DNS request" to use correct dropdown lists
# Fixed "MikroTik Firewall Rules" to use better searh, removed base level, added counters, long prefix
# Rewritten "MikroTik Live attack" to speed up and added more dropdown
# Fixed "MikroTik Resources" to give correct host number
# Changed "MikroTik System Changes" to use 30 day and 4 hour span, removed DHCP info
# Fixed "MikroTik Traffic" to use script= and some clean up
# Fixed "MikroTik uPnP" script name, added ip to dropdown
# Added to ">MikroTik Uptime" dropdown menu
# Fixed "MikroTik Volt/Temperature" sorting
# Fixed "MikroTik VPN Connection" faster search
# Fixed "MikroTik Web Proxy" sorting and some code clean up
# Changed "MikroTik Wifi strength" to use script tag and some clean up
# Added "dashboard.css" to set menu color global
# Fixed "props.conf" to better handel wrong prefixed and some other changes
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 03, 2019 5:36 am

I have been paying attention to this post, very powerful chart, but the cumbersome construction and the lack of relevant knowledge have been unsuccessful. I can only temporarily use the mrtg icon inside routeros to temporarily cope with it. I hope the poster can write the deployment manual from the perspective of the technology-poor. .
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 03, 2019 3:17 pm

Its written so that a user with some knowlege should be able to set it up.
You can start by telling me what your problem is, and we may be able to help you out.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 06, 2019 6:50 am

Reinstalled splunk on ubuntu18.04, is a virtual machine under esxi, the deployment is very simple and normal, according to the steps of the top post, but the splunk dashboard can not see the task data incoming. Very strange, what else do I need to pay attention to? Please forgive my English using Google Translate, I am from China
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 06, 2019 10:43 am

After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik 
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 08, 2019 12:57 pm

After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
According to what you said carefully, but still can not receive the data, I introduced the cdb1016 log file db format, can be displayed to splunk, indicating that splunk no problem, is the data input problem, I see ros is the log The output is udp514 port, but I only see tcp listening port settings in splunk's receiving settings. Is this the reason?
1.png
2.png
3.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 08, 2019 1:36 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 09, 2019 5:51 am

There is no local listening udp514, now there is data in, but click on the meter in the Mikrotik2.7 dashboard, most of them do not have any charts, how to add or customize the dashboard you need here, for example, I want The wan's real-time or past and downstream traffic in a certain period of time, as well as the system temperature, the number of online hosts, and so on. How to do it?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 09, 2019 8:29 am

514 UDP do need to be active
Do you run it on Linux?

If so, as Root, type:
netstat -opan | grep 514
You should see one line like this:
udp        0      0 0.0.0.0:514             0.0.0.0:*                           23557/splunkd        off (0.00/0/0)
if not UDP/514 is not running.

One the mikrotik, post the output of:
/system logging export
You should see some like:
# jul/09/2019 07:26:37 by RouterOS 6.43.16
# software id = E4B6-94N8
#
# model = RouterBOARD 750G r3
# serial number = xyz
/system logging action
set 3 remote=ip_your_syslog_server
/system logging
set 0 disabled=yes
add action=remote prefix=MikroTik topics=dhcp
add action=remote prefix=MikroTik topics=hotspot
add action=remote prefix=MikroTik topics=!debug
There should be IP for your server, and prefix for all action with MikroTik. If one letter is wrong in the prefix, it will fail. See capital M and T in the MikroTik.
Last edited by Jotne on Wed Nov 17, 2021 1:23 pm, edited 1 time in total.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 1:03 pm

It’s true that I set it wrong, Mikrotik changed to MikroTik, and it should be fine, then I will report it.
 
haaroons
just joined
Posts: 1
Joined: Wed Jul 10, 2019 11:15 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 1:32 pm

Hello Jotne,
I am new to this forum.

I have install MikroTik logs 2.7.

MikroTik DNS Live usage and MikroTik DNS Live request is not working. if i do search eventtype=dns_query No item found

Do advice how to fix this.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 11:43 pm

DNS information are coming from standard logs on the router.

What do you get if you go to search window and search with the following line:
sourcetype=mikrotik earliest=-24h latest=now() | stats count by module
I do get some like this:
module		count
dhcp		12764
dns		324512
firewall	1349
ipsec		7
script		91182
upnp		308
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 6:02 am

The data is coming, some of the tables are already filled, some still have no data, such as dns, it doesn't matter, I want to know how to monitor the flow table of an interface (wan), just like mirkrotik's built-in mrtg chart, every 5 minutes, 30 minutes and so on. . . As shown
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 8:05 am

That is why I need the output of the above command.
Some data are coming from the logg.
Some are comming from scripting

Log:
-------
dhcp,dhcp_static,dns,firewall,ipsec,upnp

script:
-------
IPSEC_failed,address_list,healt,pool,resource,sysinfo,traffic,uncounted,upnp

So I guess you have some log problems. Read section 2b carefully.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 11:42 am

Splunk is too powerful. If I have multiple ccr1016, how can I transfer data to the splunk server, how do I distinguish syslogs from different mikrotik routers?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 1:29 pm

All the view for MikroTik in Splunk has a host drop down. So if you have more than one router, just select the host you like to monitor.
There is one possible problem, if you have many routers with same IP that sends log to same Splunk.
That could be solved using unique ID for each router and some small change to the code.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 6:24 am

How can I write the interface tx-bits-per-second parameter to the log and then plot it in splunk.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 8:06 am

What command do you use on the router to see this data?
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 10:07 am

What command do you use on the router to see this data?
interface monitor-traffic ether1

Search forums see scripts with such calls
  "/interface monitor-traffic ether1 once do={
:put ($"tx-bits-per-second"/1000 /1000 )
}"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 16, 2019 8:12 pm

It can be done.
I do use IP accounting to see the traffic going trough the router.
This way are more generic and does work without any modification.
If you monitor one and one interface, this has to be adopted for each setup.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 5:03 am

{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX (($speedRX/1000)/1000);
:set $mbpsTX (($speedTX/1000)/1000);
:put "$iname RX:$mbpsRX Mbps TX:$mbpsTX Mbps";
}
}
I found the script for this post available, but after running it is all interfaces, I don't want all interfaces, only a few interfaces are needed, for example, I only need ether1, ether2, how to modify the script, and how can I get it? Let him display in the log, I use the splunk search call, and display it as 14.5Mbps instead of 14528. I hope to get everyone's help.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 9:29 am

Info
It seems that data you get from monitor are just moment blink of data going through the interface. So it will fly up and down for every time you run it. If it would be like cisco, average last 5 min, it would be perfect to rune every 5 min. Not sure if it are useful at as is.


If you have not renamed interface
:foreach interface in=[/interface find] do={
To
:foreach interface in=[/interface find where (name~"^ether1\$" || name~"^ether2\$") ] do={
or use regex
:foreach interface in=[/interface find where name~"^ether[12]\$" ] do={
Anchor ^ \$ are used to distinguish ether1 from ether11 etc.

Edit
You can use ID instead of name, so you can change from:
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
to
:set $monitor [/interface monitor-traffic $interface as-value once]
PS2, no need to declare variables, use them directly
do not divide data by 1000 two times, let splunk do that, so you do not loose any resolution
use equal sign for splunk to read data directly
you do not need semicolon behind each line ;

So final script could be some like this
:foreach interface in=[/interface find where name~"^ether[12]\$"] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 11:47 am

Your script, the regular expression method, no success without any output, it doesn't matter, my code is as follows
I want to know how to search for rich charts, and there are 14.8Mbps and 14833 display problems. This is not important. The important thing is how splunk draws charts.

mycode
{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "WAN-ether2 down RX=$mbpsRX Kbps";
:log info "WAN-ether2 up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"adsl-tx") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "adsl-tx down RX=$mbpsRX Kbps";
:log info "adsl-tx up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"bonding1") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "bonding1 down RX=$mbpsRX Kbps";
:log info "bonding1 up   TX=$mbpsTX Kbps"
}
}
After the schedule is displayed as follows
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 12:26 pm

Tested on other ccr1016 your script is successful, it should be the problem of the interface name, but it is important to draw the splunk graphics, I hope you can add to the new version.
3.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 1:38 pm

When you have multiple interface, use only one section, no a section for every interface

change
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
to
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
Test code that should output data to screen:
{
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:put "script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
}

PS, when testing cut and past on the cli, you need to wrape all script in brackets {} !!!

PS how often would you like to run the script? every 5 min. Do you know if monitor could show average 5 min data?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 1:58 pm

Try this

Add this to the Data_to_Splunk_using_Syslog script
# Get interface data (test)
# ----------------------------------
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1")(name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
Then in Splunk do this search for the last 4 hour.
sourcetype=mikrotik script=monitor| timechart avg(RX) as RX avg(TX) as TX by interface limit=10
May take some time to nice graphs.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 7:06 pm

Nice,
I have added the additional script entries and changed the inteface names to the names I use.
But............
The sourcetype entry in the search entry schould be "sourcetype=MikroTik" 8) 8)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 7:32 pm

In Splunk, search ignore case :)

Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 9:06 pm


Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.
The current "Mikrotik Traffic" overview is indeed a nice oveview.
But apart from knowing who is generating the traffic I am very interested in the amount of traffic that floats over each individual interface. And especially the WAN interface(s) and ISL interfaces. And when you see a bottleneck on one of your interfaces you can drill down to your traffic overview to identify the source of all that traffic.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 20, 2019 5:46 am

Jotne,Great, I did it according to your script, and the beautiful chart shows normal. I tried to add scripts to my multiple ccr and routerboards, so my interface has a lot of duplicate names, such as bonding1 and bridge1, how can I distinguish between them, or change the name for each interface.
4.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 20, 2019 5:59 am

Understand, add host=x.x.x.x in front of the search statement you gave to open my ccr and rb.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 20, 2019 7:10 am

Host=x.x.x.x Although this option is available, some devices have an internet connection that is a dynamic ip obtained by adsl dialing. So before the log warning, add an identity=xxxxx to distinguish the mikrotik device. After testing, it is feasible and runs very well.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 22, 2019 10:24 am

@ fengyuclub
Nice to see you are getting it to work.

@ All
Section 2c) Logging prefix has been updated with sample on how to name to logs.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 22, 2019 3:44 pm

Script in section 2f) updated to 2.9

It now support to get interface counters and you can also set modules true/false if you do not like to monitor one section.
If you do not have wifi/dhcp, you can just set them to false.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 24, 2019 8:23 am

Script in section 2f) updated to 3.0

Do now get CDP neighbors
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 25, 2019 12:46 pm

Splunk is really powerful, I see splunk have a lot of apps to install, in our China use wechat (similar to facebook, telegram) this social software, I saw this social software related app, WeChat Alert App for Splunk, I installed this App, sending test messages from wechat is successful, but I don't know much about splun's alert settings, set it many times, only a single success, can you help me?
5.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 25, 2019 1:44 pm

No need for extra app to send message. Sending email using a gmail account is easy and works well.

But there is a big issue.

If you have a free Splunk license, you do loose a lot of thing.
* Monitor and Alerting (needed for sending alerts)
* 500MB pr day maximum
* Cluster
* Universal Forwarder
* HA
* Distributed Search
* Perfomance Acceleration
* Access controll (only on user)
* LDAP
+++

This is why I have not included any Alerting in the project.

There is a workaround. You can setup an batch job that runs search from command line and do stuff from it. (I have not tested it)
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 27, 2019 4:56 am

I have Splunk Enterprise license, gmail alert can't be real-time, mobile mail client can't update mail in real time, there is a delay of about 10 minutes, so I choose wechat alert.I received some wechat alert, but some of them use the search to save as an alert, I can't receive a wechat alert, I don't know where the problem is.
6.jpg
7.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 27, 2019 8:30 am

Splunk do handle real time alerts (or close to)
https://docs.splunk.com/Documentation/S ... TimeAlerts
It should not depend of type of action you are using, starting a program, sending sms, email, wechat etc. Alerts should go out.
But you should not use to many alerts, since it will use more CPU to handle them.

Not sure what your problem is.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Aug 02, 2019 10:20 am

Updated script to 3.1

Fixed CDP, since some devices sends long version with new lines breaking up the log lines. (Cisco)

PS still have problem that line is cut in Splunk. Not sure if its MT not sending whole line, or Splunk that cuts the lines.
I do only get 278 characters.
 
User avatar
antispam
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Mon Apr 11, 2005 5:57 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Aug 06, 2019 3:34 pm

Using 2.7, it's mentioned that the "defconf: drop all not coming from LAN" rule should have the prefix 'FI_D_port-test'. When I set that, the Live Attack dashboard doesn't populate as it appears from the source in the dashboard it is searching for 'FW_Drop_all_from_WAN'. When changing this to 'FI_D_port-test' in the Live Attack dashboard source, it works. Is the FW_Drop_all_from_WAN still required?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Aug 06, 2019 4:24 pm

The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Aug 07, 2019 9:39 am

Recently, one of my ccr is a bit problematic. I can only recover from the time when the device is powered off and restarted. The top of the log in winbox can see red like "system, error System rebooted because of kernel failure" or "Out of memory condition was detected", but I can't see it in splunk search. These log messages, how can I output all the log information to splunk for easy query.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Aug 07, 2019 1:12 pm

A search like this should give all message:
sourcetype=mikrotik module=system
IF not try this:
sourcetype=mikrotik
Or at last just this
*
 
User avatar
antispam
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Mon Apr 11, 2005 5:57 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 6:04 am

The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik
Thanks for the prompt reply, that was exactly what I did to get it fixed - keep up the great work!
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 2:28 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.

One thing to be careful of if you're setting this up in an existing Splunk environment - unless you're really familiar with how things are setup, don't enable Splunk's UDP/514 input without first checking that syslog isn't already being received by something like syslog-ng or rsyslog. You could wind up with data loss or have events put into the wrong index or sourcetype.

It's also not best practices to run Splunk as root. For home use I guess you can get away with it, but for any production Splunk environment you will want to have Splunk running as a restricted user (user = splunk and group = splunk is commonly used).

When you install Splunk, you can set it to autostart on boot and also set the user if you want.
[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

For receiving syslog - I'm more familiar with syslog-ng, but I also see rsyslog being used successfully. Either one of these will work well for you.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 5:32 pm

This is already mention in section 1b)

If you install Ubuntu, (i think from 16.x), rsyslog is installed as default. But its not listening on port 514/UPD as default and you need to edit the config and restart syslog to get it running. So it should normally not be any conflict.

But in production environment I do also recommend running Splunk as a non root user, then use rsyslog to listen on 514/UDP. Then make Splunk index rsyslogs config.

If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 8:31 pm


If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.
I think that syslog-ng has an option that can be used to do this.
keep-timestamp()
Description: Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
https://www.syslog-ng.com/technical-doc ... -timestamp
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 8:50 pm

I also wanted to note that I'm not advocating that anyone switch from rsyslog or whatever they're currently using to syslog-ng unless they have good reason to do so.

I don't even really have an opinion on how they compare. I've been working with syslog-ng a bit so that's what I'm familiar with. I'm not trying to start a flame war over which to use. :-)
 
Spotegg
just joined
Posts: 4
Joined: Thu Aug 15, 2019 12:28 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Aug 15, 2019 1:39 am

Hello! I have installed Splunk with Mikrotik module. Thanks, it's great!
Is there a way to organize monitoring of Internet connection on the router. For example, there is an Internet channel on ether1, and you need to somehow download data to Splunk about when the Internet crashed on the router. Maybe there is already such a script?
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Aug 15, 2019 4:10 pm


Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Fri Aug 16, 2019 12:21 pm


Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?

Resolved! :D
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 5:35 pm

I'm having a strange problem both on my synology with docker and my windows pc also, I configured everything as described, I got many logs from my router but after a while it stops reading while counters are still increasing.
Checking via tcpdump, logs are arriving to the server but is like they are not processed.
Immagine.jpg
Someone could help me? Maybe I got something wrong, I cannot image the server is flooded by a single router.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 5:53 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 7:19 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing
Immagine.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 10:03 pm

What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 10:34 pm

What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.
Yes the script is installed, I had the 30 min windows search and no data are showed even when the script starts. The stranger thing is that I have the same problem running Splunk on two different system
Immagine.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sun Aug 18, 2019 8:39 pm

Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sun Aug 18, 2019 11:12 pm

Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?
Thanks for all your help and your time, it's very strange as you said... I opened syslog on port 5014 udp, i also checked the file props.conf on MikroTik app match the port.
I also tried as you said with echo "hello" and it's working.

I think there is some problem with timestamp because then it stops collect i got some error in internal index:
08-18-2019 19:09:43.054 +0200 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Sun Aug 18 01:28:00 2019). Context: source=udp:5014|host=192.168.1.1|syslog|
I checked around on google but I didn't find a solution
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Aug 19, 2019 8:21 am

You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Aug 19, 2019 9:30 am

You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.
It's a real riddle! :D
The server is running as a root on my last test with ubuntu but nothing is changed. Now I just changed the port from 5014 to default 514 (I used 5014 because on my synology is already kept) and seems to be working. I will keep it running to see what happen!

Thanks for all your support and time spent! ;)
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Aug 20, 2019 12:19 am

Finally it's working!
Just uncommented the option: MAX_TIMESTAMP_LOOKAHEAD = 23 "inside /opt/splunk/etc/apps/MikroTik/default/props.conf" :D
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Aug 20, 2019 8:27 am

For me it then sounds like you have a time problem. Its important that all clock is synced by using NTP.
Look at time on your router and on Splunk server. It should be within the same second.

Or date stamp is in the wrong format. Not sure how you can get this, since MT sends it in correct format.
You sends UDP directly to Splunk, not using rsyslog or syslog-ng etc?
Something adding data to your UDP packets?

From the manual
MAX_TIMESTAMP_LOOKAHEAD = <integer>
* Specifies how far (in characters) into an event Splunk should look for a timestamp.
* This constraint to timestamp extraction is applied from the point of the TIME_PREFIX-set location.
* For example, if TIME_PREFIX positions a location 11 characters into the event, and MAX_TIMESTAMP_LOOKAHEAD is set to 10, timestamp extraction will be constrained to characters 11 through 20.
So this sets where to look for the time stamp.
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 24, 2019 5:00 pm

After I posted this reply I realized that I goofed.

I didn't read the post I replied to very well. I looked at the event you posted and just looked at the timestamp at the beginning of it and completely missed that it was a Splunk internal log.

So you can probably ignore what I had posted shown below about how to handle the timestamp in your logs. It does show how to setup timestamp detection, but for the wrong logs. :-)

What I originally posted is below:

Finally it's working!
Just uncommented the option: MAX_TIMESTAMP_LOOKAHEAD = 23 "inside /opt/splunk/etc/apps/MikroTik/default/props.conf" :D

I think you could probably change it to MAX_TIMESTAMP_LOOKAHEAD = 29

The timestamp in your events is "08-18-2019 19:09:43.054 +0200" and Splunk needs to know that all of that is part of the timestamp in order to parse things properly. If you use 23 for the lookahead, Splunk may not catch the timezone.

So maybe:

MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_PREFIX = ^
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%3N %z
Last edited by stuartkoh on Sat Aug 24, 2019 5:29 pm, edited 3 times in total.
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 24, 2019 5:05 pm

BTW, Packt is giving away a free copy of a decent book on Splunk today. https://www.packtpub.com/free-learning

Implementing Splunk 7 - Third Edition

James D. Miller

Mar 2018

576 pages

What will you learn

Enrich machine-generated data and transform it into useful, meaningful insights
Perform search operations and configurations, build dashboards, and manage logs
Extend Splunk services with scripts and advanced configurations to process optimal results

You have to sign up for an account with them, but I've had one for several years and they haven't been obnoxious about sending me too much e-mail or anything. The offer expires later today (in about 10 hours from when I'm posting this) so I hope people will see this in time to take advantage of this.

My only connection with Packt is as a customer. I've bought some of their books and gotten some good free ones from them too.
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 24, 2019 5:24 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing

Sometimes Splunk will mistakenly identify the timestamp of an event as coming from the future because it misinterprets the timezone (or the sending device has the incorrect time set and actually is telling Splunk the events are from the future.

So it can be useful to run a search with "All time" selected in the time picker. If there are events "from the future" they will show up this way. You can also run an "All time (real time)" search and see events as they come in and see what the timestamps are. Just be careful with real-time searches on Splunk because they use up a lot of resources. So I only use them when I really need to, and I stop them as soon as I have what I want.

(There are some times when you need to have a real-time search running all the time for very time-critical alerting, but it's normally better for performance to change a real-time search into a scheduled one running every 5 minutes or whatever. Each real-time search basically ties up a CPU core, so if you have several real-time searches running at once you can see poor overall performance on your search heads.)

I don't know if you're getting events "from the future" or not, but it might be worth checking.
 
stuartkoh
newbie
Posts: 30
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 24, 2019 5:29 pm

For me it then sounds like you have a time problem. Its important that all clock is synced by using NTP.
Look at time on your router and on Splunk server. It should be within the same second.

Yes! What Jotne said! :-)

It is extremely important to make sure the clocks on your devices and your Splunk servers are all in sync. Having time properly synced is very important for logging in general, but especially so for things like Splunk where you'll want to correlate events from different devices.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 24, 2019 9:28 pm

I think you could probably change it to MAX_TIMESTAMP_LOOKAHEAD = 29

The timestamp in your events is "08-18-2019 19:09:43.054 +0200" and Splunk needs to know that all of that is part of the timestamp in order to parse things properly. If you use 23 for the lookahead, Splunk may not catch the timezone.

So maybe:

MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_PREFIX = ^
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%3N %z
If you see props.conf in the MikroTik app, the MAX_TIMESTAMP_LOOKAHEAD = 23 line is not active due to # in front of it
So you should only change this if it does not work.
Still not sure why you need to sett it. Maybe you have other settings that influence on how Splunk handle MikroTik data.
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sun Aug 25, 2019 11:00 am

Thanks everyone for the answers, It's seems to be working since 7 days now so I prefer to not changing anything! :D

I just uncommented the line MAX_TIMESTAMP_LOOKAHEAD = 23 and the time zone is also working properly, to be sure I also set the NTP to all devices and nothing else.

If I will have some time to spend, I will try to manage another install of splunk to try the LOOKAHEAD of 29
 
jacauc
just joined
Posts: 17
Joined: Sun Jan 30, 2011 3:49 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Aug 29, 2019 4:52 pm

I created a dropdown to select rules on the Live Attack View. All my mikrotik prefixes start with either Allow or Drop - Such as DropPings, AllowDNS etc.
Now I can quickly see the entire overview of my rules.

The updated dashboard code is below:
<form theme="dark">
  <label>MikroTik Live attack</label>
  <description>Shows form where ip comes from that tries to access your router on ports that are not opened using rule "FW_Drop_all_from_WAN"</description>
  <!--Version 
  2.4.0
  2.5.1 Change to "coalesce"
  2.7.1 Change to base searc
  2.7.2 Revritten to speed up and added more dropdown
  ######################################################
  #
  # Mikrotik Add-On for Splunk
  #
  # Copyright (C) 2019 Jotne
  # All Rights Reserved
  #
  # v2.7
  #
  ######################################################
  -->
  <search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule="*"
      | table _time host dest_port src_ip protocol rule
      | iplocation src_ip
      | replace "" with "Unknown" in City,Country
      | fields - Region
      | search 
          host="$Host$"
          protocol="$protocol$"
          dest_port="$port$"
          Country="$Country$"
          rule="$Rule$"
    </query>
  </search>
  <fieldset submitButton="false" autoRun="true">
    <input type="time">
      <label>Time span</label>
      <default>
        <earliest>rt-5m</earliest>
        <latest>rt</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Host</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="protocol">
      <label>Protocol</label>
      <search base="base_search">
        <query>
          | eval data=protocol
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="port">
      <label>Port</label>
      <search base="base_search">
        <query>
          | eval data=dest_port
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Country">
      <label>Country</label>
      <search base="base_search">
        <query>
          | eval data=Country
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Rule">
      <label>Rule</label>
      <search base="base_search">
        <query>
          | eval data=rule
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <choice value="Allow*">AnyAllow</choice>
      <choice value="Drop*">AnyDrop</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Drop*</default>
      <initialValue>Drop*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <map>
        <search base="base_search">
          <query>
            | eval info=src_ip."-".City."-".Country."-".protocol."/".dest_port
            | geostats globallimit=0 count by info
          </query>
        </search>
        <option name="height">600</option>
        <option name="refresh.display">progressbar</option>
      </map>
    </panel>
  </row>
</form>
2019-08-29_15-50-50.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 31, 2019 10:45 am

Interesting.

I may implement this in ver 2.8 :)
 
jacauc
just joined
Posts: 17
Joined: Sun Jan 30, 2011 3:49 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 31, 2019 11:01 am

Interesting.

I may implement this in ver 2.8 :)
Please do!
Do you have a beta of 2.8 yet, or is it imminent for release?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 31, 2019 10:29 pm

I am always working with some changes, and release new version when I have spare time to do it :)

Here is the working list for 2.8
# 2.8 (xx.xx.2019)
# Added interface changes
# Updated script to 2.7, added uncounted traffic and fastrack test, fixed when missing temperature
# Updated script to 2.8, fixed where system healt does not show anything (x86)
# Updated script to 2.9, get interface counters and you can also set modules true/false
# Updated script to 3.0, get CDP neigbhours
# Added new view "MikroTik Neighbor"
# Added uncounted packages to "MikroTik Traffic"
# Added board_name to "MikroTik Device List"
# Added Mikrotik interface Traffic"
# Fixed l2tp user extraction
# Fixed VPN when user info is missing
# Added access rule dropdown to "Mikrotik Live attack"
 
keattikun
just joined
Posts: 1
Joined: Fri Oct 18, 2019 1:53 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Oct 18, 2019 3:29 pm

Hi Jotne
I'm try to install Mikrotik2.7 app on splunk. It not working for me, and I able to search log on splunk that send from MK but in app show "No results found" as capture
Image

Please help me to investigate
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Oct 18, 2019 10:17 pm

You are 100% sure you have 2b correctly?
Click on Search
In search line add only a * (star) and search last 24 hour. Do you see any data?
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Nov 13, 2019 5:36 pm

Hi Jo,
Great set of dashboards! All working just fine.
In order to customize them further, when can I take out some complete dash-boards ? (pressing "Dashboards" in Splunk gives you the overview)
Don't need any Wireless or Upnp stuff and I would like to get it off the list.
Under the actions-button there is no such as a "Delete" choice.


EDIT : Ok, figured it out after untarring the app that I needed to edit some XML.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Nov 15, 2019 7:03 pm

Jotne,Great, I did it according to your script, and the beautiful chart shows normal. I tried to add scripts to my multiple ccr and routerboards, so my interface has a lot of duplicate names, such as bonding1 and bridge1, how can I distinguish between them, or change the name for each interface.
4.png
Hi can you give some hints on how to do that in Splunk (I'm running 8.0.0 FREE) ? Whatever graph I make, it seems to have a rolling time-line on the X-axis every 10-15 seconds and each 5 minutes the TX and RX values are plotted but not like this. I need to the X-axis to be fixed, 5-minutes increments to match the input coming from the script.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Nov 15, 2019 7:38 pm

You need to have unique naming or else data will be joined.
What you can do is to use some values and join them together.
Eks using host-name port-number interface name etc and create a new join name.

| eval newifname=host."-".ifname

Without seeing the real problem its not easy to give a solution.
 
blinderix
newbie
Posts: 48
Joined: Wed Apr 08, 2009 8:55 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Dec 02, 2019 10:01 pm

Jotne, Greetings for the good job!
I'm new to Splunk and keep reading.
I'm curious how to make the drop down list with host IP's to show the identity of the router instead the IP address.
Is it possible with some kind of lookup or token?

Regards,
 
blinderix
newbie
Posts: 48
Joined: Wed Apr 08, 2009 8:55 pm

Re: Using Splunk to analyse MikroTik logs 2.2 (Graphing everything)

Thu Dec 05, 2019 7:08 pm

No idea why it stops working.
I have run Splunk for years without it stopping by it self, always using Ubuntu Server.
Maybe there are some other stuff/software on you Ubuntu that kills it.

Finally there is a problem with timestamp, found on splunkd.log
08-01-2018 15:52:07.610 +0300 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Wed Aug 1 02:35:00 2018). Context: source=udp:514|host=192.168.1.1|syslog|

And it stops logging.

This is an example of what mikrotik sends to me, on another syslog server
192.168.1.1 Jul 12 00:17:05 firewall,info DROP INPUT input in:pppoe-WAN out:(unknown 0), proto TCP (SYN), 79.129.108.120:41236->79.129.36.201:7547, len 44


Any work around ?
No matter the question is old, I had the same problem. The logs from some routers stopped and had to restart the splunk each time.
I had the same timestamp error in the log file.
Yesterday I changed in props.conf : DATETIME_CONFIG = CURRENT and till now, almost 24h, everything runs smoothly.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Sat Dec 07, 2019 10:56 am

2.8 Finally released.

# 2.8 (07.12.2019)
# Added interface changes
# Updated script to 2.7, added uncounted traffic and fastrack test, fixed when missing temperature
# Updated script to 2.8, fixed where system healt does not show anything (x86)
# Updated script to 2.9, get interface counters and you can also set modules true/false
# Updated script to 3.0, get CDP neigbhours
# Added new view "MikroTik Neighbor"
# Added uncounted packages to "MikroTik Traffic"
# Added board_name to "MikroTik Device List"
# Added Mikrotik interface Traffic"
# Fixed l2tp user extraction
# Fixed VPN when user info is missing
# Added access rule dropdown to "Live attack"
# Added some color and more field to "DHCP Request"
# Fixed Span and removed time scal "System Changes"
# Fixed missing mac in "Admin Conection"
# Added more lines and color in "DHCP request"
# Added output color in "Firewall Rules"
# Added rule menu to "Live attack"
# Added "DATETIME_CONFIG = CURRENT" to props.conf
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Sat Dec 21, 2019 4:51 pm

Tested with latest Splunk 8.0.1 and it works fine.
 
jo2jo
Forum Guru
Forum Guru
Posts: 1003
Joined: Fri May 26, 2006 1:25 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Sun Dec 22, 2019 10:19 am

wow, thanks so much for all the time and effort you have put into this! it really is an excellent splunk app, thank you!

I do have your splunk app working on both of our splunk servers, when i add the "MikroTik" prefix to remote logs for a few test mikrotiks.
(our splunk servers are not the free version, but are licensed)

However we have been using splunk for a few years, and have many (over 200+) mikrotiks already sending lots of data into splunk via syslog, so im trying to avoid having to winbox into each of them and add a duplicate remote rule (with your "MikroTik" prefix).

Our mikrotiks are all configured to send logs into splunk like this:
(config from the splunk side)
udp 8002 data_input goes into splunk index "mikrotikFirewall"
udp 8004 data_input goes into splunk index "mikrotikOther"

Each of our many mikrotiks does have a prefix set (ie /system logging add action=remoteSplunk prefix=+LOCATION-NAME-FW- topics=firewall ) - (note the "+" before location-name) we ofcourse uses the prefixes to help our own manual searching within splunk

my question: Would you please point me in the right direction of what i would need to modify so that your app/views searches splunk entries with our prefix, as opposed to your "MikroTik"

Things i have tried / looked into:
\Splunk\etc\apps\MikroTik\default\props.conf - Changing the 1st line, [source::udp:514] to [source::udp:8002] - (made no difference, so i changed it back)

\Splunk\etc\apps\MikroTik\default\transforms.conf - Changing the [force_mikrotik]... REGEX = \s\+[a-zA-Z0-9\-\:]+\s - ( this regex, as all of our log prefixes do start with "+" , this also made no difference)

In splunk web gui: data Inputs->UDP -> udp:8002 , setting the source type to "mikrotik" (from our initial source_type setting of "syslog") - also did not get desired result.

(keep in mind that i did try these things together, and separately, and was restarting splunk server after any changes).

(i would have no problem setting up a 3rd, new free license splunk just for using your app, however our mikrotik splunk intake alone is much more than 500mb/day currently)

thanks again!
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Wed Dec 25, 2019 5:19 pm

Since I do use the tag to add sourcetype to the log entry, I think its there you should start.
Change all dashboard from using: sourcetype=mikrotik to use your index like this: index=mikrotikOther or index=mikrotikFirewall
 
kenkilaw
just joined
Posts: 2
Joined: Wed Feb 24, 2016 10:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Sat Mar 07, 2020 8:10 am

Good job !! I'll test it, thanks for sharing Jotne !!
 
silverblade
just joined
Posts: 4
Joined: Sat Nov 18, 2017 11:20 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Thu Mar 12, 2020 6:53 pm

Evening, everybody. One question, use splunk 8 and latest version of the app. I would like to have the results with the name of mikrotik instead of ip or dns. Can you explain please? Thank you
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Thu Mar 12, 2020 7:07 pm

Not sure what you mean. Can you give some example?
 
silverblade
just joined
Posts: 4
Joined: Sat Nov 18, 2017 11:20 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Fri Mar 13, 2020 11:38 am

Not sure what you mean. Can you give some example?
So, I always set the identity in my mikrotik but in splunk I see the ip or dns address. I'd like to see the identity on the menu or else I don't know who it is directly. If it's possible... Thank you.

Image
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Sat Mar 14, 2020 12:25 am

This is how Syslog works.
It sends log data with standard message from the MikroTik Router like this:
system,info,account MikroTik: user xxx logged out from 10.10.10.32 via winbox
system,info MikroTik: dhcp lease changed
system,info MikroTik: dhcp lease changed
system,info MikroTik: static dns entry changed by xxx
system,info MikroTik: static dns entry changed by xxx
system,info MikroTik: device changed by xxx
system,info MikroTik: device changed by xxx
It does not contain any identifier only IP.
I see three ways this can be solved.
1. You add DNS record for your IP in your local DNS server.
2. Change the "MikroTik" tag to some like "MikroTik Device=Router58". I have not tested it, so not sure what would break :)
3. Use the data from "MikroTik device list" in a subsearch to get the device name.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Tue Mar 17, 2020 10:29 am


So, I always set the identity in my mikrotik but in splunk I see the ip or dns address. I'd like to see the identity on the menu or else I don't know who it is directly. If it's possible... Thank you.
I have a solution for this in v2.9.
It reads the "MikroTik Device List" and store it inn to a KV store file every night. Then this data i later used in other view.
 
valeex87
just joined
Posts: 4
Joined: Wed Apr 01, 2020 4:11 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Thu Apr 02, 2020 1:49 pm

Hi, how can i make a script on mikrotik for check user that connects/disconnect in ovpn?
On splunk in "MikroTik VPN Connection" everything is clear and not populated, ovpn works correctly but on splunk i cant find whats my ip and the user who connected.

Thanks
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Thu Apr 02, 2020 6:48 pm

Hi
I have not used ovpn, so have not looked at it. If it logs other stuff, you you then send me a copy of the logs when a user logs inn and out using ovpn.

Do a search like this
sourcetype=mikrotik
or copy the logs from Mikrotik directly
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Wed Apr 08, 2020 1:57 pm

Script updated to version 3.3

Added script info and NTP status.
Backward compatible so you can just overwrite previous version.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.8 (Graphing everything)

Mon Apr 13, 2020 11:23 am

Script updated to 3.4

Gives more correct transfer statistics pr interface.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Tue Apr 14, 2020 1:34 pm

Splunk for MikroTik is updated to 2.9

# 2.9 (14.04.2020)
# Added KVstore DB to handel device info (identity)
# Fixed "MikroTik DNS Live usage" showing wrong count
# Fixed "MikroTik DHCP request" duplicate entity and cleaned up table
# Renamed "Mikrotik Admin connection" to "MikroTik Admin user login"
# Removed "_time" for X-Axes in dashboards
# Added NTP warning to "MikroTik System Changes"
# Added auto lookup of host to identity
# Added option to select CPU/Disk/Memory in "MikroTik Resources"
# Rearanged the menu layout
# Moved various test to the "device list" page and set the page as default


Mayor change is the new Menu layout and the KVstore. The DB makes the devices show up with their identity name and not just the IP.
This version also needs the latest version of the script (ver 3.4+) installed.

To upgrade, just download the file in the first post (section 1g), unrar and install. You can expand the tar ball and copy the files manually.
Script should be updated on the routers as well.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed Apr 15, 2020 8:55 pm

Script updated to 3.5

More compatible with Router OS v7.0 that has a problem with some function that does not work like:
/ip accounting uncounted {
	:log info message=("script=uncounted,bytes=".[get bytes].",packets=".[get packets])
And
:local model ([/system routerboard get model])
:local serial ([/system routerboard get serial-number])
This may be due to v.7 does run on a X86 VM. 6.47beta on VM does not fail, so its a but in v7.
 
jesuraja
just joined
Posts: 10
Joined: Sun Jul 23, 2017 11:06 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Sun May 10, 2020 2:42 pm

Interface Traffic Graph not sync.It's nothing. How can i troubleshoot to resolve
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Mon May 11, 2020 8:45 am

Do you see any data in other graphs?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Mon May 11, 2020 3:34 pm

An serious error found in the script in 2f. If NTP module is missing, the whole script fails. Fixed in new version 3.6.
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 3:16 pm

Hello!

I've setup as described in your tutorial.
But unfortunately there is no data available in Splunk.
After detailed investigations I have found, that there is no prefix "MikroTik" in log messages, even I defined in logging rules:
loggingrules.png
Rebooting the Routerboard did not help as well.
Splunk is completely empty...

Do you have any ideas?

/edit: Sorry, my fault. I was a little bit blind...
/solved
You do not have the required permissions to view the files attached to this post.
Last edited by verpeilo on Wed May 13, 2020 3:47 pm, edited 3 times in total.
 
User avatar
oaas
just joined
Posts: 4
Joined: Sun Feb 10, 2019 7:15 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 3:38 pm

There seems to be a kind a catch 22" situation with the "new" KVstore DB.

The KVstore fails to initially poplate as field extraction by "device lookup updater" is failing to auto extract the fields "identity, serial, model, board_name, version" while the lookup "automatic_device_lookup" is enabled. This results in missing information for "identity, serial, model, board_name, version" in the main "Device List" view (and other places).

After I disabled the lookup "automatic_device_lookup" the "device lookup updater" executed successfully populating the KVstore DB. In the end I enabled the "automatic_device_lookup", and view functionality seems to be working as expected.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 3:48 pm

You are 100% correct.
I did fix this by using coalesce
| eval identity=coalesce(identity,host)
Version 3.0 will be out soon with this fix and som other stuff as well.
PPPoE logging
IPv6 IP support in firewall
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 3:48 pm

There seems to be a kind a catch 22" situation with the "new" KVstore DB.

The KVstore fails to initially poplate as field extraction by "device lookup updater" is failing to auto extract the fields "identity, serial, model, board_name, version" while the lookup "automatic_device_lookup" is enabled. This results in missing information for "identity, serial, model, board_name, version" in the main "Device List" view (and other places).

After I disabled the lookup "automatic_device_lookup" the "device lookup updater" executed successfully populating the KVstore DB. In the end I enabled the "automatic_device_lookup", and view functionality seems to be working as expected.
Same here.
Where do you disable it?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 3:58 pm

Can you try to run i manually.
Splunk-app:Mikrotik->Reports->Click on name "device lookup updater"
Do you get any information? (Try it twice)
If not, what do you get
Splunk-app:Mikrotik->Reports-> "device lookup updater" Open in Search
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 4:11 pm

I only get the host column filled - everything else is empty
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 4:16 pm

What about this search?
sourcetype=mikrotik
            module=script
            script=sysinfo
| dedup host
| rex "version=\"(?<version>[^\"]*)\" board-name=\"(?<board_name>[^\"]*)\" model=\"(?<model>[^\"]*)\" serial=(?<serial>\S*) identity=\"(?<identity>[^\"]*)\"" 
| table  host identity serial model board_name version
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 4:52 pm

This works great!
 
User avatar
oaas
just joined
Posts: 4
Joined: Sun Feb 10, 2019 7:15 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 6:18 pm

May I also suggest you include the following in future release:
MAX_TIMESTAMP_LOOKAHEAD = 23

I see others are also refering to issues with this earlier as well, and I'm having the exact same issue with one of my devices. Unless I add this the device (cAP ac) disappears from syslog parsing after a few hours.
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 6:53 pm

Is it "normal" that Splunk is quite slow?
I have it running since 4-5 hours and the Traffic Accounting Dashboard takes 2 minutes until some results are shown...
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 7:43 pm

Is it "normal" that Splunk is quite slow?
I have it running since 4-5 hours and the Traffic Accounting Dashboard takes 2 minutes until some results are shown...
No that seems not normal...
The "Traffic" dashboard show here in about 2 seconds when selecting "Time Range = last 4 hours". I run Splunk Enterprise 8.0.1 (FREE Edition, so limited at 500MB/day indexing) on a VM on my Synology NAS, gave it 4vCPU & 2GB RAM
What are you running Splunk on ?

If you go to "Settings" on the top and then select "Monitoring Console" what stats do you see ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Wed May 13, 2020 9:01 pm

MAX_TIMESTAMP_LOOKAHEAD = 23
Added in the upcoming 3.0
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 9:30 am

Is it "normal" that Splunk is quite slow?
I have it running since 4-5 hours and the Traffic Accounting Dashboard takes 2 minutes until some results are shown...
No that seems not normal...
The "Traffic" dashboard show here in about 2 seconds when selecting "Time Range = last 4 hours". I run Splunk Enterprise 8.0.1 (FREE Edition, so limited at 500MB/day indexing) on a VM on my Synology NAS, gave it 4vCPU & 2GB RAM
What are you running Splunk on ?

If you go to "Settings" on the top and then select "Monitoring Console" what stats do you see ?
I have Splunk Enterprise 8.0.3 FREE Edition as well.

I run it on a Windows Server 2019 Standard.
4 Intel CPUs, 16GB RAM.

I can install it was well on a Linux System to compare the performance, but I don't think that the architecture has an impact like this.
stats_splunk.png
You do not have the required permissions to view the files attached to this post.
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 10:19 am

Update regarding the performance:

I installed Splunk on the same Windows Server, but in a Hyper-V Ubuntu 18.04.4 environment with only 2 CPUs and 2 GB of RAM.
==> Now it is very fast.

Seems that Splunk in Windows is slow... Hmm.

However, I am happy with this Linux VM.

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 10:42 am

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.
I do recommend using Linux for Splunk. Its created for Linux, later exported to work in Windows.
My install are usually done on Ubuntu server.

It would be possible to read the comment from an address and store and use it in Splunk, but not sure how this will work with large access lists.
My router blocks all that tries any non open port for 24 hours to access any port on my router. This creates an access list that can be between 2000 to 15000 addresses. So sending this every 5 minutes would be a big chunk.
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 11:55 am

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.
I do recommend using Linux for Splunk. Its created for Linux, later exported to work in Windows.
My install are usually done on Ubuntu server.

It would be possible to read the comment from an address and store and use it in Splunk, but not sure how this will work with large access lists.
My router blocks all that tries any non open port for 24 hours to access any port on my router. This creates an access list that can be between 2000 to 15000 addresses. So sending this every 5 minutes would be a big chunk.
Maybe you can configure it customizable if the comments should be read or not...
It would be very helpful.

/edit: Maybe you can build a function which determines the comment in DHCP leases
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 3:52 pm

If you do use Splunk as a non root (recomended) user, you need an external Syslog server.
This setup needs Splunk for Mikrotik v3.0 to read field correctly. (out soon)

This is how to set it up using Ubuntu server. Should work on most version.

rsyslog comes default with Ubuntu so no need to install any extra software.


PS do not modify these file to use other location. If you do so you will need to modify udp.conf rsyslog and input.conf splunk for every upgrade.

Copy these two files to /etc/rsyslog.d/

udp.conf (sets up rsylog to accept Sylog on udp/514)
# rsyslog.d/udp.conf
#
# This receives UDP syslog on port 514 and stores it in a reliable format
# in /data/syslog/udp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imudp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="udp_split_filename" type="list") {
  constant(value="/data/syslog/udp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="udp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="udp_split_filename"
  )
}

# setting
input(type="imudp" port="514" ruleset="udp_split")

tcp.conf (sets up rsylog to accept Sylog on tcp/1514)
# rsyslog.d/tcp.conf
#
# This receives TCP syslog on port 1514 and stores it in a reliable format
# in /data/syslog/tcp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imtcp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="tcp_split_filename" type="list") {
  constant(value="/data/syslog/tcp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="tcp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="tcp_split_filename"
  )
}

# settings
input(type="imtcp" port="1514" ruleset="tcp_split")

Create the following folders
mkdir /data
mkdir /data/syslog
mkdir /data/syslog/tcp
mkdir /data/syslog/udp
Change folder rights to syslog and restart rsyslog
chown -R syslog:syslog /data/syslog
service rsyslog restart
run ss or netstat as root user to see that rsylog is running
ss -tunlp | grep syslog
udp   UNCONN 0      0                                0.0.0.0:514        0.0.0.0:*     users:(("rsyslogd",pid=5532,fd=8))
udp   UNCONN 0      0                                   [::]:514           [::]:*     users:(("rsyslogd",pid=5532,fd=9))
tcp   LISTEN 0      25                               0.0.0.0:1514       0.0.0.0:*     users:(("rsyslogd",pid=5532,fd=6))
tcp   LISTEN 0      25                                  [::]:1514          [::]:*     users:(("rsyslogd",pid=5532,fd=7))
netstat -tunlp | grep rsyslog
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      1459/rsyslogd
tcp6       0      0 :::1514                 :::*                    LISTEN      1459/rsyslogd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1459/rsyslogd
udp6       0      0 :::514                  :::*                                1459/rsyslogd
To make Splunk reads the data, this input.conf file is needed %SplunkHome%/etc/apps/MikroTik/default
It is not included in current version 2.9, but will be included in 3.0. So you need to create it and restart Splunk.
[monitor:///data/syslog/udp/.../*.log]
sourcetype = mikrotik
host_segment=4

[monitor:///data/syslog/tcp/.../*.log]
sourcetype = mikrotik
host_segment=4
Test your server:
echo '<14>sourcehost message text' | nc -v -u -w 0 127.0.0.1 514
This should create a folder /data/syslog/udp/127.0.0.1 with a *.log file
Last edited by Jotne on Sun Feb 21, 2021 9:12 am, edited 6 times in total.
 
verpeilo
just joined
Posts: 9
Joined: Wed May 13, 2020 12:08 pm

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 6:59 pm

For what is it?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Thu May 14, 2020 7:39 pm

Did not understand the question.
My post above clearly stats that if you run Splunk as a non root user you need an external server to receive sysloge.
At least if you uses port below 1024, Splunk will not work. So Mikrotik send udp/515->rsyslog server <- Splunk reads the file.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Fri May 15, 2020 9:08 am

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.
What commend do you like?
Name of host are resolved when Splunk uses DNS to find its name.
There are several places a comment could exist.
* DHCP comments
* DNS comments
* IP address lists comments
All this can set comment for an IP

To get all information with comment for all ip addresses in the address list and send it using syslog:
{
	:foreach interface in=[/ip firewall address-list find] do={
		:local output [ip firewall address-list get $interface]
		:set ( "$output"->"script" ) "addresses"
		:log info message= "$output"
	}
}
 
markwien
Frequent Visitor
Frequent Visitor
Posts: 69
Joined: Sun Jul 22, 2018 10:49 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Fri May 22, 2020 12:33 pm

If you do use Splunk as a non root user, you need an external Syslog server.
This setup needs Splunk for Mikrotik v3.0 to read field correctly. (out soon)

This is how to set it up using Ubuntu server. Should work on most version.

rsyslog comes default with Ubuntu so no need to install any extra software.

Copy these two files to /etc/rsyslog.d/

udp.conf (sets up rsylog to accept Sylog on udp/514)
# rsyslog.d/11_hemit_tcp.conf
#
# This receives UDP syslog on port 514 and stores it in a reliable format
# in /data/syslog/udp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imudp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="udp_split_filename" type="list") {
  constant(value="/data/syslog/udp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="udp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="udp_split_filename"
  )
}

# setting
input(type="imudp" port="514" ruleset="udp_split")

tcp.conf (sets up rsylog to accept Sylog on tcp/1514)
# rsyslog.d/11_hemit_tcp.conf
#
# This receives TCP syslog on port 1514 and stores it in a reliable format
# in /data/syslog/tcp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imtcp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="tcp_split_filename" type="list") {
  constant(value="/data/syslog/tcp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="tcp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="tcp_split_filename"
  )
}

# settings
input(type="imtcp" port="1514" ruleset="tcp_split")

Create the following folders
mkdir /data
mkdir /data/syslog
mkdir /data/syslog/tcp
mkdir /data/syslog/udp
Change folder rights to syslog and restart rsyslog
chown -R syslog:syslog /data/syslog
service rsyslog restart
run netstat to see that rsylog is running
netstat -tuopan | grep 514
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      -                    off (0.00/0/0)
tcp6       0      0 :::1514                 :::*                    LISTEN      -                    off (0.00/0/0)
udp        0      0 0.0.0.0:514             0.0.0.0:*                           -                    off (0.00/0/0)
udp6       0      0 :::514                  :::*                                -                    off (0.00/0/0)
To maks Splunk reads the data, this input.conf file is needed %SplunkHome%/etc/apps/MikroTik/default (included in v3.0)
[monitor:///data/syslog/udp/.../*.log]
sourcetype = mikrotik
host_segment=4

[monitor:///data/syslog/tcp/.../*.log]
sourcetype = mikrotik
host_segment=4
Test your server:
echo '<14>sourcehost message text' | nc -v -u -w 0 127.0.0.1 514
This should create a folder /data/syslog/udp/127.0.0.1 with a *.log file

hi i made installation exactly as your tutorial, but syslog files dont get read by splunk.
i am wondering how user splunk can read /data/ dir if the syslog dir runs under user syslog ....

do i miss something?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Fri May 22, 2020 1:58 pm

Splunk reads data since rights on the files are set by syslog so Splunk can read them and inputs.conf tells it to read them.
-rw-r--r-- 1 syslog syslog 2520311 May 22 12:56 20200522-12.log
The "r" in all three places makes all able to read the files.

To test, log in as the Splunk user:
sudo su - splunk
Do you see any folder/files under /data/syslog/udp ?
try to cat them

IF you do not see any files, there may be several problems
* Local firewall
* Syslog not set up correctly
* Nothing sending to udp/514

Splunk looks at the inputs.conf and see that it should read the files. (so if there are no inputs.conf in MikroTik/Default, you need to create it.
It will be included in upcoming v3.0

If all is setup correctly try in Splunk to search for
index=_internal source="*splunkd.log" NOT INFO
and see what errors/warnings you have.

PS no need to quote the whole post. Use "Post Reply" button below post to reply, or quote only needed part.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Sat May 23, 2020 2:36 pm

Script updated to 3.8
Fixed so that some information only get collected every hour, even if scripts run every 5 min.
This is to not flod system with duplicate information.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Sun May 24, 2020 1:19 pm

Script updated to 3.9

Gives better command history for Router OS v7 or newer.
Fixed better handling with NTP/SNTP information

PS script can be updated without update Splunk software.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Sun May 24, 2020 11:52 pm

Did an change in what to debug settings in first post

Change from:
/system logging add action=logserver prefix=MikroTik topics=!debug
to:
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet
This reduces overall DNS logging by 80%
So I do suggest that you add this, specially if you have a lots of DNS requests.
 
User avatar
fmarais007
newbie
Posts: 26
Joined: Thu Jan 11, 2018 9:16 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Mon May 25, 2020 5:02 pm

Hi,

Thank you for all the hard work! This is really a nice tool.

Can you please assist.
I'm getting the following error. I'm running it on a Windows Server 2019 VM: Error in 'outputlookup' command: External command based lookup 'device_lookup' is not available because KV Store initialization has failed. Contact your system administrator.
Splunk-KV Store Failed-20200525.jpg
All the searches I do give me Linux commands to run, and I'm not sure if MongoD is even running?
Any help will be appreciated thank you.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Mon May 25, 2020 6:36 pm

I have see KV store error due to certificate error on splunk. Other than that, I do not know what can be wrong. I do suggest that you install Splunk from scratch on an Ubuntu 18.04 or 20.04 server. Works every time I have tried.
I do use less than an hour to install Ubuntu/Splunk and Mikrotik plugin and get all up and running.
 
User avatar
fmarais007
newbie
Posts: 26
Joined: Thu Jan 11, 2018 9:16 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Mon May 25, 2020 8:14 pm

Hi,

OK I will do that. Sucks that the only answer is to try another OS, but oh well.

Another thing, is it possible to monitor several MikroTik's via this system, or is it only designed for a single device?

Thanks
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Mon May 25, 2020 8:35 pm

It can monitor as many as you like. Only limit is the amount of data logged to Splunk. For free you get 500MB, that should be ok for a small to medium system, depending on what you select to log. DNS eats lots of log space.

What OS did you try? I do in first post recommend Ubuntu.
Ubuntu running on VmWare workstation on Windows server works fin as well, but I do recommend a dedicated server.
 
User avatar
fmarais007
newbie
Posts: 26
Joined: Thu Jan 11, 2018 9:16 am

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Tue May 26, 2020 10:54 am

Hi,

No worries, I got it running on Ubuntu 20.04

So I once again tried to see the devices reporting but no data.
I then realized it only runs every morning at 01:00 so I tried to run it manually.
Even though I have two devices sending logs to Splunk successfully (I can see my firewall traffic, but can't filter by host), It's not showing any results:
Splunk-DeviceNoResults-20200526.jpg
No KV errors this time :D

How do I force the refresh of devices reporting to my server? I tried:
Splunk-ForceRun-20200526.jpg
If that is the correct method, why is it not appearing?
Thanks for all the help.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Tue May 26, 2020 1:41 pm

I am working with v3.0 and there the kv search is updated. Try this:

This should just give the devices:
sourcetype=mikrotik
module=script
script=sysinfo
| dedup host
Then if that gives lines, run this updated version.
sourcetype=mikrotik
module=script
script=sysinfo
| dedup host
| rex "version=\"(?<version>[^\"]*)\" board-name=\"(?<board_name>[^\"]*)\" model=\"(?<model>[^\"]*)\" serial=(?<serial>\S*) identity=\"(?<identity>[^\"]*)\"" 
| table  host identity serial model board_name version
| fillnull value="-"
| outputlookup device_lookup
 
User avatar
Rhoos
just joined
Posts: 22
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Tue May 26, 2020 8:21 pm

Hi Jotne,

Thank you for this magnificent work!
I have a small problem with the file "MikroTik2.9.spl.rar" for a few days I have tried without success to decompress the information, it always gives me an error using Winrar!
"The archive is either in unknown format or damaged"
Could you please check!
Thanks for your help and for the whole project.

Ricardo
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Tue May 26, 2020 8:46 pm

Downloaded the file from this forum and expanded it using 7-zip (Winrar should do as well), so file seems fine,
 
User avatar
Rhoos
just joined
Posts: 22
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 2.9 (Graphing everything)

Tue May 26, 2020 10:55 pm

Thanks success, before making the query, use Winrar on two different computers, downloading a new file on each one without success, I followed your advice of "7-zip" and extracted it without any problem!
Ricardo
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed May 27, 2020 10:27 pm

Splunk for MikroTik updated to v3.0

Mayor changes is the PPPoE view and support for IPv6 in the MikroTik Firewall Rules module

To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9)

# 3.0 (27.05.2020)
# Fixed missing identity before device are logged to index
# Added PPPoE view
# Added IPv6 support for firewall
# Added support for using external rsyslog server
# Updated "Device List" to update KV-Store automatically when run
# Added dhcp_name to "MikroTik DHCP to Static"
# Change script and dashboard for "MikroTik Neighbor"
# Added more infor for the "Mikrotik Wifi connection"
# Updated script to 3.5, Better RouterOS v7 handling
# Updated script to 3.6, NTP info
# Updated script to 3.8, Change some colelction to get info every hour, not every 5 min (CDP++)
# Updated script to 3.9, to get better command history fro v7 RouterOS, fixed better NTP/SNTP handling
# Added minspan=5m to some graphs
# Fixed "MikroTik DNS Live usage" to work without dns packets log
 
User avatar
fmarais007
newbie
Posts: 26
Joined: Thu Jan 11, 2018 9:16 am

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Fri May 29, 2020 9:14 am

Hello,

I was wondering if you can help with a previous post regarding CAPsMAN setup for your application.
Two of the charts uses the name of the interface for the data, but the problem comes in where if there are spaces in the interface it stops displaying the name before the space.
eg. if my interface name is "Client-A cAP-1" it only displays data for "Client-A", which means it misses data for all the other AP's.

This is the graph i'm referring to:

Splunk-CapGraphError-20200528.jpg
The chart uses the following search:
sourcetype=mikrotik counter>0  |chart values(counter) by name
Which gives these results when there are multiple AP's
Splunk-CapGraphErrorResult-20200529.jpg
And thereby displaying no results in the actual graph.


Would it be possible to either help me to define the search better, or perhaps integrate the OP's idea into your application?

Many thanks
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Fri May 29, 2020 9:25 am

This is some wrong:
sourcetype=mikrotik counter>0  |chart values(counter) by name
Buts not easy to fix when I do not see the real log data.
Sent you a private message.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 10:37 am

Splunk for MikroTik updated to v3.0

Mayor changes is the PPPoE view and support for IPv6 in the MikroTik Firewall Rules module

To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9)
Hi,
Can 2 versions co-exist somehow ? I'm currently running the 2.7 version in my Spunk which is modified a bit.
Would be great if I load the app/dashboard and it sits next to the existing one as "Mikrotik 3.0" or something.

Thanks!

EDIT : Hmm, doesn't seem to be, in the SPL-file "Mikrotik" seems hardcoded somehow internally because even if I rename the SPL-file to let's say "Mikrotik3_0.spl" before loading the app it still says I already have it running...
EDIT2 : It was along time since I played with it, but I've untarred the SPL-files and adapted the app.conf so the "id" now is "MikroTik3" in stead of "MikroTik" and then tarred it again. Now it imports next to my existing 2.7-dashboard and all seems fine.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 4:22 pm

You can copy all files you have modified to another folder.
Remove all MikroTik files, install 3.0, then restore your files.

Its also possible to use 7-zip/winrar to extract all the files from 3.0 manuall, then add one by one.

If your edit is interesting for other, you could send me them, and I could add it in v3.0
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 4:27 pm

If your edit is interesting for other, you could send me them, and I could add it in v3.0
I only removed some items not applicable for me at all, so no real enhancements.
I'm now using your supplied 3.0 Splunk Dashboard and it looks good enough for me ! I'm going to leave it as-is.
Thanks!
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 6:02 pm

I'm seeing some weird values on in the DHCP-section -> the "DHCP Pool Information" seems to give a faulty % value (eg. 450%)
Looking at the performed query for this :

litsearch (sourcetype=mikrotik module=script script=pool) | eval percent=used*100/total, host_name=coalesce(identity,host) | fields keepcolorder=t "_time" "host" "host_name" "percent" "pool" "total" "used"

There is something wrong with this "total" value. In my case, the pool is a defined /24 in size and almost everything is reserved.
Looking at the Mikrotik script output it only seems to be transmitting following to the Splunk -> script=pool pool=Pool 1 used=45 total=10


# Collect DHCP Pool information
# ----------------------------------
if ($DHCP and $run) do={
/ip pool {
:local poolname
:local pooladdresses
:local poolused
:local minaddress
:local maxaddress
:local findindex
# Iterate through IP Pools
:foreach pool in=[find] do={
:set poolname [get $pool name]
:set pooladdresses 0
:set poolused 0
# Iterate through current pool's IP ranges
:foreach range in=[:toarray [get $pool range]] do={
# Get min and max addresses
:set findindex [:find [:tostr $range] "-"]
:if ([:len $findindex] > 0) do={
:set minaddress [:pick [:tostr $range] 0 $findindex]
:set maxaddress [:pick [:tostr $range] ($findindex + 1) [:len [:tostr $range]]]
} else={
:set minaddress [:tostr $range]
:set maxaddress [:tostr $range]
}
# Calculate number of ip in one range
:set pooladdresses ($maxaddress - $minaddress)
# /foreach range
}
# Test if pools is used in DHCP or VPN and show leases used
:local dname [/ip dhcp-server find where address-pool=$poolname]
:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
:set poolused [:len [used find pool=[:tostr $poolname]]]
} else={
# DHCP server found, count leases
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
:set poolused [:len [/ip dhcp-server lease find where server=$dname]]}
# Send data
:log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")
# /foreach pool
}
# /ip pool
}
}
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 8:50 pm

There may be some wrong with that part. Its on part of the script that is not made by me ;)

For me it looks correct
/ip pool print
 # NAME                                                                    RANGES                         
 0 DHCP-Pool-vlan1-Home                                                    10.10.10.55-10.10.11.254
Then the script shows this:
script,info MikroTik: script=pool pool=DHCP-Pool-vlan1-Home used=158 total=455
Can you post the output of
/ip pool print

Who is online

Users browsing this forum: No registered users and 7 guests