Community discussions

 
Halfeez92
newbie
Posts: 36
Joined: Tue Oct 30, 2012 12:58 pm
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 7:08 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.
Ok thanks for the help. Already delete the duplicate device.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Jun 10, 2019 5:49 pm

Updated section 2c regarding Log prefix.

NB Do not use more than 20 charters, or else it start to clip other part of the log
firewall,info MikroTik: 123456789012345678901234567890 : in:ether1-Wan ...
firewall,info MikroTik: 1234567890123456789012345 forwa: in:ether1-Wan ...
firewall,info MikroTik: 12345678901234567890123 forward: in:ether1-Wan...
firewall,info MikroTik: 12345678901234567890 forward: in:ether1-Wan ...
As you see here the chain word forward is eat'n up by the prefix.
MT is this a bug???
If not, set a warning in the gui :)
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 13, 2019 1:54 pm

Updated section 2f)

Script updated to collect and show how many dynamic/static address lists entry there are.
Eks output
script,info MikroTik: script=address_lists list=rdp_stage2 dynamic=24 static=0
script,info MikroTik: script=address_lists list=rdp_stage1 dynamic=28 static=0
script,info MikroTik: script=address_lists list=ftp_stage2 dynamic=1 static=0
script,info MikroTik: script=address_lists list=ftp_stage1 dynamic=1 static=0
script,info MikroTik: script=address_lists list=black_list_rdp dynamic=42 static=0
script,info MikroTik: script=address_lists list=black_list_ftp dynamic=1 static=0
script,info MikroTik: script=address_lists list=Whitelist_IP dynamic=3 static=2
script,info MikroTik: script=address_lists list=Router dynamic=0 static=1
script,info MikroTik: script=address_lists list=IPSEC dynamic=1 static=0
script,info MikroTik: script=address_lists list=FW_Block_user_try_unkown_port dynamic=1089 static=0
script,info MikroTik: script=address_lists list=Clients dynamic=0 static=2
script,info MikroTik: script=address_lists list=Blocked dynamic=1 static=7
This will later be used in its own graph to see variation in the lists.

PS only one IP en the ssh black list black_list_ssh is due to that I do not use default port.

You can update script only and wait for new Mikrotik Splunk app to be updated later.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 20, 2019 9:59 am

Hello Jotne,

I want to upgrade my Splunk version 7.2 environment tot Splunk 7.3

Is the mikrotik app compatible with Splunk 7.3?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 20, 2019 1:43 pm

Yes, I do try to not use anything special in the APP so it should be compatible with all new version.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jun 21, 2019 9:29 pm

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
pidde
just joined
Posts: 1
Joined: Fri Aug 24, 2012 5:22 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Jun 23, 2019 2:59 am

Hi!

Must say you did a great work with this app!
Is it possible to add option82 to dhcpserver part?
And is it also possible decode the option82 from hex?
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jun 25, 2019 10:34 am

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
When I look at the current script under 2f I only see the "# Collect DHCP Pool information" part.

It seems the rest of the script is missing.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jun 25, 2019 1:09 pm

You are 100% correct. Copy past error.

Fixed.

PS It's getting closer to the release of v 2.7 of Splunk for MikroTik
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jun 28, 2019 2:10 pm

Script to get information on the router is upgraded to 2.6 section 2f

Simpler DHCP calculation.
Fixed comment so it start on the beginning of the line.
Fixed Script names
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 01, 2019 1:15 pm

Upgraded to 2.7

There are a lot of new changes to the app as listed below, so its a larger upgrade.
Simplest way to upgrade, if you have not made changes your self, remove (uninstall) previous version, install new version.
Please report any problems back to this thread, and I will try to fixed.

PS If you do upgrade, you also need to upgrade script in section 2f (fist post) on all router you like to get data from.
Just cut/past the script over the old one.

PS2 File is found under section 1g first post

Request to changes are also welcome :)

What new:
# 2.7 (01.07.2019)
# New view added "Address Lists Counters"
# Changes most view to use "Base Search"
# Changed "MikroTik DHCP request" to use stats and fixed host flaw
# Changed "MikroTik System Changes" to use 30 day and 4 hour span and maxspan in transaction
# Removed changes to "DHCP leases" in "MikroTik System Changes"
# Added search in dropdown for "MikroTik DNS Live usage"
# Added Time picker for "MikroTik Device List"
# Speeded up "MikroTik Remote Connection"
# Fixed wrong timestamp of packets logged
# Changed "MikroTik DHCP request" to use stats and fixed host flaw and maxspan in trnsaction
# Added search in dropdown for "MikroTik DNS Live usage" and added IP to client and change sorting
# Fixed "MikroTik DNS request" to use correct dropdown lists
# Fixed "MikroTik Firewall Rules" to use better searh, removed base level, added counters, long prefix
# Rewritten "MikroTik Live attack" to speed up and added more dropdown
# Fixed "MikroTik Resources" to give correct host number
# Changed "MikroTik System Changes" to use 30 day and 4 hour span, removed DHCP info
# Fixed "MikroTik Traffic" to use script= and some clean up
# Fixed "MikroTik uPnP" script name, added ip to dropdown
# Added to ">MikroTik Uptime" dropdown menu
# Fixed "MikroTik Volt/Temperature" sorting
# Fixed "MikroTik VPN Connection" faster search
# Fixed "MikroTik Web Proxy" sorting and some code clean up
# Changed "MikroTik Wifi strength" to use script tag and some clean up
# Added "dashboard.css" to set menu color global
# Fixed "props.conf" to better handel wrong prefixed and some other changes
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 03, 2019 5:36 am

I have been paying attention to this post, very powerful chart, but the cumbersome construction and the lack of relevant knowledge have been unsuccessful. I can only temporarily use the mrtg icon inside routeros to temporarily cope with it. I hope the poster can write the deployment manual from the perspective of the technology-poor. .
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 03, 2019 3:17 pm

Its written so that a user with some knowlege should be able to set it up.
You can start by telling me what your problem is, and we may be able to help you out.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 06, 2019 6:50 am

Reinstalled splunk on ubuntu18.04, is a virtual machine under esxi, the deployment is very simple and normal, according to the steps of the top post, but the splunk dashboard can not see the task data incoming. Very strange, what else do I need to pay attention to? Please forgive my English using Google Translate, I am from China
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 06, 2019 10:43 am

After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik 
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 08, 2019 12:57 pm

After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
According to what you said carefully, but still can not receive the data, I introduced the cdb1016 log file db format, can be displayed to splunk, indicating that splunk no problem, is the data input problem, I see ros is the log The output is udp514 port, but I only see tcp listening port settings in splunk's receiving settings. Is this the reason?
1.png
2.png
3.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 08, 2019 1:36 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 09, 2019 5:51 am

There is no local listening udp514, now there is data in, but click on the meter in the Mikrotik2.7 dashboard, most of them do not have any charts, how to add or customize the dashboard you need here, for example, I want The wan's real-time or past and downstream traffic in a certain period of time, as well as the system temperature, the number of online hosts, and so on. How to do it?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 09, 2019 8:29 am

514 UDP do need to be active
Do you run it on Linux?

If so, as Root, type:
netstat -opan | grep 514
You should see one line like this:
udp        0      0 0.0.0.0:514             0.0.0.0:*                           23557/splunkd        off (0.00/0/0)
if not UDP/514 is not running.

One the mikrotik, post the output of:
/system logging export
You should see some like:
# jul/09/2019 07:26:37 by RouterOS 6.43.16
# software id = E4B6-94N8
#
# model = RouterBOARD 750G r3
# serial number = 6F3806E0A160
/system logging action
set 3 remote=ip_your_syslog_server
/system logging
set 0 disabled=yes
add action=remote prefix=MikroTik topics=dhcp
add action=remote prefix=MikroTik topics=hotspot
add action=remote prefix=MikroTik topics=!debug
There should be IP for your server, and prefix for all action with MikroTik. If one letter is wrong in the prefix, it will fail. See capital M and T in the MikroTik.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 1:03 pm

It’s true that I set it wrong, Mikrotik changed to MikroTik, and it should be fine, then I will report it.
 
haaroons
just joined
Posts: 1
Joined: Wed Jul 10, 2019 11:15 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 1:32 pm

Hello Jotne,
I am new to this forum.

I have install MikroTik logs 2.7.

MikroTik DNS Live usage and MikroTik DNS Live request is not working. if i do search eventtype=dns_query No item found

Do advice how to fix this.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 11:43 pm

DNS information are coming from standard logs on the router.

What do you get if you go to search window and search with the following line:
sourcetype=mikrotik earliest=-24h latest=now() | stats count by module
I do get some like this:
module		count
dhcp		12764
dns		324512
firewall	1349
ipsec		7
script		91182
upnp		308
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 6:02 am

The data is coming, some of the tables are already filled, some still have no data, such as dns, it doesn't matter, I want to know how to monitor the flow table of an interface (wan), just like mirkrotik's built-in mrtg chart, every 5 minutes, 30 minutes and so on. . . As shown
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 8:05 am

That is why I need the output of the above command.
Some data are coming from the logg.
Some are comming from scripting

Log:
-------
dhcp,dhcp_static,dns,firewall,ipsec,upnp

script:
-------
IPSEC_failed,address_list,healt,pool,resource,sysinfo,traffic,uncounted,upnp

So I guess you have some log problems. Read section 2b carefully.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 11:42 am

Splunk is too powerful. If I have multiple ccr1016, how can I transfer data to the splunk server, how do I distinguish syslogs from different mikrotik routers?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 1:29 pm

All the view for MikroTik in Splunk has a host drop down. So if you have more than one router, just select the host you like to monitor.
There is one possible problem, if you have many routers with same IP that sends log to same Splunk.
That could be solved using unique ID for each router and some small change to the code.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 6:24 am

How can I write the interface tx-bits-per-second parameter to the log and then plot it in splunk.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 8:06 am

What command do you use on the router to see this data?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 10:07 am

What command do you use on the router to see this data?
interface monitor-traffic ether1

Search forums see scripts with such calls
  "/interface monitor-traffic ether1 once do={
:put ($"tx-bits-per-second"/1000 /1000 )
}"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 16, 2019 8:12 pm

It can be done.
I do use IP accounting to see the traffic going trough the router.
This way are more generic and does work without any modification.
If you monitor one and one interface, this has to be adopted for each setup.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 5:03 am

{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX (($speedRX/1000)/1000);
:set $mbpsTX (($speedTX/1000)/1000);
:put "$iname RX:$mbpsRX Mbps TX:$mbpsTX Mbps";
}
}
I found the script for this post available, but after running it is all interfaces, I don't want all interfaces, only a few interfaces are needed, for example, I only need ether1, ether2, how to modify the script, and how can I get it? Let him display in the log, I use the splunk search call, and display it as 14.5Mbps instead of 14528. I hope to get everyone's help.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 9:29 am

Info
It seems that data you get from monitor are just moment blink of data going through the interface. So it will fly up and down for every time you run it. If it would be like cisco, average last 5 min, it would be perfect to rune every 5 min. Not sure if it are useful at as is.


If you have not renamed interface
:foreach interface in=[/interface find] do={
To
:foreach interface in=[/interface find where (name~"^ether1\$" || name~"^ether2\$") ] do={
or use regex
:foreach interface in=[/interface find where name~"^ether[12]\$" ] do={
Anchor ^ \$ are used to distinguish ether1 from ether11 etc.

Edit
You can use ID instead of name, so you can change from:
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
to
:set $monitor [/interface monitor-traffic $interface as-value once]
PS2, no need to declare variables, use them directly
do not divide data by 1000 two times, let splunk do that, so you do not loose any resolution
use equal sign for splunk to read data directly
you do not need semicolon behind each line ;

So final script could be some like this
:foreach interface in=[/interface find where name~"^ether[12]\$"] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 11:47 am

Your script, the regular expression method, no success without any output, it doesn't matter, my code is as follows
I want to know how to search for rich charts, and there are 14.8Mbps and 14833 display problems. This is not important. The important thing is how splunk draws charts.

mycode
{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "WAN-ether2 down RX=$mbpsRX Kbps";
:log info "WAN-ether2 up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"adsl-tx") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "adsl-tx down RX=$mbpsRX Kbps";
:log info "adsl-tx up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"bonding1") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "bonding1 down RX=$mbpsRX Kbps";
:log info "bonding1 up   TX=$mbpsTX Kbps"
}
}
After the schedule is displayed as follows
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 12:26 pm

Tested on other ccr1016 your script is successful, it should be the problem of the interface name, but it is important to draw the splunk graphics, I hope you can add to the new version.
3.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 1:38 pm

When you have multiple interface, use only one section, no a section for every interface

change
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
to
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
Test code that should output data to screen:
{
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:put "script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
}

PS, when testing cut and past on the cli, you need to wrape all script in brackets {} !!!

PS how often would you like to run the script? every 5 min. Do you know if monitor could show average 5 min data?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 1:58 pm

Try this

Add this to the Data_to_Splunk_using_Syslog script
# Get interface data (test)
# ----------------------------------
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1")(name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
Then in Splunk do this search for the last 4 hour.
sourcetype=mikrotik script=monitor| timechart avg(RX) as RX avg(TX) as TX by interface limit=10
May take some time to nice graphs.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 7:06 pm

Nice,
I have added the additional script entries and changed the inteface names to the names I use.
But............
The sourcetype entry in the search entry schould be "sourcetype=MikroTik" 8) 8)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 7:32 pm

In Splunk, search ignore case :)

Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 19, 2019 9:06 pm


Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.
The current "Mikrotik Traffic" overview is indeed a nice oveview.
But apart from knowing who is generating the traffic I am very interested in the amount of traffic that floats over each individual interface. And especially the WAN interface(s) and ISL interfaces. And when you see a bottleneck on one of your interfaces you can drill down to your traffic overview to identify the source of all that traffic.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 20, 2019 5:46 am

Jotne,Great, I did it according to your script, and the beautiful chart shows normal. I tried to add scripts to my multiple ccr and routerboards, so my interface has a lot of duplicate names, such as bonding1 and bridge1, how can I distinguish between them, or change the name for each interface.
4.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 20, 2019 5:59 am

Understand, add host=x.x.x.x in front of the search statement you gave to open my ccr and rb.
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 20, 2019 7:10 am

Host=x.x.x.x Although this option is available, some devices have an internet connection that is a dynamic ip obtained by adsl dialing. So before the log warning, add an identity=xxxxx to distinguish the mikrotik device. After testing, it is feasible and runs very well.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 22, 2019 10:24 am

@ fengyuclub
Nice to see you are getting it to work.

@ All
Section 2c) Logging prefix has been updated with sample on how to name to logs.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 22, 2019 3:44 pm

Script in section 2f) updated to 2.9

It now support to get interface counters and you can also set modules true/false if you do not like to monitor one section.
If you do not have wifi/dhcp, you can just set them to false.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 24, 2019 8:23 am

Script in section 2f) updated to 3.0

Do now get CDP neighbors
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 25, 2019 12:46 pm

Splunk is really powerful, I see splunk have a lot of apps to install, in our China use wechat (similar to facebook, telegram) this social software, I saw this social software related app, WeChat Alert App for Splunk, I installed this App, sending test messages from wechat is successful, but I don't know much about splun's alert settings, set it many times, only a single success, can you help me?
5.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 25, 2019 1:44 pm

No need for extra app to send message. Sending email using a gmail account is easy and works well.

But there is a big issue.

If you have a free Splunk license, you do loose a lot of thing.
* Monitor and Alerting (needed for sending alerts)
* 500MB pr day maximum
* Cluster
* Universal Forwarder
* HA
* Distributed Search
* Perfomance Acceleration
* Access controll (only on user)
* LDAP
+++

This is why I have not included any Alerting in the project.

There is a workaround. You can setup an batch job that runs search from command line and do stuff from it. (I have not tested it)
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 27, 2019 4:56 am

I have Splunk Enterprise license, gmail alert can't be real-time, mobile mail client can't update mail in real time, there is a delay of about 10 minutes, so I choose wechat alert.I received some wechat alert, but some of them use the search to save as an alert, I can't receive a wechat alert, I don't know where the problem is.
6.jpg
7.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 27, 2019 8:30 am

Splunk do handle real time alerts (or close to)
https://docs.splunk.com/Documentation/S ... TimeAlerts
It should not depend of type of action you are using, starting a program, sending sms, email, wechat etc. Alerts should go out.
But you should not use to many alerts, since it will use more CPU to handle them.

Not sure what your problem is.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1223
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Aug 02, 2019 10:20 am

Updated script to 3.1

Fixed CDP, since some devices sends long version with new lines breaking up the log lines. (Cisco)

PS still have problem that line is cut in Splunk. Not sure if its MT not sending whole line, or Splunk that cuts the lines.
I do only get 278 characters.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 

Who is online

Users browsing this forum: No registered users and 1 guest