Community discussions

 
Halfeez92
newbie
Posts: 34
Joined: Tue Oct 30, 2012 12:58 pm
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 7:08 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.
Ok thanks for the help. Already delete the duplicate device.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Jun 10, 2019 5:49 pm

Updated section 2c regarding Log prefix.

NB Do not use more than 20 charters, or else it start to clip other part of the log
firewall,info MikroTik: 123456789012345678901234567890 : in:ether1-Wan ...
firewall,info MikroTik: 1234567890123456789012345 forwa: in:ether1-Wan ...
firewall,info MikroTik: 12345678901234567890123 forward: in:ether1-Wan...
firewall,info MikroTik: 12345678901234567890 forward: in:ether1-Wan ...
As you see here the chain word forward is eat'n up by the prefix.
MT is this a bug???
If not, set a warning in the gui :)
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 13, 2019 1:54 pm

Updated section 2f)

Script updated to collect and show how many dynamic/static address lists entry there are.
Eks output
script,info MikroTik: script=address_lists list=rdp_stage2 dynamic=24 static=0
script,info MikroTik: script=address_lists list=rdp_stage1 dynamic=28 static=0
script,info MikroTik: script=address_lists list=ftp_stage2 dynamic=1 static=0
script,info MikroTik: script=address_lists list=ftp_stage1 dynamic=1 static=0
script,info MikroTik: script=address_lists list=black_list_rdp dynamic=42 static=0
script,info MikroTik: script=address_lists list=black_list_ftp dynamic=1 static=0
script,info MikroTik: script=address_lists list=Whitelist_IP dynamic=3 static=2
script,info MikroTik: script=address_lists list=Router dynamic=0 static=1
script,info MikroTik: script=address_lists list=IPSEC dynamic=1 static=0
script,info MikroTik: script=address_lists list=FW_Block_user_try_unkown_port dynamic=1089 static=0
script,info MikroTik: script=address_lists list=Clients dynamic=0 static=2
script,info MikroTik: script=address_lists list=Blocked dynamic=1 static=7
This will later be used in its own graph to see variation in the lists.

PS only one IP en the ssh black list black_list_ssh is due to that I do not use default port.

You can update script only and wait for new Mikrotik Splunk app to be updated later.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 36
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 20, 2019 9:59 am

Hello Jotne,

I want to upgrade my Splunk version 7.2 environment tot Splunk 7.3

Is the mikrotik app compatible with Splunk 7.3?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jun 20, 2019 1:43 pm

Yes, I do try to not use anything special in the APP so it should be compatible with all new version.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jun 21, 2019 9:29 pm

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
pidde
just joined
Posts: 1
Joined: Fri Aug 24, 2012 5:22 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Jun 23, 2019 2:59 am

Hi!

Must say you did a great work with this app!
Is it possible to add option82 to dhcpserver part?
And is it also possible decode the option82 from hex?
 
zandhaas
newbie
Posts: 36
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jun 25, 2019 10:34 am

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.
When I look at the current script under 2f I only see the "# Collect DHCP Pool information" part.

It seems the rest of the script is missing.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jun 25, 2019 1:09 pm

You are 100% correct. Copy past error.

Fixed.

PS It's getting closer to the release of v 2.7 of Splunk for MikroTik
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jun 28, 2019 2:10 pm

Script to get information on the router is upgraded to 2.6 section 2f

Simpler DHCP calculation.
Fixed comment so it start on the beginning of the line.
Fixed Script names
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 01, 2019 1:15 pm

Upgraded to 2.7

There are a lot of new changes to the app as listed below, so its a larger upgrade.
Simplest way to upgrade, if you have not made changes your self, remove (uninstall) previous version, install new version.
Please report any problems back to this thread, and I will try to fixed.

PS If you do upgrade, you also need to upgrade script in section 2f (fist post) on all router you like to get data from.
Just cut/past the script over the old one.

PS2 File is found under section 1g first post

Request to changes are also welcome :)

What new:
# 2.7 (01.07.2019)
# New view added "Address Lists Counters"
# Changes most view to use "Base Search"
# Changed "MikroTik DHCP request" to use stats and fixed host flaw
# Changed "MikroTik System Changes" to use 30 day and 4 hour span and maxspan in transaction
# Removed changes to "DHCP leases" in "MikroTik System Changes"
# Added search in dropdown for "MikroTik DNS Live usage"
# Added Time picker for "MikroTik Device List"
# Speeded up "MikroTik Remote Connection"
# Fixed wrong timestamp of packets logged
# Changed "MikroTik DHCP request" to use stats and fixed host flaw and maxspan in trnsaction
# Added search in dropdown for "MikroTik DNS Live usage" and added IP to client and change sorting
# Fixed "MikroTik DNS request" to use correct dropdown lists
# Fixed "MikroTik Firewall Rules" to use better searh, removed base level, added counters, long prefix
# Rewritten "MikroTik Live attack" to speed up and added more dropdown
# Fixed "MikroTik Resources" to give correct host number
# Changed "MikroTik System Changes" to use 30 day and 4 hour span, removed DHCP info
# Fixed "MikroTik Traffic" to use script= and some clean up
# Fixed "MikroTik uPnP" script name, added ip to dropdown
# Added to ">MikroTik Uptime" dropdown menu
# Fixed "MikroTik Volt/Temperature" sorting
# Fixed "MikroTik VPN Connection" faster search
# Fixed "MikroTik Web Proxy" sorting and some code clean up
# Changed "MikroTik Wifi strength" to use script tag and some clean up
# Added "dashboard.css" to set menu color global
# Fixed "props.conf" to better handel wrong prefixed and some other changes
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 03, 2019 5:36 am

I have been paying attention to this post, very powerful chart, but the cumbersome construction and the lack of relevant knowledge have been unsuccessful. I can only temporarily use the mrtg icon inside routeros to temporarily cope with it. I hope the poster can write the deployment manual from the perspective of the technology-poor. .
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Jul 03, 2019 3:17 pm

Its written so that a user with some knowlege should be able to set it up.
You can start by telling me what your problem is, and we may be able to help you out.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 06, 2019 6:50 am

Reinstalled splunk on ubuntu18.04, is a virtual machine under esxi, the deployment is very simple and normal, according to the steps of the top post, but the splunk dashboard can not see the task data incoming. Very strange, what else do I need to pay attention to? Please forgive my English using Google Translate, I am from China
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Jul 06, 2019 10:43 am

After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik 
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 08, 2019 12:57 pm

After starting Splunk, go to Search & Reporting menu. Add following search:
sourcetype=mikrotik
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.
According to what you said carefully, but still can not receive the data, I introduced the cdb1016 log file db format, can be displayed to splunk, indicating that splunk no problem, is the data input problem, I see ros is the log The output is udp514 port, but I only see tcp listening port settings in splunk's receiving settings. Is this the reason?
1.png
2.png
3.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 08, 2019 1:36 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 09, 2019 5:51 am

There is no local listening udp514, now there is data in, but click on the meter in the Mikrotik2.7 dashboard, most of them do not have any charts, how to add or customize the dashboard you need here, for example, I want The wan's real-time or past and downstream traffic in a certain period of time, as well as the system temperature, the number of online hosts, and so on. How to do it?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 09, 2019 8:29 am

514 UDP do need to be active
Do you run it on Linux?

If so, as Root, type:
netstat -opan | grep 514
You should see one line like this:
udp        0      0 0.0.0.0:514             0.0.0.0:*                           23557/splunkd        off (0.00/0/0)
if not UDP/514 is not running.

One the mikrotik, post the output of:
/system logging export
You should see some like:
# jul/09/2019 07:26:37 by RouterOS 6.43.16
# software id = E4B6-94N8
#
# model = RouterBOARD 750G r3
# serial number = 6F3806E0A160
/system logging action
set 3 remote=ip_your_syslog_server
/system logging
set 0 disabled=yes
add action=remote prefix=MikroTik topics=dhcp
add action=remote prefix=MikroTik topics=hotspot
add action=remote prefix=MikroTik topics=!debug
There should be IP for your server, and prefix for all action with MikroTik. If one letter is wrong in the prefix, it will fail. See capital M and T in the MikroTik.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 1:03 pm

It’s true that I set it wrong, Mikrotik changed to MikroTik, and it should be fine, then I will report it.
 
haaroons
just joined
Posts: 1
Joined: Wed Jul 10, 2019 11:15 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 1:32 pm

Hello Jotne,
I am new to this forum.

I have install MikroTik logs 2.7.

MikroTik DNS Live usage and MikroTik DNS Live request is not working. if i do search eventtype=dns_query No item found

Do advice how to fix this.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Jul 11, 2019 11:43 pm

DNS information are coming from standard logs on the router.

What do you get if you go to search window and search with the following line:
sourcetype=mikrotik earliest=-24h latest=now() | stats count by module
I do get some like this:
module		count
dhcp		12764
dns		324512
firewall	1349
ipsec		7
script		91182
upnp		308
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 6:02 am

The data is coming, some of the tables are already filled, some still have no data, such as dns, it doesn't matter, I want to know how to monitor the flow table of an interface (wan), just like mirkrotik's built-in mrtg chart, every 5 minutes, 30 minutes and so on. . . As shown
1.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 8:05 am

That is why I need the output of the above command.
Some data are coming from the logg.
Some are comming from scripting

Log:
-------
dhcp,dhcp_static,dns,firewall,ipsec,upnp

script:
-------
IPSEC_failed,address_list,healt,pool,resource,sysinfo,traffic,uncounted,upnp

So I guess you have some log problems. Read section 2b carefully.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 11:42 am

Splunk is too powerful. If I have multiple ccr1016, how can I transfer data to the splunk server, how do I distinguish syslogs from different mikrotik routers?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Fri Jul 12, 2019 1:29 pm

All the view for MikroTik in Splunk has a host drop down. So if you have more than one router, just select the host you like to monitor.
There is one possible problem, if you have many routers with same IP that sends log to same Splunk.
That could be solved using unique ID for each router and some small change to the code.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 6:24 am

How can I write the interface tx-bits-per-second parameter to the log and then plot it in splunk.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 8:06 am

What command do you use on the router to see this data?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
fengyuclub
newbie
Posts: 42
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Jul 15, 2019 10:07 am

What command do you use on the router to see this data?
interface monitor-traffic ether1

Search forums see scripts with such calls
  "/interface monitor-traffic ether1 once do={
:put ($"tx-bits-per-second"/1000 /1000 )
}"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1131
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Jul 16, 2019 8:12 pm

It can be done.
I do use IP accounting to see the traffic going trough the router.
This way are more generic and does work without any modification.
If you monitor one and one interface, this has to be adopted for each setup.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 

Who is online

Users browsing this forum: No registered users and 1 guest