Community discussions

 
Hunty
just joined
Posts: 12
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:00 pm

I've attached two screenshot
You do not have the required permissions to view the files attached to this post.
 
Hunty
just joined
Posts: 12
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:03 pm

please note that I've inserted "Mikrotik" under System/Logging
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:13 pm

All apps needs to be in
$SPLUNK_HOME/etc/apps
So on windows you should have:
C:\Program Files\Splunk\etc\apps\MikroTik
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
Hunty
just joined
Posts: 12
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:14 pm

All apps needs to be in
$SPLUNK_HOME/etc/apps
So on windows you should have:
C:\Program Files\Splunk\etc\apps\MikroTik
Yes
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:16 pm

Uninstall everything.
Install follow post #1 step by step.

You should also see
C:\Program Files\Splunk\etc\apps\MikroTik\default
C:\Program Files\Splunk\etc\apps\MikroTik\metadata
etc
Not
C:\Program Files\Splunk\etc\apps\MikroTik\MikroTik\default
If that does not work, I will try to do an install my self from the #1 post and test it.

PS no need to quote post above you.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
Hunty
just joined
Posts: 12
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:23 pm

are you sure?
I've attached a screenshot of the content of the folder
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:43 pm

You have restarted Splunk after app install?
All looks correct.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
Hunty
just joined
Posts: 12
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:50 pm

Yes several times
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 8:55 pm

You have installed Splunk free as in post #1, or do you use Splunk before to some else?
I have seen problem with installed version that using other index.

From the picture above, it does not seem that splunk does the filed extraction.

If you like, I can try teamviewer to see what is wrong.
Not able to post a private message to you, so post an email so I can get in touch with you.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
Hunty
just joined
Posts: 12
Joined: Mon May 28, 2018 11:37 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Nov 15, 2018 9:08 pm

thanks for your help, but I'll try tomorrow with the linux VM but I've to solve first why the 514 port is not available even if I followed your guide to install the app with a non root user
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.4 (Graphing everything)

Tue Nov 20, 2018 12:20 pm

2.4 Released

Nearly all code are rewritten to get better speed and make it cleaner.
Dark Theme makes a big visual change.

# v2.4 (20.11.2018)
# Updated "MikroTik Hotspot login/logout information" to show IP
# Fixed when inn interface= unknown
# Updated view 2.4 to handel more hits
# Updated "MikroTik DNS" to not view revers lookup "site!=*.in-addr.arpa"
# Rewritten "Microtik Traffic" Error in all calculation
# Fixed data rounding and fixed typo
# Fixed formating in "MikroTik Remote Connection"
# Set permission view the view to show in app only
# Added System Changes as a new default menu
# Fixed missing host in "MikroTik Uptime"
# Added Host to "MikroTik Traffic"
# Added view "MikroTik Wifi strength"
# Added view "MikroTik System Changes"
# Dark theme needs >=7.2
# Removed global time (use default time)
# Removed searchWhenChanged="true" (default)
# Cleaned code
# Fixed error in "13. OSCam config changes"
# Added Sprakline to "MikroTik Device List"
2.4 Device list.jpg
.
2.4 System Changes.jpg
.
2.4 Traffic.jpg
You do not have the required permissions to view the files attached to this post.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
jareckib12
just joined
Posts: 1
Joined: Fri Jan 12, 2018 1:04 am

Re: Using Splunk to analyse MikroTik logs 2.4 (Graphing everything)

Sat Nov 24, 2018 12:06 am

Hi,
First - thx for update.
Second - in MikroTik DNS request view, client filtering does not work. When selecting any item in addition to "any" does not show any results.

Jarecki
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sun Nov 25, 2018 10:40 am

Good catch,

I have updated to 2.5

# 2.5 (25.11.2018)
# Change all "if" test to use "coalesce"
# Fixed error in "MikroTik DNS request"
# Moved more to base search
# Removed some code not needed in "MikroTik Web Proxy"
# Fixed error with src_port in dest_ip dropdown in "MikroTik Firewall Rules"
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
sherics
just joined
Posts: 6
Joined: Sun Nov 25, 2018 10:02 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sun Nov 25, 2018 9:26 pm

Hello,

I have installed it completely and except the Traffic, everything work.

In the traffic I see just few MBs, even if I download 500MB or 1GB, it does not shows up there, just few % of the downloaded amount.

I do not have a public IP on my internal network, the public IP is on the WAN port, ether1, as a standard home router, other clients are on WiFi on first VLAN and 2 computers on second VLAN.

Do you have an idea what is wrong?

Thank you.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 6:08 pm

I did download an 1GB file from here: http://www.ovh.net/files/
And it showed up correctly.

Do you have Fasttrack on?
If so try to disable it, it may be that packed are not accounted when Fasttrack is on.
https://www.youtube.com/watch?v=6LaqhDm6PHI
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 8:45 pm

Hello Jotne and the whole community.
First I want to tell you, good job, really good jobs, and thanks for sharing with us Jotne.

Secondly I have a question, in version 2.4 I see in the record that wrote "List of devices" this function indicates that it already supports multi-router log ?, in such case as it is identified in each module to which router belongs each record?
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 9:32 pm

You are correct, it does support as many routers as you like to add.
Going away from SNMP to Syslog only was driven by the simpler way to do thing.
With SNMP, you need to set up the monitor system to request SNMP from the device.
This is ok for singel router ans small system.
But if you like to monitor a router across public internet, you end up in a security risk by open for SNMP.

Whit using script and Syslog this is a one way communication. All data are sent from the device to the monitor system.
No need to open ports. Same script for all routers. No need to configure any configuration on the monitoring system for each router.

I have four routers/host that sends log to my sentral log server.

On every view you can select host to view only that host.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 9:41 pm

Great, you're right, forget that each module has a drop-down menu Hosts. I'm going to try it and anything I write. Thanks again.
 
sherics
just joined
Posts: 6
Joined: Sun Nov 25, 2018 10:02 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 10:59 pm

I did download an 1GB file from here: http://www.ovh.net/files/
And it showed up correctly.

Do you have Fasttrack on?
If so try to disable it, it may be that packed are not accounted when Fasttrack is on.
https://www.youtube.com/watch?v=6LaqhDm6PHI
Well, I forgot about fastrack... without fastrack it works now, but unfortunately without fastrack my router is on 95-99% CPU while I download/upload anything; and the speed is lowered for 300mbit/s... With fastrack enable, the cpu is approx on 70% on full gigabit connection, about 90MB/s real speed. Well, after 4 years, I think, I need to purchase a more powerul and new router :)
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Nov 26, 2018 11:22 pm

It may also be that you could configure your router to use hardware offloading. Depending on type and software version.
But old boxes do have less power so upgrade may be the only option.

Its a good point to now that traffic monitoring does not work when fast track is enabled, so I will mention that in the first post.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
MSandoval
newbie
Posts: 26
Joined: Thu Mar 01, 2018 3:32 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Nov 30, 2018 3:45 pm

Hello everyone, I have a problem with module MikroTik_Traffic section Public IP. when reviewing this, I found a small error when declaring the variable host, in this case that variable is capitalized Host, it does that the section does not work, changing this I achieved that it works correctly.
<title>Public IP</title>
        <search base="base_search">
          <query>
            search
              Host="$Host$"         >>>   change with host="$Host$"
            | eval ip_in=if("$direction$"=="in",src_address,dst_address)
            | eval ip_out=if("$direction$"!="in",src_address,dst_address)
Image
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Nov 30, 2018 5:28 pm

Thanks for the feedback :)

It will be fixed in 2.6. For others you can edit det file and correct the typo.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Dec 07, 2018 4:18 am

Hello there,

This is my first time using splunk and I have no result on dashboard anyway also I did every step on the post #1, any idea why this happen?
The logs already show up on the splunk but the MikroTik app dashboard have no result at all.

Thanks and appreciate your help.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Fri Dec 07, 2018 11:51 pm

/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug
I would guess you have typed wrong prefix. Any other word than MikroTik would brake the index of the data.
Make sure its 100% equal with capital M and K

Cut and Past is the best option to get it correct.

Do a search like this in Splunk, change to your MikroTik Routers IP, what is the output?
index=* host=192.168.88.1 | rex "^\S+\s(?<prefix>\S+)\s" | stats count by prefix
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 1:23 am

Hello,

I did copy and paste that command on cli.

The result prefix search on attachment
Search MT.JPG
And then I found something that on the search section if I remove module=xxx then I got the result on the dashboard.
For the example on the device list dashboard I use this
No Module.JPG
instead of your originally script
With Module.JPG
I think that module=xx didn't work on my splunk search. Any idea?
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 9:38 am

Strange.

Can you post output of sourcetype=mikrotik script=sysinfo
Make sure you have Smart Mode selected (see circle on picture)
Click the arrow to expand one post so I see the extraction. >
.
test_output.jpg
You do not have the required permissions to view the files attached to this post.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 10:33 am

Hi Jotne,

Here is the output and just different from yours.
test_output_1.JPG
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 12:26 pm

I see two strange things.
1. It seems that Splunk does not handle the date/time correctly since its shown within your event.
2.I do not see the information from router that shows where it comes from and type (ipsec/DNS/DHCP) (debug packets)

Is this a clean Splunk installation, followed the steps above?

You are running on a 951G a common box, I have a 941 and 750Gr3 and some other.
Your RouterOS software 6.43.4 is the same as I do run, so should be ok

Can you post the last lines of the output on the Router of /log print and /log print detail
Just cut and paste the line, so I do see how it looks like.

On mine
11:21:32 script,info script=pool pool=default-dhcp used=1 total=245
and
time=11:21:32 topics=script,info
message="script=pool pool=default-dhcp used=1 total=245"
I do miss the stuff in bold from your logg message and would like to see how it looks like on the router to compare what Splunk sees.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 12:52 pm

Hi Jotne,

Yes, this is fresh install splunk and I did several time remove my VM and install again to make sure that.

Here is the output
/log print

17:47:19 firewall,info FW_INTERNAL forward: in:PJX out:BRX-LAN, proto TCP (ACK), 10.99.100.102:7332->10.121.61.108:52380, len 40

/log print detail

time=17:48:19 topics=firewall,info message="FW_INTERNAL forward: in:PJX out:BRX-LAN, proto TCP (ACK), 10.99.100.102:7332->10.121.61.108:52380, len 40"
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 1:09 pm

This looks correct, so it have to be some wrong with Splunk implementation since message looks different there.
Several other has used this, so should not be an big error in the code.

If you tyoe index=* in splunk, do you see any message that have the module tag coming from the router?

Like this
firewall,info
PS If you set time to: real time 1-minute window you should see data live as they arrive.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 1:28 pm

Unfortunately I didn't see that message on my splunk,
test_output_2.JPG
Any idea what is happening on my splunk?
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 2:10 pm

You have some strange in your message that I have not see with other: RTZPKN02

Can you post this? /system logging export

How did you install the files in Splunk?

Why do you get Des 9 in your log, I am still at Des 8?
Your logs has two different time stamp.
See if all clock is equal everywhere. Router, Computer ++
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 2:23 pm

Sorry the Dec 9 is from date server, I already change the NTP :D

Here is the output
 /system logging export
# dec/08/2018 19:21:37 by RouterOS 6.43.4
# software id = 29W1-FTPT
#
# model = 951G-2HnD
# serial number = 642E05A9020A
/system logging action
add name=syslog remote=10.99.100.77 remote-port=7514 src-address=10.122.82.200 \
    target=remote
add bsd-syslog=yes name=logserver remote=10.100.10.105 src-address=\
    10.122.82.200 target=remote
/system logging
add action=syslog disabled=yes topics=info,error,interface,warning
add action=logserver prefix=MikroTik topics=dhcp
add action=logserver prefix=MikroTik topics=!debug
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 2:44 pm

Seems that you do not have a clean install.
You are logging to several system at the same time.
It should work.

Try this: Remove all logg line and add this:
/system logging action
add name=myserver remote=10.100.10.105 target=remote
/system logging
add action=myserver prefix=MikroTik topics=!debug
add action=myserver prefix=MikroTik topics=dhcp
Is this your Splunk server? 10.100.10.105
If not, do you relay your message (rslyslog or other server)?
Do you send your log message passing trough several routers?
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 3:45 pm

Seems that you do not have a clean install.
You are logging to several system at the same time.
It should work.

Try this: Remove all logg line and add this:
/system logging action
add name=myserver remote=10.100.10.105 target=remote
/system logging
add action=myserver prefix=MikroTik topics=!debug
add action=myserver prefix=MikroTik topics=dhcp
Is this your Splunk server? 10.100.10.105
If not, do you relay your message (rslyslog or other server)?
Do you send your log message passing trough several routers?
Ok I will reinstall my splunk VM again and change all log line and I will tell you the result

And yes my log message passing through several routers.
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 3:46 pm

But is this your Splunk server? 10.100.10.105
Or do you send data to an rsyslog or other syslog server, that then sends it to your Splunk server?
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
ariefwido
just joined
Posts: 8
Joined: Thu Dec 06, 2018 10:51 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Sat Dec 08, 2018 5:30 pm

Hi Jotne,

It seems I found the problem, the problem is marking the BSD Syslog on log remote action.
test_output_3.JPG
Finally the result is come.

Thanks and very appreciate your help.
You do not have the required permissions to view the files attached to this post.
 
WeWiNet
Frequent Visitor
Frequent Visitor
Posts: 78
Joined: Thu Sep 27, 2018 4:11 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Tue Dec 11, 2018 1:53 pm

Hi Jotne,

Wanted to say thank you, very nice job.
Also to highlight that this tutorial works perfect on MacOS 10.14.
I just followed your tutorial and installed it with the Splunk Enterprise version
and all is working perfect (Ok I had to restart my machine once as splunk did not launch first time correctly).

I now try to make sense out of all that data and nice graphs ... :-)

PS: How can you know how much data you log per day (which is the limitation of the free version)?
WeWiNet

**
MTCNA
hapac2, map, hap-lite
 
User avatar
Jotne
Forum Veteran
Forum Veteran
Topic Author
Posts: 707
Joined: Sat Dec 24, 2016 11:17 am

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Tue Dec 11, 2018 2:52 pm

Thanks.

You find license information her:

Settings->Licensing
There you see this for free version
Licensed daily volume 500 MB

Select:
Usage-Report->Previous 30 days

Here you will see how much of the license you use each day, last 30 days.
.
Use Splunk to monitor your MikroTik Router

MikroTik->Splunk

Who is online

Users browsing this forum: No registered users and 1 guest