Community discussions

 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 7:07 pm

Below an example of the local 192.168.0.1 address
_time	rule	chain	in_if	out_if	src_mac	protocol	src_ip	src_port	dest_ip	dest_port	City	Country
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	42597	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	57660	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	56630	192.168.0.1	53	Unknown	
And I wil try the solution with an extra Drop rule in the firewall.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 7:34 pm

It looks like your router tries to resolve DNS on it self and get blocked.
From router console try this.
:put [/resolve mikrotik.com]
You should get an IP as result, like 159.148.147.196
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.5 (Graphing everything)

Mon Jan 21, 2019 8:20 pm

I get the same IP as a result biut perhaps My PI-Hole implementation has something to do with it.

I'm using PI-Hole as an "Ad blocker for my Internal network"
And for this I'm using DHCP option 6 to force all internal clients to go to the PI-Hole server for the DNS resolving.


By the way I changed the "Drop all from not coming from LAN" rule.
I replaced the "In Interface list" from !LAN to "In Interface" Ether1-WAN.

This seemed to have resolved my issue.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 8:43 am

2.6 released

# 2.6 (22.01.2019)
# Added information about fast track in "traffic monitor"
# Fixed typo in Traffic view. Added fast track info
# Changed to checkbox in "DNS Request"
# Added better sparkline "in Device List"
# Added identity to "Device List"
# Updated script to get identity
# Removed parentheses from services from "MikroTik uPnP"
# Added ip to client drop-down list to "MikroTik uPnP"
# Added more disk info to "MikroTik Resources"
# Changed to last 12 hour instead of 4 in "MikroTik DNS Live usage"
# Changed to sort by count in "Sort by count"
# Added timeline dashboard to "DNS Request"
# Fixed public IP speed by reducing lookup in "Traffic"
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 11:05 am

I see some strange things happen.

I have added three devices to the Splunk Mikrotik environment.

1. RB750Gr3 as a router. (sending over UDP 514)
2. HAPac2 configured as a switch (Accesspoint) (sending over UDP 515)
3. Mikrotik CHR as Dude server. (sending over UDP 516)

Everything seems to log all information to splunk but after somtime the data of the HAPac2 is not examind any more by Splunk.
After restarting the splunk server Everything is OK again for a short time.
The Router and the DUDE server have no issues.

When i check the Splunkd.log file I see a lot "Failed to parse timestamp" messages for the HAPac2 syslog.
01-22-2019 09:46:45.504 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:515|host=192.168.0.8|syslog|
What can be wrong?

This morning I updated to version 2.6.
But I had this problem before. So it is not version related.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 11:40 am

It may be that the props filter only look at UDP:514 and syslog.
When data comming in on UDP:515 it will not see that its MikroTik data.

You can fix this by edit etc/apps/MikroTik/default/props.conf and add
[source::udp:515]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik

[source::udp:516]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik
But my questioon to you is, why use more than on UDP?
I do see noe good reason to use on port for each device. Send all to UDP/514
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 1:10 pm

I started with using port 514 for all 3 mikrotik devices.
At that moment I had the same problem. No data in visible in Splunk.
After that I changed to port 515 and restarted splunk. And yes I saw data in Splunk. but some time later Splunk stopped showing data in the graphs.
Then I restarted splunk again and yes Splunk is showing data for an hour or so.
The Router and the Dude device are showing Up as expected.

See the picture below:
2019-01-22 11_52_06-MikroTik Wifi strength _ Splunk 7.png
At the moment I changed all 3 devices back to UDP port 514. With the same result as before.

I still see below messages in the splunkd.log file saying it suppresses messages:
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|\n                                1295 similar messages suppressed.  First occurred at: Tue Jan 22 11:57:40 2019
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.345 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.348 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.356 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 1:19 pm

Do examine time on all your devices. It must be in sync.
Do use NTP on all devices to make sure time is ok.
Last edited by Jotne on Tue Jan 22, 2019 3:53 pm, edited 1 time in total.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Jan 22, 2019 1:52 pm

The router is used as the timeserver for my local environment.
the HAPac2, the Dude server and the Splunk server synchronize time with the router and all have the same time and date.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 9:57 am

It may have something to do that you have used different UDP ports. I may not recognize the message correctly.
You may try to start over and follow the example step by step.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 7:10 pm

I made some progress.

After an other look at the messages in the splunkd.log file
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
I focused on the MAX_TIMESTAMP_LOOKAHEAD option. according to the default props.conf file this option default to 32 for syslog events.
Looking at the mikrotik log events the consists of 19 characters (excluding the mili seconds).

To change the default 32 to 19 I added the MAX_TIMESTAMP_LOOKAHEAD option to the "/opt/splunk/etc/apps/MikroTik/default/props.conf " file and restarted Splunk.
[syslog]
TRANSFORMS-force_mikrotik = force_mikrotik
MAX_TIMESTAMP_LOOKAHEAD = 19
After this change I do not see the above message in the Splunkd.log file anymore. And more important, the Hapac2 is logging events for more as 3 hours now. This is already an hour longer as before (max 2 hours).

I will keep an eye on the Mikrotik splunk environment to see if everything keeps running.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 9:11 pm

Interesting. Have not used much time in my splunkd.log, but have the same problem as you,
But only in one of 4 routers. Other are ok.

Tried bot 19 and 23 but still get samme message.
01-23-2019 20:08:40.136 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event (Wed Jan 23 20:08:39 2019). Context: source=udp:514|host=193.1.1.100|syslog|
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Jan 23, 2019 10:41 pm

At my site it was 1 out of 3 that failed and I was missing information for that router.
Are you also missing data?
After the change my failing router is still visible in Splunk so for mee it seems the solution.
But I did not check the log files that come from the routers. Do you now were I can find them?
Perhaps it has something to do with too many events during a short time period.

We need to debug this.

Regards Peter
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Jan 24, 2019 8:12 am

I do get event from all routers. To see if you get from one specific router use search and type host=1.2.3.4 (change to your IP)
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
Egert143
just joined
Posts: 11
Joined: Tue Apr 24, 2018 4:05 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jan 25, 2019 3:38 pm

Hello

Could i get instructions how to create splunk source type manualy ? I have splunk light (paid) and it doesent support apps (as far as i know).

Current problem is that source and dest addres fields are merged with port numbers.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Jan 25, 2019 10:07 pm

I have no idea on how to use Splunk Light.
In normal Splunk, source type based on the source it comes from udp:514

props.conf
[source::udp:514]
TRANSFORMS-force_mikrotik = force_mikrotik
transforms.conf
[force_mikrotik]
DEST_KEY =  MetaData:Sourcetype
REGEX =  \sMikroTik:\s
FORMAT =  sourcetype::mikrotik
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
Egert143
just joined
Posts: 11
Joined: Tue Apr 24, 2018 4:05 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Jan 26, 2019 10:07 pm

And how would i turn 123.123.123.123:1234->12.34.45.67:80 to Source Address = 123.123.123.123 Source Port = 1234 Dest Address = 12.34.45.67 Dest Port 80 So they would be searchable ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Jan 26, 2019 11:02 pm

The traffic solution are based on that you have private ip inside your net and public on the outside.
Private IPv4 addresses
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16


But if you like to log other IP and know what is inside/outside, you have to modify the Splunk files.

Edit:
MikroTik Traffic
Replace all
 | search (ip_in="10.0.0.0/8" OR ip_in="172.16.0.0/12" OR ip_in="192.168.0.0/16")
with
 | search ip_in="12.34.45.0/8"
That if you like 12.34.45.0/8 to be your inside net.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Feb 01, 2019 8:41 am

Where can I find the link to download MikroTik2.6 spl? Thanks.
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Feb 02, 2019 9:47 pm

In the first post of this topic :)
Or the below link
download/file.php?id=35231
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 04, 2019 9:40 am

Thanks zandhaas - I got it downloaded.

I also installed everything per our topic owner Jotne's procedure but cannot get the data flow from MikroTik to Splunk, after verifying port 514 is open. Upon diving into some details, I suspect it's due to the lack of SSL of my MikronTik (192.168.88.1 shows "Not secure") - anyone know if this is the root cause? If yes what is the easiest way to enable SSL under RouterOS v6.40.8 and Win10? Appreciate any tips there.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 05, 2019 10:55 am

There are no certificates involved in the transaction. All data are sent using UDP/514 Syslog (not encrypted).

In Splunk search, type only a * and do a search for the last 24 hour. Do you see any data at all?
Make sure you follow all steps in the first post 1 by 1.
Do you have any deviation? Using a clean Splunk install? Windows firewall opened if you run on Windows?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Feb 06, 2019 7:16 am

Thanks Jotne. Now I see the data (events) through the Splunk search, though MikroTik2.6 app still not sees the data yet and I am still debugging.

BTW the Splunk observed event entry looks like - do you see any anomaly there?

2/5/19
9:09:54.000 PM
Feb 5 21:09:54 router.lan Feb 5 21:09:54 MikroTik MikroTik: Router = 192.168.88.1
host = router.lan
source = udp:514
sourcetype = mikrotik
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Feb 06, 2019 8:22 am

Can you post some example line from search in Splunk that shows what you got in the log from using * search?

Do you have tagget all packet with MikroTik? This will fail Mikrotik since its not the same name.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 6:19 am

Hi Jonte, here're three snapshots

1. Splunk Event entry sample from the MikroTik UDP feed - great if you can help review the "Host", "Source", "Sourcetype" field to see if they are right for the MikroTik2.6 App
Splunk Event Entry from UDP and MikroTik.png
2. Splunk UDP input setting
Splunk UDP Input Setting.png
3. MikroTik2.6 App snapshot (system change search, with no data found while the Splunk search gives items like above)
Splunk MikroTik 2.6 App Lauch Snapshot.png
Thanks
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 8:27 am

1. Is this the only type of event you see?

Here are some example on how they should look like: (various modules)
firewall,info MikroTik: NAT_Web_server dstnat: in:ether1-Wan out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 91.12.58.49:49145->92.220.200.251:80, len 60
dhcp,debug,packet MikroTik:     Parameter-List = Subnet-Mask,Router,Domain-Server,Domain-Name,NETBIOS-Name-Server,Static-Route
dns,packet MikroTik: --- sending reply to 10.10.10.244:53720:
script,info MikroTik: script=health voltage=24 V temperature=42 
wireless,info MikroTik: 04:62:73:xx:xx:21@wlan1 established connection on 2437000, SSID GjestenettHMN
ipsec MikroTik: invalied encryption algorithm=6.
interface,info MikroTik: ether1 link up (speed 100M, full duplex)
Have you followed tutorial in post#1?
Do you use Splunk for other stuff?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 10:26 am

Hi I followed your first post but skipped 2c~2e (FW/NAT/Traffic logging since not sure about the detailed steps). I did have Home Monitor app before that affected the MikroTik data inputs, and I have it removed so the data inputs seems right (though not complete if without 2c~2e). The question I have is that, even with incomplete but valid data (say only DHCP request part), should MikroTik2.6 App see them and populate some view right? But now it seems the app does not pick up anything and I am not sure if the app has access to the log. Thanks.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Feb 07, 2019 11:52 am

You should get DHCP and other stuff from the router if you skipped 2c-2e.
Thats why I asked about how the log lines looks like.
You could use a search for host=192.168.88.1 and post some line.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 9:17 am

Thanks again Jotne. Here's a screenshot. Seems the Splunk events have the right contents, but the format is different from yours.
Splunk MikroTik 2.6 Event Snapshots.png
Basically, before the identifier "MikroTik", there are timestamps and another "MikroTik", but without the log field name like "dns,packet" as in your snapshots.
I copied the MikroTik scripts exactly, so do you think I missed something on the Splunk side? My Splunk version is 7.2.3.
You do not have the required permissions to view the files attached to this post.
 
zandhaas
newbie
Posts: 38
Joined: Tue Dec 11, 2018 11:02 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 9:44 am

Are you sure your "router script" is complete?

I had problems getting my data visible in splunk to.
It turned out that I missed the last "}" in the Router script.
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 10:04 am

Hi Jotne ~ some progress - for some reason, the "Module" field picks up part of the timestamp (the Month) since their is no syslog field name for some reason (the event item format difference I mentioned). After tweaking the Volt/Temperature code (removing the module key from the search), I was able to get that view right. Encouraged and will see how to get the module field right in the first place - help appreciated.
Splunk MikroTik 2.6 Volt_n_Temp.png
You do not have the required permissions to view the files attached to this post.
 
Larsa
Member Candidate
Member Candidate
Posts: 119
Joined: Sat Aug 29, 2015 7:40 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 12:04 pm

Since I'm not a Splunk expert I wonder if anyone has some bright ideas how to optimize Splunk / Mongodb?

We have about 15.5 million entries and the reports are getting really slow to produce. In a regular SQL database you can run a "Query Execution Plan" and then add indexes to columns that performs table scans. Is there an equivalent way in Splunk or any other way to optimize the environment? We're running Splunk with 12 cores, 20 Gb ram and SSD which ought to be sufficient.

Any suggestions are welcome!
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 6:41 pm

@Larsa
Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.

I do recommend that you start a thread about your problem over here:
https://answers.splunk.com/index.html
Last edited by Jotne on Mon Feb 11, 2019 6:54 pm, edited 1 time in total.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Feb 11, 2019 6:51 pm

@JieYu2001

There are some wrong with extraction of the data in the Splunk or the format that your MT Router sends it.
In List view in Splunk your should not see time and date in the Event space, only in Time column.
In your view, I do not see it only one time extra, but two times in front of the data. This breaks all view.
You get it to work since you adjusted to view to accept your wrong data.
source=udp:514 and sourcetype=mikrotik looks correct.

I would recommend you to start over.
Clean Install of Splunk, remove all connection to Splunk in your router.

@zandhaas
You do not need the script to get data inn to splun, so it could also be removed to rule out problems.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
JieYu2001
just joined
Posts: 8
Joined: Fri Feb 01, 2019 8:36 am

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 12, 2019 9:26 am

Thanks Jotne - the issue is resolved. In the MK Logging setting, I checked "BSD Syslog" which caused issue (still don't know why since that is the correct syslog protocol supported in Splunk). Uncheck it and things look fine now.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 12, 2019 3:08 pm

There was nothing in the first post telling you to select it so not sure why you did it.
Will update post #1 to say not to select it.
Good you find out what was wrong :)
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
Larsa
Member Candidate
Member Candidate
Posts: 119
Joined: Sat Aug 29, 2015 7:40 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Feb 12, 2019 9:07 pm

Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.I do recommend that you start a thread about your problem over here: https://answers.splunk.com/index.html

Thanks for the suggestion, I'll report back if I find out an appropriate solution!
 
oaas
just joined
Posts: 1
Joined: Sun Feb 10, 2019 7:15 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Fri Feb 15, 2019 5:41 pm

Great work!

Had some issues with parsing messages from one cAP ac where the messages suddenly dropped due to "Failed to parse timestamp" warning messages.

Seems it got solved by adding
TIME_FORMAT = %b/%d/%Y %H:%M:%S
to the props.conf file.

Please consider adding this to future releases.

/Thanks
 
frankcale
just joined
Posts: 9
Joined: Sat Nov 03, 2018 6:39 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 11:28 am

Hi, Can u pls help with displaying Vlan info
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 11:59 am

Not sure what you asks for.
A list of Vlan on the router?
Traffic going trough Vlan?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
frankcale
just joined
Posts: 9
Joined: Sat Nov 03, 2018 6:39 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 4:59 pm

Hi, Can u include vlan traffic monitoring and if possible protocols like youtube, torrent, updates, etc
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sun Feb 17, 2019 7:30 pm

Protocol are complicated to monitor due to https, near to impossible.
Vlan can be monitored used SNMP or you can use script and syslog to send data.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Tue Mar 05, 2019 12:19 pm

Updated 1a to mention that you need an account at splunk.com to download software.
Account is free to create.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
ithelp
just joined
Posts: 2
Joined: Sun Aug 16, 2015 9:41 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Mar 06, 2019 6:18 am

Hi, thanks for this magnificent explanation.
Can you give me on how to see the PPP and PPPOE information from the log?
I've already configure it on the rules tab, but nothing shows on any dashboard.
Thanks,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Wed Mar 06, 2019 8:09 am

I do not have PPP nor PPPOE so I can not easily make log for it.

But if you could post 3-4 pages of logs that involves PPP and PPPOE output I could have look at it.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
cavin12
just joined
Posts: 1
Joined: Thu Mar 07, 2019 12:29 pm
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Thu Mar 07, 2019 12:32 pm

thanks for such a clear presentation for the newbies to understand, appreciate the efforts.
 
neutronlaser
Member Candidate
Member Candidate
Posts: 193
Joined: Thu Jan 18, 2018 5:18 pm

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Mar 16, 2019 8:07 pm

Price is ridiculous.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Sat Mar 16, 2019 9:40 pm

500MB/day for free is ridiculous much to pay.

But I do agree that if you pay retail price for Splunk and need eks 500GB/day, price is high.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
Halfeez92
newbie
Posts: 36
Joined: Tue Oct 30, 2012 12:58 pm
Contact:

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 9:25 am

Hi how can I remove the MikroTik device list in the splunk dashboard view? I have multiple same devices showing up because I forgot to disable NAT and enable routing. Now it have 2 same devices with different IP
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1154
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.6 (Graphing everything)

Mon Apr 29, 2019 1:14 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 

Who is online

Users browsing this forum: No registered users and 1 guest