Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) 💾 🛠 💻 📊

zandhaas · Mon Jan 21, 2019 7:07 pm

Below an example of the local 192.168.0.1 address

_time	rule	chain	in_if	out_if	src_mac	protocol	src_ip	src_port	dest_ip	dest_port	City	Country
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	42597	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	57660	192.168.0.1	53	Unknown	 
2019-01-21 17:56:20	FW_Drop_all_from_WAN	input	(unknown 1)	(unknown 0)	na	UDP	192.168.0.1	56630	192.168.0.1	53	Unknown

And I wil try the solution with an extra Drop rule in the firewall.

Jotne · Mon Jan 21, 2019 7:34 pm

It looks like your router tries to resolve DNS on it self and get blocked.
From router console try this.

:put [/resolve mikrotik.com]

You should get an IP as result, like 159.148.147.196

zandhaas · Mon Jan 21, 2019 8:20 pm

I get the same IP as a result biut perhaps My PI-Hole implementation has something to do with it.

I'm using PI-Hole as an "Ad blocker for my Internal network"
And for this I'm using DHCP option 6 to force all internal clients to go to the PI-Hole server for the DNS resolving.

By the way I changed the "Drop all from not coming from LAN" rule.
I replaced the "In Interface list" from !LAN to "In Interface" Ether1-WAN.

This seemed to have resolved my issue.

Jotne · Tue Jan 22, 2019 8:43 am

2.6 released

# 2.6 (22.01.2019)
# Added information about fast track in "traffic monitor"
# Fixed typo in Traffic view. Added fast track info
# Changed to checkbox in "DNS Request"
# Added better sparkline "in Device List"
# Added identity to "Device List"
# Updated script to get identity
# Removed parentheses from services from "MikroTik uPnP"
# Added ip to client drop-down list to "MikroTik uPnP"
# Added more disk info to "MikroTik Resources"
# Changed to last 12 hour instead of 4 in "MikroTik DNS Live usage"
# Changed to sort by count in "Sort by count"
# Added timeline dashboard to "DNS Request"
# Fixed public IP speed by reducing lookup in "Traffic"

zandhaas · Tue Jan 22, 2019 11:05 am

I see some strange things happen.

I have added three devices to the Splunk Mikrotik environment.

1. RB750Gr3 as a router. (sending over UDP 514)
2. HAPac2 configured as a switch (Accesspoint) (sending over UDP 515)
3. Mikrotik CHR as Dude server. (sending over UDP 516)

Everything seems to log all information to splunk but after somtime the data of the HAPac2 is not examind any more by Splunk.
After restarting the splunk server Everything is OK again for a short time.
The Router and the DUDE server have no issues.

When i check the Splunkd.log file I see a lot "Failed to parse timestamp" messages for the HAPac2 syslog.

01-22-2019 09:46:45.504 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:515|host=192.168.0.8|syslog|

What can be wrong?

This morning I updated to version 2.6.
But I had this problem before. So it is not version related.

Jotne · Tue Jan 22, 2019 11:40 am

It may be that the props filter only look at UDP:514 and syslog.
When data comming in on UDP:515 it will not see that its MikroTik data.

You can fix this by edit etc/apps/MikroTik/default/props.conf and add

[source::udp:515]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik

[source::udp:516]
TRANSFORMS-dns=remove_dns_query,remove_dns_answer
TRANSFORMS-force_mikrotik = force_mikrotik

But my questioon to you is, why use more than on UDP?
I do see noe good reason to use on port for each device. Send all to UDP/514

zandhaas · Tue Jan 22, 2019 1:10 pm

I started with using port 514 for all 3 mikrotik devices.
At that moment I had the same problem. No data in visible in Splunk.
After that I changed to port 515 and restarted splunk. And yes I saw data in Splunk. but some time later Splunk stopped showing data in the graphs.
Then I restarted splunk again and yes Splunk is showing data for an hour or so.
The Router and the Dude device are showing Up as expected.

See the picture below:

2019-01-22 11_52_06-MikroTik Wifi strength _ Splunk 7.png

At the moment I changed all 3 devices back to UDP port 514. With the same result as before.

I still see below messages in the splunkd.log file saying it suppresses messages:

01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|\n                                1295 similar messages suppressed.  First occurred at: Tue Jan 22 11:57:40 2019
01-22-2019 12:02:41.342 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.345 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.348 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.351 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.352 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|
01-22-2019 12:02:41.356 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|

Jotne · Tue Jan 22, 2019 1:19 pm

Do examine time on all your devices. It must be in sync.
Do use NTP on all devices to make sure time is ok.

zandhaas · Tue Jan 22, 2019 1:52 pm

The router is used as the timeserver for my local environment.
the HAPac2, the Dude server and the Splunk server synchronize time with the router and all have the same time and date.

Jotne · Wed Jan 23, 2019 9:57 am

It may have something to do that you have used different UDP ports. I may not recognize the message correctly.
You may try to start over and follow the example step by step.

zandhaas · Wed Jan 23, 2019 7:10 pm

I made some progress.

After an other look at the messages in the splunkd.log file

01-22-2019 12:02:41.350 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Tue Jan 22 00:20:00 2019). Context: source=udp:514|host=192.168.0.8|syslog|

I focused on the MAX_TIMESTAMP_LOOKAHEAD option. according to the default props.conf file this option default to 32 for syslog events.
Looking at the mikrotik log events the consists of 19 characters (excluding the mili seconds).

To change the default 32 to 19 I added the MAX_TIMESTAMP_LOOKAHEAD option to the "/opt/splunk/etc/apps/MikroTik/default/props.conf " file and restarted Splunk.

[syslog]
TRANSFORMS-force_mikrotik = force_mikrotik
MAX_TIMESTAMP_LOOKAHEAD = 19

After this change I do not see the above message in the Splunkd.log file anymore. And more important, the Hapac2 is logging events for more as 3 hours now. This is already an hour longer as before (max 2 hours).

I will keep an eye on the Mikrotik splunk environment to see if everything keeps running.

Jotne · Wed Jan 23, 2019 9:11 pm

Interesting. Have not used much time in my splunkd.log, but have the same problem as you,
But only in one of 4 routers. Other are ok.

Tried bot 19 and 23 but still get samme message.

01-23-2019 20:08:40.136 +0100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event (Wed Jan 23 20:08:39 2019). Context: source=udp:514|host=193.1.1.100|syslog|

zandhaas · Wed Jan 23, 2019 10:41 pm

At my site it was 1 out of 3 that failed and I was missing information for that router.
Are you also missing data?
After the change my failing router is still visible in Splunk so for mee it seems the solution.
But I did not check the log files that come from the routers. Do you now were I can find them?
Perhaps it has something to do with too many events during a short time period.

We need to debug this.

Regards Peter

Jotne · Thu Jan 24, 2019 8:12 am

I do get event from all routers. To see if you get from one specific router use search and type host=1.2.3.4 (change to your IP)

Egert143 · Fri Jan 25, 2019 3:38 pm

Hello

Could i get instructions how to create splunk source type manualy ? I have splunk light (paid) and it doesent support apps (as far as i know).

Current problem is that source and dest addres fields are merged with port numbers.

Jotne · Fri Jan 25, 2019 10:07 pm

I have no idea on how to use Splunk Light.
In normal Splunk, source type based on the source it comes from udp:514

props.conf

[source::udp:514]
TRANSFORMS-force_mikrotik = force_mikrotik

transforms.conf

[force_mikrotik]
DEST_KEY =  MetaData:Sourcetype
REGEX =  \sMikroTik:\s
FORMAT =  sourcetype::mikrotik

Egert143 · Sat Jan 26, 2019 10:07 pm

And how would i turn 123.123.123.123:1234->12.34.45.67:80 to Source Address = 123.123.123.123 Source Port = 1234 Dest Address = 12.34.45.67 Dest Port 80 So they would be searchable ?

Jotne · Sat Jan 26, 2019 11:02 pm

The traffic solution are based on that you have private ip inside your net and public on the outside.
Private IPv4 addresses
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

But if you like to log other IP and know what is inside/outside, you have to modify the Splunk files.

Edit:
MikroTik Traffic
Replace all

 | search (ip_in="10.0.0.0/8" OR ip_in="172.16.0.0/12" OR ip_in="192.168.0.0/16")

with

 | search ip_in="12.34.45.0/8"

That if you like 12.34.45.0/8 to be your inside net.

JieYu2001 · Fri Feb 01, 2019 8:41 am

Where can I find the link to download MikroTik2.6 spl? Thanks.

zandhaas · Sat Feb 02, 2019 9:47 pm

In the first post of this topic

Or the below link
download/file.php?id=35231

JieYu2001 · Mon Feb 04, 2019 9:40 am

Thanks zandhaas - I got it downloaded.

I also installed everything per our topic owner Jotne's procedure but cannot get the data flow from MikroTik to Splunk, after verifying port 514 is open. Upon diving into some details, I suspect it's due to the lack of SSL of my MikronTik (192.168.88.1 shows "Not secure") - anyone know if this is the root cause? If yes what is the easiest way to enable SSL under RouterOS v6.40.8 and Win10? Appreciate any tips there.

Jotne · Tue Feb 05, 2019 10:55 am

There are no certificates involved in the transaction. All data are sent using UDP/514 Syslog (not encrypted).

In Splunk search, type only a * and do a search for the last 24 hour. Do you see any data at all?
Make sure you follow all steps in the first post 1 by 1.
Do you have any deviation? Using a clean Splunk install? Windows firewall opened if you run on Windows?

JieYu2001 · Wed Feb 06, 2019 7:16 am

Thanks Jotne. Now I see the data (events) through the Splunk search, though MikroTik2.6 app still not sees the data yet and I am still debugging.

BTW the Splunk observed event entry looks like - do you see any anomaly there?

2/5/19
9:09:54.000 PM
Feb 5 21:09:54 router.lan Feb 5 21:09:54 MikroTik MikroTik: Router = 192.168.88.1
host = router.lan
source = udp:514
sourcetype = mikrotik

Jotne · Wed Feb 06, 2019 8:22 am

Can you post some example line from search in Splunk that shows what you got in the log from using * search?

Do you have tagget all packet with MikroTik? This will fail Mikrotik since its not the same name.

JieYu2001 · Thu Feb 07, 2019 6:19 am

Hi Jonte, here're three snapshots

1. Splunk Event entry sample from the MikroTik UDP feed - great if you can help review the "Host", "Source", "Sourcetype" field to see if they are right for the MikroTik2.6 App

Splunk Event Entry from UDP and MikroTik.png

2. Splunk UDP input setting

Splunk UDP Input Setting.png

3. MikroTik2.6 App snapshot (system change search, with no data found while the Splunk search gives items like above)

Splunk MikroTik 2.6 App Lauch Snapshot.png

Thanks

Jotne · Thu Feb 07, 2019 8:27 am

1. Is this the only type of event you see?

Here are some example on how they should look like: (various modules)

firewall,info MikroTik: NAT_Web_server dstnat: in:ether1-Wan out:(unknown 0), src-mac 00:05:00:01:00:01, proto TCP (SYN), 91.12.58.49:49145->92.220.200.251:80, len 60
dhcp,debug,packet MikroTik:     Parameter-List = Subnet-Mask,Router,Domain-Server,Domain-Name,NETBIOS-Name-Server,Static-Route
dns,packet MikroTik: --- sending reply to 10.10.10.244:53720:
script,info MikroTik: script=health voltage=24 V temperature=42 
wireless,info MikroTik: 04:62:73:xx:xx:21@wlan1 established connection on 2437000, SSID GjestenettHMN
ipsec MikroTik: invalied encryption algorithm=6.
interface,info MikroTik: ether1 link up (speed 100M, full duplex)

Have you followed tutorial in post#1?
Do you use Splunk for other stuff?

JieYu2001 · Thu Feb 07, 2019 10:26 am

Hi I followed your first post but skipped 2c~2e (FW/NAT/Traffic logging since not sure about the detailed steps). I did have Home Monitor app before that affected the MikroTik data inputs, and I have it removed so the data inputs seems right (though not complete if without 2c~2e). The question I have is that, even with incomplete but valid data (say only DHCP request part), should MikroTik2.6 App see them and populate some view right? But now it seems the app does not pick up anything and I am not sure if the app has access to the log. Thanks.

Jotne · Thu Feb 07, 2019 11:52 am

You should get DHCP and other stuff from the router if you skipped 2c-2e.
Thats why I asked about how the log lines looks like.
You could use a search for host=192.168.88.1 and post some line.

JieYu2001 · Mon Feb 11, 2019 9:17 am

Thanks again Jotne. Here's a screenshot. Seems the Splunk events have the right contents, but the format is different from yours.

Splunk MikroTik 2.6 Event Snapshots.png

Basically, before the identifier "MikroTik", there are timestamps and another "MikroTik", but without the log field name like "dns,packet" as in your snapshots.
I copied the MikroTik scripts exactly, so do you think I missed something on the Splunk side? My Splunk version is 7.2.3.

zandhaas · Mon Feb 11, 2019 9:44 am

Are you sure your "router script" is complete?

I had problems getting my data visible in splunk to.
It turned out that I missed the last "}" in the Router script.

JieYu2001 · Mon Feb 11, 2019 10:04 am

Hi Jotne ~ some progress - for some reason, the "Module" field picks up part of the timestamp (the Month) since their is no syslog field name for some reason (the event item format difference I mentioned). After tweaking the Volt/Temperature code (removing the module key from the search), I was able to get that view right. Encouraged and will see how to get the module field right in the first place - help appreciated.

Splunk MikroTik 2.6 Volt_n_Temp.png

Larsa · Mon Feb 11, 2019 12:04 pm

Since I'm not a Splunk expert I wonder if anyone has some bright ideas how to optimize Splunk / Mongodb?

We have about 15.5 million entries and the reports are getting really slow to produce. In a regular SQL database you can run a "Query Execution Plan" and then add indexes to columns that performs table scans. Is there an equivalent way in Splunk or any other way to optimize the environment? We're running Splunk with 12 cores, 20 Gb ram and SSD which ought to be sufficient.

Any suggestions are welcome!

Jotne · Mon Feb 11, 2019 6:41 pm

@Larsa
Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.

I do recommend that you start a thread about your problem over here:
https://answers.splunk.com/index.html

Jotne · Mon Feb 11, 2019 6:51 pm

@JieYu2001

There are some wrong with extraction of the data in the Splunk or the format that your MT Router sends it.
In List view in Splunk your should not see time and date in the Event space, only in Time column.
In your view, I do not see it only one time extra, but two times in front of the data. This breaks all view.
You get it to work since you adjusted to view to accept your wrong data.
source=udp:514 and sourcetype=mikrotik looks correct.

I would recommend you to start over.
Clean Install of Splunk, remove all connection to Splunk in your router.

@zandhaas
You do not need the script to get data inn to splun, so it could also be removed to rule out problems.

JieYu2001 · Tue Feb 12, 2019 9:26 am

Thanks Jotne - the issue is resolved. In the MK Logging setting, I checked "BSD Syslog" which caused issue (still don't know why since that is the correct syslog protocol supported in Splunk). Uncheck it and things look fine now.

Jotne · Tue Feb 12, 2019 3:08 pm

There was nothing in the first post telling you to select it so not sure why you did it.
Will update post #1 to say not to select it.
Good you find out what was wrong

Larsa · Tue Feb 12, 2019 9:07 pm

Not sure if I could help with this. But when you have a lot of data, its sometime better to do a summary indexes that is based of for example 1 hour reports. Then you get less data to search trough.I do recommend that you start a thread about your problem over here: https://answers.splunk.com/index.html

Thanks for the suggestion, I'll report back if I find out an appropriate solution!

oaas · Fri Feb 15, 2019 5:41 pm

Great work!

Had some issues with parsing messages from one cAP ac where the messages suddenly dropped due to "Failed to parse timestamp" warning messages.

Seems it got solved by adding

TIME_FORMAT = %b/%d/%Y %H:%M:%S

to the props.conf file.

Please consider adding this to future releases.

/Thanks

frankcale · Sun Feb 17, 2019 11:28 am

Hi, Can u pls help with displaying Vlan info

Jotne · Sun Feb 17, 2019 11:59 am

Not sure what you asks for.
A list of Vlan on the router?
Traffic going trough Vlan?

frankcale · Sun Feb 17, 2019 4:59 pm

Hi, Can u include vlan traffic monitoring and if possible protocols like youtube, torrent, updates, etc

Jotne · Sun Feb 17, 2019 7:30 pm

Protocol are complicated to monitor due to https, near to impossible.
Vlan can be monitored used SNMP or you can use script and syslog to send data.

Jotne · Tue Mar 05, 2019 12:19 pm

Updated 1a to mention that you need an account at splunk.com to download software.
Account is free to create.

ithelp · Wed Mar 06, 2019 6:18 am

Hi, thanks for this magnificent explanation.
Can you give me on how to see the PPP and PPPOE information from the log?
I've already configure it on the rules tab, but nothing shows on any dashboard.
Thanks,

Jotne · Wed Mar 06, 2019 8:09 am

I do not have PPP nor PPPOE so I can not easily make log for it.

But if you could post 3-4 pages of logs that involves PPP and PPPOE output I could have look at it.

neutronlaser · Sat Mar 16, 2019 8:07 pm

Price is ridiculous.

Jotne · Sat Mar 16, 2019 9:40 pm

500MB/day for free is ridiculous much to pay.

But I do agree that if you pay retail price for Splunk and need eks 500GB/day, price is high.

Halfeez92 · Mon Apr 29, 2019 9:25 am

Hi how can I remove the MikroTik device list in the splunk dashboard view? I have multiple same devices showing up because I forgot to disable NAT and enable routing. Now it have 2 same devices with different IP

Jotne · Mon Apr 29, 2019 1:14 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:

your search | delete

PS this just mark data as deleted so they does not who up in logs. It does not remove any data.

Halfeez92 · Mon Apr 29, 2019 7:08 pm

I am not sure what you mean. All MT send their IP when sending syslog, not the identity name.
So if you select the host drop down in each view, it shows what IP logs comes from.

If its data that are already been logged in splunk you like to remove, do a search for what to remove and then add delete.
Like his:
Code: Select all
your search | delete
PS this just mark data as deleted so they does not who up in logs. It does not remove any data.

Ok thanks for the help. Already delete the duplicate device.

Jotne · Mon Jun 10, 2019 5:49 pm

Updated section 2c regarding Log prefix.

NB Do not use more than 20 charters, or else it start to clip other part of the log

firewall,info MikroTik: 123456789012345678901234567890 : in:ether1-Wan ...
firewall,info MikroTik: 1234567890123456789012345 forwa: in:ether1-Wan ...
firewall,info MikroTik: 12345678901234567890123 forward: in:ether1-Wan...
firewall,info MikroTik: 12345678901234567890 forward: in:ether1-Wan ...

As you see here the chain word forward is eat'n up by the prefix.
MT is this a bug???
If not, set a warning in the gui

Jotne · Thu Jun 13, 2019 1:54 pm

Updated section 2f)

Script updated to collect and show how many dynamic/static address lists entry there are.
Eks output

script,info MikroTik: script=address_lists list=rdp_stage2 dynamic=24 static=0
script,info MikroTik: script=address_lists list=rdp_stage1 dynamic=28 static=0
script,info MikroTik: script=address_lists list=ftp_stage2 dynamic=1 static=0
script,info MikroTik: script=address_lists list=ftp_stage1 dynamic=1 static=0
script,info MikroTik: script=address_lists list=black_list_rdp dynamic=42 static=0
script,info MikroTik: script=address_lists list=black_list_ftp dynamic=1 static=0
script,info MikroTik: script=address_lists list=Whitelist_IP dynamic=3 static=2
script,info MikroTik: script=address_lists list=Router dynamic=0 static=1
script,info MikroTik: script=address_lists list=IPSEC dynamic=1 static=0
script,info MikroTik: script=address_lists list=FW_Block_user_try_unkown_port dynamic=1089 static=0
script,info MikroTik: script=address_lists list=Clients dynamic=0 static=2
script,info MikroTik: script=address_lists list=Blocked dynamic=1 static=7

This will later be used in its own graph to see variation in the lists.

PS only one IP en the ssh black list black_list_ssh is due to that I do not use default port.

You can update script only and wait for new Mikrotik Splunk app to be updated later.

zandhaas · Thu Jun 20, 2019 9:59 am

Hello Jotne,

I want to upgrade my Splunk version 7.2 environment tot Splunk 7.3

Is the mikrotik app compatible with Splunk 7.3?

Jotne · Thu Jun 20, 2019 1:43 pm

Yes, I do try to not use anything special in the APP so it should be compatible with all new version.

Jotne · Fri Jun 21, 2019 9:29 pm

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.

pidde · Sun Jun 23, 2019 2:59 am

Hi!

Must say you did a great work with this app!
Is it possible to add option82 to dhcpserver part?
And is it also possible decode the option82 from hex?

zandhaas · Tue Jun 25, 2019 10:34 am

Updated section 2f)

Updated script to v2.4 and fixed reserved DHCP leases to be taken inn to account.

When I look at the current script under 2f I only see the "# Collect DHCP Pool information" part.

It seems the rest of the script is missing.

Jotne · Tue Jun 25, 2019 1:09 pm

You are 100% correct. Copy past error.

Fixed.

PS It's getting closer to the release of v 2.7 of Splunk for MikroTik

Jotne · Fri Jun 28, 2019 2:10 pm

Script to get information on the router is upgraded to 2.6 section 2f

Simpler DHCP calculation.
Fixed comment so it start on the beginning of the line.
Fixed Script names

Jotne · Mon Jul 01, 2019 1:15 pm

Upgraded to 2.7

There are a lot of new changes to the app as listed below, so its a larger upgrade.
Simplest way to upgrade, if you have not made changes your self, remove (uninstall) previous version, install new version.
Please report any problems back to this thread, and I will try to fixed.

PS If you do upgrade, you also need to upgrade script in section 2f (fist post) on all router you like to get data from.
Just cut/past the script over the old one.

PS2 File is found under section 1g first post

Request to changes are also welcome

What new:
# 2.7 (01.07.2019)
# New view added "Address Lists Counters"
# Changes most view to use "Base Search"
# Changed "MikroTik DHCP request" to use stats and fixed host flaw
# Changed "MikroTik System Changes" to use 30 day and 4 hour span and maxspan in transaction
# Removed changes to "DHCP leases" in "MikroTik System Changes"
# Added search in dropdown for "MikroTik DNS Live usage"
# Added Time picker for "MikroTik Device List"
# Speeded up "MikroTik Remote Connection"
# Fixed wrong timestamp of packets logged
# Changed "MikroTik DHCP request" to use stats and fixed host flaw and maxspan in trnsaction
# Added search in dropdown for "MikroTik DNS Live usage" and added IP to client and change sorting
# Fixed "MikroTik DNS request" to use correct dropdown lists
# Fixed "MikroTik Firewall Rules" to use better searh, removed base level, added counters, long prefix
# Rewritten "MikroTik Live attack" to speed up and added more dropdown
# Fixed "MikroTik Resources" to give correct host number
# Changed "MikroTik System Changes" to use 30 day and 4 hour span, removed DHCP info
# Fixed "MikroTik Traffic" to use script= and some clean up
# Fixed "MikroTik uPnP" script name, added ip to dropdown
# Added to ">MikroTik Uptime" dropdown menu
# Fixed "MikroTik Volt/Temperature" sorting
# Fixed "MikroTik VPN Connection" faster search
# Fixed "MikroTik Web Proxy" sorting and some code clean up
# Changed "MikroTik Wifi strength" to use script tag and some clean up
# Added "dashboard.css" to set menu color global
# Fixed "props.conf" to better handel wrong prefixed and some other changes

fengyuclub · Wed Jul 03, 2019 5:36 am

I have been paying attention to this post, very powerful chart, but the cumbersome construction and the lack of relevant knowledge have been unsuccessful. I can only temporarily use the mrtg icon inside routeros to temporarily cope with it. I hope the poster can write the deployment manual from the perspective of the technology-poor. .

Jotne · Wed Jul 03, 2019 3:17 pm

Its written so that a user with some knowlege should be able to set it up.
You can start by telling me what your problem is, and we may be able to help you out.

fengyuclub · Sat Jul 06, 2019 6:50 am

Reinstalled splunk on ubuntu18.04, is a virtual machine under esxi, the deployment is very simple and normal, according to the steps of the top post, but the splunk dashboard can not see the task data incoming. Very strange, what else do I need to pay attention to? Please forgive my English using Google Translate, I am from China

1.png

Jotne · Sat Jul 06, 2019 10:43 am

After starting Splunk, go to Search & Reporting menu. Add following search:

sourcetype=mikrotik

and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.

fengyuclub · Mon Jul 08, 2019 12:57 pm

After starting Splunk, go to Search & Reporting menu. Add following search:
Code: Select all
sourcetype=mikrotik
and set last 24 hour.
Do you then see any data?
If not try to just use a * (star) and last 24 hour.
If you do not see any data, make sure
Router is sending data to correct IP/Port.
Splunk is listening on correct IP/port
No local firewall (Windows/Linux) are blocking incoming data.

According to what you said carefully, but still can not receive the data, I introduced the cdb1016 log file db format, can be displayed to splunk, indicating that splunk no problem, is the data input problem, I see ros is the log The output is udp514 port, but I only see tcp listening port settings in splunk's receiving settings. Is this the reason?

1.png

2.png

3.jpg

Jotne · Mon Jul 08, 2019 1:36 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.

fengyuclub · Tue Jul 09, 2019 5:51 am

There is no local listening udp514, now there is data in, but click on the meter in the Mikrotik2.7 dashboard, most of them do not have any charts, how to add or customize the dashboard you need here, for example, I want The wan's real-time or past and downstream traffic in a certain period of time, as well as the system temperature, the number of online hosts, and so on. How to do it?

Jotne · Tue Jul 09, 2019 8:29 am

514 UDP do need to be active
Do you run it on Linux?

If so, as Root, type:

netstat -opan | grep 514

You should see one line like this:

udp        0      0 0.0.0.0:514             0.0.0.0:*                           23557/splunkd        off (0.00/0/0)

if not UDP/514 is not running.

One the mikrotik, post the output of:

/system logging export

You should see some like:

# jul/09/2019 07:26:37 by RouterOS 6.43.16
# software id = E4B6-94N8
#
# model = RouterBOARD 750G r3
# serial number = xyz
/system logging action
set 3 remote=ip_your_syslog_server
/system logging
set 0 disabled=yes
add action=remote prefix=MikroTik topics=dhcp
add action=remote prefix=MikroTik topics=hotspot
add action=remote prefix=MikroTik topics=!debug

There should be IP for your server, and prefix for all action with MikroTik. If one letter is wrong in the prefix, it will fail. See capital M and T in the MikroTik.

fengyuclub · Thu Jul 11, 2019 1:03 pm

It’s true that I set it wrong, Mikrotik changed to MikroTik, and it should be fine, then I will report it.

haaroons · Thu Jul 11, 2019 1:32 pm

Hello Jotne,
I am new to this forum.

I have install MikroTik logs 2.7.

MikroTik DNS Live usage and MikroTik DNS Live request is not working. if i do search eventtype=dns_query No item found

Do advice how to fix this.

Jotne · Thu Jul 11, 2019 11:43 pm

DNS information are coming from standard logs on the router.

What do you get if you go to search window and search with the following line:

sourcetype=mikrotik earliest=-24h latest=now() | stats count by module

I do get some like this:

module		count
dhcp		12764
dns		324512
firewall	1349
ipsec		7
script		91182
upnp		308

fengyuclub · Fri Jul 12, 2019 6:02 am

The data is coming, some of the tables are already filled, some still have no data, such as dns, it doesn't matter, I want to know how to monitor the flow table of an interface (wan), just like mirkrotik's built-in mrtg chart, every 5 minutes, 30 minutes and so on. . . As shown

1.png

Jotne · Fri Jul 12, 2019 8:05 am

That is why I need the output of the above command.
Some data are coming from the logg.
Some are comming from scripting

Log:
-------
dhcp,dhcp_static,dns,firewall,ipsec,upnp

script:
-------
IPSEC_failed,address_list,healt,pool,resource,sysinfo,traffic,uncounted,upnp

So I guess you have some log problems. Read section 2b carefully.

fengyuclub · Fri Jul 12, 2019 11:42 am

Splunk is too powerful. If I have multiple ccr1016, how can I transfer data to the splunk server, how do I distinguish syslogs from different mikrotik routers?

Jotne · Fri Jul 12, 2019 1:29 pm

All the view for MikroTik in Splunk has a host drop down. So if you have more than one router, just select the host you like to monitor.
There is one possible problem, if you have many routers with same IP that sends log to same Splunk.
That could be solved using unique ID for each router and some small change to the code.

fengyuclub · Mon Jul 15, 2019 6:24 am

How can I write the interface tx-bits-per-second parameter to the log and then plot it in splunk.

Jotne · Mon Jul 15, 2019 8:06 am

What command do you use on the router to see this data?

fengyuclub · Mon Jul 15, 2019 10:07 am

What command do you use on the router to see this data?

interface monitor-traffic ether1

Search forums see scripts with such calls

  "/interface monitor-traffic ether1 once do={
:put ($"tx-bits-per-second"/1000 /1000 )
}"

Jotne · Tue Jul 16, 2019 8:12 pm

It can be done.
I do use IP accounting to see the traffic going trough the router.
This way are more generic and does work without any modification.
If you monitor one and one interface, this has to be adopted for each setup.

fengyuclub · Fri Jul 19, 2019 5:03 am

{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX (($speedRX/1000)/1000);
:set $mbpsTX (($speedTX/1000)/1000);
:put "$iname RX:$mbpsRX Mbps TX:$mbpsTX Mbps";
}
}

I found the script for this post available, but after running it is all interfaces, I don't want all interfaces, only a few interfaces are needed, for example, I only need ether1, ether2, how to modify the script, and how can I get it? Let him display in the log, I use the splunk search call, and display it as 14.5Mbps instead of 14528. I hope to get everyone's help.

Jotne · Fri Jul 19, 2019 9:29 am

Info
It seems that data you get from monitor are just moment blink of data going through the interface. So it will fly up and down for every time you run it. If it would be like cisco, average last 5 min, it would be perfect to rune every 5 min. Not sure if it are useful at as is.

If you have not renamed interface

:foreach interface in=[/interface find] do={

To

:foreach interface in=[/interface find where (name~"^ether1\$" || name~"^ether2\$") ] do={

or use regex

:foreach interface in=[/interface find where name~"^ether[12]\$" ] do={

Anchor ^ \$ are used to distinguish ether1 from ether11 etc.

Edit
You can use ID instead of name, so you can change from:

:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];

to

:set $monitor [/interface monitor-traffic $interface as-value once]

PS2, no need to declare variables, use them directly
do not divide data by 1000 two times, let splunk do that, so you do not loose any resolution
use equal sign for splunk to read data directly
you do not need semicolon behind each line ;

So final script could be some like this

:foreach interface in=[/interface find where name~"^ether[12]\$"] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}

fengyuclub · Fri Jul 19, 2019 11:47 am

Your script, the regular expression method, no success without any output, it doesn't matter, my code is as follows
I want to know how to search for rich charts, and there are 14.8Mbps and 14833 display problems. This is not important. The important thing is how splunk draws charts.

mycode

{
:local iname;
:local monitor;
:local speedRX;
:local speedTX;
:local mbpsRX;
:local mbpsTX;
:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "WAN-ether2 down RX=$mbpsRX Kbps";
:log info "WAN-ether2 up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"adsl-tx") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "adsl-tx down RX=$mbpsRX Kbps";
:log info "adsl-tx up   TX=$mbpsTX Kbps"
}
:foreach interface in=[/interface find where (name~"bonding1") ] do={
:delay 100ms;
:set $iname [/interface get $interface name];
:set $monitor [/interface monitor-traffic $iname as-value once];
:set $speedRX ($monitor->"rx-bits-per-second");
:set $speedTX ($monitor->"tx-bits-per-second");
:set $mbpsRX ($speedRX/1000);
:set $mbpsTX ($speedTX/1000);
:put "$iname RX=$mbpsRX Kbps TX=$mbpsTX Kbps";
:log info "bonding1 down RX=$mbpsRX Kbps";
:log info "bonding1 up   TX=$mbpsTX Kbps"
}
}

After the schedule is displayed as follows

1.png

fengyuclub · Fri Jul 19, 2019 12:26 pm

Tested on other ccr1016 your script is successful, it should be the problem of the interface name, but it is important to draw the splunk graphics, I hope you can add to the new version.

3.png

Jotne · Fri Jul 19, 2019 1:38 pm

When you have multiple interface, use only one section, no a section for every interface

change

:foreach interface in=[/interface find where (name~"WAN-ether2") ] do={

to

:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={

Test code that should output data to screen:

{
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:put "script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}
}

PS, when testing cut and past on the cli, you need to wrape all script in brackets {} !!!

PS how often would you like to run the script? every 5 min. Do you know if monitor could show average 5 min data?

Jotne · Fri Jul 19, 2019 1:58 pm

Try this

Add this to the Data_to_Splunk_using_Syslog script

# Get interface data (test)
# ----------------------------------
:foreach interface in=[/interface find where (name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1")(name~"WAN-ether2" || name~"adsl-tx" || name~"bonding1") ] do={
	:delay 100ms
	:local iname [/interface get $interface name]
	:local monitor [/interface monitor-traffic $interface as-value once]
	:local speedRX ($monitor->"rx-bits-per-second")
	:local speedTX ($monitor->"tx-bits-per-second")
	:log info message="script=monitor interface=$iname RX=$speedRX bps TX=$speedTX bps"
	}

Then in Splunk do this search for the last 4 hour.

sourcetype=mikrotik script=monitor| timechart avg(RX) as RX avg(TX) as TX by interface limit=10

May take some time to nice graphs.

zandhaas · Fri Jul 19, 2019 7:06 pm

Nice,
I have added the additional script entries and changed the inteface names to the names I use.
But............
The sourcetype entry in the search entry schould be "sourcetype=MikroTik"

Jotne · Fri Jul 19, 2019 7:32 pm

In Splunk, search ignore case

Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.

zandhaas · Fri Jul 19, 2019 9:06 pm

Even if this works, I like better the view in Splunk MikroTik Traffic, that uses accounting for creating the graphs.
There you can see who is generating the traffic, compare to only see what interface traffic goes in/out.

The current "Mikrotik Traffic" overview is indeed a nice oveview.
But apart from knowing who is generating the traffic I am very interested in the amount of traffic that floats over each individual interface. And especially the WAN interface(s) and ISL interfaces. And when you see a bottleneck on one of your interfaces you can drill down to your traffic overview to identify the source of all that traffic.

fengyuclub · Sat Jul 20, 2019 5:46 am

Jotne，Great, I did it according to your script, and the beautiful chart shows normal. I tried to add scripts to my multiple ccr and routerboards, so my interface has a lot of duplicate names, such as bonding1 and bridge1, how can I distinguish between them, or change the name for each interface.

4.png

fengyuclub · Sat Jul 20, 2019 5:59 am

Understand, add host=x.x.x.x in front of the search statement you gave to open my ccr and rb.

fengyuclub · Sat Jul 20, 2019 7:10 am

Host=x.x.x.x Although this option is available, some devices have an internet connection that is a dynamic ip obtained by adsl dialing. So before the log warning, add an identity=xxxxx to distinguish the mikrotik device. After testing, it is feasible and runs very well.

Jotne · Mon Jul 22, 2019 10:24 am

@ fengyuclub
Nice to see you are getting it to work.

@ All
Section 2c) Logging prefix has been updated with sample on how to name to logs.

Jotne · Mon Jul 22, 2019 3:44 pm

Script in section 2f) updated to 2.9

It now support to get interface counters and you can also set modules true/false if you do not like to monitor one section.
If you do not have wifi/dhcp, you can just set them to false.

Jotne · Wed Jul 24, 2019 8:23 am

Script in section 2f) updated to 3.0

Do now get CDP neighbors

fengyuclub · Thu Jul 25, 2019 12:46 pm

Splunk is really powerful, I see splunk have a lot of apps to install, in our China use wechat (similar to facebook, telegram) this social software, I saw this social software related app, WeChat Alert App for Splunk, I installed this App, sending test messages from wechat is successful, but I don't know much about splun's alert settings, set it many times, only a single success, can you help me?

5.png

Jotne · Thu Jul 25, 2019 1:44 pm

No need for extra app to send message. Sending email using a gmail account is easy and works well.

But there is a big issue.

If you have a free Splunk license, you do loose a lot of thing.
* Monitor and Alerting (needed for sending alerts)
* 500MB pr day maximum
* Cluster
* Universal Forwarder
* HA
* Distributed Search
* Perfomance Acceleration
* Access controll (only on user)
* LDAP
+++

This is why I have not included any Alerting in the project.

There is a workaround. You can setup an batch job that runs search from command line and do stuff from it. (I have not tested it)

fengyuclub · Sat Jul 27, 2019 4:56 am

I have Splunk Enterprise license, gmail alert can't be real-time, mobile mail client can't update mail in real time, there is a delay of about 10 minutes, so I choose wechat alert.I received some wechat alert, but some of them use the search to save as an alert, I can't receive a wechat alert, I don't know where the problem is.

6.jpg

7.jpg

Jotne · Sat Jul 27, 2019 8:30 am

Splunk do handle real time alerts (or close to)
https://docs.splunk.com/Documentation/S ... TimeAlerts
It should not depend of type of action you are using, starting a program, sending sms, email, wechat etc. Alerts should go out.
But you should not use to many alerts, since it will use more CPU to handle them.

Not sure what your problem is.

Jotne · Fri Aug 02, 2019 10:20 am

Updated script to 3.1

Fixed CDP, since some devices sends long version with new lines breaking up the log lines. (Cisco)

PS still have problem that line is cut in Splunk. Not sure if its MT not sending whole line, or Splunk that cuts the lines.
I do only get 278 characters.

antispam · Tue Aug 06, 2019 3:34 pm

Using 2.7, it's mentioned that the "defconf: drop all not coming from LAN" rule should have the prefix 'FI_D_port-test'. When I set that, the Live Attack dashboard doesn't populate as it appears from the source in the dashboard it is searching for 'FW_Drop_all_from_WAN'. When changing this to 'FI_D_port-test' in the Live Attack dashboard source, it works. Is the FW_Drop_all_from_WAN still required?

Jotne · Tue Aug 06, 2019 4:24 pm

The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:

<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test

Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik

fengyuclub · Wed Aug 07, 2019 9:39 am

Recently, one of my ccr is a bit problematic. I can only recover from the time when the device is powered off and restarted. The top of the log in winbox can see red like "system, error System rebooted because of kernel failure" or "Out of memory condition was detected", but I can't see it in splunk search. These log messages, how can I output all the log information to splunk for easy query.

Jotne · Wed Aug 07, 2019 1:12 pm

A search like this should give all message:

sourcetype=mikrotik module=system

IF not try this:

sourcetype=mikrotik

Or at last just this

antispam · Sat Aug 10, 2019 6:04 am

The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
Code: Select all
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik

Thanks for the prompt reply, that was exactly what I did to get it fixed - keep up the great work!

stuartkoh · Sat Aug 10, 2019 2:28 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.

One thing to be careful of if you're setting this up in an existing Splunk environment - unless you're really familiar with how things are setup, don't enable Splunk's UDP/514 input without first checking that syslog isn't already being received by something like syslog-ng or rsyslog. You could wind up with data loss or have events put into the wrong index or sourcetype.

It's also not best practices to run Splunk as root. For home use I guess you can get away with it, but for any production Splunk environment you will want to have Splunk running as a restricted user (user = splunk and group = splunk is commonly used).

When you install Splunk, you can set it to autostart on boot and also set the user if you want.

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

For receiving syslog - I'm more familiar with syslog-ng, but I also see rsyslog being used successfully. Either one of these will work well for you.

Jotne · Sat Aug 10, 2019 5:32 pm

This is already mention in section 1b)

If you install Ubuntu, (i think from 16.x), rsyslog is installed as default. But its not listening on port 514/UPD as default and you need to edit the config and restart syslog to get it running. So it should normally not be any conflict.

But in production environment I do also recommend running Splunk as a non root user, then use rsyslog to listen on 514/UDP. Then make Splunk index rsyslogs config.

If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.

stuartkoh · Sat Aug 10, 2019 8:31 pm

If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.

I think that syslog-ng has an option that can be used to do this.

keep-timestamp()
Description: Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.

https://www.syslog-ng.com/technical-doc ... -timestamp

stuartkoh · Sat Aug 10, 2019 8:50 pm

I also wanted to note that I'm not advocating that anyone switch from rsyslog or whatever they're currently using to syslog-ng unless they have good reason to do so.

I don't even really have an opinion on how they compare. I've been working with syslog-ng a bit so that's what I'm familiar with. I'm not trying to start a flame war over which to use.

Spotegg · Thu Aug 15, 2019 1:39 am

Hello! I have installed Splunk with Mikrotik module. Thanks, it's great!
Is there a way to organize monitoring of Internet connection on the router. For example, there is an Internet channel on ether1, and you need to somehow download data to Splunk about when the Internet crashed on the router. Maybe there is already such a script?

ferdytao · Thu Aug 15, 2019 4:10 pm

Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.

Script: manuel_export_dhcp_splunk

:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";

This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?

ferdytao · Fri Aug 16, 2019 12:21 pm

Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.

Script: manuel_export_dhcp_splunk

:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";

This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?

Resolved!

ferdytao · Sat Aug 17, 2019 5:35 pm

I'm having a strange problem both on my synology with docker and my windows pc also, I configured everything as described, I got many logs from my router but after a while it stops reading while counters are still increasing.
Checking via tcpdump, logs are arriving to the server but is like they are not processed.

Immagine.jpg

Someone could help me? Maybe I got something wrong, I cannot image the server is flooded by a single router.

Jotne · Sat Aug 17, 2019 5:53 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.

ferdytao · Sat Aug 17, 2019 7:19 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.

Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing

Immagine.jpg

Jotne · Sat Aug 17, 2019 10:03 pm

What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.

ferdytao · Sat Aug 17, 2019 10:34 pm

What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.

Yes the script is installed, I had the 30 min windows search and no data are showed even when the script starts. The stranger thing is that I have the same problem running Splunk on two different system

Immagine.jpg

Jotne · Sun Aug 18, 2019 8:39 pm

Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window

"hello"

Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :

echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514

Linux should respond some like this:

Connection to 127.0.0.1 514 port [udp/syslog] succeeded!

On Splunk, you should get a message like this:

Aug 18 19:32:53 127.0.0.1  test hello

If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?

ferdytao · Sun Aug 18, 2019 11:12 pm

Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
Code: Select all
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
Code: Select all
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Code: Select all
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Code: Select all
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?

Thanks for all your help and your time, it's very strange as you said... I opened syslog on port 5014 udp, i also checked the file props.conf on MikroTik app match the port.
I also tried as you said with echo "hello" and it's working.

I think there is some problem with timestamp because then it stops collect i got some error in internal index:

08-18-2019 19:09:43.054 +0200 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Sun Aug 18 01:28:00 2019). Context: source=udp:5014|host=192.168.1.1|syslog|

I checked around on google but I didn't find a solution

Jotne · Mon Aug 19, 2019 8:21 am

You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.

ferdytao · Mon Aug 19, 2019 9:30 am

You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.

It's a real riddle!

The server is running as a root on my last test with ubuntu but nothing is changed. Now I just changed the port from 5014 to default 514 (I used 5014 because on my synology is already kept) and seems to be working. I will keep it running to see what happen!

Thanks for all your support and time spent!

ferdytao · Tue Aug 20, 2019 12:19 am

Finally it's working!
Just uncommented the option: MAX_TIMESTAMP_LOOKAHEAD = 23 "inside /opt/splunk/etc/apps/MikroTik/default/props.conf"

Jotne · Tue Aug 20, 2019 8:27 am

For me it then sounds like you have a time problem. Its important that all clock is synced by using NTP.
Look at time on your router and on Splunk server. It should be within the same second.

Or date stamp is in the wrong format. Not sure how you can get this, since MT sends it in correct format.
You sends UDP directly to Splunk, not using rsyslog or syslog-ng etc?
Something adding data to your UDP packets?

From the manual

MAX_TIMESTAMP_LOOKAHEAD = <integer>
* Specifies how far (in characters) into an event Splunk should look for a timestamp.
* This constraint to timestamp extraction is applied from the point of the TIME_PREFIX-set location.
* For example, if TIME_PREFIX positions a location 11 characters into the event, and MAX_TIMESTAMP_LOOKAHEAD is set to 10, timestamp extraction will be constrained to characters 11 through 20.

So this sets where to look for the time stamp.

stuartkoh · Sat Aug 24, 2019 5:00 pm

After I posted this reply I realized that I goofed.

I didn't read the post I replied to very well. I looked at the event you posted and just looked at the timestamp at the beginning of it and completely missed that it was a Splunk internal log.

So you can probably ignore what I had posted shown below about how to handle the timestamp in your logs. It does show how to setup timestamp detection, but for the wrong logs.

What I originally posted is below:

Finally it's working!
Just uncommented the option: MAX_TIMESTAMP_LOOKAHEAD = 23 "inside /opt/splunk/etc/apps/MikroTik/default/props.conf"

I think you could probably change it to MAX_TIMESTAMP_LOOKAHEAD = 29

The timestamp in your events is "08-18-2019 19:09:43.054 +0200" and Splunk needs to know that all of that is part of the timestamp in order to parse things properly. If you use 23 for the lookahead, Splunk may not catch the timezone.

So maybe:

MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_PREFIX = ^
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%3N %z

stuartkoh · Sat Aug 24, 2019 5:05 pm

BTW, Packt is giving away a free copy of a decent book on Splunk today. https://www.packtpub.com/free-learning

Implementing Splunk 7 - Third Edition

James D. Miller

Mar 2018

576 pages

What will you learn

Enrich machine-generated data and transform it into useful, meaningful insights
Perform search operations and configurations, build dashboards, and manage logs
Extend Splunk services with scripts and advanced configurations to process optimal results

You have to sign up for an account with them, but I've had one for several years and they haven't been obnoxious about sending me too much e-mail or anything. The offer expires later today (in about 10 hours from when I'm posting this) so I hope people will see this in time to take advantage of this.

My only connection with Packt is as a customer. I've bought some of their books and gotten some good free ones from them too.

stuartkoh · Sat Aug 24, 2019 5:24 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing

Sometimes Splunk will mistakenly identify the timestamp of an event as coming from the future because it misinterprets the timezone (or the sending device has the incorrect time set and actually is telling Splunk the events are from the future.

So it can be useful to run a search with "All time" selected in the time picker. If there are events "from the future" they will show up this way. You can also run an "All time (real time)" search and see events as they come in and see what the timestamps are. Just be careful with real-time searches on Splunk because they use up a lot of resources. So I only use them when I really need to, and I stop them as soon as I have what I want.

(There are some times when you need to have a real-time search running all the time for very time-critical alerting, but it's normally better for performance to change a real-time search into a scheduled one running every 5 minutes or whatever. Each real-time search basically ties up a CPU core, so if you have several real-time searches running at once you can see poor overall performance on your search heads.)

I don't know if you're getting events "from the future" or not, but it might be worth checking.

stuartkoh · Sat Aug 24, 2019 5:29 pm

For me it then sounds like you have a time problem. Its important that all clock is synced by using NTP.
Look at time on your router and on Splunk server. It should be within the same second.

Yes! What Jotne said!

It is extremely important to make sure the clocks on your devices and your Splunk servers are all in sync. Having time properly synced is very important for logging in general, but especially so for things like Splunk where you'll want to correlate events from different devices.

Jotne · Sat Aug 24, 2019 9:28 pm

I think you could probably change it to MAX_TIMESTAMP_LOOKAHEAD = 29

The timestamp in your events is "08-18-2019 19:09:43.054 +0200" and Splunk needs to know that all of that is part of the timestamp in order to parse things properly. If you use 23 for the lookahead, Splunk may not catch the timezone.

So maybe:

MAX_TIMESTAMP_LOOKAHEAD = 29
TIME_PREFIX = ^
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%3N %z

If you see props.conf in the MikroTik app, the MAX_TIMESTAMP_LOOKAHEAD = 23 line is not active due to # in front of it
So you should only change this if it does not work.
Still not sure why you need to sett it. Maybe you have other settings that influence on how Splunk handle MikroTik data.

ferdytao · Sun Aug 25, 2019 11:00 am

Thanks everyone for the answers, It's seems to be working since 7 days now so I prefer to not changing anything!

I just uncommented the line MAX_TIMESTAMP_LOOKAHEAD = 23 and the time zone is also working properly, to be sure I also set the NTP to all devices and nothing else.

If I will have some time to spend, I will try to manage another install of splunk to try the LOOKAHEAD of 29

jacauc · Thu Aug 29, 2019 4:52 pm

I created a dropdown to select rules on the Live Attack View. All my mikrotik prefixes start with either Allow or Drop - Such as DropPings, AllowDNS etc.
Now I can quickly see the entire overview of my rules.

The updated dashboard code is below:

<form theme="dark">
  <label>MikroTik Live attack</label>
  <description>Shows form where ip comes from that tries to access your router on ports that are not opened using rule "FW_Drop_all_from_WAN"</description>
  <!--Version 
  2.4.0
  2.5.1 Change to "coalesce"
  2.7.1 Change to base searc
  2.7.2 Revritten to speed up and added more dropdown
  ######################################################
  #
  # Mikrotik Add-On for Splunk
  #
  # Copyright (C) 2019 Jotne
  # All Rights Reserved
  #
  # v2.7
  #
  ######################################################
  -->
  <search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule="*"
      | table _time host dest_port src_ip protocol rule
      | iplocation src_ip
      | replace "" with "Unknown" in City,Country
      | fields - Region
      | search 
          host="$Host$"
          protocol="$protocol$"
          dest_port="$port$"
          Country="$Country$"
          rule="$Rule$"
    </query>
  </search>
  <fieldset submitButton="false" autoRun="true">
    <input type="time">
      <label>Time span</label>
      <default>
        <earliest>rt-5m</earliest>
        <latest>rt</latest>
      </default>
    </input>
    <input type="dropdown" token="Host">
      <label>Host</label>
      <search base="base_search">
        <query>
          | eval data=host
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="protocol">
      <label>Protocol</label>
      <search base="base_search">
        <query>
          | eval data=protocol
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="port">
      <label>Port</label>
      <search base="base_search">
        <query>
          | eval data=dest_port
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Country">
      <label>Country</label>
      <search base="base_search">
        <query>
          | eval data=Country
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>*</default>
    </input>
    <input type="dropdown" token="Rule">
      <label>Rule</label>
      <search base="base_search">
        <query>
          | eval data=rule
          | stats count by data
          | eval info=data ." (".count.")" 
          | sort -count
        </query>
      </search>
      <choice value="*">Any</choice>
      <choice value="Allow*">AnyAllow</choice>
      <choice value="Drop*">AnyDrop</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>data</fieldForValue>
      <default>Drop*</default>
      <initialValue>Drop*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <map>
        <search base="base_search">
          <query>
            | eval info=src_ip."-".City."-".Country."-".protocol."/".dest_port
            | geostats globallimit=0 count by info
          </query>
        </search>
        <option name="height">600</option>
        <option name="refresh.display">progressbar</option>
      </map>
    </panel>
  </row>
</form>

2019-08-29_15-50-50.jpg

Jotne · Sat Aug 31, 2019 10:45 am

Interesting.

I may implement this in ver 2.8

jacauc · Sat Aug 31, 2019 11:01 am

Interesting.

I may implement this in ver 2.8

Please do!
Do you have a beta of 2.8 yet, or is it imminent for release?

Jotne · Sat Aug 31, 2019 10:29 pm

I am always working with some changes, and release new version when I have spare time to do it

Here is the working list for 2.8
# 2.8 (xx.xx.2019)
# Added interface changes
# Updated script to 2.7, added uncounted traffic and fastrack test, fixed when missing temperature
# Updated script to 2.8, fixed where system healt does not show anything (x86)
# Updated script to 2.9, get interface counters and you can also set modules true/false
# Updated script to 3.0, get CDP neigbhours
# Added new view "MikroTik Neighbor"
# Added uncounted packages to "MikroTik Traffic"
# Added board_name to "MikroTik Device List"
# Added Mikrotik interface Traffic"
# Fixed l2tp user extraction
# Fixed VPN when user info is missing
# Added access rule dropdown to "Mikrotik Live attack"

keattikun · Fri Oct 18, 2019 3:29 pm

Hi Jotne
I'm try to install Mikrotik2.7 app on splunk. It not working for me, and I able to search log on splunk that send from MK but in app show "No results found" as capture

Please help me to investigate

Jotne · Fri Oct 18, 2019 10:17 pm

You are 100% sure you have 2b correctly?
Click on Search
In search line add only a * (star) and search last 24 hour. Do you see any data?

jvanhambelgium · Wed Nov 13, 2019 5:36 pm

Hi Jo,
Great set of dashboards! All working just fine.
In order to customize them further, when can I take out some complete dash-boards ? (pressing "Dashboards" in Splunk gives you the overview)
Don't need any Wireless or Upnp stuff and I would like to get it off the list.
Under the actions-button there is no such as a "Delete" choice.

EDIT : Ok, figured it out after untarring the app that I needed to edit some XML.

jvanhambelgium · Fri Nov 15, 2019 7:03 pm

Jotne，Great, I did it according to your script, and the beautiful chart shows normal. I tried to add scripts to my multiple ccr and routerboards, so my interface has a lot of duplicate names, such as bonding1 and bridge1, how can I distinguish between them, or change the name for each interface.
4.png

Hi can you give some hints on how to do that in Splunk (I'm running 8.0.0 FREE) ? Whatever graph I make, it seems to have a rolling time-line on the X-axis every 10-15 seconds and each 5 minutes the TX and RX values are plotted but not like this. I need to the X-axis to be fixed, 5-minutes increments to match the input coming from the script.

Jotne · Fri Nov 15, 2019 7:38 pm

You need to have unique naming or else data will be joined.
What you can do is to use some values and join them together.
Eks using host-name port-number interface name etc and create a new join name.

| eval newifname=host."-".ifname

Without seeing the real problem its not easy to give a solution.

blinderix · Mon Dec 02, 2019 10:01 pm

Jotne, Greetings for the good job!
I'm new to Splunk and keep reading.
I'm curious how to make the drop down list with host IP's to show the identity of the router instead the IP address.
Is it possible with some kind of lookup or token?

Regards,

blinderix · Thu Dec 05, 2019 7:08 pm

No idea why it stops working.
I have run Splunk for years without it stopping by it self, always using Ubuntu Server.
Maybe there are some other stuff/software on you Ubuntu that kills it.

Finally there is a problem with timestamp, found on splunkd.log
08-01-2018 15:52:07.610 +0300 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Wed Aug 1 02:35:00 2018). Context: source=udp:514|host=192.168.1.1|syslog|

And it stops logging.

This is an example of what mikrotik sends to me, on another syslog server
192.168.1.1 Jul 12 00:17:05 firewall,info DROP INPUT input in:pppoe-WAN out:(unknown 0), proto TCP (SYN), 79.129.108.120:41236->79.129.36.201:7547, len 44

Any work around ?

No matter the question is old, I had the same problem. The logs from some routers stopped and had to restart the splunk each time.
I had the same timestamp error in the log file.
Yesterday I changed in props.conf : DATETIME_CONFIG = CURRENT and till now, almost 24h, everything runs smoothly.

Jotne · Sat Dec 07, 2019 10:56 am

2.8 Finally released.

# 2.8 (07.12.2019)
# Added interface changes
# Updated script to 2.7, added uncounted traffic and fastrack test, fixed when missing temperature
# Updated script to 2.8, fixed where system healt does not show anything (x86)
# Updated script to 2.9, get interface counters and you can also set modules true/false
# Updated script to 3.0, get CDP neigbhours
# Added new view "MikroTik Neighbor"
# Added uncounted packages to "MikroTik Traffic"
# Added board_name to "MikroTik Device List"
# Added Mikrotik interface Traffic"
# Fixed l2tp user extraction
# Fixed VPN when user info is missing
# Added access rule dropdown to "Live attack"
# Added some color and more field to "DHCP Request"
# Fixed Span and removed time scal "System Changes"
# Fixed missing mac in "Admin Conection"
# Added more lines and color in "DHCP request"
# Added output color in "Firewall Rules"
# Added rule menu to "Live attack"
# Added "DATETIME_CONFIG = CURRENT" to props.conf

Jotne · Sat Dec 21, 2019 4:51 pm

Tested with latest Splunk 8.0.1 and it works fine.

jo2jo · Sun Dec 22, 2019 10:19 am

wow, thanks so much for all the time and effort you have put into this! it really is an excellent splunk app, thank you!

I do have your splunk app working on both of our splunk servers, when i add the "MikroTik" prefix to remote logs for a few test mikrotiks.
(our splunk servers are not the free version, but are licensed)

However we have been using splunk for a few years, and have many (over 200+) mikrotiks already sending lots of data into splunk via syslog, so im trying to avoid having to winbox into each of them and add a duplicate remote rule (with your "MikroTik" prefix).

Our mikrotiks are all configured to send logs into splunk like this:
(config from the splunk side)
udp 8002 data_input goes into splunk index "mikrotikFirewall"
udp 8004 data_input goes into splunk index "mikrotikOther"

Each of our many mikrotiks does have a prefix set (ie /system logging add action=remoteSplunk prefix=+LOCATION-NAME-FW- topics=firewall ) - (note the "+" before location-name) we ofcourse uses the prefixes to help our own manual searching within splunk

my question: Would you please point me in the right direction of what i would need to modify so that your app/views searches splunk entries with our prefix, as opposed to your "MikroTik"

Things i have tried / looked into:
\Splunk\etc\apps\MikroTik\default\props.conf - Changing the 1st line, [source::udp:514] to [source::udp:8002] - (made no difference, so i changed it back)

\Splunk\etc\apps\MikroTik\default\transforms.conf - Changing the [force_mikrotik]... REGEX = \s\+[a-zA-Z0-9\-\:]+\s - ( this regex, as all of our log prefixes do start with "+" , this also made no difference)

In splunk web gui: data Inputs->UDP -> udp:8002 , setting the source type to "mikrotik" (from our initial source_type setting of "syslog") - also did not get desired result.

(keep in mind that i did try these things together, and separately, and was restarting splunk server after any changes).

(i would have no problem setting up a 3rd, new free license splunk just for using your app, however our mikrotik splunk intake alone is much more than 500mb/day currently)

thanks again!

Jotne · Wed Dec 25, 2019 5:19 pm

Since I do use the tag to add sourcetype to the log entry, I think its there you should start.
Change all dashboard from using: sourcetype=mikrotik to use your index like this: index=mikrotikOther or index=mikrotikFirewall

kenkilaw · Sat Mar 07, 2020 8:10 am

Good job !! I'll test it, thanks for sharing Jotne !!

silverblade · Thu Mar 12, 2020 6:53 pm

Evening, everybody. One question, use splunk 8 and latest version of the app. I would like to have the results with the name of mikrotik instead of ip or dns. Can you explain please? Thank you

Jotne · Thu Mar 12, 2020 7:07 pm

Not sure what you mean. Can you give some example?

silverblade · Fri Mar 13, 2020 11:38 am

Not sure what you mean. Can you give some example?

So, I always set the identity in my mikrotik but in splunk I see the ip or dns address. I'd like to see the identity on the menu or else I don't know who it is directly. If it's possible... Thank you.

Jotne · Sat Mar 14, 2020 12:25 am

This is how Syslog works.
It sends log data with standard message from the MikroTik Router like this:

system,info,account MikroTik: user xxx logged out from 10.10.10.32 via winbox
system,info MikroTik: dhcp lease changed
system,info MikroTik: dhcp lease changed
system,info MikroTik: static dns entry changed by xxx
system,info MikroTik: static dns entry changed by xxx
system,info MikroTik: device changed by xxx
system,info MikroTik: device changed by xxx

It does not contain any identifier only IP.
I see three ways this can be solved.
1. You add DNS record for your IP in your local DNS server.
2. Change the "MikroTik" tag to some like "MikroTik Device=Router58". I have not tested it, so not sure what would break

3. Use the data from "MikroTik device list" in a subsearch to get the device name.

Jotne · Tue Mar 17, 2020 10:29 am

So, I always set the identity in my mikrotik but in splunk I see the ip or dns address. I'd like to see the identity on the menu or else I don't know who it is directly. If it's possible... Thank you.

I have a solution for this in v2.9.
It reads the "MikroTik Device List" and store it inn to a KV store file every night. Then this data i later used in other view.

valeex87 · Thu Apr 02, 2020 1:49 pm

Hi, how can i make a script on mikrotik for check user that connects/disconnect in ovpn?
On splunk in "MikroTik VPN Connection" everything is clear and not populated, ovpn works correctly but on splunk i cant find whats my ip and the user who connected.

Thanks

Jotne · Thu Apr 02, 2020 6:48 pm

Hi
I have not used ovpn, so have not looked at it. If it logs other stuff, you you then send me a copy of the logs when a user logs inn and out using ovpn.

Do a search like this

sourcetype=mikrotik

or copy the logs from Mikrotik directly

Jotne · Wed Apr 08, 2020 1:57 pm

Script updated to version 3.3

Added script info and NTP status.
Backward compatible so you can just overwrite previous version.

Jotne · Mon Apr 13, 2020 11:23 am

Script updated to 3.4

Gives more correct transfer statistics pr interface.

Jotne · Tue Apr 14, 2020 1:34 pm

Splunk for MikroTik is updated to 2.9

# 2.9 (14.04.2020)
# Added KVstore DB to handel device info (identity)
# Fixed "MikroTik DNS Live usage" showing wrong count
# Fixed "MikroTik DHCP request" duplicate entity and cleaned up table
# Renamed "Mikrotik Admin connection" to "MikroTik Admin user login"
# Removed "_time" for X-Axes in dashboards
# Added NTP warning to "MikroTik System Changes"
# Added auto lookup of host to identity
# Added option to select CPU/Disk/Memory in "MikroTik Resources"
# Rearanged the menu layout
# Moved various test to the "device list" page and set the page as default

Mayor change is the new Menu layout and the KVstore. The DB makes the devices show up with their identity name and not just the IP.
This version also needs the latest version of the script (ver 3.4+) installed.

To upgrade, just download the file in the first post (section 1g), unrar and install. You can expand the tar ball and copy the files manually.
Script should be updated on the routers as well.

Jotne · Wed Apr 15, 2020 8:55 pm

Script updated to 3.5

More compatible with Router OS v7.0 that has a problem with some function that does not work like:

/ip accounting uncounted {
	:log info message=("script=uncounted,bytes=".[get bytes].",packets=".[get packets])

And

:local model ([/system routerboard get model])
:local serial ([/system routerboard get serial-number])

This may be due to v.7 does run on a X86 VM. 6.47beta on VM does not fail, so its a but in v7.

jesuraja · Sun May 10, 2020 2:42 pm

Interface Traffic Graph not sync.It's nothing. How can i troubleshoot to resolve

Jotne · Mon May 11, 2020 8:45 am

Do you see any data in other graphs?

Jotne · Mon May 11, 2020 3:34 pm

An serious error found in the script in 2f. If NTP module is missing, the whole script fails. Fixed in new version 3.6.

verpeilo · Wed May 13, 2020 3:16 pm

Hello!

I've setup as described in your tutorial.
But unfortunately there is no data available in Splunk.
After detailed investigations I have found, that there is no prefix "MikroTik" in log messages, even I defined in logging rules:

loggingrules.png

Rebooting the Routerboard did not help as well.
Splunk is completely empty...

Do you have any ideas?

/edit: Sorry, my fault. I was a little bit blind...
/solved

oaas · Wed May 13, 2020 3:38 pm

There seems to be a kind a catch 22" situation with the "new" KVstore DB.

The KVstore fails to initially poplate as field extraction by "device lookup updater" is failing to auto extract the fields "identity, serial, model, board_name, version" while the lookup "automatic_device_lookup" is enabled. This results in missing information for "identity, serial, model, board_name, version" in the main "Device List" view (and other places).

After I disabled the lookup "automatic_device_lookup" the "device lookup updater" executed successfully populating the KVstore DB. In the end I enabled the "automatic_device_lookup", and view functionality seems to be working as expected.

Jotne · Wed May 13, 2020 3:48 pm

You are 100% correct.
I did fix this by using coalesce

| eval identity=coalesce(identity,host)

Version 3.0 will be out soon with this fix and som other stuff as well.
PPPoE logging
IPv6 IP support in firewall

verpeilo · Wed May 13, 2020 3:48 pm

There seems to be a kind a catch 22" situation with the "new" KVstore DB.

The KVstore fails to initially poplate as field extraction by "device lookup updater" is failing to auto extract the fields "identity, serial, model, board_name, version" while the lookup "automatic_device_lookup" is enabled. This results in missing information for "identity, serial, model, board_name, version" in the main "Device List" view (and other places).

After I disabled the lookup "automatic_device_lookup" the "device lookup updater" executed successfully populating the KVstore DB. In the end I enabled the "automatic_device_lookup", and view functionality seems to be working as expected.

Same here.
Where do you disable it?

Jotne · Wed May 13, 2020 3:58 pm

Can you try to run i manually.

Splunk-app:Mikrotik->Reports->Click on name "device lookup updater"

Do you get any information? (Try it twice)
If not, what do you get

Splunk-app:Mikrotik->Reports-> "device lookup updater" Open in Search

verpeilo · Wed May 13, 2020 4:11 pm

I only get the host column filled - everything else is empty

Jotne · Wed May 13, 2020 4:16 pm

What about this search?

sourcetype=mikrotik
            module=script
            script=sysinfo
| dedup host
| rex "version=\"(?<version>[^\"]*)\" board-name=\"(?<board_name>[^\"]*)\" model=\"(?<model>[^\"]*)\" serial=(?<serial>\S*) identity=\"(?<identity>[^\"]*)\"" 
| table  host identity serial model board_name version

verpeilo · Wed May 13, 2020 4:52 pm

This works great!

oaas · Wed May 13, 2020 6:18 pm

May I also suggest you include the following in future release:
MAX_TIMESTAMP_LOOKAHEAD = 23

I see others are also refering to issues with this earlier as well, and I'm having the exact same issue with one of my devices. Unless I add this the device (cAP ac) disappears from syslog parsing after a few hours.

verpeilo · Wed May 13, 2020 6:53 pm

Is it "normal" that Splunk is quite slow?
I have it running since 4-5 hours and the Traffic Accounting Dashboard takes 2 minutes until some results are shown...

jvanhambelgium · Wed May 13, 2020 7:43 pm

Is it "normal" that Splunk is quite slow?
I have it running since 4-5 hours and the Traffic Accounting Dashboard takes 2 minutes until some results are shown...

No that seems not normal...
The "Traffic" dashboard show here in about 2 seconds when selecting "Time Range = last 4 hours". I run Splunk Enterprise 8.0.1 (FREE Edition, so limited at 500MB/day indexing) on a VM on my Synology NAS, gave it 4vCPU & 2GB RAM
What are you running Splunk on ?

If you go to "Settings" on the top and then select "Monitoring Console" what stats do you see ?

Jotne · Wed May 13, 2020 9:01 pm

MAX_TIMESTAMP_LOOKAHEAD = 23

Added in the upcoming 3.0

verpeilo · Thu May 14, 2020 9:30 am

Is it "normal" that Splunk is quite slow?
I have it running since 4-5 hours and the Traffic Accounting Dashboard takes 2 minutes until some results are shown...
No that seems not normal...
The "Traffic" dashboard show here in about 2 seconds when selecting "Time Range = last 4 hours". I run Splunk Enterprise 8.0.1 (FREE Edition, so limited at 500MB/day indexing) on a VM on my Synology NAS, gave it 4vCPU & 2GB RAM
What are you running Splunk on ?

If you go to "Settings" on the top and then select "Monitoring Console" what stats do you see ?

I have Splunk Enterprise 8.0.3 FREE Edition as well.

I run it on a Windows Server 2019 Standard.
4 Intel CPUs, 16GB RAM.

I can install it was well on a Linux System to compare the performance, but I don't think that the architecture has an impact like this.

stats_splunk.png

verpeilo · Thu May 14, 2020 10:19 am

Update regarding the performance:

I installed Splunk on the same Windows Server, but in a Hyper-V Ubuntu 18.04.4 environment with only 2 CPUs and 2 GB of RAM.
==> Now it is very fast.

Seems that Splunk in Windows is slow... Hmm.

However, I am happy with this Linux VM.

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.

Jotne · Thu May 14, 2020 10:42 am

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.

I do recommend using Linux for Splunk. Its created for Linux, later exported to work in Windows.
My install are usually done on Ubuntu server.

It would be possible to read the comment from an address and store and use it in Splunk, but not sure how this will work with large access lists.
My router blocks all that tries any non open port for 24 hours to access any port on my router. This creates an access list that can be between 2000 to 15000 addresses. So sending this every 5 minutes would be a big chunk.

verpeilo · Thu May 14, 2020 11:55 am

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.
I do recommend using Linux for Splunk. Its created for Linux, later exported to work in Windows.
My install are usually done on Ubuntu server.

It would be possible to read the comment from an address and store and use it in Splunk, but not sure how this will work with large access lists.
My router blocks all that tries any non open port for 24 hours to access any port on my router. This creates an access list that can be between 2000 to 15000 addresses. So sending this every 5 minutes would be a big chunk.

Maybe you can configure it customizable if the comments should be read or not...
It would be very helpful.

/edit: Maybe you can build a function which determines the comment in DHCP leases

Jotne · Thu May 14, 2020 3:52 pm

If you do use Splunk as a non root (recomended) user, you need an external Syslog server.
This setup needs Splunk for Mikrotik v3.0 to read field correctly. (out soon)

This is how to set it up using Ubuntu server. Should work on most version.

rsyslog comes default with Ubuntu so no need to install any extra software.

PS do not modify these file to use other location. If you do so you will need to modify udp.conf rsyslog and input.conf splunk for every upgrade.

Copy these two files to /etc/rsyslog.d/

udp.conf (sets up rsylog to accept Sylog on udp/514)

# rsyslog.d/udp.conf
#
# This receives UDP syslog on port 514 and stores it in a reliable format
# in /data/syslog/udp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imudp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="udp_split_filename" type="list") {
  constant(value="/data/syslog/udp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="udp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="udp_split_filename"
  )
}

# setting
input(type="imudp" port="514" ruleset="udp_split")

tcp.conf (sets up rsylog to accept Sylog on tcp/1514)

# rsyslog.d/tcp.conf
#
# This receives TCP syslog on port 1514 and stores it in a reliable format
# in /data/syslog/tcp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imtcp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="tcp_split_filename" type="list") {
  constant(value="/data/syslog/tcp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="tcp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="tcp_split_filename"
  )
}

# settings
input(type="imtcp" port="1514" ruleset="tcp_split")

Create the following folders

mkdir /data
mkdir /data/syslog
mkdir /data/syslog/tcp
mkdir /data/syslog/udp

Change folder rights to syslog and restart rsyslog

chown -R syslog:syslog /data/syslog
service rsyslog restart

run ss or netstat as root user to see that rsylog is running

ss -tunlp | grep syslog
udp   UNCONN 0      0                                0.0.0.0:514        0.0.0.0:*     users:(("rsyslogd",pid=5532,fd=8))
udp   UNCONN 0      0                                   [::]:514           [::]:*     users:(("rsyslogd",pid=5532,fd=9))
tcp   LISTEN 0      25                               0.0.0.0:1514       0.0.0.0:*     users:(("rsyslogd",pid=5532,fd=6))
tcp   LISTEN 0      25                                  [::]:1514          [::]:*     users:(("rsyslogd",pid=5532,fd=7))

netstat -tunlp | grep rsyslog
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      1459/rsyslogd
tcp6       0      0 :::1514                 :::*                    LISTEN      1459/rsyslogd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1459/rsyslogd
udp6       0      0 :::514                  :::*                                1459/rsyslogd

To make Splunk reads the data, this input.conf file is needed %SplunkHome%/etc/apps/MikroTik/default
It is not included in current version 2.9, but will be included in 3.0. So you need to create it and restart Splunk.

[monitor:///data/syslog/udp/.../*.log]
sourcetype = mikrotik
host_segment=4

[monitor:///data/syslog/tcp/.../*.log]
sourcetype = mikrotik
host_segment=4

Test your server:

echo '<14>sourcehost message text' | nc -v -u -w 0 127.0.0.1 514

This should create a folder /data/syslog/udp/127.0.0.1 with a *.log file

verpeilo · Thu May 14, 2020 6:59 pm

For what is it?

Jotne · Thu May 14, 2020 7:39 pm

Did not understand the question.
My post above clearly stats that if you run Splunk as a non root user you need an external server to receive sysloge.
At least if you uses port below 1024, Splunk will not work. So Mikrotik send udp/515->rsyslog server <- Splunk reads the file.

Jotne · Fri May 15, 2020 9:08 am

Other question:
Is there an option, to give comments to the IP-Adresses in Traffic Throughput Monitor?
Sometimes the IPs are solved to Hostnames, but I would be happy to define comments as well.

What commend do you like?
Name of host are resolved when Splunk uses DNS to find its name.
There are several places a comment could exist.
* DHCP comments
* DNS comments
* IP address lists comments
All this can set comment for an IP

To get all information with comment for all ip addresses in the address list and send it using syslog:

{
	:foreach interface in=[/ip firewall address-list find] do={
		:local output [ip firewall address-list get $interface]
		:set ( "$output"->"script" ) "addresses"
		:log info message= "$output"
	}
}

markwien · Fri May 22, 2020 12:33 pm

If you do use Splunk as a non root user, you need an external Syslog server.
This setup needs Splunk for Mikrotik v3.0 to read field correctly. (out soon)

This is how to set it up using Ubuntu server. Should work on most version.

rsyslog comes default with Ubuntu so no need to install any extra software.

Copy these two files to /etc/rsyslog.d/

udp.conf (sets up rsylog to accept Sylog on udp/514)

# rsyslog.d/11_hemit_tcp.conf
#
# This receives UDP syslog on port 514 and stores it in a reliable format
# in /data/syslog/udp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imudp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="udp_split_filename" type="list") {
  constant(value="/data/syslog/udp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="udp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="udp_split_filename"
  )
}

# setting
input(type="imudp" port="514" ruleset="udp_split")

tcp.conf (sets up rsylog to accept Sylog on tcp/1514)

# rsyslog.d/11_hemit_tcp.conf
#
# This receives TCP syslog on port 1514 and stores it in a reliable format
# in /data/syslog/tcp/:
#
#  - A subdirectory is created for each logging host.
#    The directory name is the source IP address of the received log message.
#    The log filename is made up of date and hour of file creation.
#    A new log file is created for each logging host every hour.
#
#  - Each log message is prefixed with a locally generated timestamp.
#
#  - Then the raw, undecoded text of the incoming syslog message is written.
#    If the message is missing <PRI> then a default <PRI> is prepended.
#    This ensures that both old tyle (RFC-3164) and new style (RFC-5424)
#    syslog messages can be recognised and parsed from the log files without
#    risk of loosing information. The raw messages also include the original
#    host name and time stamp from the sender.
#
#  - Each written log message is guaranteed to end in a single newline
#    character.
#

# Stupidly this is set to "on" in the default rsyslog.conf file
# Do our best to negate the effect.
$RepeatedMsgReduction off

module(load="imtcp")


# format
template(name="RawFormat" type="list") {
  property(name="timegenerated" dateformat="rfc3339")
  constant(value=" ")
  constant(value="<")
  property(name="pri")
  constant(value=">")
  property(name="rawmsg-after-pri" droplastlf="on")
  constant(value="\n")
}

# file name
template(name="tcp_split_filename" type="list") {
  constant(value="/data/syslog/tcp/")
  property(name="fromhost-ip")
  constant(value="/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
  constant(value="-")
  property(name="$hour")
  #property(name="$minute")
  constant(value=".log")
}

# rule set
ruleset(name="tcp_split") {
  action(type="omfile"
    template="RawFormat"
    createDirs="on"
    dirCreateMode="0755"
    fileCreateMode="0644"
    dynaFile="tcp_split_filename"
  )
}

# settings
input(type="imtcp" port="1514" ruleset="tcp_split")

Create the following folders

mkdir /data
mkdir /data/syslog
mkdir /data/syslog/tcp
mkdir /data/syslog/udp

Change folder rights to syslog and restart rsyslog

chown -R syslog:syslog /data/syslog
service rsyslog restart

run netstat to see that rsylog is running

netstat -tuopan | grep 514
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      -                    off (0.00/0/0)
tcp6       0      0 :::1514                 :::*                    LISTEN      -                    off (0.00/0/0)
udp        0      0 0.0.0.0:514             0.0.0.0:*                           -                    off (0.00/0/0)
udp6       0      0 :::514                  :::*                                -                    off (0.00/0/0)

To maks Splunk reads the data, this input.conf file is needed %SplunkHome%/etc/apps/MikroTik/default (included in v3.0)

[monitor:///data/syslog/udp/.../*.log]
sourcetype = mikrotik
host_segment=4

[monitor:///data/syslog/tcp/.../*.log]
sourcetype = mikrotik
host_segment=4

Test your server:

echo '<14>sourcehost message text' | nc -v -u -w 0 127.0.0.1 514

This should create a folder /data/syslog/udp/127.0.0.1 with a *.log file

hi i made installation exactly as your tutorial, but syslog files dont get read by splunk.
i am wondering how user splunk can read /data/ dir if the syslog dir runs under user syslog ....

do i miss something?

Jotne · Fri May 22, 2020 1:58 pm

Splunk reads data since rights on the files are set by syslog so Splunk can read them and inputs.conf tells it to read them.

-rw-r--r-- 1 syslog syslog 2520311 May 22 12:56 20200522-12.log

The "r" in all three places makes all able to read the files.

To test, log in as the Splunk user:

sudo su - splunk

Do you see any folder/files under /data/syslog/udp ?
try to cat them

IF you do not see any files, there may be several problems
* Local firewall
* Syslog not set up correctly
* Nothing sending to udp/514

Splunk looks at the inputs.conf and see that it should read the files. (so if there are no inputs.conf in MikroTik/Default, you need to create it.
It will be included in upcoming v3.0

If all is setup correctly try in Splunk to search for

index=_internal source="*splunkd.log" NOT INFO

and see what errors/warnings you have.

PS no need to quote the whole post. Use "Post Reply" button below post to reply, or quote only needed part.

Jotne · Sat May 23, 2020 2:36 pm

Script updated to 3.8
Fixed so that some information only get collected every hour, even if scripts run every 5 min.
This is to not flod system with duplicate information.

Jotne · Sun May 24, 2020 1:19 pm

Script updated to 3.9

Gives better command history for Router OS v7 or newer.
Fixed better handling with NTP/SNTP information

PS script can be updated without update Splunk software.

Jotne · Sun May 24, 2020 11:52 pm

Did an change in what to debug settings in first post

Change from:

/system logging add action=logserver prefix=MikroTik topics=!debug

to:

/system logging add action=logserver prefix=MikroTik topics=!debug,!packet

This reduces overall DNS logging by 80%
So I do suggest that you add this, specially if you have a lots of DNS requests.

fmarais007 · Mon May 25, 2020 5:02 pm

Hi,

Thank you for all the hard work! This is really a nice tool.

Can you please assist.
I'm getting the following error. I'm running it on a Windows Server 2019 VM: Error in 'outputlookup' command: External command based lookup 'device_lookup' is not available because KV Store initialization has failed. Contact your system administrator.

Splunk-KV Store Failed-20200525.jpg

All the searches I do give me Linux commands to run, and I'm not sure if MongoD is even running?
Any help will be appreciated thank you.

Jotne · Mon May 25, 2020 6:36 pm

I have see KV store error due to certificate error on splunk. Other than that, I do not know what can be wrong. I do suggest that you install Splunk from scratch on an Ubuntu 18.04 or 20.04 server. Works every time I have tried.
I do use less than an hour to install Ubuntu/Splunk and Mikrotik plugin and get all up and running.

fmarais007 · Mon May 25, 2020 8:14 pm

Hi,

OK I will do that. Sucks that the only answer is to try another OS, but oh well.

Another thing, is it possible to monitor several MikroTik's via this system, or is it only designed for a single device?

Thanks

Jotne · Mon May 25, 2020 8:35 pm

It can monitor as many as you like. Only limit is the amount of data logged to Splunk. For free you get 500MB, that should be ok for a small to medium system, depending on what you select to log. DNS eats lots of log space.

What OS did you try? I do in first post recommend Ubuntu.
Ubuntu running on VmWare workstation on Windows server works fin as well, but I do recommend a dedicated server.

fmarais007 · Tue May 26, 2020 10:54 am

Hi,

No worries, I got it running on Ubuntu 20.04

So I once again tried to see the devices reporting but no data.
I then realized it only runs every morning at 01:00 so I tried to run it manually.
Even though I have two devices sending logs to Splunk successfully (I can see my firewall traffic, but can't filter by host), It's not showing any results:

Splunk-DeviceNoResults-20200526.jpg

No KV errors this time

How do I force the refresh of devices reporting to my server? I tried:

Splunk-ForceRun-20200526.jpg

If that is the correct method, why is it not appearing?
Thanks for all the help.

Jotne · Tue May 26, 2020 1:41 pm

I am working with v3.0 and there the kv search is updated. Try this:

This should just give the devices:

sourcetype=mikrotik
module=script
script=sysinfo
| dedup host

Then if that gives lines, run this updated version.

sourcetype=mikrotik
module=script
script=sysinfo
| dedup host
| rex "version=\"(?<version>[^\"]*)\" board-name=\"(?<board_name>[^\"]*)\" model=\"(?<model>[^\"]*)\" serial=(?<serial>\S*) identity=\"(?<identity>[^\"]*)\"" 
| table  host identity serial model board_name version
| fillnull value="-"
| outputlookup device_lookup

Rhoos · Tue May 26, 2020 8:21 pm

Hi Jotne,

Thank you for this magnificent work!
I have a small problem with the file "MikroTik2.9.spl.rar" for a few days I have tried without success to decompress the information, it always gives me an error using Winrar!
"The archive is either in unknown format or damaged"
Could you please check!
Thanks for your help and for the whole project.

Ricardo

Jotne · Tue May 26, 2020 8:46 pm

Downloaded the file from this forum and expanded it using 7-zip (Winrar should do as well), so file seems fine,

Rhoos · Tue May 26, 2020 10:55 pm

Thanks success, before making the query, use Winrar on two different computers, downloading a new file on each one without success, I followed your advice of "7-zip" and extracted it without any problem!
Ricardo

Jotne · Wed May 27, 2020 10:27 pm

Splunk for MikroTik updated to v3.0

Mayor changes is the PPPoE view and support for IPv6 in the MikroTik Firewall Rules module

To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9)

# 3.0 (27.05.2020)
# Fixed missing identity before device are logged to index
# Added PPPoE view
# Added IPv6 support for firewall
# Added support for using external rsyslog server
# Updated "Device List" to update KV-Store automatically when run
# Added dhcp_name to "MikroTik DHCP to Static"
# Change script and dashboard for "MikroTik Neighbor"
# Added more infor for the "Mikrotik Wifi connection"
# Updated script to 3.5, Better RouterOS v7 handling
# Updated script to 3.6, NTP info
# Updated script to 3.8, Change some colelction to get info every hour, not every 5 min (CDP++)
# Updated script to 3.9, to get better command history fro v7 RouterOS, fixed better NTP/SNTP handling
# Added minspan=5m to some graphs
# Fixed "MikroTik DNS Live usage" to work without dns packets log

fmarais007 · Fri May 29, 2020 9:14 am

Hello,

I was wondering if you can help with a previous post regarding CAPsMAN setup for your application.
Two of the charts uses the name of the interface for the data, but the problem comes in where if there are spaces in the interface it stops displaying the name before the space.
eg. if my interface name is "Client-A cAP-1" it only displays data for "Client-A", which means it misses data for all the other AP's.

This is the graph i'm referring to:

Splunk-CapGraphError-20200528.jpg

The chart uses the following search:

sourcetype=mikrotik counter>0  |chart values(counter) by name

Which gives these results when there are multiple AP's

Splunk-CapGraphErrorResult-20200529.jpg

And thereby displaying no results in the actual graph.

Would it be possible to either help me to define the search better, or perhaps integrate the OP's idea into your application?

Many thanks

Jotne · Fri May 29, 2020 9:25 am

This is some wrong:

sourcetype=mikrotik counter>0  |chart values(counter) by name

Buts not easy to fix when I do not see the real log data.
Sent you a private message.

jvanhambelgium · Wed Jun 03, 2020 10:37 am

Splunk for MikroTik updated to v3.0

Mayor changes is the PPPoE view and support for IPv6 in the MikroTik Firewall Rules module

To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9)

Hi,
Can 2 versions co-exist somehow ? I'm currently running the 2.7 version in my Spunk which is modified a bit.
Would be great if I load the app/dashboard and it sits next to the existing one as "Mikrotik 3.0" or something.

Thanks!

EDIT : Hmm, doesn't seem to be, in the SPL-file "Mikrotik" seems hardcoded somehow internally because even if I rename the SPL-file to let's say "Mikrotik3_0.spl" before loading the app it still says I already have it running...
EDIT2 : It was along time since I played with it, but I've untarred the SPL-files and adapted the app.conf so the "id" now is "MikroTik3" in stead of "MikroTik" and then tarred it again. Now it imports next to my existing 2.7-dashboard and all seems fine.

Jotne · Wed Jun 03, 2020 4:22 pm

You can copy all files you have modified to another folder.
Remove all MikroTik files, install 3.0, then restore your files.

Its also possible to use 7-zip/winrar to extract all the files from 3.0 manuall, then add one by one.

If your edit is interesting for other, you could send me them, and I could add it in v3.0

jvanhambelgium · Wed Jun 03, 2020 4:27 pm

If your edit is interesting for other, you could send me them, and I could add it in v3.0

I only removed some items not applicable for me at all, so no real enhancements.
I'm now using your supplied 3.0 Splunk Dashboard and it looks good enough for me ! I'm going to leave it as-is.
Thanks!

jvanhambelgium · Wed Jun 03, 2020 6:02 pm

I'm seeing some weird values on in the DHCP-section -> the "DHCP Pool Information" seems to give a faulty % value (eg. 450%)
Looking at the performed query for this :

litsearch (sourcetype=mikrotik module=script script=pool) | eval percent=used*100/total, host_name=coalesce(identity,host) | fields keepcolorder=t "_time" "host" "host_name" "percent" "pool" "total" "used"

There is something wrong with this "total" value. In my case, the pool is a defined /24 in size and almost everything is reserved.
Looking at the Mikrotik script output it only seems to be transmitting following to the Splunk -> script=pool pool=Pool 1 used=45 total=10

# Collect DHCP Pool information
# ----------------------------------
if ($DHCP and $run) do={
/ip pool {
:local poolname
:local pooladdresses
:local poolused
:local minaddress
:local maxaddress
:local findindex
# Iterate through IP Pools
:foreach pool in=[find] do={
:set poolname [get $pool name]
:set pooladdresses 0
:set poolused 0
# Iterate through current pool's IP ranges
:foreach range in=[:toarray [get $pool range]] do={
# Get min and max addresses
:set findindex [:find [:tostr $range] "-"]
:if ([:len $findindex] > 0) do={
:set minaddress [:pick [:tostr $range] 0 $findindex]
:set maxaddress [:pick [:tostr $range] ($findindex + 1) [:len [:tostr $range]]]
} else={
:set minaddress [:tostr $range]
:set maxaddress [:tostr $range]
}
# Calculate number of ip in one range
:set pooladdresses ($maxaddress - $minaddress)
# /foreach range
}
# Test if pools is used in DHCP or VPN and show leases used
:local dname [/ip dhcp-server find where address-pool=$poolname]
:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
:set poolused [:len [used find pool=[:tostr $poolname]]]
} else={
# DHCP server found, count leases
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
:set poolused [:len [/ip dhcp-server lease find where server=$dname]]}
# Send data
:log info message=("script=pool pool=$poolname used=$poolused total=$pooladdresses")
# /foreach pool
}
# /ip pool
}
}

Jotne · Wed Jun 03, 2020 8:50 pm

There may be some wrong with that part. Its on part of the script that is not made by me

For me it looks correct

/ip pool print
 # NAME                                                                    RANGES                         
 0 DHCP-Pool-vlan1-Home                                                    10.10.10.55-10.10.11.254

Then the script shows this:

script,info MikroTik: script=pool pool=DHCP-Pool-vlan1-Home used=158 total=455

Can you post the output of

/ip pool print