Community discussions

MikroTik App
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 9:39 pm

OK, I think we hit a special case here ;-)
Usually all my systems at home receive a reserved DHCP-entry ("lease"), so my "pool" is actually very small and your script is correct to this extend...
I forgot how small I made it.

The script summarizes ; script=pool pool=Pool 1 used=45 total=10

The pool indeed is actually only 10 IP's big...and the "used" are 45 leases I have statically configured for various devices based on their MAC.

[myuser@gateway] > /ip pool print
# NAME RANGES
0 Pool 1 172.29.45.190-172.29.45.200
[myuser@gateway] >


Then I think the Splunk logic needs to be addressed somewhat? The "used" in my case are not really a part of the start-stop range of the pool.
Splunk seems a bit confused perhaps with "45" used on a total of "10" ?


EDIT : I always believed I had to craft my pool "outside" the range I configure "static" for device. (which is > 90% of them). But apparently I can simply make my pool a real /24 (eg 192.168.1.2 -> 192.168.1.253 and the *.254 would be Mikrotik) and even in that space IP's that I configured "static" for certains MAC's would not be handed out. So perhaps the way I configured the DHCP is not really according to the way it should be) => Let me try to fix that by changing my pool first and test again...

EDIT2 : Yep it fixed it. I already experimented in Splunk by changing the "divider" [total] with a fixed value of 254 and then percentage indeed became more realistic (eg. 16%). So yeah, I will leave my pool configured as it probably should ,the whole /24 minus the gateway-IP.

Sorry to waste your time over this!
Last edited by jvanhambelgium on Wed Jun 03, 2020 10:05 pm, edited 1 time in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 10:01 pm

Interesting. I see this is a way you can handle DHCP, and it will confuse the system.
Its not easy to take inn to account every possibility.

In my work (20000 + computers 2500+ servers), we have only DHCP, and all server IP are within the DHCP scope. But we to convert DHCP leases to static for all that needs fixed IP. We found that this way will give less work for the team working with IP.

Will have a look at it, but not sure if its possible to solve. How to see if an reserved IP is within the DHCP scope or not?
[:len [/ip dhcp-server lease find where server=$dname]]
This part gets all lease the DHCP server and does not care where in the range they are.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 10:11 pm

In case you missed my edits ;

I always believed I had to craft my pool "outside" the range I configure "static" for device. (which is > 90% of them). But apparently I can simply make my pool a real /24 (eg 192.168.1.2 -> 192.168.1.253 and the *.254 would be Mikrotik) and even in that space IP's that I configured "static" for certains MAC's would not be handed out. So perhaps the way I configured the DHCP is not really according to the way it should be) => Let me try to fix that by changing my pool first and test again...

....Yep it fixed it. I already experimented in Splunk by changing the "divider" [total] with a fixed value of 254 and then percentage indeed became more realistic (eg. 16%). So yeah, I will leave my pool configured as it probably should ,the whole /24 minus the gateway-IP.

Sorry to waste your time over this!
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 03, 2020 10:15 pm

Sorry to waste your time over this!
No problem. You have not done anything wrong, just in another way. :)
I will add a comment about in the DHCP view, that if you add static release outside the pool,but within the subnet, i will give wrong number.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Thu Jun 04, 2020 8:24 am

Script updated to 4.0

Removed double stuff
Added write-sector-total

PS script can be updated without update Splunk software.

Here is an example view on write sector increase last 10 hour that will be included in Splunk for MikroTik 3.1
* 10.10.10.1 hEX 6.45.9 (will have a look at this after upgrade to 6.47, but will wait for at least 6.47.1
* 10.10.10.140 VmWare x86 6.47
* 10.10.10.153 VmWare x86 7.0Beta5
sector.jpg
You do not have the required permissions to view the files attached to this post.
 
anwarkollam
just joined
Posts: 1
Joined: Mon May 27, 2019 12:39 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Mon Jun 08, 2020 9:01 pm

I am facing issue. Spkunk stop logging after a while (around 1 - 2 hours). I tried with ubuntu and windows. Re installed so many times. Issue not solved yet. monitor will start, if i restart splunk service. So i schedule a cron job on ubuntu to restart splunk service every 30 min. Is there any other option to stable service? I have installed Ubuntu 16.04 on Vaphere 5.5.
Last edited by anwarkollam on Tue Jun 09, 2020 10:57 pm, edited 1 time in total.
 
mger
just joined
Posts: 1
Joined: Sat Jun 13, 2020 11:33 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 14, 2020 1:08 pm

tried 3.0 version with hAP ac2 mikrotik, routeros 6.47 , splunk enterperise v.8.0.4.1 on windows 10 v.2004
i don't get it why MikroTik DNS request shows only my my main computer's dns requests which are generated by services itself: nvidia,windows updates., teamviewer.,sharepoint,onedrive and nothing more what i search through web with my computer or mobile phone which are on the same network,

DNS Servers on my windows machine /
8.8.8.8
192.168.0.1 router dns
1.1.1.1

am i correct, computer must use only router's local DNS server or what?
EDIT: when i set only router's dns server it seems it now logs all websites i visit.

also mikrotik uptime always show "1" and only one dot..and volt/temperatures module no data is showing.
Last edited by mger on Sun Jun 14, 2020 6:10 pm, edited 2 times in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Thu Jun 18, 2020 12:19 pm

To see DNS loogs, your router needs to be the one and only DNS server.

Up time 1 is one day so it show 1. It also takes time (days) to get graphs for up time, so have a look after some days :)
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Thu Jun 18, 2020 6:43 pm

Did you ever considered extending your (already) very nice dashboard(s) with some NETFLOW information to gain more insights in the traffic + protocol distribution.
(bit like the "accounting" section on your dashboard, but with more info)
I'm currently playing around with the PMACCT-packages and writing out some CSV-style files. (other formats possible too like json)

I'm absolutely no Splunk expert, but I'm going to try to add such CSV (as test) to my splunk and visualize something from it.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Fri Jun 19, 2020 12:20 am

I'm currently playing around with the PMACCT-packages and writing out some CSV-style files. (other formats possible too like json)
This is interesting. CSV is perfect, and better than json since its smaller.
Splunk app do show traffic accounting using the accounting on the Router it self and sends it using syslog.

Problem with neflow is that it can not be sent with the Syslog packages, so we need to add a new port to listen to, not just one.

MT has decided to remove accounting on the router. Removed in v7 Beta 8
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Fri Jun 19, 2020 9:33 pm

Do you have experience with the "Splunk Stream" (app) ??

https://splunkbase.splunk.com/app/1809/

This could natively ingest & decode Netflow
""Capture Flow-type records, including NetFlow v5, v9, jFlow, and sFlow, and IPFIX, and send Flow Records directly into your Indexers, with optional filtering and aggregation.""

https://maddosaurus.github.io/2018/05/2 ... d-netflows

Sorry to pollute your Splunk topic with this.
 
colin
Frequent Visitor
Frequent Visitor
Posts: 74
Joined: Mon May 11, 2015 11:11 am

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 3:45 pm

Oh my god, why didn't I discover it until now. It's exactly what i want. Thank you so much.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 4:43 pm

Have not had time to look at much yet, but it look possible som complicated to set up. It have to much possibility, not sure of saved format is ok.
I would like a small program that listen for netflow and save them one line at a time. Then Splunk can index it.

System we have to day with just sending accounting data using syslog with rest of the data works quick and easy and no need for extra port etc.

But I will investigate it and see if its the road to go.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 6:13 pm

Have not had time to look at much yet, but it look possible som complicated to set up. It have to much possibility, not sure of saved format is ok.
I would like a small program that listen for netflow and save them one line at a time. Then Splunk can index it.

System we have to day with just sending accounting data using syslog with rest of the data works quick and easy and no need for extra port etc.

But I will investigate it and see if its the road to go.
Should be pretty straightforward but it depends a bit on what we want to achieve. Let's say for Netflow v9, Splunk created following ready-to-use fields.
I had installed the "app" from a (zip) archive that I downloaded and then needed to edit 2 config-files and make sure the "netflow" was enabled in the GUI config-part on your Splunk. The dashboard that come with the app for some reason do not handle "netflow" well and produce no usable results or give "no data"
If you really want a separate binary I think the PMACCT "packages", including the "nfacctd" (Netflow Accounting deamon" is a very solid one. With that you can produce CSV's and format what fields you want in there. But then you need to get these files also to your Splunk.

bytes_in: 3979
dest_ip: 172.217.168.206
dest_mac: 6c:3b:6b:20:22:b6
dest_mask: 0
dest_port: 443
endtime: 2020-06-21T14:54:04.840000Z
event_name: netFlowData
exporter_ip: 172.29.45.254
exporter_time: 2020-Jun-21 14:55:06
exporter_uptime: 811908030
flow_end_rel: 811846870
flow_start_rel: 811802820
input_snmpidx: 15
netflow_version: 9
nexthop_addr: 172.217.168.206
observation_domain_id: 0
output_snmpidx: 14
packets_in: 8
post_src_mac: 00:00:00:00:00:00
protoid: 17
seqnumber: 21718
src_ip: 172.29.45.4
src_mac: d0:50:99:84:01:36
src_mask: 0
src_port: 42751
tcp_flags: 0
timestamp: 2020-06-21T14:53:20.790000Z
tos: 0

I think the challenge is more

1) Getting & grouping results all together in a certain time-window so you can accurately calculate howmuch traffic was done per 1minute or 5minutes or so.
2) Depending on the amount of interfaces you collect Netflow from (eg. "all" Mikrotik interfaces vs only your PPPoE "Internet" interface) it might become confusing what flow is related to what direction. I've seen flows with a "dest_ip" of my WAN-IP (eg. DNS replies coming backup from Cloudflare or something) so they are part of a NAT transaction.

Like below, the dest_ip is my WAN and the field "nexthop_addr" (value 172.29.45.7) is effectively a PC on my LAN.
So I can image things get hairy if not counted correctly etc.

bytes_in: 88
dest_ip: 91.119.127.160
dest_mac: 00:00:00:00:00:00
dest_mask: 0
dest_port: 46924
endtime: 2020-06-21T15:04:23.570000Z
event_name: netFlowData
exporter_ip: 172.29.45.254
exporter_time: 2020-Jun-21 15:05:25
exporter_uptime: 812527030
flow_end_rel: 812465600
flow_start_rel: 812465600
input_snmpidx: 14
netflow_version: 9
nexthop_addr: 172.29.45.7
observation_domain_id: 0
output_snmpidx: 15
packets_in: 1
post_src_mac: 6c:3b:6b:20:22:b6
protoid: 6
seqnumber: 21863
src_ip: 13.59.106.231
src_mac: 20:e0:9c:6b:71:43
src_mask: 0
src_port: 30999
tcp_flags: 146
timestamp: 2020-06-21T15:04:23.570000Z
tos: 0
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 7:30 pm

I got it up and running. Some more complicated when Splunk do not run as an admin (what I do recommend to do).
Not sure why there are so many low number on source port like 443. That is normal destination port. Will examine it, make a SPL search that graph it and post it here.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 8:46 pm

In the meantime I searched some already existing dashboards and got some hits on Github.
I adapted the XML since my netflow is not sitting in the main-index and some of the names of the fields where different. etc.etc

However , there are some issues.
In 1 of these dashboard the field "bytes_out" is used which seems not existing in the Mikrotik v9 template. Only "bytes_in" exist.
I guess we need to adapt the logic to clearly identify what is "out" and what is "in" (a bit like Jotne did on his current syslog-based accounting dashboard)
It seems traffic directed at your WAN-IP (so dest_ip = WAN) seems to have in the field nexthop_addr: container the inside address. I guess this traffic is part of the NAT-session so these bytes need to be counted also.
Also the dashboard with white backdrop has no selectors and seem "statically" set (eg. past 60min) without dropdown menus.

So yeah ... still some work I think but ...
For the fancy visualization in the first screen on the bottom, you might need to install an app -> https://splunkbase.splunk.com/app/3767/


Image


Image
Image
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 9:08 pm

For anyone that wants to give a crack at it, see below the links to the XML templates that make up these dashboards in Splunk.

http://vanham-franck.be/pics/splunk/spl ... plate1.xml

http://vanham-franck.be/pics/splunk/spl ... plate2.xml



PS : Perhaps now is good time to file another bug with Mikrotik on the Netflow IPFIX which is not really OK.
With Splunk, I simply seem unable to ingest it (Stream App does support it)
However, back when I was testing last week with "pmacct" / "nfacctd" it turned out the Mikrotik has some timing-fields not present/incorrect so all my flow have a START but they all have 1970-Epoch as END

Example below of CSV-capture, the TIMESTAMP_END is always 1970-01-01 ... yeah ... not really usefull...

SRC_IP,DST_IP,SRC_PORT,DST_PORT,PROTOCOL,TIMESTAMP_START,TIMESTAMP_END,PACKETS,BYTES
172.29.45.250,176.9.168.180,38310,232,tcp,2020-06-19 12:12:58.000000,1970-01-01 01:00:00.000000,3,208
176.9.168.180,91.179.157.160,232,38310,tcp,2020-06-19 12:12:58.000000,1970-01-01 01:00:00.000000,2,208
172.29.45.249,216.58.214.3,36286,443,tcp,2020-06-19 12:12:58.000000,1970-01-01 01:00:00.000000,2,178
216.58.214.3,91.179.157.160,443,36286,tcp,2020-06-19 12:12:58.000000,1970-01-01 01:00:00.000000,3,231
13.59.106.231,91.179.157.160,30999,46463,tcp,2020-06-19 12:13:01.000000,1970-01-01 01:00:00.000000,1,60
172.29.45.199,195.238.28.228,56639,443,tcp,2020-06-19 12:13:01.000000,1970-01-01 01:00:00.000000,7,2918
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 9:41 pm

Maybe this should be an add on module for the MikroTik app since it would involve lots of extra stuff.

Using wan IP as a trigger is not good enough, since this will change for many user and then you need to have some sort of auto update.

But after looking at input_snmpidx and output_snmpidx (input/output SNMP interface index) we may have a solution on how this works.

input_snmpidx=2 output_snmpidx=1 Traffic going from inside to outside host
input_snmpidx=1 output_snmpidx=2 Traffic returning from outside host
input_snmpidx=2 output_snmpidx=2 Traffic going from inside oust to inside server using hairpin nat
This may be wrong, but I think I am no correct track.

I did tested the dashboard from git and they work fine. But I think they also mix whats source and destination port. I can see that 443 is top on both source port and dest post, they are part of returning packets when your request that it will go back to the same port.

in bytes og out bytes shows the same data, just renamed name :)
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Sun Jun 21, 2020 11:36 pm

in bytes og out bytes shows the same data, just renamed name :)
On the dashboard/XML I posted ? Because I did that, since there is no "bytes_out" I simply put for temporary the same "bytes_in" also ;-)
So indeed solid grouping must be done to clearly identify what is IN en what is OUT.
Also some filtering you did in your syslog-based dashboard, to exclude RFC1918 IP-space from when making some top-10 of public destinations etc.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Mon Jun 22, 2020 12:08 am

I will look at it. Should be doable to separate input/output like I did no the accounting dashboard. Maybe by looking at public/private net.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Mon Jun 22, 2020 12:13 am

Maybe this should be an add on module for the MikroTik app since it would involve lots of extra stuff.

Using wan IP as a trigger is not good enough, since this will change for many user and then you need to have some sort of auto update.

But after looking at input_snmpidx and output_snmpidx (input/output SNMP interface index) we may have a solution on how this works.

input_snmpidx=2 output_snmpidx=1 Traffic going from inside to outside host
input_snmpidx=1 output_snmpidx=2 Traffic returning from outside host
input_snmpidx=2 output_snmpidx=2 Traffic going from inside oust to inside server using hairpin nat
This may be wrong, but I think I am no correct track.

I did tested the dashboard from git and they work fine. But I think they also mix whats source and destination port. I can see that 443 is top on both source port and dest post, they are part of returning packets when your request that it will go back to the same port.

in bytes og out bytes shows the same data, just renamed name :)
Good analysis. I think that is correct. I'm now testing with hairpin-NAT session and indeed input_snmpidx = output_snmpidx (in my case value of 15) which is my "WAN" with public IP
Now everybody will have different values so I'm not sure how you would abstract this.
I've also found some output_snmpidx=0 values and they seem ALL "Broadcast" traffic, either destination_ip = 255.255.255.255 of my subnet_broascast X.Y.Z.255 at least for the OUTPUT_snmpidx=0 becasue for the INPUT_snmpidx=0 (I also have it) no broadcasts there.
All a bit odd for the moment.

I've also opened a ticket to start the IPFIX discussion again. I want to find out if the RouterOS IPFIX implementation is buggy in the timestamp-area.
For the moment I had no luck that Splunk/Stream-app would ingest this, might need to look at it again but the config docs say no difference exist between configurating it for v5/v9 or ipfix, for both the "netflow" stream should simply be enabled.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Mon Jun 22, 2020 9:05 am

Some more investigation. Snmpidx are the interfaces on the router.

You can get it using SNMP like this:
snmpwalk -v2C -c public 10.10.10.1  ifname
IF-MIB::ifName.1 = STRING: ether1
IF-MIB::ifName.2 = STRING: Bridge1
IF-MIB::ifName.3 = STRING: ether3
IF-MIB::ifName.4 = STRING: ether4
IF-MIB::ifName.5 = STRING: ether5
IF-MIB::ifName.6 = STRING: pptp-in1
IF-MIB::ifName.8 = STRING: ether2
IF-MIB::ifName.12 = STRING: VLAN20
You can also get it using:
/interface print
Flags: D - dynamic, X - disabled, R - running, S - slave
# NAME TYPE ACTUAL-MTU L2MTU MAX-L2MTU MAC-ADDRESS
0 R ;;; WAN
ether1 ether 1500 1596 2026 6C:3B:6B:88:34:3E
1 RS ;;; Cisco C3560CX
ether2 ether 1500 1596 2026 6C:3B:6B:88:34:3F
2 S ;;; Test VLAN 20
ether3 ether 1500 1596 2026 6C:3B:6B:88:34:40
3 RS ;;; Windows server
ether4 ether 1500 1596 2026 6C:3B:6B:88:34:41
4 RS ;;; Linux server
ether5 ether 1500 1596 2026 6C:3B:6B:88:34:42
5 R ;;; Main Bridge
Bridge1 bridge 1500 1596 6C:3B:6B:88:34:3F
6 R ;;; Sokkel
VLAN20 vlan 1500 1592 6C:3B:6B:88:34:3F
7 pptp-in1 pptp-in


[xxxx] > /interface print oid
Flags: D - dynamic, X - disabled, R - running, S - slave
0 R ;;; WAN
name=.1.3.6.1.2.1.2.2.1.2.1 actual-mtu=.1.3.6.1.2.1.2.2.1.4.1 mac-address=.1.3.6.1.2.1.2.2.1.6.1 admin-status=.1.3.6.1.2.1.2.2.1.7.1 oper-status=.1.3.6.1.2.1.2.2.1.8.1 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.1
packets-in=.1.3.6.1.2.1.31.1.1.1.7.1 discards-in=.1.3.6.1.2.1.2.2.1.13.1 errors-in=.1.3.6.1.2.1.2.2.1.14.1 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.1 packets-out=.1.3.6.1.2.1.31.1.1.1.11.1
discards-out=.1.3.6.1.2.1.2.2.1.19.1 errors-out=.1.3.6.1.2.1.2.2.1.20.1

1 RS ;;; Cisco C3560CX
name=.1.3.6.1.2.1.2.2.1.2.8 actual-mtu=.1.3.6.1.2.1.2.2.1.4.8 mac-address=.1.3.6.1.2.1.2.2.1.6.8 admin-status=.1.3.6.1.2.1.2.2.1.7.8 oper-status=.1.3.6.1.2.1.2.2.1.8.8 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.8
packets-in=.1.3.6.1.2.1.31.1.1.1.7.8 discards-in=.1.3.6.1.2.1.2.2.1.13.8 errors-in=.1.3.6.1.2.1.2.2.1.14.8 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.8 packets-out=.1.3.6.1.2.1.31.1.1.1.11.8
discards-out=.1.3.6.1.2.1.2.2.1.19.8 errors-out=.1.3.6.1.2.1.2.2.1.20.8

2 S ;;; Test VLAN 20
name=.1.3.6.1.2.1.2.2.1.2.3 actual-mtu=.1.3.6.1.2.1.2.2.1.4.3 mac-address=.1.3.6.1.2.1.2.2.1.6.3 admin-status=.1.3.6.1.2.1.2.2.1.7.3 oper-status=.1.3.6.1.2.1.2.2.1.8.3 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.3
packets-in=.1.3.6.1.2.1.31.1.1.1.7.3 discards-in=.1.3.6.1.2.1.2.2.1.13.3 errors-in=.1.3.6.1.2.1.2.2.1.14.3 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.3 packets-out=.1.3.6.1.2.1.31.1.1.1.11.3
discards-out=.1.3.6.1.2.1.2.2.1.19.3 errors-out=.1.3.6.1.2.1.2.2.1.20.3

3 RS ;;; Balder Windows server
name=.1.3.6.1.2.1.2.2.1.2.4 actual-mtu=.1.3.6.1.2.1.2.2.1.4.4 mac-address=.1.3.6.1.2.1.2.2.1.6.4 admin-status=.1.3.6.1.2.1.2.2.1.7.4 oper-status=.1.3.6.1.2.1.2.2.1.8.4 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.4
packets-in=.1.3.6.1.2.1.31.1.1.1.7.4 discards-in=.1.3.6.1.2.1.2.2.1.13.4 errors-in=.1.3.6.1.2.1.2.2.1.14.4 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.4 packets-out=.1.3.6.1.2.1.31.1.1.1.11.4
discards-out=.1.3.6.1.2.1.2.2.1.19.4 errors-out=.1.3.6.1.2.1.2.2.1.20.4

4 RS ;;; Varg Linux server
name=.1.3.6.1.2.1.2.2.1.2.5 actual-mtu=.1.3.6.1.2.1.2.2.1.4.5 mac-address=.1.3.6.1.2.1.2.2.1.6.5 admin-status=.1.3.6.1.2.1.2.2.1.7.5 oper-status=.1.3.6.1.2.1.2.2.1.8.5 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.5
packets-in=.1.3.6.1.2.1.31.1.1.1.7.5 discards-in=.1.3.6.1.2.1.2.2.1.13.5 errors-in=.1.3.6.1.2.1.2.2.1.14.5 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.5 packets-out=.1.3.6.1.2.1.31.1.1.1.11.5
discards-out=.1.3.6.1.2.1.2.2.1.19.5 errors-out=.1.3.6.1.2.1.2.2.1.20.5

5 R ;;; Main Bridge
name=.1.3.6.1.2.1.2.2.1.2.2 actual-mtu=.1.3.6.1.2.1.2.2.1.4.2 mac-address=.1.3.6.1.2.1.2.2.1.6.2 admin-status=.1.3.6.1.2.1.2.2.1.7.2 oper-status=.1.3.6.1.2.1.2.2.1.8.2 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.2
packets-in=.1.3.6.1.2.1.31.1.1.1.7.2 discards-in=.1.3.6.1.2.1.2.2.1.13.2 errors-in=.1.3.6.1.2.1.2.2.1.14.2 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.2 packets-out=.1.3.6.1.2.1.31.1.1.1.11.2
discards-out=.1.3.6.1.2.1.2.2.1.19.2 errors-out=.1.3.6.1.2.1.2.2.1.20.2
6 R ;;; Sokkel
name=.1.3.6.1.2.1.2.2.1.2.12 actual-mtu=.1.3.6.1.2.1.2.2.1.4.12 mac-address=.1.3.6.1.2.1.2.2.1.6.12 admin-status=.1.3.6.1.2.1.2.2.1.7.12 oper-status=.1.3.6.1.2.1.2.2.1.8.12
bytes-in=.1.3.6.1.2.1.31.1.1.1.6.12 packets-in=.1.3.6.1.2.1.31.1.1.1.7.12 discards-in=.1.3.6.1.2.1.2.2.1.13.12 errors-in=.1.3.6.1.2.1.2.2.1.14.12 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.12
packets-out=.1.3.6.1.2.1.31.1.1.1.11.12 discards-out=.1.3.6.1.2.1.2.2.1.19.12 errors-out=.1.3.6.1.2.1.2.2.1.20.12

7 name=.1.3.6.1.2.1.2.2.1.2.6 actual-mtu=.1.3.6.1.2.1.2.2.1.4.6 mac-address=.1.3.6.1.2.1.2.2.1.6.6 admin-status=.1.3.6.1.2.1.2.2.1.7.6 oper-status=.1.3.6.1.2.1.2.2.1.8.6 bytes-in=.1.3.6.1.2.1.31.1.1.1.6.6
packets-in=.1.3.6.1.2.1.31.1.1.1.7.6 discards-in=.1.3.6.1.2.1.2.2.1.13.6 errors-in=.1.3.6.1.2.1.2.2.1.14.6 bytes-out=.1.3.6.1.2.1.31.1.1.1.10.6 packets-out=.1.3.6.1.2.1.31.1.1.1.11.6
discards-out=.1.3.6.1.2.1.2.2.1.19.6 errors-out=.1.3.6.1.2.1.2.2.1.20.6


Any you here see that line 6 in Interface and OID shows VLAN20 and .12 after all OID so ifindex=12

IfIndex=0 seems to be the router it self. Since I do not like SNMP since it goes the other way, it does not work behind NAT/Firewall, I will use syslog and a script to store the ifindex and name in an KV store database for use with the MikroTik app if it possible,
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Mon Jun 22, 2020 11:20 am

I've taken Wireshark captures of both IPFIX & v9 streams, starting with the exchange of the templates etc describing all the fields.
I have the impression that the Splunk Stream does not utilize ALL available "fields". I'm going to see if the "dictionary" contains these fields.
Probably you can "add" them. I've seen that. Its a matter of mapping the code (eg. 225,226,... to the correct type (eg. string, IP, integer, ...)

Field (20/23): postNATSourceIPv4Address
Type: postNATSourceIPv4Address (225)
Length: 4
Field (21/23): postNATDestinationIPv4Address
Type: postNATDestinationIPv4Address (226)
Length: 4
Field (22/23): postNAPTSourceTransportPort
Type: postNAPTSourceTransportPort (227)
Length: 2
Field (23/23): postNAPTDestinationTransportPort
Type: postNAPTDestinationTransportPort (228)
Length: 2

I've done the same using the PMACCT-package by simply creating an "primitives" files for the correct mappings and this worked fine.


Cisco NetFlow/IPFIX
Version: 9
Count: 7
SysUptime: 873590.040000000 seconds
Timestamp: Jun 22, 2020 10:03:08.000000000 CEST
CurrentSecs: 1592812988
FlowSequence: 22
SourceId: 0
FlowSet 1 [id=256] (7 flows)
FlowSet Id: (Data) (256)
FlowSet Length: 532
[Template Frame: 1]
Flow 1
[Duration: 0.000000000 seconds (switched)]
StartTime: 873528.130000000 seconds
EndTime: 873528.130000000 seconds
Packets: 1
Octets: 86
InputInt: 15
OutputInt: 14
SrcAddr: 172.29.45.4
DstAddr: 195.238.2.21
Protocol: UDP (17)
IP ToS: 0x00
SrcPort: 51020 (51020)
DstPort: 53 (53)
NextHop: 195.238.2.21
DstMask: 0
SrcMask: 0
TCP Flags: 0x00
Destination Mac Address: Routerbo_20:22:b6 (6c:3b:6b:20:22:b6)
Source Mac Address: ASRockIn_84:01:36 (d0:50:99:84:01:36)
Post Source Mac Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
Post NAT Source IPv4 Address: 81.119.157.161 (=my public IP address on my PPPoE)
Post NAT Destination IPv4 Address: 195.238.2.21
Post NAPT Source Transport Port: 51020
Post NAPT Destination Transport Port: 53
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Tue Jun 23, 2020 9:55 am

I've posted on Splunk community this question on the NAT-fields and why there are not per-direct usable as fields in Splunk ... hopefully ...
In the mean time, it seems the approach below is a good reference to what is coming IN en what is going OUT

First of all, I've limited "Netflow" currently only on my PPPoE "interface" in stead of "all"
It seems following pattern is consistent :

OUTSIDE -> INSIDE (but destined for Mikrotik itself, eg. DNS-lookups, IPSEC tunnel termination)
dest_ip = nexthop_addr

OUTSIDE -> INSIDE (returning traffic destined for LAN-stations)
(dest_ip =! nexthop_addr)

INSIDE -> OUTSIDE
(dest_ip = nexthop_addr) + src_ip is in the same range*** as "exporter_ip"

*** This only works if you "inside LAN" containing hosts is in the same range as your bridge. Eg a single 192.168.x.y network at home. If this Mikrotik is sitting somewhere in between other networks I think this will not work. The "exporter_ip" (field that you can manually set in RouterOS and if not set it will use the IP of the exiting interface on its way to the target) is then completely not related to the endhosts consuming data.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Tue Jun 23, 2020 12:12 pm

Look at this table
line	_time			src_ip		s_port	dest_ip		d_port	next_ip		byte		pacet	prot	in_if	out_if
1	2020-06-23 10:50:11.280	193.212.a.a	42744	92.220.b.b	514	10.10.10.50	3903	 	35	17	1	2	0
2	2020-06-23 10:50:00.570	193.212.a.a	22	92.220.b.b	55774	10.10.10.32	1312380	 	2191	6	1	2	24
3	2020-06-23 10:50:00.540	10.10.10.32	55774	193.212.a.a	22	92.220.200.1	1074672	 	9631	6	2	1	24
193.212.a.a an linux server
92.220.b.b my public IP
92.220.200.1 ISP gateway
10.10.10.32 inside PC
10.10.10.50 Syslog server

1 outside ether1 interface
2 inside bridge interface

Line 3:
I do an ssh to the linux server on port 22, coming from bridge going out on interface ether1 1074672 sent

Line 2:
Linux server reply with data coming from port 22 (part of previous session) going in on ehter1 and out on bridge 1312380 recieved

Line 1:
Linux server sends an udp syslog packed to my syslog server to port 514.

My problem is that in line 1 and 2 the src/dest port are swapped. How do I know that line 2 is part of an previous session?
How to see what is different on line 1 and 2 and now the correct port order.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Tue Jun 23, 2020 5:36 pm

After talking more than one hour with a super spesialist in Netflow, I do start to get the grip on how things works.

There are no way you can se in a Netflow packets, if its traffic returning from an started inside session or if it some from outside starting to sending inn data. You can look at ports and say that all ports below 1024 are destination ports, rest are source port. This will help some but will fail for all application using high ports like Minecraft that uses port 25565 as default listening port.
What you can see with Netflow is how much traffic going inn or out and from what IP to what IP. Ports however are not solvable.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Tue Jun 23, 2020 6:49 pm

After talking more than one hour with a super spesialist in Netflow, I do start to get the grip on how things works.

There are no way you can se in a Netflow packets, if its traffic returning from an started inside session or if it some from outside starting to sending inn data. You can look at ports and say that all ports below 1024 are destination ports, rest are source port. This will help some but will fail for all application using high ports like Minecraft that uses port 25565 as default listening port.
What you can see with Netflow is how much traffic going inn or out and from what IP to what IP. Ports however are not solvable.
Hmm, to get some accounting in place I don't think the what-packets-are-part-of-what-session is really helpfull/important. You only need to make sure that those flow-records within that time-frame (eg. per 60-second , 300-second) are grouped & counted together to get some IN / OUT "totals".
For the ports, I would already be happy if I get a graph in Splunk visualizing all destination-ports grouped by "external" or "internal".
So you can select "Internal Traffic" or "External Traffic" and have visibility on dest-ports to learn if any abnormal services might be there that you do not expect.

I don't understand your statement "ports however are not solvable" . You CAN filter all records related to OUTBOUND and you can filter on DST_PORT so you can get all externa/Internet targetted systems and dest-ports visible not ?? For INBOUND this is a bit harder, because the "dest_port" value might not be the same as the "src_port" initiated by the inside host. There is NA(P)T in between hence the 4 NAT/NAPT extra fields would be usefull.

I really hope there is a way to get the 4 fields visible in Splunk that ARE in a v9 flow-record :
I can't find them! Really weird.

Post NAT Source IPv4 Address: 81.120.157.162 (=my public IP address on my PPPoE, don't worry not my real one)
Post NAT Destination IPv4 Address: 195.238.2.21 (public ISP DNS servers)
Post NAPT Source Transport Port: 51020
Post NAPT Destination Transport Port: 53
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Tue Jun 23, 2020 6:53 pm

What you can see with Netflow is how much traffic going inn or out and from what IP to what IP. Ports however are not solvable.
I would not say that, in a previous project we had a global deployed Riverbed solution with a very large Netflow collector appliance (taking in millions of flows per day from over the whole globe)
You could perfectly drill down and visualize communications from any IP to any IP and display what applications/ports where at play between them.
But again, it was no free plugin ;-) but more of a 6-digit appliance.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Tue Jun 23, 2020 7:52 pm

It's clear that the 2 Github examples of dashboard have some errors in them.
Example the one with a pie-graph "Top Destination IP's" I see large chunck that has MY own public IP address which does not make sense and this is because of NAT and just returning traffic.
Sure the "dest_ip" field in the packets IS my Mikrotik public IP, but logic should be there to know that Splunk should not look at it in this context.

Splunk can "learn" what the public/NAT IP is by comparing "dest_ip' with "nexthop_addr" . If you find any records where these 2 are the same, then store the "dst_ip" value because that is a public IP used for NA(P)T.
Then later you can make sure you exclude this because it does not contribute to the aspect of "top destination IP's" (for calculating VOLUMES you obviously need this in some way, as a large chunck of return traffic hits this public IP, but then the "nexthop_addr" will reveal the real internal host to which this traffic belongs). IF the "nexthop_addr" = "dst_ip" AND src_ip =! RFC1918 space then this traffic is destined for Mikrotik itself (DNS lookups, IPSEC tunnels traffic etc) but for VOLUMES should be counted too actually.

I'm not 100% with my above claim but it looks like it, I only don't know how to pull this off in Splunk ;-(

I'm going to check if I can find some Splunk expertise within my company to ask some questions on this. I know we do, only not sure they are willing to help out ;-)


EDIT : My statement is NOT correct at all ... back to the drawing board...

EDIT2 : Wouldn't it be simpler to extended your script and obtain the IP-addresses associated with interfaces (eg. PPPoE or others) and get them into Splunk ? In addition, the pre-req could be that users must add the keyword "WAN" on the interface-description to you know directly what is the external/outside interface. That is not really THAT much of a problem I guess since your script requires some modifications/config anyway. This is easy for everyone.
Another pre-req could be Netflow should only be activated on the WAN-interface. Let's keep it simple to start with.
Once there you can obtain the current "WAN" interface-IP my making the query in Splunk at least you don't need to SNMP interface-ID stuff anymore?. Any record with dst_ip = WAN-IP is clearly "inbound" (could be DNAT portmapping traffic but also regular returning packets from an inside started session, doesn't matter for accounting purposes just count the bytes in a given time).
Then also you can count everything =! (NOT) equal to the WAN-IP and this will give you "upload" traffic. If you want to "split" traffic generated by Mikrotik add the "src_ip=WAN_IP" to the query.

So suppose WAN=92.178.157.120
Eg. source="stream:netflow" dest_ip!="92.178.157.120" src_ip="92.178.157.120" => For my dataset this returns packets related to EGRESS activity Mikrotik itself, so DNS/IPSEC/DDNS/NTP updates and I also got hits on an IP SMTP of my provider when Mikrotik sends out MAIL.
This can be added to the package of other EGRESS traffic (caused by internal hosts) that can be found with.

source="stream:netflow" dest_ip!="92.178.157.120" src_ip!="92.178.157.120" and cross-checking with the retrieved "src_ip" list only list "internal" hosts indeed.
So these combined would be total OUTPUT on the link I guess.

For INBOUND/DOWNLOAD, the logic is a bit different.
source="stream:netflow" dest_ip="92.178.157.120" nexthop_addr="92.178.157.120" gives me only records with src_ports like DNS/NTP/IPSEC to for sure traffic inbound to the Mikrotik. Pretty sure DNAT would go also under this. (need to test this)

And then finally the "bulk" of download traffic coming back from Internet for clients on the local LAN would be
source="stream:netflow" dest_ip="92.178.157.120" nexthop_addr!="92.178.157.120" => When I make this query my nexthop_addr list only contains all my LAN-stations receiving this returning traffic.

That would take care of an "accounting" alternative I guess for Inbound/Outbound at a high level.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.0 (Graphing everything)

Wed Jun 24, 2020 8:45 am

I will make a view that shows total traffic in/out, what IP it does come from and what IP it goes to. That is not the problem.
What I would like to know is what port is used, there i were the problem lays.

Look at line 1 and line 2 in the above post.
Both comes from same IP 193.212.a.a, both goes to same outside IP on my router 92.220.b.b. One will go to 10.10.10.50 and other to 10.10.10.32. So they will be counted as inn traffic.

Problem is that line 2 is part of an ongoing session starting from inside, showing port swapped around. Line 1 has started from the outside and are going trough a nat hole and has the port correct direction. There are no way I can setup a program to get this correct connected to the Port since they look the same. You can guess that line 2 are part of some, since it has source_port 22 and line 1 is a starting of some, since port are 514. But what if source port is 25565? It can be a new session going inn to dest_port 3389 (Rdp) or it can be return traffic for someone plays minecraft and connected to a server on the outside using port 25565.

Give me some days, and I will create some test views.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Jun 25, 2020 2:13 pm

Splunk for MikroTik updated to v3.1

Mayor changes is the CAPsMAN view

If you like to use the CAPsMAN, update script to 4.1 and add capsmann script fond in section 2f first post:


To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9)
# 3.1 (25.06.2020)
# Added CAPsMAN view and extraction
# Added limit=0 to "MikroTik DHCP pool information"
# Added dhcp server to "MikroTik DHCP request"
# Added pool selection to "MikroTik DHCP pool information"
# Added information about static release in "MikroTik DHCP pool information"
# Updated script to 4.0 removed double information and added write-sector information
# Added Sector Writes to "Mikrotik Resources"
# Updeated KV search in "Mikrotik Device List" to not overwrite all data
# Fixed missing host in "Mikrotik Uptime"
# Fixed KV update and change names
# Added src_ip counter in "Mikrotik DNS Live usage"
# Added name_id for mac in "MikroTik Wifi connection" and "MikroTik Wifi strength"
# Added sort by host/module and hostname in "MikroTik Log Size"
# Added free text search to "MikroTik Firewall Rules"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Jun 25, 2020 4:01 pm

Script updated to 4.1 to get CAPsMANN inforamtion.

Read section 2f) if you like to use CAPsMANN function.
 
robsgax
newbie
Posts: 27
Joined: Wed Apr 17, 2019 10:26 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 10, 2020 7:44 am

Script updated to 4.1 to get CAPsMANN inforamtion.

Read section 2f) if you like to use CAPsMANN function.
Where is version 4.1??, i only see ver 4.0 on the OP.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 10, 2020 8:23 am

Where is version 4.1??, i only see ver 4.0 on the OP.
Fixed :)
 
robsgax
newbie
Posts: 27
Joined: Wed Apr 17, 2019 10:26 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Tue Jul 14, 2020 8:23 pm

I just noticied this:
Splunk for MikroTik updated to v3.1

Mayor changes is the CAPsMAN view

If you like to use the CAPsMAN, update script to 4.1 and add capsmann script fond in section 2f first post:
and this code on the script:
# Test if CAPsMANN is installed, if yes, run it
# ----------------------------------
:do {
	:if ([:len [/caps-man registration-table find]] > 0 and $CAPsMANN) do={
		/system script run capsman
	}
} on-error={}
where is the capsman script?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Tue Jul 14, 2020 11:38 pm

A good question :)
Added to first post.
 
robsgax
newbie
Posts: 27
Joined: Wed Apr 17, 2019 10:26 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 15, 2020 1:45 am

thanks for the hard work

is there a way to make the script output not be reflected on the memory or disk log?, only send it to the remote splunk server?

were having some isp issues and i need to use the log a lot, but its filled with the splunk script output, i did put a usb memory and sent the log to disk too, because with only the memory log, i lost the values that i need every 5 or 10 minutes, but with the disk log its still too big, i did separate the logs by lines, but still to big., here's an image of my current situation, 30 files of 8192 lines just for the last 2 days.
SNAG 2020-07-14 0000.png
is there a way to hide it?
You do not have the required permissions to view the files attached to this post.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 15, 2020 1:04 pm

Hello Jotne,

Have you ever considered using a dockerized Splunk Environment?
I lately tested this but did not get any mikrotik information in Splunk.

My "normal" Splunk envirnoment is working.
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 15, 2020 1:31 pm

Hello Jotne,

Have you ever considered using a dockerized Splunk Environment?
I lately tested this but did not get any mikrotik information in Splunk.

My "normal" Splunk envirnoment is working.
I'm actually have my splunk environment on docker working perfectly.


Inviato dal mio SM-G950F utilizzando Tapatalk

 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 15, 2020 1:56 pm

I'm actually have my splunk environment on docker working perfectly.


Inviato dal mio SM-G950F utilizzando Tapatalk
do you use the official Splunk image?
and do you have a separate rsyslog environment or is that not neccesary?
and how do you start the Splunk container?

thanks in advance.
 
ferdytao
newbie
Posts: 29
Joined: Mon Sep 26, 2016 8:51 am

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 15, 2020 3:11 pm

Yes I'm using the official splunk image with internal syslog stored on local volume (I'm using Docker on my Synology NAS).

Here is my config:
docker run -d --net host -v /volume3/docker/Splunk/etc:/opt/splunk/etc -v /volume3/docker/Splunk/var:/opt/splunk/var -v /etc/localtime:/etc/localtime:ro  -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=Password" --name splunk splunk/splunk:latest
You have to use host network (--net host) or macvlan, otherwise you will not see the single client's ip but the natted address. If you don't want use --net host you need to correctly map the ports.
Last edited by ferdytao on Wed Jul 15, 2020 3:11 pm, edited 1 time in total.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 15, 2020 4:34 pm

Thank you I going to try this somewhere in the next day's.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Jul 16, 2020 11:35 am

is there a way to make the script output not be reflected on the memory or disk log?, only send it to the remote splunk server?
I do not see those files on my disk. Can you download one of them to your PC and list whats in the file?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Jul 16, 2020 11:41 am

Thank you I going to try this somewhere in the next day's.
Should work as long as data gets inn to Splunk and are tagged correctly "MikroTik"
 
robsgax
newbie
Posts: 27
Joined: Wed Apr 17, 2019 10:26 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Jul 16, 2020 6:38 pm

is there a way to make the script output not be reflected on the memory or disk log?, only send it to the remote splunk server?
I do not see those files on my disk. Can you download one of them to your PC and list whats in the file?
the files are showing on my disk because i have a rule that send the logs there, we need to analyze some things on the logs for my isp, but the script is making the logs grow a lot in size,
/system logging action
add disk-file-count=31 disk-file-name=disk1/logs/log disk-lines-per-file=8192 \
    name=disk1 target=disk
/system logging
set 3 action=memory
add action=disk1 topics=critical
add action=disk1 topics=error
add action=disk1 topics=info
add action=disk1 topics=warning
add action=disk1 topics=wireless,debug
add action=disk1 topics=e-mail,debug
add action=disk1 topics=caps,debug
    
and here's a log with the lines that are generated from the splunk script
log.0.txt
as you see, in 10 minutes more than 3000 lines are filled
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Jul 16, 2020 11:50 pm

the files are showing on my disk because i have a rule that send the logs there
You have selected to write the logs to your disk so it will write it there. I do not understand the problem. Just remove the log to the disk?
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 12:25 am

Jotne,
Did you spend some time in looking on the Netflow story with Splunk ? Possible integration into your current application/set of dashboards ?
 
robsgax
newbie
Posts: 27
Joined: Wed Apr 17, 2019 10:26 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 1:07 am

the files are showing on my disk because i have a rule that send the logs there
You have selected to write the logs to your disk so it will write it there. I do not understand the problem. Just remove the log to the disk?

the thing is the logs are filled with info from the splunk script, what i was asking is that there is a way to be able to ommit that info on the memory and disk log, only send it to the remote syslog. if i dont sent it to disk, i am sure that it will not be sent to disk, only to memory, but i choose to send it to disk because my isp and i need to analyze the logs and they are filled with the script output, and the memory log gets filled every 10 or 15 minutes, that is what im trying to hide, how i can make that?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 8:45 am

The point with the script is to send all information using syslog. If you selet that log should be sent to disk, it will also go there. As far as I know, you can not split the logg saying that some should go to disk, some to memory and some to disk.

I still do not understand why you need logs to disk. Its 10 times better to get all to Splunk, and then analyse what you are looking for there. Disk is a limited resource on the routers so it will fill up quickly.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 8:58 am

Yes I'm using the official splunk image with internal syslog stored on local volume (I'm using Docker on my Synology NAS).
With "internal syslog" you mean the Synology syslog. In the Splunk container I do not see a syslog.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 9:05 am

Yes I'm using the official splunk image with internal syslog stored on local volume (I'm using Docker on my Synology NAS).
With "internal syslog" you mean the Synology syslog. In the Splunk container I do not see a syslog.
You don't need to "look" for any Syslog in Splunk. Syslog is just 1 of many ingress channels for data into Splunk. Offcourse you need to make it possible for syslog messages to arrive in Splunk so expose some ports etc.
That script "tags" all messages coming from the Mikrotik with the label "MikroTik" and basically in Splunk you can simply enter the keywork MikroTik in the search-bar and you'll everything related to it...
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 11:32 am

Yes I'm using the official splunk image with internal syslog stored on local volume (I'm using Docker on my Synology NAS).

Here is my config:
docker run -d --net host -v /volume3/docker/Splunk/etc:/opt/splunk/etc -v /volume3/docker/Splunk/var:/opt/splunk/var -v /etc/localtime:/etc/localtime:ro  -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=Password" --name splunk splunk/splunk:latest
You have to use host network (--net host) or macvlan, otherwise you will not see the single client's ip but the natted address. If you don't want use --net host you need to correctly map the ports.

It looks like I'm getting data into Splunk.
I will check this weekend if everthing is complete.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 11:41 am

Du a search like this to see if any data comes inn to splunk.
index=*
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Jul 17, 2020 12:31 pm

I see a lot of information so it seems OK but I have the check this weekend if everything is complete.
 
robsgax
newbie
Posts: 27
Joined: Wed Apr 17, 2019 10:26 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sat Jul 18, 2020 12:52 am

The point with the script is to send all information using syslog. If you selet that log should be sent to disk, it will also go there. As far as I know, you can not split the logg saying that some should go to disk, some to memory and some to disk.

I still do not understand why you need logs to disk. Its 10 times better to get all to Splunk, and then analyse what you are looking for there. Disk is a limited resource on the routers so it will fill up quickly.
i need logs to disk because, again, were having trouble with my connection, my isp need to see the logs, and memory log is too small, that's why we send them to disk, so they can analize over the course of 4 or 5 days and do what they need to do to fix our issues, if i deny, they wont fix anything, until i send them the logs.
again, the script send everything to memory, disk, remote, and all log options that i have, what im asking is if we can route the script output only to the remote log route. bypassing memory, disk or another medium, if you go to system, loggin, actions, those are the destination of the logs, and with the rules, you can tell what goes where, and in the script you can tell where to log, for example,

:log info message="script=ntp status=$([/system ntp client get status])"

thats one line of the script, can i change that, so instead of send the log to info message, and it goes acording to the rules, to memory, disk, remote, etc etc? im not very versatile with scripts and mikrotiks,

that what im asking, if it can be done.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sat Jul 18, 2020 11:51 am

Since my script log events as info and you have this:
add action=disk1 topics=critical
add action=disk1 topics=error
add action=disk1 topics=info
You do tell that all info log should go to the disk as well.

Why can you not give your ISP access to your Splunk? They will then get the same log as you store to disk. At the same time you do not wear out the small router flash.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 22, 2020 9:32 am

Would it be possible to allow more then 20-characters on a firewall-rule index in Splunk ?? Increase it to 25 or so ?
For some rules in Splunk where my label exceed 20-chars, I get :

too_long_Prefix_max_20_characters


Especially some custom NAT/Portknock rules that contain a somewhat larger label..
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Wed Jul 22, 2020 8:43 pm

Problem is that if you do use longer name, RouterOS starts to chop off characters. So to solve this MikroTik needs to modify the RouterOS.
This is why I in first post added sample on how to name the filter rules to have some contoll.
 
User avatar
oaas
just joined
Posts: 4
Joined: Sun Feb 10, 2019 7:15 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sun Aug 02, 2020 12:26 pm

Any plans for adapting the script to the upcoming RouterOS 7.x version?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sun Aug 02, 2020 3:10 pm

It did work with 7.0 beta, have not had time to look at 7.1
Most negative thing with the new >= 7.0 beta 8 is that they have removed accounting.
We now have to use Netflow to log detailed data.
This gives around 10 times larger logs, and need extra port not just syslog port.
Much more complicated setup.

Rest should work.
 
ingus16
newbie
Posts: 29
Joined: Sun Jan 27, 2013 11:44 am

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sun Sep 20, 2020 2:16 pm

Did this solution work with splunk linux docker version as well ? In my case, splunk receives mikrotik syslog data but in this plugin shows no devices
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sun Sep 20, 2020 3:53 pm

Did this solution work with splunk linux docker version as well ? In my case, splunk receives mikrotik syslog data but in this plugin shows no devices
All message need to be tagged "MikroTik", so message should look like this using this search: index=* (section 2b)
dns MikroTik: done query: #3083521 dns name does not exist
You can also try this search:
index=* | eval status=if(match(_raw, "MikroTik"), "ok", "error") | stats count by host status
It should give a list of all host sending logs to Splunk, with "ok" behind the host that sends logs with "MikroTik" in it.

PS One letter written wrong gives problems.
 
horcsct
just joined
Posts: 3
Joined: Thu Dec 03, 2020 5:59 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Thu Dec 17, 2020 6:09 am

Dear Sir,
Thanks for Tool. I get below error (log/splunkd.log) and after that logging stopped.

WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Thu Dec 17 04:11:00 2020). Context: source=udp:514|host=xxx.xxx.xxx.x|syslog|

Thanks for your help.

EDIT
I add two below lines under [syslog] in etc/apps/MikroTik/default/props.conf and problem solved till now :-)
MAX_TIMESTAMP_LOOKAHEAD = 23
DATETIME_CONFIG = CURRENT
 
horcsct
just joined
Posts: 3
Joined: Thu Dec 03, 2020 5:59 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Dec 18, 2020 3:20 pm

I have AD DNS which forward DNS requests to MikroTik. Now the Splunk logs two DNS requests, one for AD DNS server and one for client. How I can exclude AD DNS requests?
Thanks
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Dec 18, 2020 6:17 pm

You could try this:
/system logging
add action=logserver prefix=MikroTik topics=dhcp
add action=logserver prefix=MikroTik topics=!debug,!dns
To exclude DNS logs from MT.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Fri Dec 18, 2020 7:31 pm

Dear Sir,
Thanks for Tool. I get below error (log/splunkd.log) and after that logging stopped.

WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Thu Dec 17 04:11:00 2020). Context: source=udp:514|host=xxx.xxx.xxx.x|syslog|

Thanks for your help.

EDIT
I add two below lines under [syslog] in etc/apps/MikroTik/default/props.conf and problem solved till now :-)
MAX_TIMESTAMP_LOOKAHEAD = 23
DATETIME_CONFIG = CURRENT
Perhaps some more tuning parameters to consider.

https://www.sicherevielfalt.de/blog/the ... nce-boost/
 
User avatar
j2sw
Member Candidate
Member Candidate
Posts: 131
Joined: Mon Sep 04, 2006 5:42 am
Location: Indiana
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sat Dec 19, 2020 1:44 pm

Awesome post! I have pushed this out to my blog as I think it is a very helpful tool!
 
horcsct
just joined
Posts: 3
Joined: Thu Dec 03, 2020 5:59 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sun Dec 20, 2020 9:09 pm

You could try this:
/system logging
add action=logserver prefix=MikroTik topics=dhcp
add action=logserver prefix=MikroTik topics=!debug,!dns
To exclude DNS logs from MT.
The AD DNS did not forward clients name (all clients are the domain name) so I want keep MikroTik DNS logs and exclude domain DNS logs.
Thanks.
 
Niffchen
newbie
Posts: 38
Joined: Thu Mar 22, 2018 1:36 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Sat Feb 06, 2021 11:51 pm

I am using your great tooll for some weeks but I have some problems ... sometimes.
I have one "RB4011iGS+", 4 "hAP ac" and 1 "wAP ac". All systems are performing very well and there are no issues.
But sometimes the hAP acs seem to stop sending data to my Splunk host. Than I am missing some DHCP data of the host which has paused (that is all I recognized at the moment) and at the dashboard "MikroTik Device List" I can see that there are no more "Uptime" messages for the device.
Today it happened after configuring different wifi devices and testing all wifi netowrks with all device connected to this hAP ac. Before this testing everything seems to be ok, now there are no more "Uptime" values. It looks curious ...

Do you have any ideas whats going on?

Thank you very much,
Jens
 
roe1974
Member Candidate
Member Candidate
Posts: 150
Joined: Mon Dec 31, 2018 2:14 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.1 (Graphing everything)

Mon Feb 15, 2021 11:53 am

Hi :-)
Thanks for the great description/instructions.
Does this also work with Splunk's cloud solution ?
If yes ...how ?
thx
greetings Richard
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sun Feb 21, 2021 10:26 am

Splunk for MikroTik updated to v3.2

This version contains most tweaks and fixes.

To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9+)

# 3.2 (21.02.2021)
# Fixed DHCP extractbiondue to change in 6.48 log format.
# Fixed error in numer of mac pr host in "MikroTik Wifi strength"
# Added more info in "MikroTik Accouning Traffic"
# Added list in "MikroTik Admin user login"
# Added Source Port in "MikroTik Firewall Rules"
# Added more info in "MikroTik Log Size"
# Added logout and added client_id info. Fixed sorting "MikroTik PPPoE Connection"
# Fixed Time in "MikroTik System Changes"
# Added multi selection in graphs, moved legends in "MikroTik uPnP"
# Fixed typo in "MikroTik VPN Connection"
 
MattMiTi
just joined
Posts: 6
Joined: Wed Apr 17, 2019 10:32 am

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Feb 24, 2021 10:21 am

THANKS!!!
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Feb 24, 2021 7:28 pm

Thanks!
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Mar 26, 2021 5:22 pm

Hi Jotne,

a couple of days ago I discoverd that running splunk in docker on my Synology NAS was way easier than I ever thought ...

Everything runs smooth but I have 1 question about NTP/SNTP, all my "ntp slave" devices run SNTP instead of NTP, NTP only runs on my borderrouter ...
All devices run fine with SNTP but why does splunk signal that SNTP is not correct ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Mar 26, 2021 11:36 pm

All devices run fine with SNTP but why does splunk signal that SNTP is not correct ?
An error in the script was found.
Updated script to version 4.2

You can just change the NTP part of the script to make it work:

# Get NTP status
# ----------------------------------
:do {
	:log info message="script=ntp status=$([/system ntp client get status])" 
} on-error={
	:if ([:len [/system ntp client get last-update-from]]>0) do={
		:log info message="script=ntp status=synchronized"
	} else={
		:log info message="script=ntp status=not-synchronized"
	}
}
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Mar 29, 2021 11:20 am

tnx, I updated the script ;-)

btw, I ran into the 500MB free licence limit the 2nd day it was running ...
As I am running dude with SNMP monitoring,
the rule
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet
to log everything produces a LOT of snmp log traffic.

I had to change it to
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Mar 29, 2021 11:56 am

Thanks for info, I will remove snmp for the main post, since we do not need this type of log, since its already logged by the SNMP
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Mar 29, 2021 12:12 pm

I noticed your change on the main topic.
you need to modify the line for the webinterface ;-)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Mar 29, 2021 12:43 pm

Not sure what you mean, I change the 2b section.
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Mar 29, 2021 12:48 pm

2b) Then select what modules to log.
I do suggest that you send all DHCP logs including debug and all other logs that are not debug.
It is very important to name the prefix like this "MikroTik" and not "mikrotik" or some other.
Splunk uses the MikroTik prefix to find out what type of syslog data that is coming to it.
Uppercase T and uppercase M, rest are lowercase
Web gui:
System->Logging->Rules->Add new->Topics:dhcp->Prefix:MikroTik->action:your syslog server->Ok
System->Logging->Rules->Add new->Topics:!debug->Prefix:MikroTik->action:your syslog server->Ok
last line should match CLI ...
System->Logging->Rules->Add new->Topics:!debug,!packet,!snmp->Prefix:MikroTik->action:your syslog server->Ok
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Mar 29, 2021 1:19 pm

Thanks ageing.

Fixed. I need a cup of coffee :)
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 12:01 am

After a long time not using your tool I decided to install it again as a docker container on my Synology NAS.
I have a RB4011 Router and a seperate HAPac2 access point. I configured both devices to send all information to Splunk.
This worked for several hours but then the HAPac2 is not sending any information to splunk anymore. When I search in splunk for that host no information is found.
The script runs every 5 minutes but no info in Splunk. The logging is configured as it should (It did work for a couple of hours).
Has someone seen this also?? What can be the source for this issue??
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 8:01 am

If you pass more than 500MB/day on free license, it will stop showing new data, not stop receiving data. If once device can send data and its shown in Splunk, splunk is ok. It may be some blocking your data, or device it self does not send data. Look at the config and see if all are correct.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 12:05 pm

For the licensing,go to "Settings" -> "Licensing" and there you will see howmuch MBytes you've consumed today.
If it worked for a couple of hours then I suspect the HAPac2 ?
If you go to the "Apps" then "Search" and then you have this button "Data Summary" where you can see the activity for the different datasources.
What does it say ?
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 12:24 pm

It's not the license limit. Yesterday I used < 5% of the 500MB. And yes the RB4011 continued to work after the HAPac2 stopped.
In the "data summary" I can see that the HAPac2 did send messages from around 15:00 in the afternoon (the time I added it to Splunk) until 17:37. After that nothing anymore.

This morning I added a new syslog data input to Splunk with a different port for the HAPac2. After I changed the port on the HAPac2 it immediately started sending log data again. I made this change at 08:00. And until now 11:10 the HAPac2 is still sending data. That's already 30 minutes longer then yesterday :).

Fingers crossed.
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 12:29 pm

Like the devil is playing with it.
5 minutes after I wrote the previous post the HAPac2 stopped again with sending log data to Splunk.

Yesterday it lasted 2,5 hours today 3 hours. So I have made little progress :(
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 12:46 pm

Like the devil is playing with it.
5 minutes after I wrote the previous post the HAPac2 stopped again with sending log data to Splunk.

Yesterday it lasted 2,5 hours today 3 hours. So I have made little progress :(
Then clearly a bug on the RouterOS of that box ?
Can you check the logging, if you don't see any output then the script is not running.
Perhaps "re-create" it ??

Try another RouterOS release ? Hard to believe basic script execution/scheduling would be f*cked up, but with RouterOS you can expect everything ;-)
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Apr 16, 2021 1:13 pm

Both devices are running at 6.48.1
The script is running I see messages coming in the log when I manual start the script.

I tried again to change the UDP port backup to the port I used yesterday. And immediatly I got a message in Splunk from the HAPac2 saying I changed the log action.
	
4/16/21 12:01:06.000 PM	system,info MikroTik: log action changed by admin
host = 192.168.0.8 source = udp:1514 sourcetype = mikrotik
 
zandhaas
Frequent Visitor
Frequent Visitor
Posts: 73
Joined: Tue Dec 11, 2018 11:02 pm
Location: The Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sat Apr 17, 2021 8:33 pm

Yesterday I changed several things on the HAPac2:
1. Updated the device to version 6.48.2 (also upgraded the router firmware)
2. Removed the Splunk script and created it again.
3. removed the splunk remote logging action
4. edited the default remote logging action to send the syslog messages to the splunk server.

After all these changes the HAPac2 is sending log messages to the splunk server for more as 24 hours now.

So again fingers crossed.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon May 03, 2021 1:32 pm

To not fill up internal logs with firewall logs etc, turn off info log to memory (max 999 lines) /system logging set 0 disabled=yes PS Hotspot is not needed if you do not use it.
Is there a way to not log the "firewall logs" into the memory without disabling system logging? I need system logging for info/debug/errors like interfaces going down etc.
Last edited by DarkNate on Mon May 03, 2021 2:31 pm, edited 1 time in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon May 03, 2021 2:22 pm

You can try to enable info logging and add that firewall should not be included, like this:
.
logging.jpg
You do not have the required permissions to view the files attached to this post.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon May 03, 2021 2:34 pm

You can try to enable info logging and add that firewall should not be included, like this:
.
logging.jpg
Thanks, that works well and makes more sense than disabling it completely, I'd suggest putting that in the original guidepost itself.

So basically I got Splunk up and working on a DigitalOcean droplet instance.
  • What can I do to ensure MikroTik to Splunk Server communication is encrypted and not sent in plaintext?
  • Is there a secure (HTTPS) way for me to expose the Live Attack Dashboard on my site?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon May 03, 2021 2:50 pm

I'd suggest putting that in the original guidepost itself.
A good Ide, I will add that.
What can I do to ensure MikroTik to Splunk Server communication is encrypted and not sent in plaintext?
Since MikroTik does not support TLS syslog (please add), the only workaround I do see is to send log to a local Rsyslog (with TLS support) that sends it to an external Syslog server using TLS
https://medium.com/poka-techblog/securi ... 862326c154
Is there a secure (HTTPS) way for me to expose the Live Attack Dashboard on my site?
You can set up Splunk to use HTTPS or add a proxy server (HAProxy) in front. Create a read only user that only sees that dashboard.
You can also make Splunk send data (eks. each 5 min) to annoter web site. (Have not tried this)
Also look at Rest API or Embed scheduled reports
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon May 03, 2021 3:54 pm

I'd suggest putting that in the original guidepost itself.
A good Ide, I will add that.
What can I do to ensure MikroTik to Splunk Server communication is encrypted and not sent in plaintext?
Since MikroTik does not support TLS syslog (please add), the only workaround I do see is to send log to a local Rsyslog (with TLS support) that sends it to an external Syslog server using TLS
https://medium.com/poka-techblog/securi ... 862326c154
Is there a secure (HTTPS) way for me to expose the Live Attack Dashboard on my site?
You can set up Splunk to use HTTPS or add a proxy server (HAProxy) in front. Create a read only user that only sees that dashboard.
You can also make Splunk send data (eks. each 5 min) to annoter web site. (Have not tried this)
Also look at Rest API or Embed scheduled reports
The external Syslog setup looks complicated to me, with too much overhead.

Noticed a flaw with your app, if the MikroTik is resolved using DDNS (IP>Cloud), Splunk Dashboard still reports the old IP address as "Host".
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon May 03, 2021 6:12 pm

It reports the syslog sending IP. Host is the host field in Splunk for the incoming logs.

PS no need to quote the whole post above you. Use Post Reply button under the post.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue May 04, 2021 10:23 am

I dropped this idea, due to plaintext syslog from MikroTik. I can't be bothered with gymnastic workarounds for this one. In 5 minutes of plaintext logs over the internet, I saw direct attacks dropped by the firewall that was destined for my internal subnets. So yeah...
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue May 04, 2021 1:21 pm

I have naver seen any problems with my pain text syslog, but TLS would be a good enhancement.
You can set access list on who can send syslog to your server and also monitor when you get new hosts trying to send syslog message.

One reason that I do not see many wrong attempts, is that I have a rule that blocks an IP for 24 hour if it tries one port that are not open in my router. So if some tries example SQL port 1433, he will be blocked for all port that are open as well. including syslog/web +++
Access list have around 7000 entries all time.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue May 04, 2021 10:50 pm

I have naver seen any problems with my pain text syslog, but TLS would be a good enhancement.
You can set access list on who can send syslog to your server and also monitor when you get new hosts trying to send syslog message.

One reason that I do not see many wrong attempts, is that I have a rule that blocks an IP for 24 hour if it tries one port that are not open in my router. So if some tries example SQL port 1433, he will be blocked for all port that are open as well. including syslog/web +++
Access list have around 7000 entries all time.

You're getting the wrong idea. The issue is MITM snooping. Plaintext exposes my internal network for free.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue May 04, 2021 11:16 pm

Ahh it was that you mean. This is why I like to use DoH. I not like all inn the middle can look at all my DNS request.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 999
Joined: Fri Jun 26, 2020 4:37 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed May 05, 2021 2:41 pm

Ahh it was that you mean. This is why I like to use DoH. I not like all inn the middle can look at all my DNS request.
How would DoH encrypt Syslog's plaintext which works on IP:Port Basis after the initial DNS lookup regardless?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed May 05, 2021 2:47 pm

It was just a comparison. DNS unencrypted. Syslog unencrypted. Syslog-TLS encrypted. DoH - DNS encrypted.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 9:45 am

Script updated to 4.3

Script now gets firmware information from the Router Board.
Will be added to upcoming 3.3 app.

To upgrade: Select the script data from section 2f in the first post and edit srcipt Data_to_Splunk_using_Syslog ont the router, replace all data.

This is not a needed upgrade, just to get more information. Everything will work if you do not upgrade or miks older 4.2 and 4.3 script. (Just miss the firmware info)
 
leosedf
just joined
Posts: 16
Joined: Sun Nov 08, 2009 1:34 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 3:37 pm

Tried to create an account and i got a message due to US guidelines etc you have to contact us..
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 3:53 pm

Tried to create an account and i got a message due to US guidelines etc you have to contact us..
You probably where too honoust when telling in what country you live ?
Sound a bit like an export-restriction of Splunk "technology" to certain countries ?

And yeah, these days plenty of countries are on the US "blacklist" when it comes to this kind of stuff.
 
leosedf
just joined
Posts: 16
Joined: Sun Nov 08, 2009 1:34 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 4:33 pm

I was honest.
Is UK on the black list??
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 5:05 pm

Never seen that before, If for some reason, you can not download, I can make a like to it.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 5:06 pm

I was honest.
Is UK on the black list??
Nah would surprise me ;-)
Not using any funny VPN service ?
Or perhaps your ISP used an IP-block formerly used by North-Korea ;-)

Perhaps you should open a request with Splunk and ask them why they don't let you download the software ?
 
leosedf
just joined
Posts: 16
Joined: Sun Nov 08, 2009 1:34 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri May 28, 2021 6:08 pm

Oh!!!
Activated now, it's going to be a complicated but fun weekend.
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sat May 29, 2021 10:01 am

@jotne
I did enroll 4.3 to a router and that stopped display "uptime" in the devicelist in spl3.2 ...
Did a rollback to 4.2
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sat May 29, 2021 11:10 am

I did for some hour have a 4.3 script posted with a small error.
Try copy it again.

Do this search, you should see uptime for every 5 min.
module=script script=resource | table _time host uptime
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sat May 29, 2021 11:55 am

Screenshot 2021-05-29 at 10.53.57.png
that search gives me no results ... (I have 6 devices reporting)
You do not have the required permissions to view the files attached to this post.
 
pixture08
just joined
Posts: 5
Joined: Mon Jun 07, 2021 3:32 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 6:07 am

Hi there,

I'm using this and very thankful to this app. Currently, my splunk is installed on my local machine (personal PC), and I'm planning to deploy Splunk on Windows VM to access it anywhere. I was able to set it up access the web UI publicly. However, I'm not getting any data from my Mikrotik. I've open the port 8000 and 514 and still wasn't able to get any data. Is there any guide for my use case?

Thank you very much and Cheers from Philippines!
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 10:21 am

Here is how I test if syslog do sends data to Splunk server with IP 192.168.0.50

From a linux server (192.168.0.10) use the following command.
echo '<14>_sourcehost_ messagetext' | nc -v -u -w 0 192.168.0.50 514
Then on the Splunk web console do a search like this:
host="192.168.0.10"
or just
*
You should then from the test server see:
Jun  8 09:14:57 192.168.0.10_sourcehost_ messagetext
If you do see nothing, syslog may not work, you have some local firewall on the server (iptables)
 
pixture08
just joined
Posts: 5
Joined: Mon Jun 07, 2021 3:32 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 12:39 pm

Hi,

I'm not sure how you will test it using Windows Machine. I've tried to set up Splunk on my local PC (with ip of 192.168.0.3), update the mikrotik logserver to that IP and can confirm that I'm getting data from mikrotik. However, if I change to the Windows VM (ip - 32.x.x.x), I didn't receive any syslog. I can confirm that both 8000 and 514 port is open. Firewall was already turned off as well.
Here is how I test if syslog do sends data to Splunk server with IP 192.168.0.50

From a linux server (192.168.0.10) use the following command.
echo '<14>_sourcehost_ messagetext' | nc -v -u -w 0 192.168.0.50 514
Then on the Splunk web console do a search like this:
host="192.168.0.10"
or just
*
You should then from the test server see:
Jun  8 09:14:57 192.168.0.10_sourcehost_ messagetext
If you do see nothing, syslog may not work, you have some local firewall on the server (iptables)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 1:15 pm

For Splunk, Linux is the best option, but it works in Windows as well. (Install VmWare Workstation on your Windows and add a Ubuntu 20.04 to use with Splunk.

As far as I know there are no easy way to send udp packets from Windows.
To use NetCat (nc) you need a linux device for testing, it can be a raspberry pi.
You need just for sending data to your Windows Splunk server.

PS no need to quote the whole message above you. Use the Post Reply button below the post.
 
pixture08
just joined
Posts: 5
Joined: Mon Jun 07, 2021 3:32 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 1:27 pm

Thank you! I'll try to install a Linux VM instance instead. Very noted on quoting reply as well. Cheers!
 
pixture08
just joined
Posts: 5
Joined: Mon Jun 07, 2021 3:32 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 3:35 pm

Hi Jotne,

I tried to test if my VM windows instance cannot received UDP packets from other public ip address using packet sender app. As per testing, I could receive UDP packets (please see attached testing here https://ibb.co/7QjVH5X). Is there a way to check/troubleshoot the Splunk app? I've tried searching on this and cannot find any best answer :((
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 3:49 pm

There has to be something that blocks the UDP packets, or Splunk does not listen on UDP.
Still not sure how you run Splunk. On Windows or on Linux?

If splunk runs on Windows and you have open port 514 in Windows Splunk setup, as an administrator run the following command from CMD
netstat -toan
You should see that your Windows listening on port 514 udp
UDP 0.0.0.0:514
PS Mikrotik app has nothing to do if you receive syslog or not.

To se in Splunk if you get any data, use search and search for
index=*
Try just for test to see internal log on splunk as well
index=_internal
But I do suggest you should install Splunk on Linux as a non root user.
 
pixture08
just joined
Posts: 5
Joined: Mon Jun 07, 2021 3:32 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 8:10 pm

I've figured it out and happy to share it to the community!

To people who use Windows VM instance (mine is Windows server 2012 r2)
1. After installing Splunk, it will not automatically listen to UDP port 514
2. to confirm, you need to open a cmd to C:\Program Files\Splunk\bin, then type splunk add udp 514 -sourcetype syslog
3. If Splunk is listening to the said port, it should response with "Listening for port input on the following UDP ports: 514" --> (You're good to go)
4. if you received a response "Splunk is not listening for input on any UDP input." Then proceed reading this.
5. On the same CMD path, encode splunk add udp 514 -sourcetype syslog. It will return a response of "Listening to UDP input on port 514."
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 08, 2021 9:42 pm

No need for command line.

If you run Splunk in Windows or as root in Linux (not recommended), you can do:

Settings->Data Inputs->UDP->New Local UDP
Port: 514 -> Next
Select Source Type: Operating system-> Syslog
Review->Submit

Then you should be good to go.

BUT As I do recommend using Linux and not as Root, Splunk it self then can not listen for port < 1024. So that is way in this thread there are a description of to use Rsyslog (listen in udp:514) that gets the data and Splunk reads it log files.
 
David1234
Forum Guru
Forum Guru
Posts: 1424
Joined: Sun Sep 18, 2011 7:00 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jun 16, 2021 4:16 pm

great guide
I need some first time help

I have done "almost" everything you said

this is what I have done in the login
/system logging action
set 3 remote=192.168.1.0
/system logging
add action=remote prefix=MikroTik topics=account
add action=remote prefix=MikroTik topics=critical
for now I want to see all the login attempts made to my router

I can see in the "MikroTik Admin user login" all the login (good and bad)
so I guess this working , now? right

I have also added the script - but I can't see any data on router voltage\temp
the script in running without any problems - so why I don't see in the the Splunk?

I can't see any other data also in no page - everything is No results found. (even on router uptime, temp\voltage )
why is it?
the accounting is working
/ip accounting
set enabled=yes threshold=2560
do I need to enable something in the firewall so the server can read this data or something ?
**also why do I need to enable the accounting? , the data is send out from the router and not from the Splunk to the router - or I missunderstand soemthing?
Thanks ,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jun 16, 2021 5:48 pm

Remember with this:
add action=remote prefix=MikroTik topics=account
add action=remote prefix=MikroTik topics=critical
You only get all account and critical logs and nothing else from the internal logs.

That is what I do use:
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=hotspot
This gets all dhcp logs including debug dhcp packets.
This gets all hotspot logs including debug hotspot packets.
With the !, get all other logs that are not debug, packet and snmp.

Voltage and temperature are only on devices that do supports it.
Accounting is used to get all the packet flow from the router to Splunk.
Accounting is gone in v7, so there netflow has to be used. Not implemented in my solution, yet.
 
David1234
Forum Guru
Forum Guru
Posts: 1424
Joined: Sun Sep 18, 2011 7:00 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jun 16, 2021 6:02 pm

OK
I just saw you answer
I thought that on the RBM33G there is temp\voltage , but now I see there isn't any
 :put [/system health get cpu-temperature ] 

I can see the device now , I looked somewhere else

so if I want to see only spesific things I can only make logs for them ?
is there any where a WiKi or some explain about what each page can show and how he read it ?
for example -
this is what I get in the Device list
18.14.45.136	Test_Router	A2FFFFF71F31	RBM33G	RBM33G	6.48.2 (stable)	1	
is there any way to change the first IP to some local variable to be sent?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jun 16, 2021 7:51 pm

You can send anything to logs, in for example this form.
:log info message="This is a test"
Then in splunk you should be able to see this by search for:
"This is a test"
 
David1234
Forum Guru
Forum Guru
Posts: 1424
Joined: Sun Sep 18, 2011 7:00 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Thu Jun 17, 2021 2:59 pm

great
I will try to see now what can I do with this

thank you !

*** I will ask if I will have more questions
 
David1234
Forum Guru
Forum Guru
Posts: 1424
Joined: Sun Sep 18, 2011 7:00 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sun Jun 20, 2021 1:16 pm

more quetsions:
for my testing I only want to be able to see the MikroTik Admin user login block\OK
I have setup a loggin like this:
/system logging action
add name=Splunk remote=12.15.15.14 target=remote
/system logging
add disabled=yes topics=ovpn,debug
add action=DavidServer disabled=yes topics=warning
add action=Splunk prefix=MikroTik topics=account
add action=Splunk prefix=MikroTik topics=critical
when I put a wrong password I can see in the router logs "error login failure for user admin from 8.19.166.6 via winbox"
and on the splunk page I can see the line:
_time	host	user	system	user_ip	eventtype	host_name
2021-06-20 12:56:50	9.13.10.118	admin	winbox	 8.19.166.6	adm_user_login_failure	9.13.10.118
2 questions (help)
why I can't see the name I setup in the identity for this router in the host name \ host user system?

is there any way to make the login send more information ? for example the MAC of the ethernet and put it in the host name?
I'm asing because my devices use dynamic IP - so the router IP give me nothing.

Thanks ,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Sun Jun 20, 2021 11:38 pm

Everything is already logged in menu Overview ->MikroTik Admin user login.
There you see both logged inn and blocked user.

You have changed from the default log action setup and do loose a lot of information since you do only log some.
Change back to what is posted inn first post and you get all back.

This one logs all except: !debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
 
nikc
Member Candidate
Member Candidate
Posts: 208
Joined: Wed Jul 13, 2016 6:05 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 29, 2021 12:13 pm

1st up, amazing effort, love this, so easy to setup and use, thank you.

I have a question though ....

A lot of my nodes are LTE based devices, so they tend to change IP address quite frequently and as a result of that if i want to look at traffic utlisiation for the device, i need to collate all the stats from the various IPs.

Is there any way to aggregate the data at hostname level opposed to IP ?

Thanks
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Tue Jun 29, 2021 6:58 pm

The app do use IP as hostname and do sort everything after that, so when IP changes, it will see it as a new device.

What can be done is to tag all packed with an id so that all router can be identified even after IP change.

Example, change from:
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=hotspot
To:
add action=logserver prefix="MikroTik: sys=22" topics=dhcp
add action=logserver prefix="MikroTik: sys=22" topics=!debug,!packet
add action=logserver prefix="MikroTik: sys=22" topics=hotspot
Use different number or name for the sys to distinguish each ruter.
Take care not use to long ID, since it eats of the max packet size.

Then you need to adapt the Splunk app to use sys instead of host
Its some work to do, but it will give you separate logs for each router based by the sys name.
 
David1234
Forum Guru
Forum Guru
Posts: 1424
Joined: Sun Sep 18, 2011 7:00 pm

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Jul 05, 2021 9:50 am

another question
I notice that my all my device are sending data to another syslog system
so I have connected to the log file of it and sendting the data to the Splunk
it's working - I get the data - but not as mikrotik (just as regular udp syslog )
is there anything I can change in the setting of the Splunk-Mikrotik to read that data?

this is what I'm getting on the splunk :
SEVERE 05/07 06:24:06 wrong message: 3.13.12.118 <27>Jul  5 09:24:06 Router script error: error - contact MikroTik support and send a supout file (2)
can I change the reading somehow so it will enter the splunk as Mikrotik ?

Thanks ,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Mon Jul 05, 2021 2:01 pm

If I understand you, you do not get the "MikroTik" prefix?
system,info,account MikroTik: sys=22: user hidden logged out from 1.2.3.4 via winbox
The whole app with field extraction etc are based on this tag, so If that changes or are removed, lots of the stuff inn the app must be rewritten.
Possible, but you need to try to figure it out.
 
eb123456
just joined
Posts: 23
Joined: Sat Jul 24, 2021 6:20 am
Location: California

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jul 28, 2021 7:32 am

Thank you so much Jotne this is an amazing tool you put together!
I am new to Mikrotik, just got the router 2 weeks ago.
I am wondering how you guys get the hostname to show in the screenshots? I have the IP address instead?
Image
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jul 28, 2021 2:54 pm

Thank you so much Jotne this is an amazing tool you put together!
I am new to Mikrotik, just got the router 2 weeks ago.
I am wondering how you guys get the hostname to show in the screenshots? I have the IP address instead?
Image
My Splunk host (=VM) just resolves these.
If you have a rather "static" small home-network, just add them to /etc/hosts or something on the Splunk host so your internal entries are resolved too.
If you run a Pihole or other DNS internal just let Splunk use that one and add the entries over there.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Wed Jul 28, 2021 6:34 pm

I am wondering how you guys get the hostname to show in the screenshots? I have the IP address instead?
Some scripts run only once every day, so if you wait one day, it should be ok.

PS post image on the forum using Attachements below the post instead of posting a link.
 
eb123456
just joined
Posts: 23
Joined: Sat Jul 24, 2021 6:20 am
Location: California

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Thu Jul 29, 2021 5:56 am

Thank you editing the /etc/hosts does provide immediate name resolution.

I need to do some digging though: on the splunk host I seem to be experiencing local DNS issues:

me@docker01:~/scripts$ host 192.168.88.24
Host 24.88.168.192.in-addr.arpa. not found: 3(NXDOMAIN)
me@docker01:~/scripts$ ping DESKTOP-MESF01
ping: DESKTOP-MESF01: Temporary failure in name resolution
me@docker01:~/scripts$ ping DESKTOP-MESF01.local
PING DESKTOP-MESF01.local (192.168.88.24) 56(84) bytes of data.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Thu Jul 29, 2021 8:35 am

Ah ok, you run Splunk in a container.
DNS resolving in a container is slightly different so I read. Perhaps you could try info below ?
On linux systems, DNS resolution happens using /etc/resolv.conf file, check this file inside your container, if it has invalid DNS, then your container won't be able to resolve hostnames.

Docker uses a property set in /etc/docker/daemon.json file(on host) for populating entries in /etc/resolv.conf inside container.
update the value of this property in /etc/docker/daemon.json file in host machine:

{
"dns": ["8.8.8.8"]
}
Note: This change requires restarting docker to take effect, also existing containers have to be removed and created again.

Restarting Docker:

sudo systemctl restart docker
removing containers:

sudo docker stop <container-name/id>
sudo docker rm <container-name/id>
 
eb123456
just joined
Posts: 23
Joined: Sat Jul 24, 2021 6:20 am
Location: California

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Jul 30, 2021 1:33 am

oh I should have mentioned that the command line output I shared were directly from the docker host that runs the docker container. The DNS issues materialize on the physical machine and for some reasons name resolution only happen with the .local domain.

Anyway - I did not figure it out yet, but I am happy with the /etc/hosts solution. I actually find it better than DNS resolution because I can give more friendly name to my devices.

For those who are contemplating running splunk in docker, here is my script:
docker stop splunk-server 
docker rm splunk-server 
docker run -d \
-p 8000:8000 \
--name splunk-server \
-v /data/syslog/udp:/data/syslog/udp \
-v /data/syslog/tcp:/data/syslog/tcp \
-v /data/splunk/etc:/opt/splunk/etc \
-v /data/splunk/var:/opt/splunk/var \
-v /etc/localtime:/etc/localtime:ro \
-v /etc/hosts:/etc/hosts:ro \
-v ~/conf/input.conf:${SPLUNK_HOME}/etc/apps/MikroTik/default/input.conf \
-e SPLUNK_START_ARGS='--accept-license' \
-e SPLUNK_PASSWORD='XXXXXXXXXXX' \
--restart always \
splunk/splunk:latest
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.2 (Graphing everything)

Fri Aug 13, 2021 8:18 am

# Script version 4.4
# 4.4 Removed on-error from wifi
# 4.4 Removed on-error from board info

Some small changes to error handling, no need for upgrade.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 12:17 pm

Splunk for MikroTik updated to v3.3

New file found under section 1g) at the first post

This version contains lots of tweaks and fixes, and some new KV store

To upgrade, delete the folder /splunk/etc/app/Mikrotik
Then install the unpacked spl (use winrar/winzip) file, install app from
"Manage app" -> "Install app from file"

To get the most out of this version, upgrade the script (not needed) on the router to latest version. (3.9+)

# 3.3 (13.08.2021)
# New KV store (used to store info about Devices and DHCP info
# Changed from table to fields in base search for all dashboard to speed up searches.
# Added more DNS extraction
# Updated script to 4.4 (Added firmware info)
# Added frimware info "MikroTik Device List"
# Fixed lookup "MikroTik Wifi strength"
# Added new DoH dashboard "MikroTik DoH information"
# Change table format "MikroTik Interface Changes"
# Fixed typo in eventtype "MikroTik VPN Connections"
# Added user local ip "MikroTik VPN Connections"
# Fixed missing search in 2b, duplicate entry in user-name, added 1b login info "MikroTik VPN Connections"
# Fixed to not show false logged in user "MikroTik VPN Connections"
# Fixed html formatting in "Mikrotik DHCP to Static"
# Fixed missing City in "Mikrotik Firewall Rules"
# Moved "Mikrotik uPnP" to Connections menu
# Added table view to "Mikrotik uPnP"
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 1:41 pm

Thanks for the update Jotne.
I seems I've lost data on the "Overview" > "Device List" is not populating (as in "no result found")
All other tabs seems to work find, data is still coming in normally.
I'm running it on a RB3011 / 6.47.x release

If I look in the output on the console while the script is running, I'm only seeing this line : " script=version ver=4.4" where I normally would need to see more like

:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial identity=\"$identity\" factory-firmware=\"$ffirmware\" current-firmware=\"$cfirmware\" upgrade-firmware=\"$ufirmware\""



The previous script had this section

# Collect system information
# ----------------------------------
:local model
:local serial
if ($SystemInformation and $run) do={
:local version ([/system resource get version])
:local board ([/system resource get board-name])
:do {
:set model ([/system routerboard get model])
:set serial ([/system routerboard get serial-number])
} on-error={
:set model na
:set serial na
}
:local identity ([/system identity get name])
:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial identity=\"$identity\""
}

While you new script has this one


# Collect system information
# ----------------------------------
:local model na
:local serial na
:local ffirmware na
:local cfirmware na
:local ufirmware na
:if ($SystemInformation and $run) do={
:local version ([/system resource get version])
:local board ([/system resource get board-name])
:if ([/system routerboard get routerboard]) do={
/system routerboard
:set model ([get model])
:set serial ([get serial-number])
:set ffirmware ([get factory-firmware])
:set cfirmware ([get current-firmware])
:set ufirmware ([get upgrade-firmware])
}
:local identity ([/system identity get name])
:log info message="script=sysinfo version=\"$version\" board-name=\"$board\" model=\"$model\" serial=$serial identity=\"$identity\" factory-firmware=\"$ffirmware\" current-firmware=\"$cfirmware\" upgrade-firmware=\"$ufirmware\""
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 4:07 pm

If I can, I want suggest something...

list is from -> to
:local hour [:pick ([/system clock print as-value ]->"time") 0 2]
:local hour [:pick [/sys clock get time] 0 2]

	:foreach logline in=[/ip firewall nat find dynamic=yes] do={
	:foreach logline in=[/ip firewall nat find where dynamic=yes and comment~"^upnp "] do={

	:if (([/system health get]~"state=disabled" || [/system health get]="")=false) do={
	:if (!([/system health get]~"(state=disabled|^\$)")) do={
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 4:47 pm

@ jvanhambelgium

Some script runes only once a day and since some part is changed to use KV store, it may take a day before all is populated.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 5:01 pm

:foreach logline in=[/ip firewall nat find where dynamic=yes and comment~"^upnp "] do={
This I do not understand. Can there be other dynamic nat than upnp lines, and why should I include the comments test.
Other line are fine, thanks :)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 5:13 pm

the comment can distinguish the upnp rules from the other dynamics
yes, dynamic SMTP from HotSpot, and other related dynamic hotspot rules, walled garden rules, etc.
for example

print code

/ip firewall nat> pri
Flags: X - disabled, I - invalid, D - dynamic 
 0  D chain=dstnat action=jump jump-target=hotspot hotspot=from-client 

 1  D chain=hotspot action=jump jump-target=pre-hotspot 

 2  D chain=hotspot action=redirect to-ports=64872 protocol=udp dst-port=53 

 3  D chain=hotspot action=redirect to-ports=64872 protocol=tcp dst-port=53 

 4  D chain=hotspot action=redirect to-ports=64873 protocol=tcp hotspot=local-dst dst-port=80 

 5  D chain=hotspot action=redirect to-ports=64875 protocol=tcp hotspot=local-dst dst-port=443 

 6  D chain=hotspot action=jump jump-target=hs-unauth protocol=tcp hotspot=!auth 

 7  D chain=hotspot action=jump jump-target=hs-auth protocol=tcp hotspot=auth 

 9  D chain=hs-unauth action=return dst-address=179.44.232.13 

10  D ;;; www.paypal.com
      chain=hs-unauth action=return dst-address=2.17.140.171 

11  D ;;; www.paypal.com
      chain=hs-unauth action=return dst-address=2.20.40.117 
[...]
Last edited by rextended on Fri Aug 13, 2021 5:21 pm, edited 3 times in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 5:18 pm

another hint:

# from
        :set ( $array->$listname ) 1
[...]
    :foreach k,v in=$array do={


# to
        :if (!($array ~ $listname)) do={ :set array ($array , $listname) }
[...]
    :foreach k in=$array do={
# only k without v
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 5:34 pm

WARNING: BREAKED COMPATIBILITY
missing : before some occurrences of if
(no matter is missing on root, but better if you put : on every if also on root to be sure...)
	if ([/ip accounting get enabled]=yes) do={

		if ($listdynamic = true) do={

		if ($i = $cmd) do={:set $f 1}
		if ($f<>1) do={

missing : before some occurrences of foreach
		foreach logline in=[/ip accounting snapshot find] do={

	foreach i in=[/system history find] do={
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 6:31 pm

anoter two fix:

replace all this
:do {
    :if ([:len [/caps-man registration-table find]] > 0 and $CAPsMANN) do={
        
    }
} on-error={}

with only this
:if ( ([:len [/interface find where type="cap"]] > 0) and $CAPsMANN) do={ /system script run capsman }



and replace all this
:do {
} on-error={
    :if ([:len [/system ntp client get last-update-from]]>0) do={
        :log info message="script=ntp status=synchronized"
    } else={
        :log info message="script=ntp status=not-synchronized"
    }
}

with this:
:local ntpstatus ""
:if ([:len [/system package find where !disabled and name=ntp]] > 0) do={
    :set ntpstatus [/system ntp client get status]
} else={
    :if ([:typeof [/system ntp client get last-update-from]] = "nil") do={
        :set ntpstatus "using-local-clock"
    } else={
        :set ntpstatus "synchronized"
    }
}
:log info message="script=ntp status=$ntpstatus" 
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 7:24 pm

I do not check (for now) the "Collect DHCP Pool information" because is really complicated and I must have more time.

I hope you like my effort to help you :P
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 9:24 pm

I do own you several beers for helping out 🍺🍺🍺🍺🍺
DHCP part is the only part I have taken from some other ;)

Added all parts to the script. Even found some local variable without :
Not 100% sure on what I do with the uPnP script. One option is to use your solution, other is to rename Dashboard to Dynamic NAT, and to show all types of NAT added to the Router. Do the other dynamic nat has comment set?

Comments for dynamic nat are some strange. In Winbox overview, you do see the comment, but when you open a rule, the comment button is missing and you can not see the comment.
/ip firewall nat print does not show to command for dynamic lines.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Fri Aug 13, 2021 9:56 pm

Very thanks for the beers ;)

I just notice also one :do without the : , :set variable with $ in front of variable, and what you already have noticed for global and local
apparently new routeros 6.48+ ignore if the : are present or not, but previous version get an error when the script is on "subdirectory" like "/ip address" and : are omitted.
# *** missing : on do {
# Get uncounted data
		do {
			/ip accounting uncounted {

# *** remove $ from first $value in front of ":set"
				:set $value [:pick $value 0 $newline]

I rewrite this directly, too much errors
:if ([:tonum [:pick [/system resource get version] 0 1]] > 6 and $CmdHistory) do={
#       v missing : on both global and local
	:global cmd
	:local f 0
	:foreach i in=[/system history find] do={
#               v missing : on both if    v removed $ from $f
		:if ($i = $cmd) do={ :set f 1 }
#               v       v replaced <> with != but not really an error if v7 accept <>
		:if ($f != 1) do={
			:log info message="StartCMD"
			:log info message=[/system history get $i]
			:log info message="EndCMD"
		}
	}
#       v missing :          v missing /
	:global cmd  [:pick [/system history find] 0]
}

About the dynamic NAT, can be one from 3 types:

this match upnp rule because all upnp rule are "upnp <MACHINE-IP>: <SERVICE>" real example: "upnp 192.168.0.163: DIRECT_REVERSE_SERVER"
[/ip firewall nat find where dynamic=yes and comment~"^upnp "]

this match dynamic rules added from hotspot
[/ip firewall nat find where dynamic=yes and chain="hotspot"]

this match dynamic rules added from walled garden
[/ip firewall nat find where dynamic=yes and chain="hs-unauth"]
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Sat Aug 14, 2021 12:26 am

@ jvanhambelgium

Some script runes only once a day and since some part is changed to use KV store, it may take a day before all is populated.
Fixed now ;-)
Sorry for jumping the gun on this 8)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything)

Sat Aug 14, 2021 11:02 am

# Script version 4.5
# fixed missing :
# Simplifyed some commands
# Changed foreach loop
# Changed NTP section
# Changes CAPsMANN section
# Removed $ in set command
# Fixed v 7 module missing : / and canged test
# Added dynamic nat types

Mostly bug fixes. A big thanks to rextended
PS If you upgrade script to 4.5, you will lose uPnP info. Will be fixed in upcoming 3.4 with new dynamic nat rule view.

Do any use the Web Proxy view, or is Web Proxy no longer used by anyone due to HTTPS etc? If so I will remove it.
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Wed Aug 25, 2021 10:15 am

Hi,
This is great work.
But - Well, something is not working for me.
I have a CHR Mikrotik running the script, and a virtual Debian running Rsyslog + Universal Forwarder.
The UF delivers to a Splunk Enterprise (Free) index MikrotikInfo with the Mikrotik App attached.
Data is coming into the index, but I see nothing in the app. In Data -> indexes I see the event counter increasing.

Where did I take a wrong turn ?

Regards from Denmark
Klaus
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Wed Aug 25, 2021 11:35 am

Normal its problem that the tag is wrong.
It need to be MikroTik with uppercase M and T

Try following search:
index=* sourcetype=mikrotik
if no data, try:
index=* host=<ip of your router>
if no data try
index=*
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Wed Aug 25, 2021 2:54 pm

I am able to find events, using "index=* sourcetype=MikroTik"
In the index I am able to find all the Neighbor units.
Nothing in the App
This is how I monitor:
bin/splunk add monitor /var/log/remotelogs/192.168.1.126/script,info.log -index MikrotikInfo -sourcetype MikroTik
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Wed Aug 25, 2021 10:11 pm

I guess the index (MikrotikInfo ) you store the MikroTik data is not set a default search index for you.

If you do not see event here:
sourcetype=MikroTik
But this is ok
index=* sourcetype=MikroTik
If you user is part of the admin role, try this:

Settings->Roles->Admin->Indexes
Find the index (MikrotikInfo ) where you store the MikroTik data and in column Default and mark it.
You may need to log out/inn.

If this work, post it here and I will update this info in top post so that other user not using default index will get it to work.
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Thu Aug 26, 2021 10:46 am

Getting closer.
After setting the index MikrotikInfo to "default", and restarting splunk (!), I am now able to see my syslogsrv in the devicelist as a host.
But no devices are listed.
I have set the routerID to "MikroTikCHR". (A virtual CHR for testing purposes)

Then I started tailing the syslog file on syslogsrv. This came up after I rebooted the router:
2021-08-26T09:26:46.302962+02:00 192.168.1.126 script,info script=version ver=4.5
2021-08-26T09:26:46.305321+02:00 192.168.1.126 script,info script=resource free_memory=54 MB total_memory=96 MB free_hdd_space=26 MB total_hdd_space=63 MB cpu_load=1 uptime=00:02:52 write-sect-total=3577
2021-08-26T09:26:46.307561+02:00 192.168.1.126 script,info script=ntp status=started
2021-08-26T09:26:46.307938+02:00 192.168.1.126 script,info script=traffic,fasttrack=0
2021-08-26T09:26:46.318521+02:00 192.168.1.126 script,info .id=*1;comment=;name=ether1;rx-byte=154737;rx-drop=0;rx-error=0;rx-packet=1532;tx-byte=385454;tx-drop=0;tx-error=0;tx-packet=1508;tx-queue-drop=0;script=if_traffic
2021-08-26T09:26:46.320081+02:00 192.168.1.126 script,info script=sysinfo version="6.48.4 (stable)" board-name="CHR" model="na" serial=na identity="MikroTikCHR" factory-firmware="na" current-firmware="na" upgrade-firmware="na"
2021-08-26T09:26:46.321610+02:00 192.168.1.126 script,info script=neighbor nid=3 .id="*3"
2021-08-26T09:26:46.321711+02:00 192.168.1.126 script,info script=neighbor nid=3 address="192.168.1.1"
2021-08-26T09:26:46.321820+02:00 192.168.1.126 script,info script=neighbor nid=3 address4="192.168.1.1"
<------ the rest deleted ------>
In the index MikrotikInfo this event appears: :-)
26/08/2021
09:26:46.320
2021-08-26T09:26:46.320081+02:00 192.168.1.126 script,info script=sysinfo version="6.48.4 (stable)" board-name="CHR" model="na" serial=na identity="MikroTikCHR" factory-firmware="na" current-firmware="na" upgrade-firmware="na" host = syslogsrvsource = /var/log/remotelogs/192.168.1.126/script,info.logsourcetype = MikroTik

So we have the identity string in the index, but it is not shown in the device list.
If you would like the entire logfile, I will happy to share it.

Regards
/Klaus
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Thu Aug 26, 2021 2:10 pm

Here is how data looks in my logs in Splunk
script,info MikroTik: script=pool pool=DHCP-Pool-vlan20-Guest used=50 total=190
script,info MikroTik: script=pool pool=VPN-pool used=0 total=18
script,info MikroTik: script=pool pool=DHCP-Pool-vlan1-Home used=252 total=455
script,info MikroTik: script=neighbor nid=3 version="6.48.1 (stable)"
script,info MikroTik: script=neighbor nid=3 uptime="05:07:14"
You can see the MikroTik tag. For me it seem that you do not have it in your logs.

Its set in section 2b
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=hotspot
What do you see with this:
index=* host 192.168.1.126 |  table index host sourcetype source _raw
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Thu Aug 26, 2021 11:21 pm

# Script version 4.6
# fixed Wifi script not working after upgrading RouterOS to 6.48.4

To upgrade, just cut/past the script to all router.
If you do not use Wifi, not need for upgrade.
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Fri Aug 27, 2021 9:42 am

Here is how data looks in my logs in Splunk
script,info MikroTik: script=pool pool=DHCP-Pool-vlan20-Guest used=50 total=190
script,info MikroTik: script=pool pool=VPN-pool used=0 total=18
script,info MikroTik: script=pool pool=DHCP-Pool-vlan1-Home used=252 total=455
script,info MikroTik: script=neighbor nid=3 version="6.48.1 (stable)"
script,info MikroTik: script=neighbor nid=3 uptime="05:07:14"
You can see the MikroTik tag. For me it seem that you do not have it in your logs.

Its set in section 2b
/system logging add action=logserver prefix=MikroTik topics=dhcp
/system logging add action=logserver prefix=MikroTik topics=!debug,!packet,!snmp
/system logging add action=logserver prefix=MikroTik topics=hotspot
What do you see with this:
index=* host 192.168.1.126 |  table index host sourcetype source _raw
My bad - It is a case of RTFM. I did not put the MikroTik prefix in the action.
I cleaned out all old indexfiles and data, and started over, to remove old data.
I am now able to se that a logfile exists in the Mikrotik app.
I will hack on :-)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Fri Aug 27, 2021 12:59 pm

That was my first reply to you :)
Normal its problem that the tag is wrong.
It need to be MikroTik with uppercase M and T
and it need to be present :)
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Mon Aug 30, 2021 2:26 pm

Well, sometimes you need to start over. I made a new Splunk enterprise server, and a rsyslog/forwarder server.
I am getting data into the MikroTik3.3 app.
Nice :-)

Thank you for your patience, and effort.

Regards from Denmark
Klaus
 
kaydee
just joined
Posts: 10
Joined: Fri Apr 12, 2019 1:02 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Sat Sep 04, 2021 12:41 pm

Version 3.3 13.08.2021


1g) Download the Splunk spl file:
MikroTik3.3.rar


Is this file corrupted just for me or everyone else?
can you share md5?
Thanks
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Mon Sep 06, 2021 3:48 pm

File is ok. Downloaded on an other computer from other location. Extracted fine.
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Wed Sep 08, 2021 10:43 am

Hi, I am back :-)
I started all over, and made a new Debian based Splunk Enterprise server and a new Rsyslog/UF server.
My logfiles are forwarded to the correct index, which is connected to the MikroTik App, and it has a lot of events.
But hardly anything shows up in the app.
My syslogsrv shows up in the device list as a host, and in the index it has host=syslogsrv, which I would expect.
Nothing else shows, and the log file size claims a NULL in the left window. If I press NULL, this is requested:

sourcetype=mikrotik host=* | eval len=len(_raw), host_name=coalesce(identity,host) | eval module=if(module="script",module."-".script,module) | fields _time len module host host_name | eval len=len/1024/1024 | eval host=host_name."-".host

and these are the first two lines in the reply (5000+ events):
08/09/2021 09:27:49.022 2021-09-08T09:27:49.022737+02:00 router.lan dhcp,debug,packet MikroTik: Client-Id = 01-08-55-31-17-BD-8F
host = syslogsrv-syslogsrv
08/09/2021 09:27:49.022 2021-09-08T09:27:49.022737+02:00 router.lan dhcp,debug,packet MikroTik: Host-Name = "RB4011"
host = syslogsrv-syslogsrv
I notice the "host =" part where syslogsrv has doubled, and Host-Name="RB4011" is not present.

Going through all submenus under overview, I only found 1 line in neighbors, which was syslogsrv connected to the Router, besides what was in log file
When I change log file size sorting to "host" I get all my devices, as <devicename>-syslogsrv, and one -syslogsrv.
Looking at the index, all of them have host=syslogsrv-syslogsrv and not <hostname>-syslogsrv, which would make some sort of sense.

Do not tell me that the easy answer is 42 :-)

Well, I intend to have a nice day anyway, Hope you do too.
/Klaus
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Wed Sep 08, 2021 1:26 pm

The anser is 43.
You have an error some place in the format of the syslog packet coming inn to Splunk.
If you do search: sourcetype=Mikrotik, you should not see any date, and you see double date in _raw packet.

Your syslog:
08/09/2021 09:27:49.022 2021-09-08T09:27:49.022737+02:00 router.lan dhcp,debug,packet MikroTik: Client-Id = 01-08-55-31-17-BD-8F
Mine
dhcp,debug,packet MikroTik: dhcp-alert on Bridge1 sending discover with id 3480279547 to 255.255.255.255
Did you follow the rsyslog setup I did post. Solution should work if you follow it 100%

Also look at this:
PS Do NOT select BSD Syslog. It will mess up the logging format.

1b) PS: To install Splunk as a non root user.
Splunk setup:
viewtopic.php?p=677233#p677233
rsyslog setup
viewtopic.php?p=677233#p793342

Edit
Added section 2h) debugging in fist post.
 
klausf
just joined
Posts: 15
Joined: Sat Jan 02, 2021 5:19 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Thu Sep 09, 2021 12:16 pm

Thank you for your patience.

My Splunk Enterprise is now getting some data to show. :-)
I will now compile my notes, so even I may understand them in a few months *LOL*

Have a nice day
Klaus
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Thu Sep 23, 2021 7:30 pm

I do use Linux (Ubuntu) I have not see these problems.
So not sure why these happens.

Splunk do recommend to use Linux over Windows
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Fri Oct 08, 2021 10:41 am

# Script version 4.7
# Fixed CHR Router error in 7.1rc4
# Removed accouning section and unaccounted
# Fixed NTP to work with RouterOS > 6

To upgrade, just cut/past the script to all router.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Fri Oct 08, 2021 11:03 am

For some stupid reason, I have lost the "campsman" script.
If any one have it, please post it here :)

Edit script found. Thanks to: Francois :)
 
Sysxp
just joined
Posts: 4
Joined: Tue Oct 19, 2021 9:17 pm

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Tue Oct 19, 2021 11:03 pm

Wow, this thing is really funny!
I decided to try how it works, set everything up using the OP tutorial, and started checking through menus. When I reach "DNS -> Mikrotik DNS Requests", my Win2016 Server VM just died with 16Gb consumed memory. So I give it a whole 64 Gb! And what do you think? It dies again with about ~50+ GB mem consumption! This is just ... WOW! So powerful! :lol:
I only have 100Gb of RAM on this host, not sure shoud I give more?.. :o
I'm probably stupid, is it an expected behaviour?
But anyway, it was looking cool! Maybe I did something incorrectly?
--
Added some more - total 80Gb of RAM. Still no use, it takes all the available memory and then everything dies, even the RDP. Defenitely a bug, but a funny one, can't stop laughing feeling all that POWAH! :lol: Should I use the Linux version instead?
Image
Last edited by Sysxp on Tue Oct 19, 2021 11:36 pm, edited 3 times in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Thu Oct 21, 2021 5:27 pm

Something has to be wrong.
I do have a PC with only 16GB memory (Ubuntu), and has no problem running lots of logs.

I do recommend that run Splunk on Linux (Ubuntu), even if it works on Windows.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: &#128204; Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) &#128190; &#128736; &#128187; &#1282

Tue Nov 02, 2021 12:06 pm

New version out in new thread:
viewtopic.php?t=179960

Who is online

Users browsing this forum: No registered users and 4 guests