📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

RouterOS Useful user articles
21

So I upgraded to this latest version.
I am not familiar enough with splukt to tune this, but it is getting really sloooooooow
Any hints in how to get better performance ?
removing old data ?
some indexing and how ?
getting a warning :

Storage engine migration recommended

If your instance uses the MMAPv1 storage engine,

how do I find out if my docker instance uses this ??
free splunk license btw

22

There a LOT of interesting documents on Splunk that are easy to read.
Including some to limit the retention for example. I only keep about 30days I believe. I’m running it on a NAS, together with 5 other VM’s and 15+ containers so I have to make choices consuming nearly all 16GB that is in my NAS.


Concerning the storage-engine, see for example below

https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/MigrateKVstore

It seems I’m also using the “old” MMAPv1 but I don’t have such messages. I’m running 8.2.0 (I’m not updating to release 8.2.3 since the fixes are not interesting to me)

For licensing, go to “Settings” (on the top menu) , then “Licensing” en there you’ll see what you have, what volume of the 500MB/day you’ve used etc.

23

tnx,

I am using docker on synology. Splunk has 2GB out of 4GB availiable RAM …

running portainer to manage and watchtower to automaticly upgrade docker images when they are availiable …

I surely would like some options to autoclean log if older than xx days …
It would probably make splunk a lot faster here

24

In the bin-folder where the splunk binaries are, issue #./splunk show kvstore-status
It will provide the type of store.

This member:
backupRestoreStatus : Ready
date : Mon Nov 8 20:15:19 2021
dateSec : 1636398919.648
disabled : 0
guid : 6D5FAC74-4CDC-4F5C-AEA7-16B04EB7F9AE
oplogEndTimestamp : Mon Nov 8 20:15:17 2021
oplogEndTimestampSec : 1636398917
oplogStartTimestamp : Mon Oct 4 06:20:29 2021
oplogStartTimestampSec : 1633321229
port : 8191
replicaSet : 6D5FAC74-4CDC-4F5C-AEA7-16B04EB7F9AE
replicationStatus : KV store captain
standalone : 1
status : ready
storageEngine : mmapv1


I quickly performed migration to wiredTiger using the document-link earlier. It took only few minutes without issues.Just follow the procedure.


backupRestoreStatus : Ready
date : Mon Nov 8 20:39:04 2021
dateSec : 1636400344.346
disabled : 0
guid : 6D5FAC74-4CDC-4F5C-AEA7-16B04EB7F9AE
oplogEndTimestamp : Mon Nov 8 20:39:01 2021
oplogEndTimestampSec : 1636400341
oplogStartTimestamp : Mon Nov 8 20:32:08 2021
oplogStartTimestampSec : 1636399928
port : 8191
replicaSet : 6D5FAC74-4CDC-4F5C-AEA7-16B04EB7F9AE
replicationStatus : KV store captain
standalone : 1
status : ready
storageEngine : wiredTiger

25

Upgraded here too :wink:

and now try to find a way to limit and auto clean data to speedup things

26

Splunk community has tons of information.

https://community.splunk.com/t5/Getting-Data-In/Index-Retention-Time/m-p/495331

I’ll think you’ll find answers there to set some limits.

27

Ok, I am trying to find the correct indexes.conf…
I tried to do so in the webinterface but no luck thru setting reduction on 90 days …

finally found it in /opt/splunk/etc/system/local/indexes.conf
needed to restart splunk to get this active

28

Seems to be that there will be a working Traffic accounting for v7.x without need to use netflow.
Will be out in next version if all is ok:
http://forum.mikrotik.com/t/to-mt-keep-accounting-v7-x/151083/24

29

With the netflow you do have some more insight in the port-usage too and not just IP’s.
Ideal to possibly pick up certain abnormal “flows”
I agree there is a lot TCP/443 these days but still…

30

I know NetFlow is a much more in depth analyze tool and gives information about every packet.
My goal is to deliver some that is simple and many can use to monitor their routers.
Kid Control and IP Accounting, gives information about who is downloading/uploading, how much and when.
Should be enough for most small/medium network admins.

Will have a look at NetFlow later to see if I can get it to work in a simple way with Splunk.

31

Here is a view that combine accounting with kid control.
You can see how much data my Chromecast downloads. 18MB last 4 hour (backgrround images).
At the same time it shows device (kid) control status. If its not in any group, its just used to monitor traffic (dynamic). It can be set to a group with various status, open, blocked manually, blocked due to time limit, blocked due to rate limit etc.

PS Kid Control should be renamed to Device traffic control, since its not just kids you like to block, it may be other devices as well.

.
Kid Accounting2.jpg

32

I agree, there are many uses of this device tracking and control that extend beyond kids. I can also see potential for enhancing it even more with a few more features - just a few useful ones I have thought of:

  • The ability to be able to create a simple queue per host that includes the IPv4 address and IPv6 addresses - since “Devices” in kid control tracks this, there could easily be a “rate limit” setting in there. Note this is different from rate limiting per kid because sometimes you might want to limit per device like this.
  • The ability to dynamically place the IPv4 and IPv6 addresses for a single device or kid into an address list, that way they could be flexibly used in firewall rules.
  • Groups of “kids” could be created for things like departments of a company and used to populate address lists to allow creation of firewall rules based on department

Companies could use these features to restrict what some employees can do compared to others, and to provide an audit log of who had what IPv4 and IPv6 addresses at a given time. Kid-control has many practical use cases outside of restricting kids.

33

Next version will have better health and works better with 7.1

Here is an example on Routers giving PSU State
.
psu_state.jpg

34

Script version 4.8

Change to kid kontroll for accounting (needs to be fixed)

Fixed possibility to turn off account data

Updated health section to get all health info on old and new system to work better with 7.x

To upgrade, just cut/past the script to all router. (script found in first post)

NB If you do use accounting from 6.x, do not upgrade this script with also update the main Splunk version to minimum 3.5
This is due to change from accounting to kid control, since accounting does not work in RouterOS 7.x

35

Upgraded to 3.5

Happy Xmas :christmas_tree: :wrapped_gift:

3.5 (20.12.2021)

Changed from IP Accounting to Kid Control to get accounting data to work with 7.x RouterOS

Renamed “MikroTik Volt/Temperature” to “MikroTik Health”

Added more info to “Mikrotik Health”

Since the new app now uses Kid control to collect accounting data, you need to know the following.

  1. To use accounting, you need at Script at least on v4.8 or larger.
  2. You will no longer see historical data from old accounting.
  3. To get Kid Control data see section 2e) in first post.

Upgrade can be done by just replacing old files and restart Splunk

36

Next version will have a dashboard for Netwatch. With that you can keep track of when devices goes up and down.
It can also be used to monitor the stateless Wireguard VPN that can not be monitored as normal VPN can.
.
netwatch.jpg

37

Thanks Jotne!
Also usable for example to monitor ZeroTier participants on your “cloud” LAN.

38

Do ZeroTier work more or less like Wireguard with no logging on connecting/up/down etc?
If yes, this can be used for ZeroTier as well.

39

So it seems, my “interface” “zerotier1” is always UP ,but with Netwatch I can ping/test “remote endpoints” that also participate in the ZeroTier network.
I get “up / down” notifications through Netwatch the moment I switch on the ZeroTier VPN app on my Android phone
Off course conceptually ZeroTier is a bit different from WireGuard

40

There are many solution.

With Splunk you have 100% control of everything. You server, your setup. And free (up to 500MB/day)
Store as much data as long as you like.