Tool: Using Splunk to analyse MikroTik logs 3.3 (Graphing everything) 💾 🛠 💻 📊

RouterOS Useful user articles
461

That was my first reply to you :slight_smile:

Normal its problem that the tag is wrong.
It need to be MikroTik with uppercase M and T

and it need to be present :slight_smile:

462

Well, sometimes you need to start over. I made a new Splunk enterprise server, and a rsyslog/forwarder server.
I am getting data into the MikroTik3.3 app.
Nice :slight_smile:

Thank you for your patience, and effort.

Regards from Denmark
Klaus

463

Is this file corrupted just for me or everyone else?
can you share md5?
Thanks

464

File is ok. Downloaded on an other computer from other location. Extracted fine.

465

Hi, I am back :slight_smile:
I started all over, and made a new Debian based Splunk Enterprise server and a new Rsyslog/UF server.
My logfiles are forwarded to the correct index, which is connected to the MikroTik App, and it has a lot of events.
But hardly anything shows up in the app.
My syslogsrv shows up in the device list as a host, and in the index it has host=syslogsrv, which I would expect.
Nothing else shows, and the log file size claims a NULL in the left window. If I press NULL, this is requested:

sourcetype=mikrotik host=* | eval len=len(_raw), host_name=coalesce(identity,host) | eval module=if(module=“script”,module.“-”.script,module) | fields _time len module host host_name | eval len=len/1024/1024 | eval host=host_name.“-”.host

and these are the first two lines in the reply (5000+ events):
08/09/2021 09:27:49.022 2021-09-08T09:27:49.022737+02:00 router.lan dhcp,debug,packet MikroTik: Client-Id = 01-08-55-31-17-BD-8F
host = syslogsrv-syslogsrv
08/09/2021 09:27:49.022 2021-09-08T09:27:49.022737+02:00 router.lan dhcp,debug,packet MikroTik: Host-Name = “RB4011”
host = syslogsrv-syslogsrv
I notice the “host =” part where syslogsrv has doubled, and Host-Name=“RB4011” is not present.

Going through all submenus under overview, I only found 1 line in neighbors, which was syslogsrv connected to the Router, besides what was in log file
When I change log file size sorting to “host” I get all my devices, as -syslogsrv, and one -syslogsrv.
Looking at the index, all of them have host=syslogsrv-syslogsrv and not -syslogsrv, which would make some sort of sense.

Do not tell me that the easy answer is 42 :slight_smile:

Well, I intend to have a nice day anyway, Hope you do too.
/Klaus

466

The anser is 43.
You have an error some place in the format of the syslog packet coming inn to Splunk.
If you do search: sourcetype=Mikrotik, you should not see any date, and you see double date in _raw packet.

Your syslog:

08/09/2021 09:27:49.022 2021-09-08T09:27:49.022737+02:00 router.lan dhcp,debug,packet MikroTik: Client-Id = 01-08-55-31-17-BD-8F

Mine

dhcp,debug,packet MikroTik: dhcp-alert on Bridge1 sending discover with id 3480279547 to 255.255.255.255

Did you follow the rsyslog setup I did post. Solution should work if you follow it 100%

Also look at this:
PS Do NOT select BSD Syslog. It will mess up the logging format.

1b) PS: To install Splunk as a non root user.
Splunk setup:
http://forum.mikrotik.com/t/tool-using-splunk-to-analyse-mikrotik-logs-3-3-graphing-everything/121810/13
rsyslog setup
http://forum.mikrotik.com/t/tool-using-splunk-to-analyse-mikrotik-logs-3-3-graphing-everything/121810/13

Edit
Added section 2h) debugging in fist post.

467

Thank you for your patience.

My Splunk Enterprise is now getting some data to show. :slight_smile:
I will now compile my notes, so even I may understand them in a few months LOL

Have a nice day
Klaus

468

I do use Linux (Ubuntu) I have not see these problems.
So not sure why these happens.

Splunk do recommend to use Linux over Windows

469

Script version 4.7

Fixed CHR Router error in 7.1rc4

Removed accouning section and unaccounted

Fixed NTP to work with RouterOS > 6

To upgrade, just cut/past the script to all router.

470

For some stupid reason, I have lost the “campsman” script.
If any one have it, please post it here :slight_smile:

Edit script found. Thanks to: Francois :slight_smile:

471

Wow, this thing is really funny!
I decided to try how it works, set everything up using the OP tutorial, and started checking through menus. When I reach “DNS → Mikrotik DNS Requests”, my Win2016 Server VM just died with 16Gb consumed memory. So I give it a whole 64 Gb! And what do you think? It dies again with about ~50+ GB mem consumption! This is just … WOW! So powerful! :laughing:
I only have 100Gb of RAM on this host, not sure shoud I give more?.. :astonished:
I’m probably stupid, is it an expected behaviour?
But anyway, it was looking cool! Maybe I did something incorrectly?

Added some more - total 80Gb of RAM. Still no use, it takes all the available memory and then everything dies, even the RDP. Defenitely a bug, but a funny one, can’t stop laughing feeling all that POWAH! :laughing: Should I use the Linux version instead?

472

Something has to be wrong.
I do have a PC with only 16GB memory (Ubuntu), and has no problem running lots of logs.

I do recommend that run Splunk on Linux (Ubuntu), even if it works on Windows.

473

New version out in new thread:
http://forum.mikrotik.com/t/tool-using-splunk-to-analyse-mikrotik-logs-4-0-graphing-everything/153043/1