Community discussions

MikroTik App
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 991
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 03, 2024 5:11 pm

I'm running Splunk on a Synology too, but as a VM under Ubuntu Linux, not containerized.
Works OK in general, had 1 or 2 occasions where the 4GB assigned memory fell short and things fell apart ;-)

>> After a while the logging to splunk stops ...

Splunk generates a ton of logging messages that might give you an indication why something "stops" working. Did you check any of these ?
(with a container, you'll have to open a shell I guess)

/opt/splunk/var/log/splunk

Are you not exceeding the 500Mbytes daily limit ??
Top menu "Settings" then "Licensing" (under the "System" section)
Hi jvanhambelgium
Did you find anything could help resolve this error?
I never had an issue. This is my/a response somebody else.
Just make sure you do not exceed the 500MByte limit on daily basis or Splunk will stop logging.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 03, 2024 8:51 pm

Just a tip.
You can request a free 10GB/day license (Developer License) from Splunk. It will give you all function on Splunk with 10GB/day compare to 500MB/day and limited functions (no alerts, no cluster +++) . Only down side is that you need to request a new license every 6 month.

https://dev.splunk.com/enterprise/dev_license/
 
jult
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Sat Dec 26, 2020 1:16 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Fri Jan 05, 2024 5:04 pm

But this is a remote, off-premise, storage/processing option. Nice, but that would cost you extra data/traffic to/from your WAN as well, and I don't think that's a good idea. It would even interfere with all the intended/normal traffic.
 
mooglez
just joined
Posts: 3
Joined: Mon Jan 22, 2024 4:10 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Mon Jan 22, 2024 4:17 pm

Just installed this to try out today.

Running Splunk 9.1 on Windows 10. Currently have log events for few hours in Splunk.

When I go to the dashboard "MikroTik DNS requests", resource usage goes absolutely wild.
It's basically consuming all available RAM and CPU for ~10 minutes.

I also noticed that many of the other dashboards are also quite slow to load, but don't consume everything for a long time.
Any idea what might be going wrong here?
 
mooglez
just joined
Posts: 3
Joined: Mon Jan 22, 2024 4:10 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Tue Jan 23, 2024 3:29 pm

Search job inspector results for a "last 15 minutes" search in the "MikroTik DNS requests" dashboard:
This search has completed and has returned 118 results by scanning 243 events in 223.991 seconds

The following messages were returned by the search subsystem:

info : Search finalized.
info : The term '"dns* query from*#"' contains a wildcard in the middle of a word or string. This might cause inconsistent results if the characters that the wildcard represents include punctuation. Learn More 
(SID: admin__admin__MikroTik__RMD5ecf8a6ae83683ff5_1706015926.479) search.log Job Details Dashboard

Execution costs
Duration (seconds)	Component	Invocations	Input count	Output count
0.00	 command.eval	6	236	236
0.00	 command.fields	6	236	236
46.92	 command.lookup	3	118	118
0.05	 command.postprocess	1	118	118
0.00	 command.presort	3	118	118
0.23	 command.search	6	118	236
0.09	 command.search.expand_search	2	-	-
0.00	 command.search.calcfields	2	243	243
0.00	 command.search.evalfilter	2	243	243
0.00	 command.search.expand_search.calcfield	2	-	-
0.00	 command.search.expand_search.fieldaliaser	2	-	-
0.00	 command.search.expand_search.indexed_fields	2	-	-
0.00	 command.search.expand_search.kv	2	-	-
0.00	 command.search.expand_search.lookup	2	-	-
0.00	 command.search.expand_search.sourcetype	2	-	-
0.00	 command.search.fieldalias	2	243	243
0.00	 command.search.filter	2	243	118
0.00	 command.search.index	5	-	-
0.00	 command.search.index.usec_1_8	272	-	-
0.00	 command.search.index.usec_512_4096	2	-	-
0.17	 command.search.lookups	2	243	243
0.05	 command.search.rawdata	2	-	-
0.02	 command.search.kv	2	-	-
0.00	 command.search.parse_directives	2	-	-
0.00	 command.search.summary	3	-	-
0.00	 command.search.tags	2	118	118
0.00	 command.search.track_sourcetypes	3	-	-
0.00	 command.search.typer	2	118	118
0.00	 command.sort	1	50,000	118
0.02	 command.timeliner	1	118	118
0.08	 dispatch.check_disk_usage	5	-	-
0.00	 dispatch.createdSearchResultInfrastructure	1	-	-
0.00	 dispatch.evaluate.eval	4	-	-
0.00	 dispatch.evaluate.fields	2	-	-
0.00	 dispatch.evaluate.lookup	2	-	-
0.09	 dispatch.evaluate.search	2	-	-
0.00	 dispatch.evaluate.sort	2	-	-
37.25	 dispatch.fetch.rcp.phase_0	5	-	-
0.00	 dispatch.finalWriteToDisk	1	-	-
47.16	 dispatch.localSearch	1	-	-
176.34	 dispatch.preview.snapshot	5	-	-
0.00	 dispatch.readEventsInResults	1	-	-
47.16	 dispatch.stream.local	3	-	-
0.00	 dispatch.timeline	1	-	-
0.03	 dispatch.tmpevents	2	-	-
0.29	 dispatch.writeStatus	52	-	-
0.13	 startup.configuration	2	-	-
0.70	 startup.handoff	2	-	-
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Fri Jan 26, 2024 3:57 pm

Splunk runs much better/faster on Linux. Its created for Linux and ported to Windows.
SSD disk are nearly a must when data is growing.

You can in the script turn off modules that your do not need or that gives problem like to much DNS (but then you will not see DNS logs)

How much do you log a day? You can see that in the Splunk License info page.
 
mooglez
just joined
Posts: 3
Joined: Mon Jan 22, 2024 4:10 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Mon Jan 29, 2024 11:16 am

Splunk runs much better/faster on Linux. Its created for Linux and ported to Windows.
SSD disk are nearly a must when data is growing.

You can in the script turn off modules that your do not need or that gives problem like to much DNS (but then you will not see DNS logs)

How much do you log a day? You can see that in the Splunk License info page.
I'm currently logging about 20 to 30M a day. 425k events in the last 24h, of which 400k are DNS.
Splunk is running on an SSD.

I was mostly wondering if there was some problem with the version of Splunk (9.1.2) I am using and the latest version of the script.
But it seems that nobody else is having issues with it, so it quite probably must be something at my end then.

My main reason for sending the logs to Splunk was to get DNS and DHCP logs over to analyze, so would really not want to disable DNS module.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Tue Jan 30, 2024 11:05 am

20-30M a day is not much so a simple server should handle that. (also a windows server)
 
JosipTopic
newbie
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 31, 2024 2:49 am

Hello, i just have a question. This link on the beginig of this thread, for download splunk app for mikrotik, is that the first one(oldest)? Where can be found updated one? Thanks?
 
snowdogging
just joined
Posts: 16
Joined: Tue Dec 20, 2016 6:23 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 31, 2024 10:56 pm

still kind of works on v7.13.3

I had to remove the capsman code. Getting error: expected end of command (line 290 column 50)
Also had to set command history to false. That portion results in a hard interruption and crash.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 9:30 am

Hello, i just have a question. This link on the beginig of this thread, for download splunk app for mikrotik, is that the first one(oldest)? Where can be found updated one? Thanks?
What have the link do you refer to. The app that I have created under section 1g- IF so there are a link to download it, and also a git repository that always will be the latest updated.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 9:32 am

still kind of works on v7.13.3

I had to remove the capsman code. Getting error: expected end of command (line 290 column 50)
Also had to set command history to false. That portion results in a hard interruption and crash.
Since I do not have capsmann its som hard to test for me. Will try to look at the code and see whats going wrong.
The command history should work. Has tested it on 17.3.1, but will try 17.3.3 as well.
 
snowdogging
just joined
Posts: 16
Joined: Tue Dec 20, 2016 6:23 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 10:15 pm

Cool. Yeah I don't have capsman either so can't really help. Let me know if I can provide more detail on command history crash. I might pull the script apart to see exactly what command causes it.
 
snowdogging
just joined
Posts: 16
Joined: Tue Dec 20, 2016 6:23 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 10:19 pm

I turned command history back on and it no longer crashes. I did manually pull the code out and ran in terminal. The crash might have something to do with the missing global "cmd" on first run.

Quick questions:
* What log prefixes besides FI_D_port-test are valid. Specifically, what types besides F? Is N nat or does it not matter?
* WireGuard Errror dashboard (sp). How do I trigger this?

Impressive app btw....thanks.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Sun Feb 04, 2024 11:23 am

Something new in 7.13+ makes the CAPsMANN part fail, even if its run in a do={} group.
To fix this I have updated scripts to 5.5 where CAPsMANN has been separated to an external script.

If you do not like to update the script, just remove the CAPsMANN part of the script and it will work.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Fri Feb 09, 2024 6:40 pm

Great news. v4.0 are on the way.
Most importante change is that all loggs will be tracked by a unique serial number. This way even if you have many routers behind one single nat or routers with same name, it would be easy to separate all the devices.

To prepare for the new version, you can just run (copy/past to terminal) the log update script found in 2.a It will add the routerboard serial number to the log message. If the device does not have a serial number it will create one. You do the update and the old version will still work and you are prepared for the 4.0 version that needs the serial number to work. Logs size will increase some due to the serial number adds around 18 bytes.

Script has also been updated to 5.6 where just serial number are removed from the system info part, since its part of all messages.

Hope to release 4.0 in not to long time.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 06, 2024 3:28 am

I upgraded an rb750Gr3, upgraded v5.3 to v5.6, and then saw the scene as shown below. I understand that the serial number is not displayed, but for what purpose is the other attributes not displayed? At least the "identity" is displayed. , or where I made it rough, please tell me.
2024-03-06_09-22-32.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 14, 2024 10:56 am

Not sure why your RB750Gr3 does not show up with model etc.
Try to do a search like this last 60 min
index=*  module=script script=sysinfo OR script=version  NOT "log info" | stats values(script) by host
It should list all devices sending sysinfo.
It it does not show up, the script many not run on the router.
See that it has correct name, cut/past it from serve here to make sure its ok.
Try to run it manually.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 18, 2024 11:32 am

2024-03-18_17-26-08.png
You do not have the required permissions to view the files attached to this post.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 20, 2024 10:32 pm

@Jotne,

I am running the 5.6 scripts on a couple of hap ax3 and the info displayed in splunk is not complete ...
(picture removed)

as you can see a couple of fields are not filled ...
same for all the info from the new wifi drivers

guess you are working on those too ?
Last edited by eddieb on Fri Mar 29, 2024 10:35 am, edited 1 time in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 20, 2024 10:57 pm

I am working on v4, should not be to long before I release it.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 20, 2024 11:01 pm

Keep up the good work !
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 21, 2024 3:40 pm

btw, I still see a very strange thing ...
my gateway router (CCR1009) stops sending log info after a couple of hours working fine.
nothing arrives at the splunk machine.
all other MT devices do continue to work but, the CCR obviously did send a lot more logging in that couple of hours...
Looks like some log daemon on the CCR stops ???
all systems run 7.14.1
anyone seen this ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 21, 2024 7:16 pm

I did have a similar problem on an RB750gr3. It stopped sending scripts logs. Looking at the scheduler it seems to not be working and have wrong dates. Disabled and Enabled the scheduler and scripts starts to run,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 9:17 am

V 4.0 finally released.
Both files in first post and GIT are updated to latest version.

# 4.0 (21.03.2024)
# Changed to use serial in all dashboard
# Changed many regex due to added serial
# Changed to use MikroTik index directly without macro
# Removed host_name and use identity in all dashboard
# Change device_table script to update every hour, not every day
# Fixed form version. Should always be 1.1
# Added DHCP lookup of client name in mikrotik_accounting_traffic
# Fixed romon info extraction. Use host_name in graphs in mikrotik_admin_user_login
# Joined multipe IP for the same host, Fixed list for multiple firmware, fixed errors in varios dashboards in mikrotik_device_list
# Added Time Span and separated IP address from name with - in mikrotik_dns_request.
# Rewritten calcualtion to give correctly bps and now works with multiple hosts, Added graph to show total bytex tx/rx in mikrotik_interface_traffic

The most important change is the serial usage. This will help to identify devices if there are several devices sending syslog behind same NAT ip.

If you have not changed any files, you can just replace all files with the latest version.
Upgrade should also work.

Since this has some larger changes, there will be errors, so need your feedback on what is wrong.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 2:12 pm

upgraded to 4.0 ...
CCR still not showing up, even after restarting schedule on the CCR .
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 3:29 pm

Do you get anything from it?
It should send syslog with errors etc + the script part.

Try to search
index=* host=<ip of device>

Send me a mail on this temp mail, and I can try to help: sowoyar992@glaslack.com
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 3:37 pm

I do see some records in the search, but now I have a lot of blank pages in 4.0
actually, only the
Screenshot 2024-03-22 143651.png
shows info, all other screens are "no results found"
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 9:17 pm

What do you mean by blank pages? Image look ok.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 9:23 pm

Only that firts screen gives data,
all other screens are empty, like this one
Screenshot 2024-03-22 at 20.21.49.png
btw, I send you a friend request on the MikroTik discord
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 10:28 pm

NB with 4.0 there are some importunt changes you need to follow:

1. All routers needs serial number in their logging tags (section 2a). If not you will not get any dashboard to work. To add serial, run the script in section 2a on all routers. (cut and past to a terminal windows.
2. If you for some reason has an other system logging action other than logserver, you need to edit the serial update script in 2a to use your action name
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 10:38 pm

tnx for your support Jotne, looks like it is working now ;-)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sun Mar 24, 2024 1:55 pm

Since all routers needs to be configured to have serial number (one time job), I have updated the main start page "MikroTik device list" ot show all routers who is sending data to Splunk using only the old MikroTik tag so you can spot them and update the routers.

Its not in the main zip file, but you find it in the git.
https://github.com/Jotne/MikroTik/blob/ ... e_list.xml
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 9:26 am

Morning,
All seems to work wel, except the WIFI screens stay without data.
I am using all HAP AX3 devices and the collector on those do not send any data about the newer wifi to splunk ...
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 7:08 pm

playing with the wifi collector on one of my hapax3 ...

original :
# Sends wireless client data to log server 
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wlan]]>0) do={
	/interface wireless registration-table
	:foreach i in=[find] do={
		:log info message=".id=$i;ap=$([get $i ap]);interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=$([get $i signal-strength]);tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
	}
}

modified for wifi interfaces

:if ($Wireless && [:len [/int find where type=wifi]]>0) do={
	/interface wifi registration-table
	:foreach i in=[find] do={
		:log info message=".id=$i;ap=false;interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=-50;tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
	}
}

for now,
- ap is always false as there is no ap anymore
- signal-strength is always -50, as I did not succeed to extract a variable signal or signal-strength

The records are sent to splunk but not shown, the page must be filtering on wlan[n] and not on wifi[n]


below the record from a wlan and a wifi device
3/25/24 5:49:23.000 PM	script,info serial=673706CE0892 MikroTik: .id=*9;ap=false;interface=wlan1;mac-address=50:F4:EB:D8:C2:79;signal-strength=-71dBm@1Mbps;tx-rate=7.2Mbps-20MHz/1S/SGI;uptime=00:03:39;script=wifi
host = 192.168.x.x source = udp:514 sourcetype = mikrotik

3/25/24 5:44:37.000 PM	script,info serial=HF309F2QABF MikroTik: .id=*14A;ap=false;interface=wifi1;mac-address=E4:B2:FB:AE:E8:16;signal-strength=-50;tx-rate=650000000;uptime=00:12:55;script=wifi
host = 192.168.x.y source = udp:514 sourcetype = mikrotik
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 8:53 pm

hmm

This has to do with the new wifi/wireless separation. We have to look inn to it how to handle both system.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:32 pm

I notice a
script error: error - contact MikroTik support and send a supout file (10)
running the data-to-splunk script on some machines
setting CmdHistory to false seems to solve it
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:49 pm

Can you try to cut & past this to a terminal on a router giving problems.
{
:global cmd
:local f 0
:foreach i in=[/system history find] do={
:if ($i = $cmd) do={ :set f 1 }
:if ($f != 1) do={
:put "StartCMD"
:put [/system history get $i]
:put "EndCMD"
}
}
:global cmd  [:pick [/system history find] 0]
}
If you get no output, try to do a change. Example add in IP to an address list, then run the command again.
It should the list your changes.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:52 pm

[eddieb@hapax3-1] > {
{... :global cmd
{... :local f 0
{... :foreach i in=[/system history find] do={
{{... :if ($i = $cmd) do={ :set f 1 }
{{... :if ($f != 1) do={
{{{... :put "StartCMD"
{{{... :put [/system history get $i]
{{{... :put "EndCMD"
{{{... }
{{... }
{... :global cmd  [:pick [/system history find] 0]
{... }
interrupted
error - contact MikroTik support and send a supout file (10)
[eddieb@hapax3-1] > 
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:54 pm

btw, script piece below works ...
:log info message="test2";

:local Wireless true;

# Sends wireless client data to log server 
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wifi]]>0) do={
#                :log info message="test2 found";
	/interface wifi registration-table
	:foreach i in=[find] do={
#                                :local ap ([get $i ap]);
                                :local ap "false";
                                :local int ([get $i interface]);
                                :local mac ([get $i mac-address]);
                                :local signalstrength ([get $i signal]);
                                :local txrate ([get $i tx-rate]);
                                :local up ([get $i uptime]);
		:log info message=".id=$i;ap=$ap;interface=$int;mac-address=$mac;signal-strength=$signalstrength;tx-rate=$txrate;uptime=$up;script=wifi"
	}
}
outputs (log print)
21:54:18 script,info test2 
21:54:18 script,info .id=*6;ap=false;interface=wifi2;mac-address=4C:09:FA:10:21:CF;signal-strength=-47;tx-rate=72200000;uptime=5d06:18:42;script=wifi 
21:54:18 script,info .id=*41;ap=false;interface=wifi2;mac-address=EC:FA:BC:50:0C:91;signal-strength=-67;tx-rate=72200000;uptime=4d02:32:49;script=wifi 
21:54:18 script,info .id=*B7;ap=false;interface=wifi2;mac-address=80:7D:3A:33:11:2A;signal-strength=-65;tx-rate=65000000;uptime=2d04:32:17;script=wifi 
21:54:18 script,info .id=*136;ap=false;interface=wifi2;mac-address=C8:2B:96:4B:F3:A0;signal-strength=-57;tx-rate=65000000;uptime=07:29:11;script=wifi 
21:54:18 script,info .id=*163;ap=false;interface=wifi2;mac-address=E4:B2:FB:AE:E8:16;signal-strength=-62;tx-rate=650000000;uptime=01:07:34;script=wifi 
21:54:18 script,info .id=*166;ap=false;interface=wifi2;mac-address=F8:87:F1:2C:B3:81;signal-strength=-72;tx-rate=288200000;uptime=00:01:35;script=wifi 
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 11:02 pm

[eddieb@hapax3-1] > sys history print

error - contact MikroTik support and send a supout file (10)
might be a bug ...
even if I change something, there is no history visible and the same error
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 3:27 am

`index`
sourcetype=mikrotik
module=script
script=health
host=10.0.0.1
name=temperature
| where value>50
Error message: "Error in 'SearchParser': The search specifies a macro 'index' that cannot be found. Reasons include: the macro name is misspelled, you do not have "read" permission for the macro, or the macro has not been shared with this application. Click Settings, Advanced search, Search Macros to view macro information."
This is the script you used to send high temperature warning emails before. After upgrading to v4.0, it prompted an error. How to fix this bug?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:06 am

[eddieb@hapax3-1] > sys history print

error - contact MikroTik support and send a supout file (10)
might be a bug ...
even if I change something, there is no history visible and the same error
This is clearly a bug. What OS and HW is this router. I do recommend to make a support case and also try another image if there are on newer.

PS the correct command should start with /, so just try this as well:
/system/history/print
Last edited by Jotne on Tue Mar 26, 2024 8:13 am, edited 2 times in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:10 am

The macro `index` are no longer used. It was just to make sure to get the data if both main index and mikrotik index was used.

Try:
index=mikrotik
sourcetype=mikrotik
module=script
script=health
host=10.0.0.1
name=temperature
| where value>50
If that does not work. What is the output of:
index=* sourcetype=mikrotik | table index
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:43 am

[eddieb@hapax3-1] > sys history print

error - contact MikroTik support and send a supout file (10)
might be a bug ...
even if I change something, there is no history visible and the same error
This is clearly a bug. What OS and HW is this router. I do recommend to make a support case and also try another image if there are on newer.

PS the correct command should start with /, so just try this as well:
/system/history/print
all systems are on 7.14.1
I noticed this first on my CCR1009, and it still gives that error, even with the /system/history/print command
10 hours ago I had this same message on a HAPAX3, but for some reason it now gives "normal" output.
I'll stay on it and created SUP-148095 on this
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:55 am

btw, script piece below works ...
I added this part to the data_to_splunk script and splunk now displays the Wifi Strength graph correctly
BUT, the Wifi Connection and Wifi Error graphs stay empty.
Splunk receives the connect/disconnect messages from this "wifi" devices but seems not to parse the messages correctly
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 28, 2024 9:11 am

after disabling cmd_history it seems my CCR did not stop sending info to splunk ...
So it might have something to do with that /system/history/print crash ...
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 28, 2024 10:24 am

Everything is OK. Thank you.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 28, 2024 10:45 pm

I enabled some HAPAC behind a NAT gateway and they are showing up with there own serial.
Just the Device List in Splunk is a mess, these NATTED devices show up multiple times ...
something is wrong here (it even displays more lines that do not fit on one page ...
both devices are behind 192.168.4.1 and have different serials .... 1 is a RB750GL and 1 is a hAP ac

(picture removed, problem solved)
Last edited by eddieb on Fri Mar 29, 2024 9:38 am, edited 2 times in total.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 29, 2024 8:40 am

Found a bug, there is no data in traffic
2024-03-29_14-38-06.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 29, 2024 8:49 am

Found a bug, there is no data in traffic
You have followed the 2e settings about kid control?
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 29, 2024 9:44 am

yeah, kid-control is in place since I initially configured the devices ;-)
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:38 am

You have followed the 2e settings about kid control?
This is for sure. I submitted the bug after confirming it.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 8:05 am

What is your routerOS version and script version?

Post output of:
/ip/kid-control/print
It should show the day of week its enabled like this:
Columns: NAME, SUN, MON, TUE, WED, THU, FRI, SAT
# NAME     SUN    MON    TUE    WED    THU    FRI    SAT  
0 Monitor  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d
If that is ok, you have enabled it. Then post the output of:
/ip/kid-control/device/print detail
It should show a list of devices, some like this:
Flags: X - disabled, D - dynamic, B - blocked, L - limited; I - inactive 
 0 D  name="" mac-address=XX:XX:35:CF:3E:XX user="" ip-address=192.168.10.160 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 

 1 D  name="" mac-address=XX:XX:6B:88:34:XX user="" ip-address=192.168.10.1 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 
Last, do you see any data in splunk by this search:
index=* module=script script=kids
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 8:13 am

"MikroTik Device List" updated in git.
Changed to use serial instead of nat, to not give error while multiple routers are behind nat.

Working on handling the new wifi/wireless split.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 991
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 4:38 pm

Hi,

It seems a part of the script (v5.6) is giving me consistent error on 7.14.1

Screenshot from 2024-03-30 15-33-01.png
It seems in the section where all the access-list are processed/counted, so the section below.
Didn't change anything to the code, just copy-pasted into Winbox.
The ACL "Azure-Lab" is the first ACL I have, so it seems to process all of them correctly...so perhaps the error is in the next section or so ?
Is there a way to diagnose this better?

# Count IP in address-lists
#----------------------------------
:if ($AddressLists) do={
:local array [ :toarray "" ]
:local addrcntdyn [:toarray ""]
:local addrcntstat [:toarray ""]
:local test
:foreach id in=[/ip firewall address-list find] do={
:local rec [/ip firewall address-list get $id]
:local listname ($rec->"list")
:local listdynamic ($rec->"dynamic")
:if (!($array ~ $listname)) do={ :set array ($array , $listname) }
:if ($listdynamic = true) do={
:set ($addrcntdyn->$listname) ($addrcntdyn->$listname+1)
} else={
:set ($addrcntstat->$listname) ($addrcntstat->$listname+1)}
}
:foreach k in=$array do={
:log info message=("script=address_lists list=$k dynamic=".(($addrcntdyn->$k)+0)." static=".(($addrcntstat->$k)+0))}
}

# Get MNDP (CDP) Neighbors
# ----------------------------------
:if ($Neighbor and $run) do={
:foreach neighborID in=[/ip neighbor find] do={
:local nb [/ip neighbor get $neighborID]
:local id [:pick ("$nb"->".id") 1 99]
:foreach key,value in=$nb do={
:local newline [:find $value "\n"]
:if ([$newline]>0) do={
:set value [:pick $value 0 $newline]
}
:log info message="script=neighbor nid=$id $key=\"$value\""
}
}
}
You do not have the required permissions to view the files attached to this post.
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:34 pm

Hi,

It seems a part of the script (v5.6) is giving me consistent error on 7.14.1


Screenshot from 2024-03-30 15-33-01.png
the 5.6 script hits a system history print command which causes this error on my systems.
You can reproduce this by entering the command "system history print" in a console on that machine
I filed SUP-148095 for this ...
If you set CmdHistory to false in the collector script the error should be gone for now
(Also discussing this with @jotne on discord)
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 991
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:40 pm

Hi,

It seems a part of the script (v5.6) is giving me consistent error on 7.14.1


Screenshot from 2024-03-30 15-33-01.png
the 5.6 script hits a system history print command which causes this error on my systems.
You can reproduce this by entering the command "system history print" in a console on that machine
I filed SUP-148095 for this ...
If you set CmdHistory to false in the collector script the error should be gone for now
(Also discussing this with @jotne on discord)
Indeed, that makes things clear!
Thanks for the feedback
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:44 pm

Indeed, that makes things clear!
Thanks for the feedback
I guess it is better to file a ticket also,
despite me giving 3 subout.rif files MT support is not able to reproduce this errror ..
I guess playing with some scripts and creating a lot of cmd history makes some overflow somewhere ...
Had this on new HAPAX3 and older CCR1009 ..
 
eddieb
Member
Member
Posts: 327
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 01, 2024 11:24 am

I modified some events and now I am seeing "wifi" routers in the (dis)connect pages
Last edited by eddieb on Mon Apr 01, 2024 8:51 pm, edited 1 time in total.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 01, 2024 1:26 pm

What is your routerOS version and script version?

Post output of:
/ip/kid-control/print
It should show the day of week its enabled like this:
Columns: NAME, SUN, MON, TUE, WED, THU, FRI, SAT
# NAME     SUN    MON    TUE    WED    THU    FRI    SAT  
0 Monitor  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d
If that is ok, you have enabled it. Then post the output of:
/ip/kid-control/device/print detail
It should show a list of devices, some like this:
Flags: X - disabled, D - dynamic, B - blocked, L - limited; I - inactive 
 0 D  name="" mac-address=XX:XX:35:CF:3E:XX user="" ip-address=192.168.10.160 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 

 1 D  name="" mac-address=XX:XX:6B:88:34:XX user="" ip-address=192.168.10.1 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 
Last, do you see any data in splunk by this search:
index=* module=script script=kids
I have done the above steps, and I can see the data in the last step, but I can't see any data in "traffic --- mikrotik device traffic or interface traffic"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3297
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Apr 02, 2024 1:18 pm

The logs look like this?

script,info serial=75B70647AAAA MikroTik: .id=*5;activity=;blocked=false;bytes-down=0;bytes-up=0;disabled=false;dynamic=true;inactive=false;ip-address=192.168.10.241;limited=false;mac-address=D8:9E:CC:CC:CC:10;name=;rate-down=0;rate-up=0;script=kids;user=
Most important are the stuff in Bold. If that is wrong or missing, stuff does not work.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 104
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 08, 2024 6:08 am

I didn't read your update carefully. After following step 2a), it worked normally. Thank you.

Who is online

Users browsing this forum: sweetlilmre and 3 guests