Community discussions

MikroTik App
 
User avatar
jvanhambelgium
Forum Guru
Forum Guru
Posts: 1077
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 03, 2024 5:11 pm

I'm running Splunk on a Synology too, but as a VM under Ubuntu Linux, not containerized.
Works OK in general, had 1 or 2 occasions where the 4GB assigned memory fell short and things fell apart ;-)

>> After a while the logging to splunk stops ...

Splunk generates a ton of logging messages that might give you an indication why something "stops" working. Did you check any of these ?
(with a container, you'll have to open a shell I guess)

/opt/splunk/var/log/splunk

Are you not exceeding the 500Mbytes daily limit ??
Top menu "Settings" then "Licensing" (under the "System" section)
Hi jvanhambelgium
Did you find anything could help resolve this error?
I never had an issue. This is my/a response somebody else.
Just make sure you do not exceed the 500MByte limit on daily basis or Splunk will stop logging.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 03, 2024 8:51 pm

Just a tip.
You can request a free 10GB/day license (Developer License) from Splunk. It will give you all function on Splunk with 10GB/day compare to 500MB/day and limited functions (no alerts, no cluster +++) . Only down side is that you need to request a new license every 6 month.

https://dev.splunk.com/enterprise/dev_license/
 
jult
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Sat Dec 26, 2020 1:16 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Fri Jan 05, 2024 5:04 pm

But this is a remote, off-premise, storage/processing option. Nice, but that would cost you extra data/traffic to/from your WAN as well, and I don't think that's a good idea. It would even interfere with all the intended/normal traffic.
 
mooglez
just joined
Posts: 3
Joined: Mon Jan 22, 2024 4:10 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Mon Jan 22, 2024 4:17 pm

Just installed this to try out today.

Running Splunk 9.1 on Windows 10. Currently have log events for few hours in Splunk.

When I go to the dashboard "MikroTik DNS requests", resource usage goes absolutely wild.
It's basically consuming all available RAM and CPU for ~10 minutes.

I also noticed that many of the other dashboards are also quite slow to load, but don't consume everything for a long time.
Any idea what might be going wrong here?
 
mooglez
just joined
Posts: 3
Joined: Mon Jan 22, 2024 4:10 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Tue Jan 23, 2024 3:29 pm

Search job inspector results for a "last 15 minutes" search in the "MikroTik DNS requests" dashboard:
This search has completed and has returned 118 results by scanning 243 events in 223.991 seconds

The following messages were returned by the search subsystem:

info : Search finalized.
info : The term '"dns* query from*#"' contains a wildcard in the middle of a word or string. This might cause inconsistent results if the characters that the wildcard represents include punctuation. Learn More 
(SID: admin__admin__MikroTik__RMD5ecf8a6ae83683ff5_1706015926.479) search.log Job Details Dashboard

Execution costs
Duration (seconds)	Component	Invocations	Input count	Output count
0.00	 command.eval	6	236	236
0.00	 command.fields	6	236	236
46.92	 command.lookup	3	118	118
0.05	 command.postprocess	1	118	118
0.00	 command.presort	3	118	118
0.23	 command.search	6	118	236
0.09	 command.search.expand_search	2	-	-
0.00	 command.search.calcfields	2	243	243
0.00	 command.search.evalfilter	2	243	243
0.00	 command.search.expand_search.calcfield	2	-	-
0.00	 command.search.expand_search.fieldaliaser	2	-	-
0.00	 command.search.expand_search.indexed_fields	2	-	-
0.00	 command.search.expand_search.kv	2	-	-
0.00	 command.search.expand_search.lookup	2	-	-
0.00	 command.search.expand_search.sourcetype	2	-	-
0.00	 command.search.fieldalias	2	243	243
0.00	 command.search.filter	2	243	118
0.00	 command.search.index	5	-	-
0.00	 command.search.index.usec_1_8	272	-	-
0.00	 command.search.index.usec_512_4096	2	-	-
0.17	 command.search.lookups	2	243	243
0.05	 command.search.rawdata	2	-	-
0.02	 command.search.kv	2	-	-
0.00	 command.search.parse_directives	2	-	-
0.00	 command.search.summary	3	-	-
0.00	 command.search.tags	2	118	118
0.00	 command.search.track_sourcetypes	3	-	-
0.00	 command.search.typer	2	118	118
0.00	 command.sort	1	50,000	118
0.02	 command.timeliner	1	118	118
0.08	 dispatch.check_disk_usage	5	-	-
0.00	 dispatch.createdSearchResultInfrastructure	1	-	-
0.00	 dispatch.evaluate.eval	4	-	-
0.00	 dispatch.evaluate.fields	2	-	-
0.00	 dispatch.evaluate.lookup	2	-	-
0.09	 dispatch.evaluate.search	2	-	-
0.00	 dispatch.evaluate.sort	2	-	-
37.25	 dispatch.fetch.rcp.phase_0	5	-	-
0.00	 dispatch.finalWriteToDisk	1	-	-
47.16	 dispatch.localSearch	1	-	-
176.34	 dispatch.preview.snapshot	5	-	-
0.00	 dispatch.readEventsInResults	1	-	-
47.16	 dispatch.stream.local	3	-	-
0.00	 dispatch.timeline	1	-	-
0.03	 dispatch.tmpevents	2	-	-
0.29	 dispatch.writeStatus	52	-	-
0.13	 startup.configuration	2	-	-
0.70	 startup.handoff	2	-	-
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Fri Jan 26, 2024 3:57 pm

Splunk runs much better/faster on Linux. Its created for Linux and ported to Windows.
SSD disk are nearly a must when data is growing.

You can in the script turn off modules that your do not need or that gives problem like to much DNS (but then you will not see DNS logs)

How much do you log a day? You can see that in the Splunk License info page.
 
mooglez
just joined
Posts: 3
Joined: Mon Jan 22, 2024 4:10 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Mon Jan 29, 2024 11:16 am

Splunk runs much better/faster on Linux. Its created for Linux and ported to Windows.
SSD disk are nearly a must when data is growing.

You can in the script turn off modules that your do not need or that gives problem like to much DNS (but then you will not see DNS logs)

How much do you log a day? You can see that in the Splunk License info page.
I'm currently logging about 20 to 30M a day. 425k events in the last 24h, of which 400k are DNS.
Splunk is running on an SSD.

I was mostly wondering if there was some problem with the version of Splunk (9.1.2) I am using and the latest version of the script.
But it seems that nobody else is having issues with it, so it quite probably must be something at my end then.

My main reason for sending the logs to Splunk was to get DNS and DHCP logs over to analyze, so would really not want to disable DNS module.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Tue Jan 30, 2024 11:05 am

20-30M a day is not much so a simple server should handle that. (also a windows server)
 
JosipTopic
newbie
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 31, 2024 2:49 am

Hello, i just have a question. This link on the beginig of this thread, for download splunk app for mikrotik, is that the first one(oldest)? Where can be found updated one? Thanks?
 
snowdogging
just joined
Posts: 16
Joined: Tue Dec 20, 2016 6:23 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Jan 31, 2024 10:56 pm

still kind of works on v7.13.3

I had to remove the capsman code. Getting error: expected end of command (line 290 column 50)
Also had to set command history to false. That portion results in a hard interruption and crash.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 9:30 am

Hello, i just have a question. This link on the beginig of this thread, for download splunk app for mikrotik, is that the first one(oldest)? Where can be found updated one? Thanks?
What have the link do you refer to. The app that I have created under section 1g- IF so there are a link to download it, and also a git repository that always will be the latest updated.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 9:32 am

still kind of works on v7.13.3

I had to remove the capsman code. Getting error: expected end of command (line 290 column 50)
Also had to set command history to false. That portion results in a hard interruption and crash.
Since I do not have capsmann its som hard to test for me. Will try to look at the code and see whats going wrong.
The command history should work. Has tested it on 17.3.1, but will try 17.3.3 as well.
 
snowdogging
just joined
Posts: 16
Joined: Tue Dec 20, 2016 6:23 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 10:15 pm

Cool. Yeah I don't have capsman either so can't really help. Let me know if I can provide more detail on command history crash. I might pull the script apart to see exactly what command causes it.
 
snowdogging
just joined
Posts: 16
Joined: Tue Dec 20, 2016 6:23 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Feb 01, 2024 10:19 pm

I turned command history back on and it no longer crashes. I did manually pull the code out and ran in terminal. The crash might have something to do with the missing global "cmd" on first run.

Quick questions:
* What log prefixes besides FI_D_port-test are valid. Specifically, what types besides F? Is N nat or does it not matter?
* WireGuard Errror dashboard (sp). How do I trigger this?

Impressive app btw....thanks.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Sun Feb 04, 2024 11:23 am

Something new in 7.13+ makes the CAPsMANN part fail, even if its run in a do={} group.
To fix this I have updated scripts to 5.5 where CAPsMANN has been separated to an external script.

If you do not like to update the script, just remove the CAPsMANN part of the script and it will work.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Fri Feb 09, 2024 6:40 pm

Great news. v4.0 are on the way.
Most importante change is that all loggs will be tracked by a unique serial number. This way even if you have many routers behind one single nat or routers with same name, it would be easy to separate all the devices.

To prepare for the new version, you can just run (copy/past to terminal) the log update script found in 2.a It will add the routerboard serial number to the log message. If the device does not have a serial number it will create one. You do the update and the old version will still work and you are prepared for the 4.0 version that needs the serial number to work. Logs size will increase some due to the serial number adds around 18 bytes.

Script has also been updated to 5.6 where just serial number are removed from the system info part, since its part of all messages.

Hope to release 4.0 in not to long time.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 06, 2024 3:28 am

I upgraded an rb750Gr3, upgraded v5.3 to v5.6, and then saw the scene as shown below. I understand that the serial number is not displayed, but for what purpose is the other attributes not displayed? At least the "identity" is displayed. , or where I made it rough, please tell me.
2024-03-06_09-22-32.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 14, 2024 10:56 am

Not sure why your RB750Gr3 does not show up with model etc.
Try to do a search like this last 60 min
index=*  module=script script=sysinfo OR script=version  NOT "log info" | stats values(script) by host
It should list all devices sending sysinfo.
It it does not show up, the script many not run on the router.
See that it has correct name, cut/past it from serve here to make sure its ok.
Try to run it manually.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 18, 2024 11:32 am

2024-03-18_17-26-08.png
You do not have the required permissions to view the files attached to this post.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 20, 2024 10:32 pm

@Jotne,

I am running the 5.6 scripts on a couple of hap ax3 and the info displayed in splunk is not complete ...
(picture removed)

as you can see a couple of fields are not filled ...
same for all the info from the new wifi drivers

guess you are working on those too ?
Last edited by eddieb on Fri Mar 29, 2024 10:35 am, edited 1 time in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 20, 2024 10:57 pm

I am working on v4, should not be to long before I release it.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Wed Mar 20, 2024 11:01 pm

Keep up the good work !
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 21, 2024 3:40 pm

btw, I still see a very strange thing ...
my gateway router (CCR1009) stops sending log info after a couple of hours working fine.
nothing arrives at the splunk machine.
all other MT devices do continue to work but, the CCR obviously did send a lot more logging in that couple of hours...
Looks like some log daemon on the CCR stops ???
all systems run 7.14.1
anyone seen this ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 3.9 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 21, 2024 7:16 pm

I did have a similar problem on an RB750gr3. It stopped sending scripts logs. Looking at the scheduler it seems to not be working and have wrong dates. Disabled and Enabled the scheduler and scripts starts to run,
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 9:17 am

V 4.0 finally released.
Both files in first post and GIT are updated to latest version.

# 4.0 (21.03.2024)
# Changed to use serial in all dashboard
# Changed many regex due to added serial
# Changed to use MikroTik index directly without macro
# Removed host_name and use identity in all dashboard
# Change device_table script to update every hour, not every day
# Fixed form version. Should always be 1.1
# Added DHCP lookup of client name in mikrotik_accounting_traffic
# Fixed romon info extraction. Use host_name in graphs in mikrotik_admin_user_login
# Joined multipe IP for the same host, Fixed list for multiple firmware, fixed errors in varios dashboards in mikrotik_device_list
# Added Time Span and separated IP address from name with - in mikrotik_dns_request.
# Rewritten calcualtion to give correctly bps and now works with multiple hosts, Added graph to show total bytex tx/rx in mikrotik_interface_traffic

The most important change is the serial usage. This will help to identify devices if there are several devices sending syslog behind same NAT ip.

If you have not changed any files, you can just replace all files with the latest version.
Upgrade should also work.

Since this has some larger changes, there will be errors, so need your feedback on what is wrong.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 2:12 pm

upgraded to 4.0 ...
CCR still not showing up, even after restarting schedule on the CCR .
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 3:29 pm

Do you get anything from it?
It should send syslog with errors etc + the script part.

Try to search
index=* host=<ip of device>

Send me a mail on this temp mail, and I can try to help: sowoyar992@glaslack.com
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 3:37 pm

I do see some records in the search, but now I have a lot of blank pages in 4.0
actually, only the
Screenshot 2024-03-22 143651.png
shows info, all other screens are "no results found"
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 9:17 pm

What do you mean by blank pages? Image look ok.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 9:23 pm

Only that firts screen gives data,
all other screens are empty, like this one
Screenshot 2024-03-22 at 20.21.49.png
btw, I send you a friend request on the MikroTik discord
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 10:28 pm

NB with 4.0 there are some importunt changes you need to follow:

1. All routers needs serial number in their logging tags (section 2a). If not you will not get any dashboard to work. To add serial, run the script in section 2a on all routers. (cut and past to a terminal windows.
2. If you for some reason has an other system logging action other than logserver, you need to edit the serial update script in 2a to use your action name
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 22, 2024 10:38 pm

tnx for your support Jotne, looks like it is working now ;-)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sun Mar 24, 2024 1:55 pm

Since all routers needs to be configured to have serial number (one time job), I have updated the main start page "MikroTik device list" ot show all routers who is sending data to Splunk using only the old MikroTik tag so you can spot them and update the routers.

Its not in the main zip file, but you find it in the git.
https://github.com/Jotne/MikroTik/blob/ ... e_list.xml
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 9:26 am

Morning,
All seems to work wel, except the WIFI screens stay without data.
I am using all HAP AX3 devices and the collector on those do not send any data about the newer wifi to splunk ...
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 7:08 pm

playing with the wifi collector on one of my hapax3 ...

original :
# Sends wireless client data to log server 
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wlan]]>0) do={
	/interface wireless registration-table
	:foreach i in=[find] do={
		:log info message=".id=$i;ap=$([get $i ap]);interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=$([get $i signal-strength]);tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
	}
}

modified for wifi interfaces

:if ($Wireless && [:len [/int find where type=wifi]]>0) do={
	/interface wifi registration-table
	:foreach i in=[find] do={
		:log info message=".id=$i;ap=false;interface=$([get $i interface]);mac-address=$([get $i mac-address]);signal-strength=-50;tx-rate=$([get $i tx-rate]);uptime=$([get $i uptime]);script=wifi"
	}
}

for now,
- ap is always false as there is no ap anymore
- signal-strength is always -50, as I did not succeed to extract a variable signal or signal-strength

The records are sent to splunk but not shown, the page must be filtering on wlan[n] and not on wifi[n]


below the record from a wlan and a wifi device
3/25/24 5:49:23.000 PM	script,info serial=673706CE0892 MikroTik: .id=*9;ap=false;interface=wlan1;mac-address=50:F4:EB:D8:C2:79;signal-strength=-71dBm@1Mbps;tx-rate=7.2Mbps-20MHz/1S/SGI;uptime=00:03:39;script=wifi
host = 192.168.x.x source = udp:514 sourcetype = mikrotik

3/25/24 5:44:37.000 PM	script,info serial=HF309F2QABF MikroTik: .id=*14A;ap=false;interface=wifi1;mac-address=E4:B2:FB:AE:E8:16;signal-strength=-50;tx-rate=650000000;uptime=00:12:55;script=wifi
host = 192.168.x.y source = udp:514 sourcetype = mikrotik
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 8:53 pm

hmm

This has to do with the new wifi/wireless separation. We have to look inn to it how to handle both system.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:32 pm

I notice a
script error: error - contact MikroTik support and send a supout file (10)
running the data-to-splunk script on some machines
setting CmdHistory to false seems to solve it
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:49 pm

Can you try to cut & past this to a terminal on a router giving problems.
{
:global cmd
:local f 0
:foreach i in=[/system history find] do={
:if ($i = $cmd) do={ :set f 1 }
:if ($f != 1) do={
:put "StartCMD"
:put [/system history get $i]
:put "EndCMD"
}
}
:global cmd  [:pick [/system history find] 0]
}
If you get no output, try to do a change. Example add in IP to an address list, then run the command again.
It should the list your changes.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:52 pm

[eddieb@hapax3-1] > {
{... :global cmd
{... :local f 0
{... :foreach i in=[/system history find] do={
{{... :if ($i = $cmd) do={ :set f 1 }
{{... :if ($f != 1) do={
{{{... :put "StartCMD"
{{{... :put [/system history get $i]
{{{... :put "EndCMD"
{{{... }
{{... }
{... :global cmd  [:pick [/system history find] 0]
{... }
interrupted
error - contact MikroTik support and send a supout file (10)
[eddieb@hapax3-1] > 
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 10:54 pm

btw, script piece below works ...
:log info message="test2";

:local Wireless true;

# Sends wireless client data to log server 
# ----------------------------------
:if ($Wireless && [:len [/int find where type=wifi]]>0) do={
#                :log info message="test2 found";
	/interface wifi registration-table
	:foreach i in=[find] do={
#                                :local ap ([get $i ap]);
                                :local ap "false";
                                :local int ([get $i interface]);
                                :local mac ([get $i mac-address]);
                                :local signalstrength ([get $i signal]);
                                :local txrate ([get $i tx-rate]);
                                :local up ([get $i uptime]);
		:log info message=".id=$i;ap=$ap;interface=$int;mac-address=$mac;signal-strength=$signalstrength;tx-rate=$txrate;uptime=$up;script=wifi"
	}
}
outputs (log print)
21:54:18 script,info test2 
21:54:18 script,info .id=*6;ap=false;interface=wifi2;mac-address=4C:09:FA:10:21:CF;signal-strength=-47;tx-rate=72200000;uptime=5d06:18:42;script=wifi 
21:54:18 script,info .id=*41;ap=false;interface=wifi2;mac-address=EC:FA:BC:50:0C:91;signal-strength=-67;tx-rate=72200000;uptime=4d02:32:49;script=wifi 
21:54:18 script,info .id=*B7;ap=false;interface=wifi2;mac-address=80:7D:3A:33:11:2A;signal-strength=-65;tx-rate=65000000;uptime=2d04:32:17;script=wifi 
21:54:18 script,info .id=*136;ap=false;interface=wifi2;mac-address=C8:2B:96:4B:F3:A0;signal-strength=-57;tx-rate=65000000;uptime=07:29:11;script=wifi 
21:54:18 script,info .id=*163;ap=false;interface=wifi2;mac-address=E4:B2:FB:AE:E8:16;signal-strength=-62;tx-rate=650000000;uptime=01:07:34;script=wifi 
21:54:18 script,info .id=*166;ap=false;interface=wifi2;mac-address=F8:87:F1:2C:B3:81;signal-strength=-72;tx-rate=288200000;uptime=00:01:35;script=wifi 
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Mar 25, 2024 11:02 pm

[eddieb@hapax3-1] > sys history print

error - contact MikroTik support and send a supout file (10)
might be a bug ...
even if I change something, there is no history visible and the same error
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 3:27 am

`index`
sourcetype=mikrotik
module=script
script=health
host=10.0.0.1
name=temperature
| where value>50
Error message: "Error in 'SearchParser': The search specifies a macro 'index' that cannot be found. Reasons include: the macro name is misspelled, you do not have "read" permission for the macro, or the macro has not been shared with this application. Click Settings, Advanced search, Search Macros to view macro information."
This is the script you used to send high temperature warning emails before. After upgrading to v4.0, it prompted an error. How to fix this bug?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:06 am

[eddieb@hapax3-1] > sys history print

error - contact MikroTik support and send a supout file (10)
might be a bug ...
even if I change something, there is no history visible and the same error
This is clearly a bug. What OS and HW is this router. I do recommend to make a support case and also try another image if there are on newer.

PS the correct command should start with /, so just try this as well:
/system/history/print
Last edited by Jotne on Tue Mar 26, 2024 8:13 am, edited 2 times in total.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:10 am

The macro `index` are no longer used. It was just to make sure to get the data if both main index and mikrotik index was used.

Try:
index=mikrotik
sourcetype=mikrotik
module=script
script=health
host=10.0.0.1
name=temperature
| where value>50
If that does not work. What is the output of:
index=* sourcetype=mikrotik | table index
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:43 am

[eddieb@hapax3-1] > sys history print

error - contact MikroTik support and send a supout file (10)
might be a bug ...
even if I change something, there is no history visible and the same error
This is clearly a bug. What OS and HW is this router. I do recommend to make a support case and also try another image if there are on newer.

PS the correct command should start with /, so just try this as well:
/system/history/print
all systems are on 7.14.1
I noticed this first on my CCR1009, and it still gives that error, even with the /system/history/print command
10 hours ago I had this same message on a HAPAX3, but for some reason it now gives "normal" output.
I'll stay on it and created SUP-148095 on this
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Mar 26, 2024 8:55 am

btw, script piece below works ...
I added this part to the data_to_splunk script and splunk now displays the Wifi Strength graph correctly
BUT, the Wifi Connection and Wifi Error graphs stay empty.
Splunk receives the connect/disconnect messages from this "wifi" devices but seems not to parse the messages correctly
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 28, 2024 9:11 am

after disabling cmd_history it seems my CCR did not stop sending info to splunk ...
So it might have something to do with that /system/history/print crash ...
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 28, 2024 10:24 am

Everything is OK. Thank you.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Mar 28, 2024 10:45 pm

I enabled some HAPAC behind a NAT gateway and they are showing up with there own serial.
Just the Device List in Splunk is a mess, these NATTED devices show up multiple times ...
something is wrong here (it even displays more lines that do not fit on one page ...
both devices are behind 192.168.4.1 and have different serials .... 1 is a RB750GL and 1 is a hAP ac

(picture removed, problem solved)
Last edited by eddieb on Fri Mar 29, 2024 9:38 am, edited 2 times in total.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 29, 2024 8:40 am

Found a bug, there is no data in traffic
2024-03-29_14-38-06.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 29, 2024 8:49 am

Found a bug, there is no data in traffic
You have followed the 2e settings about kid control?
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Mar 29, 2024 9:44 am

yeah, kid-control is in place since I initially configured the devices ;-)
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:38 am

You have followed the 2e settings about kid control?
This is for sure. I submitted the bug after confirming it.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 8:05 am

What is your routerOS version and script version?

Post output of:
/ip/kid-control/print
It should show the day of week its enabled like this:
Columns: NAME, SUN, MON, TUE, WED, THU, FRI, SAT
# NAME     SUN    MON    TUE    WED    THU    FRI    SAT  
0 Monitor  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d
If that is ok, you have enabled it. Then post the output of:
/ip/kid-control/device/print detail
It should show a list of devices, some like this:
Flags: X - disabled, D - dynamic, B - blocked, L - limited; I - inactive 
 0 D  name="" mac-address=XX:XX:35:CF:3E:XX user="" ip-address=192.168.10.160 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 

 1 D  name="" mac-address=XX:XX:6B:88:34:XX user="" ip-address=192.168.10.1 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 
Last, do you see any data in splunk by this search:
index=* module=script script=kids
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 8:13 am

"MikroTik Device List" updated in git.
Changed to use serial instead of nat, to not give error while multiple routers are behind nat.

Working on handling the new wifi/wireless split.
 
User avatar
jvanhambelgium
Forum Guru
Forum Guru
Posts: 1077
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 4:38 pm

Hi,

It seems a part of the script (v5.6) is giving me consistent error on 7.14.1

Screenshot from 2024-03-30 15-33-01.png
It seems in the section where all the access-list are processed/counted, so the section below.
Didn't change anything to the code, just copy-pasted into Winbox.
The ACL "Azure-Lab" is the first ACL I have, so it seems to process all of them correctly...so perhaps the error is in the next section or so ?
Is there a way to diagnose this better?

# Count IP in address-lists
#----------------------------------
:if ($AddressLists) do={
:local array [ :toarray "" ]
:local addrcntdyn [:toarray ""]
:local addrcntstat [:toarray ""]
:local test
:foreach id in=[/ip firewall address-list find] do={
:local rec [/ip firewall address-list get $id]
:local listname ($rec->"list")
:local listdynamic ($rec->"dynamic")
:if (!($array ~ $listname)) do={ :set array ($array , $listname) }
:if ($listdynamic = true) do={
:set ($addrcntdyn->$listname) ($addrcntdyn->$listname+1)
} else={
:set ($addrcntstat->$listname) ($addrcntstat->$listname+1)}
}
:foreach k in=$array do={
:log info message=("script=address_lists list=$k dynamic=".(($addrcntdyn->$k)+0)." static=".(($addrcntstat->$k)+0))}
}

# Get MNDP (CDP) Neighbors
# ----------------------------------
:if ($Neighbor and $run) do={
:foreach neighborID in=[/ip neighbor find] do={
:local nb [/ip neighbor get $neighborID]
:local id [:pick ("$nb"->".id") 1 99]
:foreach key,value in=$nb do={
:local newline [:find $value "\n"]
:if ([$newline]>0) do={
:set value [:pick $value 0 $newline]
}
:log info message="script=neighbor nid=$id $key=\"$value\""
}
}
}
You do not have the required permissions to view the files attached to this post.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:34 pm

Hi,

It seems a part of the script (v5.6) is giving me consistent error on 7.14.1


Screenshot from 2024-03-30 15-33-01.png
the 5.6 script hits a system history print command which causes this error on my systems.
You can reproduce this by entering the command "system history print" in a console on that machine
I filed SUP-148095 for this ...
If you set CmdHistory to false in the collector script the error should be gone for now
(Also discussing this with @jotne on discord)
 
User avatar
jvanhambelgium
Forum Guru
Forum Guru
Posts: 1077
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:40 pm

Hi,

It seems a part of the script (v5.6) is giving me consistent error on 7.14.1


Screenshot from 2024-03-30 15-33-01.png
the 5.6 script hits a system history print command which causes this error on my systems.
You can reproduce this by entering the command "system history print" in a console on that machine
I filed SUP-148095 for this ...
If you set CmdHistory to false in the collector script the error should be gone for now
(Also discussing this with @jotne on discord)
Indeed, that makes things clear!
Thanks for the feedback
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Mar 30, 2024 5:44 pm

Indeed, that makes things clear!
Thanks for the feedback
I guess it is better to file a ticket also,
despite me giving 3 subout.rif files MT support is not able to reproduce this errror ..
I guess playing with some scripts and creating a lot of cmd history makes some overflow somewhere ...
Had this on new HAPAX3 and older CCR1009 ..
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 01, 2024 11:24 am

I modified some events and now I am seeing "wifi" routers in the (dis)connect pages
Last edited by eddieb on Mon Apr 01, 2024 8:51 pm, edited 1 time in total.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 01, 2024 1:26 pm

What is your routerOS version and script version?

Post output of:
/ip/kid-control/print
It should show the day of week its enabled like this:
Columns: NAME, SUN, MON, TUE, WED, THU, FRI, SAT
# NAME     SUN    MON    TUE    WED    THU    FRI    SAT  
0 Monitor  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d  0s-1d
If that is ok, you have enabled it. Then post the output of:
/ip/kid-control/device/print detail
It should show a list of devices, some like this:
Flags: X - disabled, D - dynamic, B - blocked, L - limited; I - inactive 
 0 D  name="" mac-address=XX:XX:35:CF:3E:XX user="" ip-address=192.168.10.160 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 

 1 D  name="" mac-address=XX:XX:6B:88:34:XX user="" ip-address=192.168.10.1 
      activity="" rate-down=0bps rate-up=0bps bytes-down=0 bytes-up=0 
Last, do you see any data in splunk by this search:
index=* module=script script=kids
I have done the above steps, and I can see the data in the last step, but I can't see any data in "traffic --- mikrotik device traffic or interface traffic"
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Apr 02, 2024 1:18 pm

The logs look like this?

script,info serial=75B70647AAAA MikroTik: .id=*5;activity=;blocked=false;bytes-down=0;bytes-up=0;disabled=false;dynamic=true;inactive=false;ip-address=192.168.10.241;limited=false;mac-address=D8:9E:CC:CC:CC:10;name=;rate-down=0;rate-up=0;script=kids;user=
Most important are the stuff in Bold. If that is wrong or missing, stuff does not work.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 08, 2024 6:08 am

I didn't read your update carefully. After following step 2a), it worked normally. Thank you.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Apr 22, 2024 10:58 pm

You are welcome.

If other has problem or suggestion, feel free to ask :)
 
User avatar
jvanhambelgium
Forum Guru
Forum Guru
Posts: 1077
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed May 15, 2024 10:11 pm

Jotne,
Would it be possible to start looking into an extra addition on the the "DNS" section of your Splunk app ?
Since 7.15RC3 there is the concept of "adlist" where you can put URL's to download filter-lists like a Pihole.
Currently testing on my RB3011 and it seems to at least load*** the Adlists ;-)
Your script would need little extra work to get info of that and get it into Splunk.


*** As with all (new) RouterOS features, it does for the moment not look THAT stable. I have about 2 million entries aggregated (same like in the Pihole) except that one runs flawless for months...It looks like my RB3011 seems to start over importing them etc. No crashes, no sup-outs.



Would be great if it would like the "access-lists". So all "Adlist" , with both the match-count and tot name-count entries.
Not too sure if you want it placed under the "DNS" section, or more under the "Other_View" like ACL information

[user@GATEWAY] /ip/dns/adlist> print
Flags: X - disabled
0 url="https://big.oisd.nl" ssl-verify=no match-count=0 name-count=0
1 url="https://v.firebog.net/hosts/Prigent-Crypto.txt" ssl-verify=no match-count=0 name-count=0
2 url="https://osint.digitalside.it/Threat-Int ... omains.txt" ssl-verify=no match-count=0 name-count=0
3 url="https://raw.githubusercontent.com/Steve ... ster/hosts" ssl-verify=no match-count=0 name-count=132314
4 url="https://raw.githubusercontent.com/Dande ... eHosts.txt" ssl-verify=no match-count=0 name-count=0
5 url="https://raw.githubusercontent.com/FadeM ... Spam/hosts" ssl-verify=no match-count=0 name-count=0
6 url="https://raw.githubusercontent.com/Polis ... Dhosts.txt" ssl-verify=no match-count=0 name-count=96446
7 url="https://raw.githubusercontent.com/mkb20 ... omains.txt" ssl-verify=no match-count=0 name-count=1728263
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed May 15, 2024 11:09 pm

We can for sure do some with this. But I think its better to start testing when 7.15 are released.
DNS may be the right place to put it, but if that does not right, its easy to move :)
 
FlippinTurt
just joined
Posts: 16
Joined: Sun Aug 20, 2023 10:48 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed May 22, 2024 2:36 am

One slight suggestion to the iplocation section of the firewall traffic log;
This will lookup the src ip, and if no country (IE an internal IP), will lookup the dst ip, and input that into the table where the normal Country/City goes :)
Also a good app to update splunks geoip db (https://splunkbase.splunk.com/app/5482)
| iplocation src_ip
            | eval City=if(City="","Unknown",City)
            |rename Country as srcCountry
            |rename City as srcCity
            |iplocation dest_ip
            | eval City=if(City="","Unknown",City)
            |rename Country as dstCountry
            |rename City as dstCity
            |eval "Country"= if(isnull(srcCountry), "","" + srcCountry) + if(isnull(dstCountry), "","" + dstCountry)
            |eval "City"= if(isnull(srcCity), "","" + srcCity) + if(isnull(dstCity), "","" + dstCity)
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed May 22, 2024 1:55 pm

Is not that a different database (MaxMind Database) compare to the one that standard built inn iplocation do use?
 
parabellium
just joined
Posts: 10
Joined: Wed Nov 11, 2015 10:30 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed May 29, 2024 4:01 pm

H, Jotne, thank for great job. Kid control consumes too much cpu resources.(RB951) could there be another approach in the future ?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu May 30, 2024 10:36 am

There was another monitoring system som years ago that did what we wanted. This was removed and the only solution to get information on traffic for each user is kid control.
You can invest in a more powerful router or use RouterOS on an X86 hardware.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Jun 13, 2024 6:47 am

I cannot renew my developer license according to https://dev.splunk.com/enterprise/dev_license/. I get the same error code 400 every time. I also cannot get a response when I send an email to devinfo@splunk.com. What should I do?
2024-06-13_11-34-10.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Jun 13, 2024 8:02 am

I did visit the site and for me it seems to work. After requesting a license I do get this message:
Developer License Pending Review
The developer license request is pending review. This process typically takes between 1-3 business days. Once your request is approved, your developer licence will be sent to the email address associated with your Splunk.com account.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Jun 18, 2024 4:10 am

After changing computers, accounts, and network operators, the problem still exists. I submitted an issue on the official forum, but there was no response. Is it possible to apply for developer authorization in this way, or have the rules been changed?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Jun 18, 2024 9:27 am

Since I am not working for Splunk, I can not help. Before I just requested a key and did get one.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Jun 21, 2024 11:03 am

And now I did get a 10GB key, so it did take 3 days. I think you can request a new key 14 days before it ends.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Jun 24, 2024 4:49 am

Found another solution, thanks
 
TraffTBE
just joined
Posts: 4
Joined: Thu Jul 04, 2024 2:43 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Jul 04, 2024 2:48 pm

Hello,

You may already know, but one of most important principles in Splunk is to have correct time. And event time should come form source of event.
Therefore is it possible to modify log format to include timestamp of the event?

Regards,
Tom
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Jul 10, 2024 8:17 am

A good point.

Read this thread viewtopic.php?t=124291 to see how MT (not) follows the RFC.
RFC-5424 clearly specify how time should be set in each message -> https://www.rfc-editor.org/rfc/rfc5424#section-6.2.3

We do use time from message in our work environment, since massage can be delayed on the way inn to splunk for several reasons.
Then its important to know when the actual message was created and we use the time field from the Equipment that has logged the error to set the time in the _time field.

Here is a vmware log message that do contains milli seconds:
<11>2024-07-10T05:08:43.997Z esxiip20.xyz.com vsand[8379206]: Traceback (most recent call last):
Since you always should tell Splunk what format data/time is in, I du use this page: https://strftime.net/
To make same format as log. In this case: %Y-%m-%dT%T%3%Z

And then we have a props.conf some like this:
TIME_PREFIX = <\d+>
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_FORMAT = %Y-%m-%dT%T%3%Z
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 10000
 
TraffTBE
just joined
Posts: 4
Joined: Thu Jul 04, 2024 2:43 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Jul 10, 2024 2:01 pm


[...]


Read this thread viewtopic.php?t=124291 to see how MT (not) follows the RFC.
RFC-5424 clearly specify how time should be set in each message -> https://www.rfc-editor.org/rfc/rfc5424#section-6.2.3

[...]
Lot of good points mentioned there, and I totally agree that RFC5424 should be used, that mean including TCP as transport protocol for syslog messages. RFC5424 is a TRUE syslog standard, where BSD is not, it's just an observation what's going in the wild.

In mean time I was wondering is it possible to force ROS to include kind of timestamp like with router serial. I can take any option, as before logs reach Splunk I'm using suslog-ng to play with incoming logs and "refine" them before sending on to HEC endpoint.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Jul 10, 2024 11:20 pm

Loggs that script sends to Splunk can be fitted with a time prefix. But all that RouterOS does log by it self, will not have it.

PS If some are afraid that loggs will increase with time prefix, it can be trimmed of after its saved to the _time field.
 
TraffTBE
just joined
Posts: 4
Joined: Thu Jul 04, 2024 2:43 pm

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Jul 23, 2024 1:24 pm

Loggs that script sends to Splunk can be fitted with a time prefix.
[ ... ]
Seems like best option we can have at the moment.
[ ... ]
But all that RouterOS does log by it self, will not have it.
[ ... ]
Indeed that's a shame for logs sent to remote location, taking into account that logs written to disk/memory have their timestamps.
[ ... ]
PS If some are afraid that loggs will increase with time prefix, it can be trimmed of after its saved to the _time field.
Yes, that's what syslog header should be used for, therefore into Splunk event goes only message part of syslog event, rest like timestamps, host, etc. go into meta fields.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Thu Jul 25, 2024 10:39 am

We can just hope that one day RFC5424 will be supported.
 
User avatar
fengyuclub
Member Candidate
Member Candidate
Posts: 109
Joined: Mon Dec 09, 2013 8:50 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Jul 31, 2024 11:10 am

About 4 days ago, one of my mikrotik devices stopped recording logs to splunk. I checked and found that it was only updated to the latest system 7.15.3 (stable) and set to restart at night, and then it stopped recording logs to splunk. But there are 3 other mikrotik devices that have also upgraded the system OS, but they are recording logs normally. I reset it by pressing 2e), but it still does not record. How can I troubleshoot?
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Aug 12, 2024 8:47 am

same happeing here on my gateway router ... (Running 7.15.3)
the script stops on an error and tells me to contact MT support (not done yet)

after rebooting the problem is gone for a while until (I guess, the memory is exhausted again)
1st impression was that it is happening in the kid control section of the script, need more debugging, hopefully I have some time to do that today
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Aug 13, 2024 2:48 pm

I have not had time to test on 7.15.x, but will do. If it eats up memory, MT have a problem.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Aug 13, 2024 4:12 pm

Sure,
like I said, all fingers point to the kid control section.
I rebooted 36h ago and waiting for the problem to re-occure
Keep you informed
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Aug 19, 2024 2:08 pm

Upgrade one switch in production from 7.14.2 til 7.15.3 (that have kid control). So far only good news. Less disk space used and less memory used.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Sep 03, 2024 7:29 am

same happeing here on my gateway router ... (Running 7.15.3)
the script stops on an error and tells me to contact MT support (not done yet)

after rebooting the problem is gone for a while until (I guess, the memory is exhausted again)
1st impression was that it is happening in the kid control section of the script, need more debugging, hopefully I have some time to do that today
I have now tested a 750Gr3 with a rather big config and kid control without any problem. Memory do go some up at start, but that is normal doe to the block list. But it is now back to same memory usage as 7.14.2 had.
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Sep 07, 2024 2:27 am

Just getting Splunk set up and I think I messed something up.

I have over a dozen devices and yet only 2 devices are showing up, but repeated/duplicated.

Is there a way to tell Splunk to completely rebuild the database? Or empty it and start over?

Thanks

Screenshot 2024-09-06 192557.png
You do not have the required permissions to view the files attached to this post.
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Sep 07, 2024 12:23 pm

Using an ax3 running 15.3 and the script fails to run with this error:
script error: error - contact MikroTik support and send a supout file (10)
I traced it to this section of the script causing the error:
# Test if pools is used in DHCP or VPN and show leases used
#			:local dname [/ip dhcp-server find where address-pool=$poolname]
#			:if ([:len $dname] = 0) do={
# No DHCP server found, assume VPN
#				:set poolused [:len [used find pool=[:tostr $poolname]]]
#			} else={
I am running two DHCP servers both using the sole created pool.
/ip dhcp-server
add address-pool=pool-guest disabled=yes interface=bridge lease-time=10m name=\
    defconf
add address-pool=pool-guest interface=Guest2g name=dhcp-guest2g
add address-pool=pool-guest interface=Guest5g name=dhcp-guest5g
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=1.1.1.1 gateway=10.0.0.1

/ip pool
add name=pool-guest ranges=10.0.0.10-10.0.0.252
Am I doing something wrong?
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Sep 07, 2024 12:50 pm

Also had to comment out this section of the script to eliminate the error:
# Get detailed command history RouterOS >= v7
# ----------------------------------
#:if ($train > 6 and $CmdHistory) do={
#	:global cmd
#	:local f 0
#	:foreach i in=[/system history find] do={
#		:if ($i = $cmd) do={ :set f 1 }
#		:if ($f != 1) do={
#			:log info message="StartCMD"
#			:log info message=[/system history get $i]
#			:log info message="EndCMD"
#		}
#	}
#	:global cmd  [:pick [/system history find] 0]
#}
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sun Sep 08, 2024 12:14 am

I have a script scheduled to run and it generated a log entry that I like to be able to see in the logs.

It initially logged to "info" but keeping logging enabled to memory for info produced too many log entries from the Splunk script.

I changed my script to log to "warning" just so I can see the entries.

Is there a way to show in Splunk the log entry made by my specific script so I can keep track of when it successfully runs?

It's a simple back and ftp script:

/system
:local cdate [clock get date] 
:local yyyy  [:pick $cdate 0  4]
:local MM    [:pick $cdate 5  7]
:local dd    [:pick $cdate 8 10]
:local identitydate "$[identity get name]_$yyyy-$MM-$dd"
/export show-sensitive file="$identitydate"

/tool fetch upload=yes mode=ftp ascii=no src-path="/$[$identitydate].rsc" dst-path="/mikrotik-backups/$[$identitydate].rsc" address=192.168.2.22 port=21 user=<user> password=<psswd>

/file remove "$identitydate.rsc"

:log warning ("Uploaded rsc backup to 192.168.2.22 as ".$identitydate)

 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 10:08 am

@Josephny

To start over/delete the device db, or just edit it, install "Splunk App for Lookup Editing". Open it and find device_kvstore. Here you can mark all and just remove rows or edit them.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 10:43 am

Not sure why the CMD part gives error. Try this from command line:
{
:global cmd
:local f 0
:foreach i in=[/system history find] do={
:if ($i = $cmd) do={ :set f 1 }
:if ($f != 1) do={
:put "StartCMD"
:put [/system history get $i]
:put "EndCMD"
}
}
:global cmd  [:pick [/system history find] 0]
}
Run it once. Do a config change on the router if it does not fail and run it once more and you should see the config you did added.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 7:27 pm

Found the error in DHCP part of the script. It was not made by me, so needed som time to figure out where the error is.

This part tries to find what DHCP server the pool is used in:
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
It does expect just one server, but in your case it is used in three servers, so it fails.

Will try to see if I can rewrite that part to work in those cases.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12379
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 7:31 pm

Fast fix until more than one is managed.
:local dname [/ip dhcp-server get ([find where address-pool=$poolname]->0) name]

"find" everytime return one array, "get" do not support array...

So expect error if find is directly used by get.


For sure some of my script containing that error. Is not easy too see when coding...
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 9:10 pm

Script updated to 5.7

Fixes when a pool is used in more than one DHCP server.
Since the pool is the same for one or more DHCP server we only take the first find.

Change from:
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
to:
:local dname [/ip dhcp-server get [:pick [find where address-pool=$poolname] 0] name]
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 11:12 pm

@Josephny

To start over/delete the device db, or just edit it, install "Splunk App for Lookup Editing". Open it and find device_kvstore. Here you can mark all and just remove rows or edit them.
Thank you.

I will look into this.

In the meantime, I think the scheduled reindexing solved the problem -- my devices are correct now (automagically).
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 11:14 pm

Script updated to 5.7

Fixes when a pool is used in more than one DHCP server.
Since the pool is the same for one or more DHCP server we only take the first find.

Change from:
:local dname [/ip dhcp-server get [find where address-pool=$poolname] name]
to:
:local dname [/ip dhcp-server get [:pick [find where address-pool=$poolname] 0] name]
I also discovered that it doesn't like multipl DHCP servers.

Thank you for the update.
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Mon Sep 09, 2024 11:20 pm

I just discovered that some of my wifi devices are populating Splunk with the wifi connections and some aren't.

It seems the "/interface/wireless" vs. "/interface/wifi" is the issue.

If we take the "wireless" section of your script and replace the 2 occurences of the word "wireless" with "wifi", and also remove the lookup for "ap", and also change the lookup for "signal-strength" to "signal", I think it will work. "mode" might be a substitute for "ap."

My scripting skills are very poor, so I can't even begin to put a check in the script to see what wireless package is in use and then call the correct script snippet.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Sep 10, 2024 11:49 am

Ye, the wifi/wireless mess. Not sure how to handle that. One of my problem is that I do not have both types. But will try to look inn to it.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Sep 10, 2024 11:59 am

Ye, the wifi/wireless mess. Not sure how to handle that. One of my problem is that I do not have both types. But will try to look inn to it.
I have both types and allready patched parts of my scripts, you can contact me thru discord if needed
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Tue Sep 17, 2024 12:22 pm

I am having a problem with the Netwatch reporting.

I have 8 Netwatch hosts that I am watching on an RB5009 running 7.14.2

If I cycle through disable/enable on each, only 3 are reflected in Splunk.

These are the Netwatch entries:
/tool netwatch
add comment=Netwatch-8.8.4.4-Splunk disabled=no down-script=Netwatch host=8.8.4.4 http-codes="" interval=30s name=Netwatch-8.8.4.4-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.0.11-Splunk disabled=no down-script=Netwatch host=192.168.0.11 http-codes="" interval=30s name=Netwatch-192.168.0.11-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.20.1-Splunk disabled=no down-script=Netwatch host=192.168.20.1 http-codes="" interval=30s name=Netwatch-192.168.20.1-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.30.2-Splunk disabled=no down-script=Netwatch host=192.168.30.2 http-codes="" interval=20s name=Netwatch-192.168.30.2-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.40.1-Splunk disabled=no down-script=Netwatch host=192.168.40.1 http-codes="" interval=30s name=Netwatch-192.168.40.1-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.70.1-Splunk disabled=no down-script=Netwatch host=192.168.70.1 http-codes="" interval=30s name=Netwatch-192.168.70.1-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.20.22-Splunk disabled=no down-script=Netwatch host=192.168.20.22 http-codes="" interval=30s name=Netwatch-192.168.20.22-Splunk test-script="" type=simple up-script=Netwatch
add comment=Netwatch-192.168.1.2-Splunk disabled=no down-script=Netwatch host=192.168.1.2 http-codes="" interval=30s name=Netwatch-192.168.1.2-Splunk test-script="" type=simple up-script=Netwatch

This is the script that is called by the Netwatch entries:
####################################
# Netwatch script
#
# Used as both up and down script
# Created Jotne 2021 v1.5
#
####################################
:local Host $host
/tool netwatch
:local Status [get [find where host="$Host"] status]
:local Comment [get [find where host="$Host"] comment]
:local Interval [get [find where host="$Host"] interval]
:local Since [get [find where host="$Host"] since]
:log info "script=netwatch watch_host=$Host comment=\"$Comment\" status=$Status interval=$Interval since=\"$Since\""
I created the exact same Netwatch entries on another MT device running 7.15.3 and it works (shows up in Splunk).

I wonder if the Netwatch changes between 14.2 and 15.3 is the problem?

EDIT:

I removed a bunch of disabled Netwatch entries as well as Netwatch entries to the same host, and I think it is working.

2nd EDIT:

I see that the problem is when there are multiple Netwatch instances to the same host, even if all but one are disabled. Having each Netwatch instance monitor a unique host (e.g., 1.1.1.1 or 8.8.4.4) the Netwatch Splunk script works.
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Sep 18, 2024 11:37 am

Ye, the wifi/wireless mess. Not sure how to handle that. One of my problem is that I do not have both types. But will try to look inn to it.
Any news on this?

Sure would be nice to have a table of all connections and disconnections showing the details of each client.

Thank you.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 3333
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Sep 18, 2024 1:56 pm

I am having a problem with the Netwatch reporting.
For me the netwatch script do works fine. It should send a log line each time one device goes up and down.
Since the script is very simple, it may be a config error or a bug. Try take som up/down manually and see in the logs.
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Wed Sep 18, 2024 2:55 pm

I am having a problem with the Netwatch reporting.
For me the netwatch script do works fine. It should send a log line each time one device goes up and down.
Since the script is very simple, it may be a config error or a bug. Try take som up/down manually and see in the logs.
The problem is when more than 1 Netwatch entry exists for a single host.
2nd EDIT:

I see that the problem is when there are multiple Netwatch instances to the same host, even if all but one are disabled. Having each Netwatch instance monitor a unique host (e.g., 1.1.1.1 or 8.8.4.4) the Netwatch Splunk script works.
 
eddieb
Member
Member
Posts: 347
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Fri Sep 20, 2024 12:37 pm

Ok, I finally managed to get CAPsMAN running on 7.15+ on my hap-ax3 ...
the CAPsMAN script by @jotne needed some adjustments and it sort of works here ...
- there is no "caps" interface on the capsmanager, it was on 6.xxx
- the value of "channel" is not availiable in the wifi cli ... the variable does exist but it is always empty
need a lot more attention ...
 
Josephny
Long time Member
Long time Member
Posts: 596
Joined: Tue Sep 20, 2022 12:11 am

Re: 📌 Tool: Using Splunk to analyse MikroTik logs 4.0 (Graphing everything) 💾 🛠 💻 📊

Sat Sep 21, 2024 8:34 am

I get no data showing up for Wireguard errors.

But, when I put the following in a search, I see many messages:
index=mikrotik
      module=wireguard

      | eval host_id=host_name."-".host
      | fields _time host interface public_key error host_name host_id serial

          | eval data=serial
          | stats count by data identity
          | eval info=identity." - ".data." (".count.")"
          | sort -count
    
          | eval data=interface
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
   
          | eval data=public_key
          | stats count by data
          | eval info=data." (".count.")"
          | sort -count
9/20/24
9:06:17.000 PM
wireguard,info serial=HDF0xxxx MikroTik: wireguard1: Otp5S5pvkk1ixxxxxxxxx=: Handshake for peer did not complete after 5 seconds, retrying (try 2)
host = 10.10.100.30

Who is online

Users browsing this forum: No registered users and 3 guests