Community discussions

 
User avatar
antispam
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Mon Apr 11, 2005 5:57 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Aug 06, 2019 3:34 pm

Using 2.7, it's mentioned that the "defconf: drop all not coming from LAN" rule should have the prefix 'FI_D_port-test'. When I set that, the Live Attack dashboard doesn't populate as it appears from the source in the dashboard it is searching for 'FW_Drop_all_from_WAN'. When changing this to 'FI_D_port-test' in the Live Attack dashboard source, it works. Is the FW_Drop_all_from_WAN still required?
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Tue Aug 06, 2019 4:24 pm

The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
fengyuclub
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Mon Dec 09, 2013 8:50 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Aug 07, 2019 9:39 am

Recently, one of my ccr is a bit problematic. I can only recover from the time when the device is powered off and restarted. The top of the log in winbox can see red like "system, error System rebooted because of kernel failure" or "Out of memory condition was detected", but I can't see it in splunk search. These log messages, how can I output all the log information to splunk for easy query.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Wed Aug 07, 2019 1:12 pm

A search like this should give all message:
sourcetype=mikrotik module=system
IF not try this:
sourcetype=mikrotik
Or at last just this
*
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
antispam
Frequent Visitor
Frequent Visitor
Posts: 59
Joined: Mon Apr 11, 2005 5:57 pm

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 6:04 am

The filter rule prefix was changed to be more uniform. So I may have mixed up some from script to Splunk code.
One the "Live Attack" dashboard, click Edit->Source.
There you will near the top find some like this:
<search id="base_search">
    <query>
      sourcetype=mikrotik
      module=firewall
      rule=FI_D_port-test
Make sure that you use the same name of the rule as in Splunk, or change Spluk to use same name of the rule as on the router.
Will be fixed in 2.8 of Splunk for MikroTik
Thanks for the prompt reply, that was exactly what I did to get it fixed - keep up the great work!
 
stuartkoh
just joined
Posts: 7
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 2:28 pm

It need to be UDP/514. Its there Router OS sends its syslog.

But:
If you use UDP/514, you need to run Splunk as root user. (allow ports below 1024 need root permission)
If you can not do that, there are two workaround.
1. Send syslog to other port above 1023, like 1514 for UDP syslog.
2. Set up a local syslog server like r-syslog and let Splunk read the lr-syslog log files.

PS updated original post with this information.

One thing to be careful of if you're setting this up in an existing Splunk environment - unless you're really familiar with how things are setup, don't enable Splunk's UDP/514 input without first checking that syslog isn't already being received by something like syslog-ng or rsyslog. You could wind up with data loss or have events put into the wrong index or sourcetype.

It's also not best practices to run Splunk as root. For home use I guess you can get away with it, but for any production Splunk environment you will want to have Splunk running as a restricted user (user = splunk and group = splunk is commonly used).

When you install Splunk, you can set it to autostart on boot and also set the user if you want.
[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user splunk

For receiving syslog - I'm more familiar with syslog-ng, but I also see rsyslog being used successfully. Either one of these will work well for you.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 5:32 pm

This is already mention in section 1b)

If you install Ubuntu, (i think from 16.x), rsyslog is installed as default. But its not listening on port 514/UPD as default and you need to edit the config and restart syslog to get it running. So it should normally not be any conflict.

But in production environment I do also recommend running Splunk as a non root user, then use rsyslog to listen on 514/UDP. Then make Splunk index rsyslogs config.

If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
stuartkoh
just joined
Posts: 7
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 8:31 pm


If any is interested, I have a rather complex rsyslog to handle non standard syslog packed that also add time stamp if that is missing on incoming packets.
I think that syslog-ng has an option that can be used to do this.
keep-timestamp()
Description: Specifies whether syslog-ng should accept the timestamp received from the sending application or client. If disabled, the time of reception will be used instead. This option can be specified globally, and per-source as well. The local setting of the source overrides the global option if available.
https://www.syslog-ng.com/technical-doc ... -timestamp
 
stuartkoh
just joined
Posts: 7
Joined: Tue Apr 09, 2019 2:16 pm
Location: USA

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 10, 2019 8:50 pm

I also wanted to note that I'm not advocating that anyone switch from rsyslog or whatever they're currently using to syslog-ng unless they have good reason to do so.

I don't even really have an opinion on how they compare. I've been working with syslog-ng a bit so that's what I'm familiar with. I'm not trying to start a flame war over which to use. :-)
 
Spotegg
just joined
Posts: 1
Joined: Thu Aug 15, 2019 12:28 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Thu Aug 15, 2019 1:39 am

Hello! I have installed Splunk with Mikrotik module. Thanks, it's great!
Is there a way to organize monitoring of Internet connection on the router. For example, there is an Internet channel on ether1, and you need to somehow download data to Splunk about when the Internet crashed on the router. Maybe there is already such a script?
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Thu Aug 15, 2019 4:10 pm


Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.3 (Graphing everything)

Fri Aug 16, 2019 12:21 pm


Only if your Mikrotik is used as DHCP server continue here, else ignore the following steps.
Check that each IP has a valid comment. I used the comment name as hostname.


Script: manuel_export_dhcp_splunk
:log info "export_dhcp_splunk";
:local hostname;
:local mac;

/file print file="export_dhcp_splunk.txt";

/file set "export_dhcp_splunk.txt" contents="";


:local newdata ("hostname,src_mac\r\n");
/file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);

/ip dhcp-server lease;
:log info "Entering export_dhcp_splunk loop";
:foreach i in=[find] do={
  /ip dhcp-server lease;
  :if ([:len [get $i comment]] > 0) do={
    :set hostname [get $i comment];
    :set mac [get $i mac-address];
    :local newdata ($hostname.",".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
   } else={
    :set mac [get $i mac-address];
    :local newdata ("NONE,".$mac. "\r\n");
    /file set "export_dhcp_splunk.txt" contents=([get export_dhcp_splunk.txt contents] . $newdata);
  }
}
:log info "Ended export_dhcp_splunk";
This script is not working for me, what you mean with comments? I have no comments on dhcp leases, did you comments each ip manually before?

Resolved! :D
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 5:35 pm

I'm having a strange problem both on my synology with docker and my windows pc also, I configured everything as described, I got many logs from my router but after a while it stops reading while counters are still increasing.
Checking via tcpdump, logs are arriving to the server but is like they are not processed.
Immagine.jpg
Someone could help me? Maybe I got something wrong, I cannot image the server is flooded by a single router.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 5:53 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 7:19 pm

See if your prefix is correct at section 2b. On wrong characters and it break all.

You can also do a search with only a start * and set it to last 24 hour and see what data you get.
Yes it's correct, if I do that search last packet is 2 ours ago now while the counter is increasing
Immagine.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 10:03 pm

What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sat Aug 17, 2019 10:34 pm

What does then stop? (looks correct)
You should from the scrip (if you have installed it) get data every 5 minutes.
So search for star and search for 30 min window, you should see data coming in all the time.
Yes the script is installed, I had the 30 min windows search and no data are showed even when the script starts. The stranger thing is that I have the same problem running Splunk on two different system
Immagine.jpg
You do not have the required permissions to view the files attached to this post.
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sun Aug 18, 2019 8:39 pm

Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Sun Aug 18, 2019 11:12 pm

Strange.

Are you 100% your server is listening on Syslog UDP/514?

Here is how to test.
Search for this with REAL-TIME - 1 minute window
"hello"
Then on a linux server run this command. (Change IP (local host 127.0.0.1) to your server if you do test this on an other server :
echo "<14> test hello" | nc -v -u -w 0 127.0.0.1 514
Linux should respond some like this:
Connection to 127.0.0.1 514 port [udp/syslog] succeeded!
On Splunk, you should get a message like this:
Aug 18 19:32:53 127.0.0.1  test hello
If you do not get this message, you need to examine your UDP/514.
Do you run Syslog as root user?
Does Syslog setup to listen on 514?
If Splunk does not run as root, how has you setup UDP/514? Rsyslog where Splunk reads log files?
Thanks for all your help and your time, it's very strange as you said... I opened syslog on port 5014 udp, i also checked the file props.conf on MikroTik app match the port.
I also tried as you said with echo "hello" and it's working.

I think there is some problem with timestamp because then it stops collect i got some error in internal index:
08-18-2019 19:09:43.054 +0200 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (32) characters of event. Defaulting to timestamp of previous event (Sun Aug 18 01:28:00 2019). Context: source=udp:5014|host=192.168.1.1|syslog|
I checked around on google but I didn't find a solution
 
User avatar
Jotne
Forum Guru
Forum Guru
Topic Author
Posts: 1228
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Aug 19, 2019 8:21 am

You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
ferdytao
just joined
Posts: 16
Joined: Mon Sep 26, 2016 8:51 am

Re: Using Splunk to analyse MikroTik logs 2.7 (Graphing everything)

Mon Aug 19, 2019 9:30 am

You do not have the possibility to try a test server and install Splunk as root with only follow the first post?`

Whats wrong in your case is a riddle.
It's a real riddle! :D
The server is running as a root on my last test with ubuntu but nothing is changed. Now I just changed the port from 5014 to default 514 (I used 5014 because on my synology is already kept) and seems to be working. I will keep it running to see what happen!

Thanks for all your support and time spent! ;)

Who is online

Users browsing this forum: No registered users and 1 guest