Using Splunk to analyse MikroTik logs

salehko · August 27, 2017, 6:28pm

Jotne plz contact me, I have a business offer for you about Splunk
ICQ: 716330332
Skype: salivan sha

Jotne:

Here is what I use to get some SNMP information from the MikroTik:

inputs.conf
[snmp://MikroTik-info]
communitystring = public
destination = 10.10.10.1
do_bulk_get = 0
do_get_subtree = 1
ipv6 = 0
object_names = 1.3.6.1.4.1.14988.1.1.3
port = 161
snmp_mode = attributes
snmp_version = 2C
sourcetype = mikrotik
split_bulk_output = 1
trap_rdns = 0
v3_authProtocol = usmHMACMD5AuthProtocol
v3_privProtocol = usmDESPrivProtocol
It runs every 60 seconds (default) and gets the environment tree.
http://www.mibdepot.com/cgi-bin/getmib3.cgi?win=mib_a&r=mikrotik&f=MIKROTIK-MIB&v=v2&t=tree
You can use it to get any SNMP data you like. Eks interface statistics.

PS You need SNMP input module:
https://splunkbase.splunk.com/app/1537/

Juliette64 · September 5, 2017, 2:56am

Hi Jotne, I myself am a Splunk enthusiast and I actually apply the tool at work. I’m also a new Mikrotik user as i’ve just purchased HAP AC and now, would love to play around by Splunking some router logs. I noticed that you’ve already created an app, i’m assuming it’s your private app as i’ve tried searching in the splunk app repository and was not able to find it.

Am wondering if you would help the Splunk community out by contributing and uploading your app to the Splunk app database and making it public and available.

Regards,

Eddy

Jotne · September 5, 2017, 8:33am

The Splunk app does not have the quality (yet) for sharing public. But can I send it to you, if you like.

fahmad · September 5, 2017, 2:37pm

Hey Jotne,

Can you please send me the splunk app also ?

Thank you.

Wallace33 · September 7, 2017, 9:09am

Hello

Can you graph data usage by MAC address?

Regards
James

Jotne · September 7, 2017, 7:29pm

I have not seen a way to do it. But maybe there are one way I have not seen.

Juliette64 · September 13, 2017, 1:37am

Hi Jotne, actually i changed my mind, as i said, i’m a student of the game. best learning is DIY practice, practice and more practice as opposed to copying someone’s work =)

i’m already seasoned in splunk with panels/views/search/charting etc, just wouldn’t mind a little guidance with Mikrotik as i’m new to this router

I have thus far;

enabled Splunk Data input UDP / manual source type - syslog /
added new rule for !ups as directed previously / prefixed with RouterOS / Action remote
defined default ‘remote’ in actions tab to send to my static nat’ed address, port 514, BSD Syslog checked, syslog facility = syslog

what was it that you mentioned previously that you had to turn on logging in your firewall rules? why? aren’t you already capturing every event when you defined !ups
why did you choose to setup transforms in the conf files as opposed to key-value pairs in actual Splunk settings?
i’m healthily ingesting everything however surprisingly the logs are not classified/tagged with the appropriate source event/topic triggers (ie firewall, interface, wireless, system…), is this why you mentioned you help defined certain triggers by “I give all log rule and nat rule different number. Then its easy to see in Splunk what rule that block the data.”

vhgomez78 · September 24, 2017, 12:13am

Hi, can yo send me the app? thanks !

Jotne · September 24, 2017, 7:24am

I post the files here, but there may be som work to get this working.
You need to change all places where I have internal net 10.10.10.x to your net.
SNMP plugin needs to be installed for Splunk to get SNMP data.
Also there are setting on the Mikrotik to make it send syslogs to remote host.

Setup logging external:

System-Logging-Action
add
Name: Some name of server getting Syslog
Type: remote
Remote Address: your Splunk server
Remote port: 514

Get your Mikrotik to send logging:

System-Logging-Rules
Topics: !degug
!snmp
Prefix: MikroTik
Action: remote

It may be more I have forget, så you must take this a beta

UNiXMIT · October 4, 2017, 11:30am

Looks great. Is this the Splunk Cloud? I can’t find the Mikrotik app for Splunk Cloud only for Splunk Enterprise.

Jotne · October 4, 2017, 11:44am

This is not a public app, just a home projects.
I do use Splunk Enterprice Free.

The “Splunk For Mikrotik” on Apps for Splunk Etnerprice is not mine, and its just a start of a prosjekt without any usefulness.

fahmad · October 21, 2017, 12:38pm

Hello Jotne,

You did a great job.

Thank you.

xoff · October 26, 2017, 10:38pm

Splunk Cloud doesn’t have a free tier, Splunk Enterprise (running locally on your own hardware) does, it’s 500mb/day, which is a LOT of data.

xoff · October 26, 2017, 10:39pm

I’ve tried to reach out to the author of that to no avail. If you’re interested in submitting the App, let me know, I’m a Splunk Admin professionally and have published Apps before.

Great work!

caradoc · October 27, 2017, 10:28pm

Oh, I think it’s useful as heck - all it needs is a proxy analyzer to go with the DNS analysis (preferably with a way to define domains/sites that are not supposed to be accessed through that router) and the clients that are making the requests, and you’ve got something pretty solid here.

If you were to define your networks as variables, you wouldn’t have to go modifying them all over the place, either.

Jotne · October 28, 2017, 6:56am

I will try to clean it up some. Then we can see what we do with it

caradoc · October 28, 2017, 5:21pm

I’m in the middle of rebuilding my proxy log parser - once I’ve gotten closer to what I had before my utterly failed migration, I’ll share that with you.

There are several Splunk apps that can do wonders with URL/URI parsing, but I think the URL Toolbox is probably one of the better ones.

https://splunkbase.splunk.com/app/2734/

Jotne · October 28, 2017, 5:32pm

Here is a dashboard for web proxy:

<form>
  <label>MikroTik Web Proxy</label>
  <fieldset submitButton="false">
    <input type="time" token="global_time">
      <label>Time frame</label>
      <default>
        <earliest>-60m@m</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="src_ip">
      <label>Host</label>
      <choice value="*">any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>src_ip</fieldForValue>
      <search>
        <query>sourcetype=mikrotik
         module="web-proxy"
         action="$action$"
         | lookup dnslookup clientip as src_ip OUTPUT clienthost as src_host
         | eval src_host=if(isnull(src_host),src_ip,src_host)
          ,src_ip_split=split(src_ip,".")
          ,l_ip=mvindex(src_ip_split,3)
         | top limit=0 src_host src_ip l_ip
         | eval info=src_host.".".l_ip." (".count.")" 
         | sort - count</query>
        <earliest>$global_time.earliest$</earliest>
        <latest>$global_time.latest$</latest>
      </search>
      <default>*</default>
    </input>
    <input type="dropdown" token="action">
      <label>Action</label>
      <choice value="*">any</choice>
      <fieldForLabel>info</fieldForLabel>
      <fieldForValue>action</fieldForValue>
      <search>
        <query>sourcetype=mikrotik
         module="web-proxy"
         src_ip="$src_ip$"
         | top limit=0 action
         | eval info=action." (".count.")" 
         | sort - count</query>
        <earliest>$global_time.earliest$</earliest>
        <latest>$global_time.latest$</latest>
      </search>
      <default>*</default>
    </input>
    <input type="radio" token="site" searchWhenChanged="true">
      <label>Site info</label>
      <choice value="site_info">Simple</choice>
      <choice value="url2">Medium</choice>
      <choice value="url">Full</choice>
      <default>site_info</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <search>
          <query>sourcetype=mikrotik
           module="web-proxy"
           src_ip="$src_ip$"
           action="$action$"
           | lookup dnslookup clientip as src_ip OUTPUT clienthost as src_host
           | eval src_host=if(isnull(src_host),src_ip,src_host)
            ,site_split=split(url,"/")
            ,To_count=mvcount(site_split)
            ,url2=mvindex(site_split,2)
            ,site_split2=split(url2,".")
            ,To_count2=mvcount(site_split2)
            ,site_dom=mvindex(site_split2,To_count2-1)
            ,site_info=mvindex(site_split2,To_count2-2)
            | rex field=url2 "(?<test>\d+\.\d+\.\d+\.\d+)"
            | eval site_info = if(isnotnull(test),test,site_info)
            | top src_ip src_host $site$ action limit=0</query>
          <earliest>$global_time.earliest$</earliest>
          <latest>$global_time.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">20</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="color" field="action">
          <colorPalette type="map">{"allow":#65A637,"history":#A2CC3E,"firstNew":#6DB7C6}</colorPalette>
        </format>
      </table>
    </panel>
  </row>
</form>

You need this in props.conf

EXTRACT-mikrotik_web-proxy = ^web-proxy,account\s\S+\s(?<src_ip>\d+\.\d+\.\d+\.\d+)\s(?<status>\w+)(\s(?<id>\d+))?\s(?<url>\S+)

caradoc · October 28, 2017, 8:59pm

Were you already working on that, or did you just freehand it after I mentioned it?

That’s… astonishing.

Jotne · October 28, 2017, 11:05pm

I did made it some days ago. I thing I used one hour or two to make it.
When you have created many dashboard, you are you copy the other and make some changes.
But since most of the surfing is using HTTPS, you do not get much valued data.

PS fixed script to cope with site does not have a DNS name