Suricata IDS/IPS integration with Mikrotik (now with OSSEC)

tomfisk · Wed Apr 18, 2018 11:33 pm

This is awesome...if only I could get this on a RB450G...is there?
Should work fine with RB450G. Just need to stream the packet sniffer to the suricata box and follow the installation instructions.
So, are you saying one has to have a separate Suricata box for this to work? The RB450G only has 512MB RAM. I already have a Pfsense machine in front of my 450G...was just thinking it would be cool to have at least Intrusion detection on Mikrotik.

Yep, that's the deal with this implementation. I'm not sure you could run a decent intrusion detection in a metarouter.

Faceless · Thu Apr 26, 2018 7:19 pm

Do I need calea packege to restream packets .Snort need calea. Alsa will hap ac2 4 core cPU handle suricata+ few qos+25filter rules?

tomfisk · Fri Apr 27, 2018 1:08 am

Do I need calea packege to restream packets .Snort need calea. Alsa will hap ac2 4 core cPU handle suricata+ few qos+25filter rules?

No, just stream packets with the sniffer tool to the suricata host. Yes, I don't see any problem with the ability to handle that configuration.

fosilt · Thu Aug 23, 2018 11:34 am

Maybe for anyone is useful this tool:
https://www.stamus-networks.com/open-source/

Integrate suricata + ELKS in a dashboard. I added Tomfisk's script and my MK ban IPs and I can check logs on a website. Final result is very pretty.

Hi , aarango
Do you have tried SELKs from stamus network ?

halimzhz · Sat Sep 15, 2018 7:00 pm

Dear All,

I have few question about this script:

1- I would like to know this script is running on background or i have to run with cron ?
2- Is it any output log for any activity sending to Mikrotik?

Currently i'm running with logstash + python for filtering fast.log and its very slow and too much delay

Please advice and thank you so much

tomfisk · Sun Sep 16, 2018 8:16 am

1. These scripts are running in the background and are started as a service.
2. You can get an email alert when an IP address has been blocked by changing the $email_alert variable in suricata_block.php

Dear All,

I have few question about this script:

1- I would like to know this script is running on background or i have to run with cron ?
2- Is it any output log for any activity sending to Mikrotik?

Currently i'm running with logstash + python for filtering fast.log and its very slow and too much delay

Please advice and thank you so much

halimzhz · Mon Sep 17, 2018 1:48 am

Dear Tomfisk,

Thank you so much to answer my question, actually i have so many question to ask, any possiblity if i can direct with you on Whatsapp or Skype ? Or can you enable your private message on this forum ?

Please help. Thank you so much

halimzhz · Tue Sep 18, 2018 4:17 pm

Hi,

What i understand the packet sniffer capturing from Mikrotik are the packet before the firewall rules, so is it possible to get any packet only after get thru the firewall rules ?

Please advice, TQ

tomfisk · Tue Sep 18, 2018 5:17 pm

Hi Halimzhz,

I don't think it is possible to get the packets only after they've gone through the firewall. The first firewall rule drops all packets from blocked IP addresses. I've look to see if the next rule could run the traffic through a virtual interface (possible), but then you'd have to get the traffic back into the firewall chain (I'm not sure how this would happen).

With regard to your request for some help, my day job keeps me pretty busy but I can try the best I can to provide some help. I don't see a PM option. I really don't want to post my contact info here, but if you want to share your WA I'll contact you.

Hi,

What i understand the packet sniffer capturing from Mikrotik are the packet before the firewall rules, so is it possible to get any packet only after get thru the firewall rules ?

Please advice, TQ

halimzhz · Tue Sep 18, 2018 6:06 pm

Dear Tomfisk,

Thank you so much to reply me, for your information the concept of forward packet to suricata is so nice and suricata will filter the packet with some rules, but that seem fine when you have a very minimal suricata rules, but when u filter the suricata with tons of rules for example u filter by blocklist.de, your suricata will keep receive the same packet again and again, that make the the script keep sending again and again to mikrotik and the process become slow and too much delay, that why i'm asking is it possible to get the packet just after get thru the mikrotik firewall rules, someone advice me to buy 2nd mikrotik device and make it as secondary, the first mikrotik will do a job of firewall and the secondary will do a packet sniffer process to suricata, i hope that is not good idea because i have to spend more.

Please advice. TQ so much

tomfisk · Wed Sep 19, 2018 5:36 am

I understand what you are saying. Have you looked at the number of packets that would be blocked vs. the total volume? There would be a threshold where passing the packets after the firewall would make sense. I'm not sure what that threshold would be, but I would suspect that it would have to be a "significant" volume to make a difference. If you've implemented suricata and the firewall rules then you should be able to look at the packets dropped by the firewall rule vs. the total number of packets.

If the volume is significant, I agree with the advice you received, the only real option is the get another Mikrotik to handle the post-firewall filtering.

Dear Tomfisk,

Thank you so much to reply me, for your information the concept of forward packet to suricata is so nice and suricata will filter the packet with some rules, but that seem fine when you have a very minimal suricata rules, but when u filter the suricata with tons of rules for example u filter by blocklist.de, your suricata will keep receive the same packet again and again, that make the the script keep sending again and again to mikrotik and the process become slow and too much delay, that why i'm asking is it possible to get the packet just after get thru the mikrotik firewall rules, someone advice me to buy 2nd mikrotik device and make it as secondary, the first mikrotik will do a job of firewall and the secondary will do a packet sniffer process to suricata, i hope that is not good idea because i have to spend more.

Please advice. TQ so much

halimzhz · Wed Sep 19, 2018 5:51 pm

Dear Tomfisk,

I have another idea but before that i feel so sorry because my bad english, but i will try to explain what i'm thinking about, i dont know is this possible or not, let say when the script start, the script will look first or grab from Mikrotik the list of banned ip and keep on script memory or as log, then the script will start looking into fast.log and doing filtering and before submit to Mikrotik for ban the IP, the script look first what on memory, on that way the script will not keep submitting to Mikrotik just to get the answer the ip is added or not, or maybe another way as u did on mysql database but everytime u restart the script, have to make sure the record on both Mikrotik and mysql database are clean, this is crazy, but main point is to make not too much delay on busy network

Please advice and thank you for your time

tomfisk · Thu Sep 20, 2018 6:22 am

OK, I thought you wanted to stop scanning traffic that was already blocked by a firewall rule.

So you're saying don't delete and re-add a firewall rule if it already exists for in IP address? Let me look at suricata_block.php and see if that can be added as an option.

Dear Tomfisk,

I have another idea but before that i feel so sorry because my bad english, but i will try to explain what i'm thinking about, i dont know is this possible or not, let say when the script start, the script will look first or grab from Mikrotik the list of banned ip and keep on script memory or as log, then the script will start looking into fast.log and doing filtering and before submit to Mikrotik for ban the IP, the script look first what on memory, on that way the script will not keep submitting to Mikrotik just to get the answer the ip is added or not, or maybe another way as u did on mysql database but everytime u restart the script, have to make sure the record on both Mikrotik and mysql database are clean, this is crazy, but main point is to make not too much delay on busy network

Please advice and thank you for your time

halimzhz · Fri Sep 21, 2018 3:37 am

Dear Tomfisk

I would like to ask you a favor, for your information my fast.log a look a bit different, let me show you:

09/21/2018-08:08:15.059030 [wDrop] [**] [1:207:1] Suricata Rules [**] [Classification: (null)] [Priority: 1] {TCP} xxx.xxx.xxx.xxx:36610 -> nnn.nnn.nnn.nnn:993

For your information i have 2 type of rules, first is alert and second is drop, 'alert' purpose just for monitoring and drop is what i plan to send out to Mikrotik, so for any drop rules will notify on fast.log as 'wDrop', so i need a script to monitor the line with word 'wDrop' and ignore for 'alert', actually i run this kind of script since years ago but its run by Logstash + Python, i'm so frustrated because too much delay for ban any IP because Logstash are cpu/memory hunger even my machine is dual xeon and 32GB RAM.

Seriously i willing to pay if you have time to spend for this. feel shame to put my mobile number here, but please skype me live:8f0c760bc11cde9

Thank you so much

tomfisk · Tue Sep 25, 2018 6:23 am

Hi Halimzhz,
Sorry for the delayed reply. I'm sorry but I really don't have time to be able to help you with this. This solution uses the insert trigger from barnyard2 to grab events that subsequently get processed. It only processes those rules that match what is in the sigs_to_block table in mysql. So the fast.log processing is done by barnyard2. So if you put your wDrop rule description in sigs_to_block table, it should work?

Tom

Dear Tomfisk

I would like to ask you a favor, for your information my fast.log a look a bit different, let me show you:

09/21/2018-08:08:15.059030 [wDrop] [**] [1:207:1] Suricata Rules [**] [Classification: (null)] [Priority: 1] {TCP} xxx.xxx.xxx.xxx:36610 -> nnn.nnn.nnn.nnn:993

For your information i have 2 type of rules, first is alert and second is drop, 'alert' purpose just for monitoring and drop is what i plan to send out to Mikrotik, so for any drop rules will notify on fast.log as 'wDrop', so i need a script to monitor the line with word 'wDrop' and ignore for 'alert', actually i run this kind of script since years ago but its run by Logstash + Python, i'm so frustrated because too much delay for ban any IP because Logstash are cpu/memory hunger even my machine is dual xeon and 32GB RAM.

Seriously i willing to pay if you have time to spend for this. feel shame to put my mobile number here, but please skype me live:8f0c760bc11cde9

Thank you so much

tomeks11 · Tue Nov 06, 2018 5:03 pm

Do I have to run suricata through trafr?

tomfisk · Wed Nov 07, 2018 2:18 am

Do I have to run suricata through trafr?

Nope. Haven't heard of trafr until your message.

tomeks11 · Wed Nov 07, 2018 9:15 am

Do I have to run suricata through trafr?
Nope. Haven't heard of trafr until your message.

I found this information in threads about Snort. I wonder how Suricata receives packets from the mikrotik sniffer with the Tazmen Sniffer Protocol (TZSP)

tomfisk · Wed Nov 07, 2018 10:08 am

Do I have to run suricata through trafr?
Nope. Haven't heard of trafr until your message.
I found this information in threads about Snort. I wonder how Suricata receives packets from the mikrotik sniffer with the Tazmen Sniffer Protocol (TZSP)

1. Packet sniffer on Mikrotik is used, streaming output to specific IP address.
2. tzsp2pcap is used to receive stream (/usr/local/bin/tzsp2pcap -f)
3. Output from tzsp2pcap goes to suricata (/usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r -)

tomeks11 · Wed Nov 07, 2018 1:56 pm

1. Packet sniffer on Mikrotik is used, streaming output to specific IP address.
2. tzsp2pcap is used to receive stream (/usr/local/bin/tzsp2pcap -f)
3. Output from tzsp2pcap goes to suricata (/usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r -)

Thank you. This image was not found in the description.
I made it so that the suricata reads from the interface and also worked.

So I have a suricate to run like this :
/usr/local/bin/tzsp2pcap -f | /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r -
?

tomfisk · Wed Nov 07, 2018 3:46 pm

1. Packet sniffer on Mikrotik is used, streaming output to specific IP address.
2. tzsp2pcap is used to receive stream (/usr/local/bin/tzsp2pcap -f)
3. Output from tzsp2pcap goes to suricata (/usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r -)
Thank you. This image was not found in the description.
I made it so that the suricata reads from the interface and also worked.

So I have a suricate to run like this :
/usr/local/bin/tzsp2pcap -f | /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r -
?

Yes, that's correct.

tomeks11 · Mon Nov 26, 2018 2:43 pm

Why in rule / ip firewall add action = drop chain = forward comment = "Drop any traffic going to bad actors" dst-address-list = Blocked
you have entered dst-address-list? If it was src-address-list, then the package from the aggressor would not go to the internal network and eg. the internal web server would not have to support the SYN

tomfisk · Tue Nov 27, 2018 1:51 am

Why in rule / ip firewall add action = drop chain = forward comment = "Drop any traffic going to bad actors" dst-address-list = Blocked
you have entered dst-address-list? If it was src-address-list, then the package from the aggressor would not go to the internal network and eg. the internal web server would not have to support the SYN

So this is to stop any traffic from going back to a blocked address. There is already a rule to stop any inbound traffic from a blocked address as well.

tomeks11 · Tue Nov 27, 2018 9:31 am

Why in rule / ip firewall add action = drop chain = forward comment = "Drop any traffic going to bad actors" dst-address-list = Blocked
you have entered dst-address-list? If it was src-address-list, then the package from the aggressor would not go to the internal network and eg. the internal web server would not have to support the SYN
So this is to stop any traffic from going back to a blocked address. There is already a rule to stop any inbound traffic from a blocked address as well.

The rule "ip firewall add action = drop chain = input comment =" Block bad actors" src-address-list = Blocked it does not stop the dstnat traffic eg to the web server inside the network. I checked on the web server, 'syn' packets are still coming

tomfisk · Tue Nov 27, 2018 10:24 pm

Why in rule / ip firewall add action = drop chain = forward comment = "Drop any traffic going to bad actors" dst-address-list = Blocked
you have entered dst-address-list? If it was src-address-list, then the package from the aggressor would not go to the internal network and eg. the internal web server would not have to support the SYN
So this is to stop any traffic from going back to a blocked address. There is already a rule to stop any inbound traffic from a blocked address as well.
The rule "ip firewall add action = drop chain = input comment =" Block bad actors" src-address-list = Blocked it does not stop the dstnat traffic eg to the web server inside the network. I checked on the web server, 'syn' packets are still coming

Oh...yes, I'm not attempting to block internet network traffic.

Matthew1471 · Thu Feb 21, 2019 11:37 am

Do I have to run suricata through trafr?
Nope. Haven't heard of trafr until your message.

Can't find trafr officially listed on the MikroTik downloads page either. It's referenced on the Wiki but looks pulled?

Presumably "tzsp2pcap" is the drop in replacement and MikroTik moved to the TZSP protocol..

tomfisk · Thu Feb 21, 2019 11:46 am

Do I have to run suricata through trafr?
Nope. Haven't heard of trafr until your message.
Can't find trafr officially listed on the MikroTik downloads page either. It's referenced on the Wiki but looks pulled?

Presumably "tzsp2pcap" is the drop in replacement and MikroTik moved to the TZSP protocol..

That would most likely be a correct assumption.

anav · Sat Mar 02, 2019 9:05 pm

Hi Tom,
Just trying to understand what all this work is LOL.
I gather you are using a computer with linux OS, that is performing some functions on incoming wan data?
So what is the architecture - modem to router to computer back to router?

What are you trying to stop? Presumably most people do not allow unsolicited traffic in (such traffic is dropped).
Thus are you simply scanning outgoing traffic???

What are you scanning, looking for bad IPs? suspicious packet traffic ???
Im a bit bamboozled by what is actually going on here?

Thanks in advance for answering such basic questions..........
With such info, I will know whether or not for a homeowner the investment in resources is worth it.

tomfisk · Sun Mar 03, 2019 8:49 am

Hi,

First of all, I think it is important for you to understand what a network threat detection engine, like Suricata, does. It ingests network packets, runs those packets against a set of rules, and then reports on those packets which match the rules. Suricata also provides the ability to do intrusion prevention (IPS) against those IP's where the rules matched. So in a typical installation, you'd dedicate a system that would sit on your network perimeter and do this work.

But with the Mikrotik router being more capable than most, then we can use the Mikrotik as a component in the solution. First of all, using the packet sniffer capability of the Mikrotik we can capture packets and send them to Suricata. Which packets you send is up to you, but I send only inbound packets. Suricata does it's thing and the triggered packets and the associated rules end up in a MySQL database. When this occurs, a trigger writes the needed info in a table that is constantly scanned by a php program. When this happens, the php program sends the necessary info to write a firewall entry to the Mikrotik. So now the Mikrotik now plays the role of IPS.

In the end, it depends on how much IDS/IPS your ISP does in catching the bad actors on their perimeter as to whether or not this effort is worth it. Here in Indonesia, my ISP doesn't do any IDS/IPS on their perimeter, so it all comes to my network. So in my case, it's a vital piece of keeping bad actors off my systems.

Hope this helps.

Tom

Hi Tom,
Just trying to understand what all this work is LOL.
I gather you are using a computer with linux OS, that is performing some functions on incoming wan data?
So what is the architecture - modem to router to computer back to router?

What are you trying to stop? Presumably most people do not allow unsolicited traffic in (such traffic is dropped).
Thus are you simply scanning outgoing traffic???

What are you scanning, looking for bad IPs? suspicious packet traffic ???
Im a bit bamboozled by what is actually going on here?

Thanks in advance for answering such basic questions..........
With such info, I will know whether or not for a homeowner the investment in resources is worth it.

anav · Sun Mar 03, 2019 10:31 pm

Great explanation Tom, it sounds a bit of what layer7 firewall does on the Mikrotik, looking for a pattern of packets etc..........
I have read that using layer7 rules really loads the MT CPU so what you in affect are doing is offloading such work and using the MT at the very front end and to implement the outcome(filter rules) (bad IPs). I imagine the time lag to sniff a packet and have the MT put up a rule or add IPs to an existing rule is very short?

Don't laugh but if I have my basic rules setup.
establish related
drop invalid
(my specific allow rules)
drop all else.

Q1. Do I really need all this IDS IDP, synflood, tarpit, blacklists etc etc etc........
In other words, I don't expect anybody to gain access to my router or on my devices based on unsolicited incoming traffic (inbound as you state).

However,. what I don't have control over is folks accessing sites or clicking on emails with bad stuff without knowing it.
Thus where I see the router coming into play is stopping bad outbound traffic (because returns from the bad actor will be allowed back in (established related) and thus one had to nip this before the traffic is allowed out. Thus some layer7 rules are probably a good idea here.
Q2. Does your program or methods address this aspect?

Q3. What is the difference between all the stuff you are managing and my simple technique of:
- adding raw rules to capture all probes on common ports (that I dont use, so prerouting capture has no negative effects).
- adding filter rules to capture all probes common ports I do use, but in order so after they have met my needs (for example DST port,,,,,,,,,,, - I only port forward with known WANIPs allowed source-address-list and to devices on own VLAN). Then in raw, I drop all those captured IPs. My logic good or bad is that I will stop the majority of bad actors for a set time be it 4hours or 2days and that is just as effective as any other set of blacklists etc...........
However, why bother when I already have drop rules on my router (other than stopping repeated attempts or multi port attempts at router).

Any input or clarification or guidance most appreciated as I know very little on this front and always willing to learn.

tomfisk · Mon Mar 04, 2019 5:23 am

Yes, time lag is very short. Less than 2 seconds.

Q1. Simple answer, of course, is "that depends". I do have services that I have open to the wild, so in my case I do want to stop someone trying to gain access through those services. If you don't have any services published, and are just a consumer of the internet, then certainly that removes a big reason for doing so. Just be sure that you really don't have any services published...like IOT devices.

Q2. Yes, Suricata can be used to scan outbound traffic looking for potential threats. I do some of this, minimal, but it is supported. Just make sure your filter on the sniffer is passing outbound traffic to Suricata.

Q3. You don't know what you don't know. Before I implemented this I ran suricata against my traffic to see what what happening. Like I said, in Indonesia I was seeing hundreds of hits an hour. On my server that is in the US, I was seeing maybe a dozen soft hits in a day, that the Mikrotik could handle with the basic firewall rules.

Great explanation Tom, it sounds a bit of what layer7 firewall does on the Mikrotik, looking for a pattern of packets etc..........
I have read that using layer7 rules really loads the MT CPU so what you in affect are doing is offloading such work and using the MT at the very front end and to implement the outcome(filter rules) (bad IPs). I imagine the time lag to sniff a packet and have the MT put up a rule or add IPs to an existing rule is very short?

Don't laugh but if I have my basic rules setup.
establish related
drop invalid
(my specific allow rules)
drop all else.

Q1. Do I really need all this IDS IDP, synflood, tarpit, blacklists etc etc etc........
In other words, I don't expect anybody to gain access to my router or on my devices based on unsolicited incoming traffic (inbound as you state).

However,. what I don't have control over is folks accessing sites or clicking on emails with bad stuff without knowing it.
Thus where I see the router coming into play is stopping bad outbound traffic (because returns from the bad actor will be allowed back in (established related) and thus one had to nip this before the traffic is allowed out. Thus some layer7 rules are probably a good idea here.
Q2. Does your program or methods address this aspect?

Q3. What is the difference between all the stuff you are managing and my simple technique of:
- adding raw rules to capture all probes on common ports (that I dont use, so prerouting capture has no negative effects).
- adding filter rules to capture all probes common ports I do use, but in order so after they have met my needs (for example DST port,,,,,,,,,,, - I only port forward with known WANIPs allowed source-address-list and to devices on own VLAN). Then in raw, I drop all those captured IPs. My logic good or bad is that I will stop the majority of bad actors for a set time be it 4hours or 2days and that is just as effective as any other set of blacklists etc...........
However, why bother when I already have drop rules on my router (other than stopping repeated attempts or multi port attempts at router).

Any input or clarification or guidance most appreciated as I know very little on this front and always willing to learn.

anav · Mon Mar 04, 2019 4:17 pm

Good points.
Yes I have ports open for septic device and solar device with source address list for static company IPs to access.
Having a source address list in my NAT rule renders the port invisible on scans
Yes I have iot devices but they are all on vlans and not on the same vlan and only have access to the internet.

By the way I was subscribing to MOAB (very decent service for pennies) but since i can write this off for taxes using
this service at the moment, since I know I am not savvy and the additional security shouldnt hurt..
https://axiomcyber.com/shield/

So the question becomes, is it worth it for me to wander down the suricata (ossec) route??

haj · Tue Jun 04, 2019 11:45 am

Hey,

Currently we don't have Mikrotik. We have some "homebuilt" Linux routers running our internal routing and firewalling, along with Suricata.
We block bad IPs with a couple of ipset sets and iptables rules.

How does Mikrotik's perform when they have to block a list of say 200.000 IPs?

tomfisk · Wed Jun 05, 2019 2:08 pm

Hey,

Currently we don't have Mikrotik. We have some "homebuilt" Linux routers running our internal routing and firewalling, along with Suricata.
We block bad IPs with a couple of ipset sets and iptables rules.

How does Mikrotik's perform when they have to block a list of say 200.000 IPs?

Hi Haj,

I really can't answer the question about performance. I'd pose that to one of the Mikrotik engineers.

But I would question why you'd be blocking 200,000 IPs? I have looked at my list of frequent offenders and have found that I've been able to consolidate some of the IP addresses into ranges. If you have 200,000 IP addresses then I'd assume that you could identify some pretty significant ranges and block the range rather than individual addresses.

Tom

slimprize · Fri Dec 13, 2019 3:05 am

Hi all,
Could someone please point me to a resource that shows me how to set up Suricata from scratch? I have a server running Ubuntu 19.10 and a mikrotik RB751g-2hnd router. The router is the gateway to my home network. It handles PPPOE authentication with my ISP. I already have the firewall configured and have open dns configured too. I want to add some more network layer detection and prevention now.

Pranav

tomfisk · Fri Dec 13, 2019 5:14 am

Hi all,
Could someone please point me to a resource that shows me how to set up Suricata from scratch? I have a server running Ubuntu 19.10 and a mikrotik RB751g-2hnd router. The router is the gateway to my home network. It handles PPPOE authentication with my ISP. I already have the firewall configured and have open dns configured too. I want to add some more network layer detection and prevention now.

Pranav

I guess I would start here https://redmine.openinfosecfoundation.o ... stallation. The Wiki does provide some good basic information on getting all set up. With regard to specific rules configuration, you may have to dig a little more deeply with Google to find examples that match what you want to do.

slimprize · Sat Dec 14, 2019 3:57 pm

Hi,
I have suricata setup on my Linux machine. I have enabled the Mikrotik to stream like this.
[pranav1@ConShield] /tool> sniffer
[pranav1@ConShield] /tool sniffer> print
only-headers: no
memory-limit: 100KiB
memory-scroll: yes
file-name:
file-limit: 1000KiB
streaming-enabled: yes
streaming-server: xxx.xxx.x.x
filter-stream: yes
filter-interface: airtel #This is the pppoe interface for my broadband connection
filter-mac-address:
filter-mac-protocol:
filter-ip-address:
filter-ipv6-address:
filter-ip-protocol:
filter-port:
filter-cpu:
filter-size:
filter-direction: any
filter-operator-between-entries: or
running: yes

I do not see any activity in suricata. What am I missing?
Pranav

tomfisk · Sun Dec 15, 2019 4:26 am

You have tzsp2pcap running to capture stream and send to suricata? Here are my processes on my suricata host:

snort      656     1  0 Nov18 ?        05:27:05 /usr/local/bin/tzsp2pcap -f
snort      658     1  4 Nov18 ?        1-02:37:00 /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin
snort    24966     1  8 00:01 ?        00:45:40 /usr/local/bin/barnyard2 -c /etc/suricata/barnyard2.conf -l /var/log/suricata -d /var/log/suricata -f unified2.alert -w /var/log/suricata/barnyard2.waldo -D
snort    26872     1  0 Nov21 ?        00:06:14 /usr/bin/php -f /usr/local/bin/suricata_block.php

Hi,
I have suricata setup on my Linux machine. I have enabled the Mikrotik to stream like this.
[pranav1@ConShield] /tool> sniffer
[pranav1@ConShield] /tool sniffer> print
only-headers: no
memory-limit: 100KiB
memory-scroll: yes
file-name:
file-limit: 1000KiB
streaming-enabled: yes
streaming-server: xxx.xxx.x.x
filter-stream: yes
filter-interface: airtel #This is the pppoe interface for my broadband connection
filter-mac-address:
filter-mac-protocol:
filter-ip-address:
filter-ipv6-address:
filter-ip-protocol:
filter-port:
filter-cpu:
filter-size:
filter-direction: any
filter-operator-between-entries: or
running: yes

I do not see any activity in suricata. What am I missing?
Pranav

slimprize · Sun Dec 15, 2019 5:50 pm

Hi,
I am sending the stream from the sniffer tool directly to a Linux box on which I have installed suricata. Do I need an intermediate tool?

Pranav

tomfisk · Mon Dec 16, 2019 3:07 am

Yes, the format from the sniffer stream needs to be converted with tzsp2pcap.

Hi,
I am sending the stream from the sniffer tool directly to a Linux box on which I have installed suricata. Do I need an intermediate tool?

Pranav

slimprize · Mon Dec 16, 2019 11:49 am

Hi Tom,
Thanks for confirming the use of tzsp2pcap. Is there any documentation on how to get it going? I have cloned its source and see the make file but I suspect I need to install headers etc., to build the program.

tomfisk · Mon Dec 16, 2019 1:09 pm

Hi Tom,
Thanks for confirming the use of tzsp2pcap. Is there any documentation on how to get it going? I have cloned its source and see the make file but I suspect I need to install headers etc., to build the program.

Check this blog entry for instructions on compiling [url]https://bløgg.no/2015/03/ids-with-mikrotik-and-snort/[/url]. It's pretty straight-forward once you have the build-essential installed and the required library. build-essential pulls everything needed for your platform.

slimprize · Mon Dec 16, 2019 6:29 pm

Hi Tom,
https://bløgg.no/2015/03/ids-with-mikrotik-and-snort/ did the trick in terms of getting packets. I believe my streaming is working but now, do I use snort and then send to suricata? Sorry, I remain puzzled about the pipeline here. I plan to implement the IPS functionality but will use IDS and tune it first.

What I have done so far is to have the tzsp2pcap utility log packets to a directory and have suricata read the packets from that directory.
sudo tzsp2pcap -o "/home/pranav/pcap/file_%s.pcap" -G 10
I have then run suricata like this.
sudo suricata -c /etc/suricata/suricata.yaml -r /home/pranav/pcap --pcap-file-continuous --pcap-file-delete

I am ok with the writing to disk but am open to other ways of doing this. How do I test this installation? The ideal way would be for me to run an exploit but I do not have that capability. I did try a port scan but did not see any alerts. I may not have suricata configured correctly so am happy to ask in the appropriate forum.
Pranav

tomfisk · Tue Dec 17, 2019 5:09 am

I believe that will work going through a file...here is how I start my instance of suricata:

nohup /usr/local/bin/tzsp2pcap -f | /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin &

So pipe the output of tzsp2pcap into suricata through stdin.

If you ran a port scan, and those ports are open at the mikrotik, you should see suricata fire a rule. Maybe open up one or more of the of the ports during the time you are running the port scan?

Hi Tom,
https://bløgg.no/2015/03/ids-with-mikrotik-and-snort/ did the trick in terms of getting packets. I believe my streaming is working but now, do I use snort and then send to suricata? Sorry, I remain puzzled about the pipeline here. I plan to implement the IPS functionality but will use IDS and tune it first.

What I have done so far is to have the tzsp2pcap utility log packets to a directory and have suricata read the packets from that directory.
sudo tzsp2pcap -o "/home/pranav/pcap/file_%s.pcap" -G 10
I have then run suricata like this.
sudo suricata -c /etc/suricata/suricata.yaml -r /home/pranav/pcap --pcap-file-continuous --pcap-file-delete

I am ok with the writing to disk but am open to other ways of doing this. How do I test this installation? The ideal way would be for me to run an exploit but I do not have that capability. I did try a port scan but did not see any alerts. I may not have suricata configured correctly so am happy to ask in the appropriate forum.
Pranav

slimprize · Tue Dec 17, 2019 5:28 pm

Hi Tom,
Many thanks. I tried reading the output of tzsp2pcap from the command line and suricata launched without a problem How have you defined your network? I went to /interface on my mikrotik, and specified the different address ranges I have. Something like
192.168.88.0/24, 192.168.3.0/24

I ask because I have tried port scanning and no alert has been triggered. Moreover, the stats.log file is also empty so something is not working.
Pranav

tomfisk · Wed Dec 18, 2019 2:27 am

Your sniffer streaming from the mikrotik is set up and you are seeing data? Your streaming server is your suricata host? The interface is the port connected to your ISP?

/tool sniffer set filter-interface=ether1 filter-ip-address=!1.2.3.4/32 filter-stream=yes streaming-enabled=yes streaming-server=192.168.3.1

Hi Tom,
Many thanks. I tried reading the output of tzsp2pcap from the command line and suricata launched without a problem How have you defined your network? I went to /interface on my mikrotik, and specified the different address ranges I have. Something like
192.168.88.0/24, 192.168.3.0/24

I ask because I have tried port scanning and no alert has been triggered. Moreover, the stats.log file is also empty so something is not working.
Pranav

slimprize · Wed Dec 18, 2019 4:41 pm

Hi Tom,
Your sniffer streaming from the mikrotik is set up and you are seeing data?
PL] Yes.
Your streaming server is your suricata host?
PL] Yes.
The interface is the port connected to your ISP?
PL] Ahem, I have a pppoe connection so that is the interface I have defined for sniffing. Should I define the physical port instead?

[pranav1@ConShield] /tool sniffer> print
only-headers: no
memory-limit: 100KiB
memory-scroll: yes
file-name:
file-limit: 1000KiB
streaming-enabled: yes
streaming-server: 192.168.3.2
filter-stream: yes
filter-interface: airtel
filter-mac-address:
filter-mac-protocol:
filter-ip-address:
filter-ipv6-address:
filter-ip-protocol:
filter-port:
filter-cpu:
filter-size:
filter-direction: any
filter-operator-between-entries: or
running: yes
[pranav1@ConShield] /tool sniffer>

tomfisk · Thu Dec 19, 2019 2:13 am

Hi Tom,
Your sniffer streaming from the mikrotik is set up and you are seeing data?
PL] Yes.
Your streaming server is your suricata host?
PL] Yes.
The interface is the port connected to your ISP?
PL] Ahem, I have a pppoe connection so that is the interface I have defined for sniffing. Should I define the physical port instead?

[pranav1@ConShield] /tool sniffer> print
only-headers: no
memory-limit: 100KiB
memory-scroll: yes
file-name:
file-limit: 1000KiB
streaming-enabled: yes
streaming-server: 192.168.3.2
filter-stream: yes
filter-interface: airtel
filter-mac-address:
filter-mac-protocol:
filter-ip-address:
filter-ipv6-address:
filter-ip-protocol:
filter-port:
filter-cpu:
filter-size:
filter-direction: any
filter-operator-between-entries: or
running: yes
[pranav1@ConShield] /tool sniffer>

OK, just wanted to make sure you are getting data. Which version of suricata are you running?

slimprize · Thu Dec 19, 2019 7:35 pm

Hi Tom,
I suspect the problem was that I was using the pppoe interface. https://groups.google.com/forum/#!topic ... UUNgIGqsv4
gave me a clue;
I have run the test script mentioned at the above URL and am getting alerts ever since I set the sniffer interface to ether1 on the Mikrotik.

Many thanks for your help.
Pranav

tomfisk · Fri Dec 20, 2019 3:24 am

Glad you got it working Pranav!

Hi Tom,
I suspect the problem was that I was using the pppoe interface. https://groups.google.com/forum/#!topic ... UUNgIGqsv4
gave me a clue;
I have run the test script mentioned at the above URL and am getting alerts ever since I set the sniffer interface to ether1 on the Mikrotik.

Many thanks for your help.
Pranav

G00dm4n · Sun Jun 14, 2020 3:17 am

Hi guys,

I find here some info for using Suricata IDS/IPS with Mikrotik.
I also found there's a good build from Stamus Networks who is good and stable - SELKS.
Can someone post more straight and updated manual of using Microtik together with SELKS5 or SELKS6 RC1.
I prefer we get straight to latest - SELKS6 Release Candidate 1. All is same as SELKS5, just some names and places of the scripts can be different.
(Later we can do extended manual how to build proper and efficient Suricata instlation for users who not prefer using SELKS ISO.)
Since Sniffer tool in Mikrotik is pretty straight forward to set up we can just skip this... but here it is:
tool sniffer
set file-limit=0KiB filter-interface=<THE_INTERFACE_WHICH_TRAFFIC_YOU_WILL_SEND> memory-limit=0KiB streaming-enabled=yes streaming-server=<IP_OF_YOUR_SELKS_SERVER>

The SELKS part from another side is a bit more foggy to be set.
The installation is straight forward, but nobody care to explain a good way and method how to use it together with Mikrotik.
So why we don't make it?

What I have get straight until now is:
1. Install SELKS on VM/x86/RaspberyPI B
Suggested minimum configuration is with 2 core CPU, 8GB RAM and at least 50GB Disk space.
2. Do inital SELKS setup by the scripts
2.1. First time run: selks-first-time-setup_stamus
2.2. Upgrade: selks-upgrade_stamus
3. Check and modify the configs by your needs regarding page below
https://github.com/StamusNetworks/SELKS ... time-setup
4. Install the converter from TZSP to PCAP so Surikata understand the stream from Mikrotik
4.1. Using trafr (just consider this is written ib 2004 and is 32bit)
dpkg --add-architecture i386
cd /usr/local/sbin
wget http://www.mikrotik.com/download/trafr.tgz
tar -xf /usr/local/sbin/trafr.tgz
chmod u+x trafr
chown root.root trafr
4.2. Using tzsp2pcap
apt-get install build-essential libpcap0.8-dev
cd /usr/local/sbin
git clone https://github.com/thefloweringash/tzsp2pcap
cd /usr/local/sbin/tzsp2pcap
cc -std=gnu99 -o tzsp2pcap -Wall -Wextra -pedantic -O2 -lpcap tzsp2pcap.c
mv tzsp2pcap /usr/local/sbin/
5. Start capturing the data
5.1. Using trafr
/usr/local/sbin/trafr -s | suricata -c /etc/suricata/suricata.yaml --runmode autofp -v -r /dev/stdin
5.2. Using tzsp2pcap
nohup /usr/local/bin/tzsp2pcap -f | /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin &

Here I come to the point I have to put this command in a bash script which supposed to run on startup. (will update it)

From this onward I don't get the clear picture what and how to do it.
By this I meant,
-in which mode to set Suricata initially,
-where to create files for the rules that to be send to Mikrotik router,
-what to do so capturing to start proper after reboot,
-does SELKS6 indeed need the data stream conversion,

I know there's some posts here by TomFisk but they are not very clear and have plenty of "jump here and there".
Also please put your comments and suggestions how we can do this better and more agile.
My idea is to avoid using any cumbersome hardware and keep good performance.
If we can go down to something as NUC or ALIX PC or even RaspberyPI - the smaller, the better.

tomfisk · Sun Jun 14, 2020 4:26 am

Hi,

So what problem are you trying to solve? SELKS is an IDS/IPS reporting/visualization and management platform that uses Suricata to implement network firewall rules. There is nothing inherent in Suricata to implement the firewall rules through a Microtik device. SELKS doesn't change that. That is why I developed the method described in this post.

Tom

Hi guys,

I find here some info for using Suricata IDS/IPS with Mikrotik.
I also found there's a good build from Stamus Networks who is good and stable - SELKS.
Can someone post more straight and updated manual of using Microtik together with SELKS5 or SELKS6 RC1.
I prefer we get straight to latest - SELKS6 Release Candidate 1. All is same as SELKS5, just some names and places of the scripts can be different.
(Later we can do extended manual how to build proper and efficient Suricata instlation for users who not prefer using SELKS ISO.)
Since Sniffer tool in Mikrotik is pretty straight forward to set up we can just skip this... but here it is:
tool sniffer
set file-limit=0KiB filter-interface=<THE_INTERFACE_WHICH_TRAFFIC_YOU_WILL_SEND> memory-limit=0KiB streaming-enabled=yes streaming-server=<IP_OF_YOUR_SELKS_SERVER>

The SELKS part from another side is a bit more foggy to be set.
The installation is straight forward, but nobody care to explain a good way and method how to use it together with Mikrotik.
So why we don't make it?

What I have get straight until now is:
1. Install SELKS on VM/x86/RaspberyPI B
Suggested minimum configuration is with 2 core CPU, 8GB RAM and at least 50GB Disk space.
2. Do inital SELKS setup by the scripts
2.1. First time run: selks-first-time-setup_stamus
2.2. Upgrade: selks-upgrade_stamus
3. Check and modify the configs by your needs regarding page below
https://github.com/StamusNetworks/SELKS ... time-setup
4. Install the converter from TZSP to PCAP so Surikata understand the stream from Mikrotik
4.1. Using trafr (just consider this is written ib 2004 and is 32bit)
dpkg --add-architecture i386
cd /usr/local/sbin
wget http://www.mikrotik.com/download/trafr.tgz
tar -xf /usr/local/sbin/trafr.tgz
chmod u+x trafr
chown root.root trafr
4.2. Using tzsp2pcap
apt-get install build-essential libpcap0.8-dev
cd /usr/local/sbin
git clone https://github.com/thefloweringash/tzsp2pcap
cd /usr/local/sbin/tzsp2pcap
cc -std=gnu99 -o tzsp2pcap -Wall -Wextra -pedantic -O2 -lpcap tzsp2pcap.c
mv tzsp2pcap /usr/local/sbin/
5. Start capturing the data
5.1. Using trafr
/usr/local/sbin/trafr -s | suricata -c /etc/suricata/suricata.yaml --runmode autofp -v -r /dev/stdin
5.2. Using tzsp2pcap
nohup /usr/local/bin/tzsp2pcap -f | /usr/bin/suricata -v -c /etc/suricata/suricata.yaml -r /dev/stdin &

Here I come to the point I have to put this command in a bash script which supposed to run on startup. (will update it)

From this onward I don't get the clear picture what and how to do it.
By this I meant,
-in which mode to set Suricata initially,
-where to create files for the rules that to be send to Mikrotik router,
-what to do so capturing to start proper after reboot,
-does SELKS6 indeed need the data stream conversion,

I know there's some posts here by TomFisk but they are not very clear and have plenty of "jump here and there".
Also please put your comments and suggestions how we can do this better and more agile.
My idea is to avoid using any cumbersome hardware and keep good performance.
If we can go down to something as NUC or ALIX PC or even RaspberyPI - the smaller, the better.

G00dm4n · Sun Jun 14, 2020 12:01 pm

Hi TomFisk,

I see your point.
Maybe you can help me to do step-by-step list what and how to use your methid with SELKS.
As you pointed additional components in SELKS really add lot of load.
I am also interested to use minimum install - just Suricata + necessary interfaces so this can be implemented on low power PC as RaspberyPi and NUC.
Can you help me with this?
The issue is that in the thread here you guys jump directly to solving practical issues. Even I know pretty much how the IDS works and so I miss your points why some things are done.
Also I will appreciate to know exact minimal install configuration, where to set some of files and so on.
Sorry if I look a bit crazy with my demands. But we really need more straight forward method.
I have some resources and at the moment I have time to help with testings.
So why we not try?
Maybe not only Suricata/SELKS... we can try anything useful.
I just need some guidance as I am not getting why and what to do with this Python/PHP ...

tomfisk · Tue Jun 16, 2020 12:12 pm

There is someone who used this work as the basis of a github project, https://github.com/elmaxid/ips-mikrotik-suricata

Hi TomFisk,

I see your point.
Maybe you can help me to do step-by-step list what and how to use your methid with SELKS.
As you pointed additional components in SELKS really add lot of load.
I am also interested to use minimum install - just Suricata + necessary interfaces so this can be implemented on low power PC as RaspberyPi and NUC.
Can you help me with this?
The issue is that in the thread here you guys jump directly to solving practical issues. Even I know pretty much how the IDS works and so I miss your points why some things are done.
Also I will appreciate to know exact minimal install configuration, where to set some of files and so on.
Sorry if I look a bit crazy with my demands. But we really need more straight forward method.
I have some resources and at the moment I have time to help with testings.
So why we not try?
Maybe not only Suricata/SELKS... we can try anything useful.
I just need some guidance as I am not getting why and what to do with this Python/PHP ...

zbe · Tue Jun 16, 2020 7:58 pm

As a part of my python-learning process I tried to make something similar to your fast2mikrotik but in python. All you need is python-librouteros, python-pyinotify and python-ujson.
It's reading from separate eve-log named alerts.json instead of fast.log. So you need to add that in suricata.yaml. It's also checking for router's uptime and adds whole file in case router has been up for less than 10 minutes. I'm just testing this on raspberry pi where I'm port-mirroring.
Anyway, there's probably some stupid code in here, so if anyone needs it - take it for what it's worth.

Edit: Need to add some kind of checking if api connection is already open or else it keeps opening new ones. (added api.close() for now)

#!/usr/bin/env python3

#
# Script for adding alerts from Suricata to Mikrotik routers. 
#
# In suricata.yaml add another eve-log:  
#  - eve-log:
#      enabled: yes
#      filetype: regular
#      filename: alerts.json
#      types:
#        - alert
#

import ssl
import librouteros
from librouteros import connect
from librouteros.query import Key
import ujson
import pyinotify
import re
from time import sleep
from datetime import datetime as dt, timedelta as td, timezone as tz
import os

# Edit these settings:
USERNAME = "suricata"
PASSWORD = "suricata123"
ROUTER_IP = "192.168.88.1"
TIMEOUT = "1d"
PORT = 8729  # api-ssl port
FILEPATH = os.path.abspath("/var/log/suricata/alerts.json")
ROUTER_LIST_NAME = "Suricata"
WAN_IP = "n/a"  # You can add your WAN IP if you are port-mirroring, so it doesn't get mistakenly added. (don't leave empty string)
LOCAL_IP_PREFIX = "192.168."
WHITELIST_IPS = (WAN_IP, LOCAL_IP_PREFIX, "127.0.0.1")
COMMENT_TIME_FORMAT = "%-d %b %Y %H:%M:%S.%f"  # Check datetime strftime formats

# Add all alerts from alerts.json on start?
# Setting this to True will start reading alerts.json from beginning
# and will add whole file to firewall when pyinotify is triggered.
# Just for testing purposes, i.e. not good for systemd service.
ADD_ON_START = False


class EventHandler(pyinotify.ProcessEvent):
    def process_IN_MODIFY(self, event):
        add_to_tik(read_json(FILEPATH))
        check_truncated(FILEPATH)

def check_truncated(fpath):  # Check if logrotate truncated file. (Use 'copytruncate' option for this to work I guess.)
    global last_pos
    
    if last_pos > os.path.getsize(fpath):
        last_pos = 0
    
def seek_to_end(fpath):
    global last_pos
    
    if not ADD_ON_START:
        while True:
            try:
                last_pos = os.path.getsize(fpath)
                return

            except(FileNotFoundError):
                print(f"File: {fpath} not found. Re-trying in 10 seconds..")
                sleep(10)
                continue

def read_json(fpath):
    global last_pos
    
    while True:
        try:
            with open(fpath, "r") as f:
                f.seek(last_pos)
                alerts = [ujson.loads(line) for line in f.readlines()]
                last_pos = f.tell()
                return alerts

        except(FileNotFoundError):
            print(f"File: {fpath} not found. Re-trying in 10 seconds..")
            sleep(10)
            continue
        
def add_to_tik(alerts):
    global last_pos
    global time
    
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.set_ciphers('ADH:@SECLEVEL=0')
    
    _address = Key("address")
    _id = Key(".id")
    _list = Key("list")
    
    while True:
        try:
            api = connect(username=USERNAME, password=PASSWORD, host=ROUTER_IP, ssl_wrapper=ctx.wrap_socket, port=PORT)
            break
        
        except librouteros.exceptions.TrapError as e:
            if "invalid user name or password" in str(e):
                print("Invalid username or password.")
            else:
                raise
                
        except ConnectionRefusedError:
            print("Connection refused. (api-ssl disabled in router?)")
        
        except OSError as e:
            if "[Errno 113] No route to host" in str(e):
                print("No route to host. Re-trying in 10 seconds..")
                sleep(10)
                continue
            else:
                raise

    address_list = api.path("/ip/firewall/address-list")
    resources = api.path("system/resource")

    for event in { item['src_ip'] : item for item in alerts }.values():  # Remove duplicate src_ips.
        timestamp = dt.strptime(event["timestamp"], "%Y-%m-%dT%H:%M:%S.%f%z").strftime(COMMENT_TIME_FORMAT)
        
        if event["src_ip"].startswith(WHITELIST_IPS):  # If you are source ip, then add destination ip.
            if event["dest_ip"].startswith(WHITELIST_IPS):  
                continue  # Skip adding anything if both source and destination ips are from WHITELIST_IPS. (just in case)
            
            try:
                address_list.add(list=ROUTER_LIST_NAME, address=event["dest_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: SPort: {event.get('src_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
            except librouteros.exceptions.TrapError as e:
                if "failure: already have such entry" in str(e):  # If such entry already exists, delete it and re-add.
                    for row in address_list.select(_id, _list, _address).where(_address == event["dest_ip"], _list == ROUTER_LIST_NAME):
                        address_list.remove(row[".id"])
                
                    address_list.add(list=ROUTER_LIST_NAME, address=event["dest_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: SPort: {event.get('src_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
                else:
                    raise
            
        else:  # Add source ip.
            try:
                address_list.add(list=ROUTER_LIST_NAME, address=event["src_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: DPort: {event.get('dest_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
            except librouteros.exceptions.TrapError as e:
                if "failure: already have such entry" in str(e):
                    for row in address_list.select(_id, _list, _address).where(_address == event["src_ip"], _list == ROUTER_LIST_NAME):
                        address_list.remove(row[".id"])
                        
                    address_list.add(list=ROUTER_LIST_NAME, address=event["src_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: DPort: {event.get('dest_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
                else:
                    raise
    # If router has been rebooted in past 10 minutes, add whole file, then wait for 10 minutes. (so rules don't get constantly re-added for 10 minutes)
    if check_tik_uptime(resources) and (dt.now(tz.utc) - time) / td(minutes=1) > 10:
        time = dt.now(tz.utc)
        last_pos = 0
        add_to_tik(read_json(FILEPATH))

    api.close()
        
def check_tik_uptime(resources):  # Check if router has been up for less than 10 minutes
    for row in resources:
        uptime = row["uptime"]
   
    if any(letter in uptime for letter in "wdh"):  # If "w", "d" or "h" is in uptime then router is obviously up for more than 10 minutes.
        return False
    
    if "m" in uptime:
        minutes = int(re.search("(\A|\D)(\d*)m", uptime).group(2))  # Find numbers in front of "m".
    else:
        minutes = 0
        
    if minutes >= 10:
        return False
    
    return True

if __name__ == "__main__":
    time = dt.now(tz.utc) - td(minutes=10)  # Set time to 10 minutes before now, so "(dt.now(tz.utc) - time) / td(minutes=1) > 10" is True the first time around.
    last_pos = 0
    seek_to_end(FILEPATH)

    wm = pyinotify.WatchManager()
    handler = EventHandler()
    notifier = pyinotify.Notifier(wm, handler)
    wm.add_watch(FILEPATH, pyinotify.IN_MODIFY)
    notifier.loop()

Edit 2: Ok I changed it a bit so it doesn't pollute log with 'logged in' 'logged out' - it stays connected. Hope this is last edit. :p

#!/usr/bin/env python3

#
# Script for adding alerts from Suricata to Mikrotik routers. 
#
# In suricata.yaml add another eve-log:  
#  - eve-log:
#      enabled: yes
#      filetype: regular
#      filename: alerts.json
#      types:
#        - alert
#

import ssl
import librouteros
from librouteros import connect
from librouteros.query import Key
import ujson
import pyinotify
import re
from time import sleep
from datetime import datetime as dt, timedelta as td, timezone as tz
import os

# Edit these settings:
USERNAME = "suricata"
PASSWORD = "suricata123"
ROUTER_IP = "192.168.88.1"
TIMEOUT = "1d"
PORT = 8729  # api-ssl port
FILEPATH = os.path.abspath("/var/log/suricata/alerts.json")
ROUTER_LIST_NAME = "Suricata"
WAN_IP = "n/a"  # You can add your WAN IP if you are port-mirroring, so it doesn't get mistakenly added. (don't leave empty string)
LOCAL_IP_PREFIX = "192.168."
WHITELIST_IPS = (WAN_IP, LOCAL_IP_PREFIX, "127.0.0.1")  # You can expand this list 
COMMENT_TIME_FORMAT = "%-d %b %Y %H:%M:%S.%f"  # Check datetime strftime formats

# Add all alerts from alerts.json on start?
# Setting this to True will start reading alerts.json from beginning
# and will add whole file to firewall when pyinotify is triggered.
# Just for testing purposes, i.e. not good for systemd service.
ADD_ON_START = False

class EventHandler(pyinotify.ProcessEvent):
    def process_IN_MODIFY(self, event):
        try:
            add_to_tik(read_json(FILEPATH))            
        except ConnectionError:
            connect_to_tik()
            
        check_truncated(FILEPATH)

def check_truncated(fpath):  # Check if logrotate truncated file. (Use 'copytruncate' option for this to work I guess.)
    global last_pos
    
    if last_pos > os.path.getsize(fpath):
        last_pos = 0
    
def seek_to_end(fpath):
    global last_pos
    
    if not ADD_ON_START:
        while True:
            try:
                last_pos = os.path.getsize(fpath)
                return

            except(FileNotFoundError):
                print(f"File: {fpath} not found. Re-trying in 10 seconds..")
                sleep(10)
                continue

def read_json(fpath):
    global last_pos
    
    while True:
        try:
            with open(fpath, "r") as f:
                f.seek(last_pos)
                alerts = [ujson.loads(line) for line in f.readlines()]
                last_pos = f.tell()
                return alerts

        except(FileNotFoundError):
            print(f"File: {fpath} not found. Re-trying in 10 seconds..")
            sleep(10)
            continue
        
def add_to_tik(alerts):
    global last_pos
    global time
    global api
    
    _address = Key("address")
    _id = Key(".id")
    _list = Key("list")
 
    address_list = api.path("/ip/firewall/address-list")
    resources = api.path("system/resource")

    for event in { item['src_ip'] : item for item in alerts }.values():  # Remove duplicate src_ips.
        timestamp = dt.strptime(event["timestamp"], "%Y-%m-%dT%H:%M:%S.%f%z").strftime(COMMENT_TIME_FORMAT)
        
        if event["src_ip"].startswith(WHITELIST_IPS):  # If you are source ip, then add destination ip.
            if event["dest_ip"].startswith(WHITELIST_IPS):  
                continue  # Skip adding anything if both source and destination ips are from WHITELIST_IPS. (just in case)
            
            try:
                address_list.add(list=ROUTER_LIST_NAME, address=event["dest_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: SPort: {event.get('src_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
            except librouteros.exceptions.TrapError as e:
                if "failure: already have such entry" in str(e):  # If such entry already exists, delete it and re-add.
                    for row in address_list.select(_id, _list, _address).where(_address == event["dest_ip"], _list == ROUTER_LIST_NAME):
                        address_list.remove(row[".id"])
                
                    address_list.add(list=ROUTER_LIST_NAME, address=event["dest_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: SPort: {event.get('src_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
                else:
                    raise
            
        else:  # Add source ip.
            try:
                address_list.add(list=ROUTER_LIST_NAME, address=event["src_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: DPort: {event.get('dest_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
            except librouteros.exceptions.TrapError as e:
                if "failure: already have such entry" in str(e):
                    for row in address_list.select(_id, _list, _address).where(_address == event["src_ip"], _list == ROUTER_LIST_NAME):
                        address_list.remove(row[".id"])
                        
                    address_list.add(list=ROUTER_LIST_NAME, address=event["src_ip"], comment=f"[{event['alert']['gid']}:{event['alert']['signature_id']}] {event['alert']['signature']} ::: DPort: {event.get('dest_port')}/{event['proto']} ::: timestamp: {timestamp}", timeout=TIMEOUT)
                
                else:
                    raise
    # If router has been rebooted in past 10 minutes, add whole file, then wait for 10 minutes. (so rules don't get constantly re-added for 10 minutes)
    if check_tik_uptime(resources) and (dt.now(tz.utc) - time) / td(minutes=1) > 10:
        time = dt.now(tz.utc)
        last_pos = 0
        add_to_tik(read_json(FILEPATH))


def check_tik_uptime(resources):  # Check if router has been up for less than 10 minutes
    for row in resources:
        uptime = row["uptime"]
   
    if any(letter in uptime for letter in "wdh"):  # If "w", "d" or "h" is in uptime then router is obviously up for more than 10 minutes.
        return False
    
    if "m" in uptime:
        minutes = int(re.search("(\A|\D)(\d*)m", uptime).group(2))  # Find numbers in front of "m".
    else:
        minutes = 0
        
    if minutes >= 10:
        return False
    
    return True

def connect_to_tik():
    global api
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.set_ciphers('ADH:@SECLEVEL=0')

    while True:
        try:
            api = connect(username=USERNAME, password=PASSWORD, host=ROUTER_IP, ssl_wrapper=ctx.wrap_socket, port=PORT)
            break
        
        except librouteros.exceptions.TrapError as e:
            if "invalid user name or password" in str(e):
                print("Invalid username or password.")
            else:
                raise
                
        except ConnectionRefusedError:
            print("Connection refused. (api-ssl disabled in router?)")
        
        except OSError as e:
            if "[Errno 113] No route to host" in str(e):
                print("No route to host. Re-trying in 10 seconds..")
                sleep(10)
                continue
            else:
                raise

if __name__ == "__main__":
    time = dt.now(tz.utc) - td(minutes=10)  # Set time to 10 minutes before now, so "(dt.now(tz.utc) - time) / td(minutes=1) > 10" is True the first time around.
    last_pos = 0
    seek_to_end(FILEPATH)    
    connect_to_tik()

    wm = pyinotify.WatchManager()
    handler = EventHandler()
    notifier = pyinotify.Notifier(wm, handler)
    wm.add_watch(FILEPATH, pyinotify.IN_MODIFY)
    notifier.loop()

Edit 3.1: Here is github link: https://github.com/zzbe/mikrocata Fixed/edited few things. Added option to add other lists you might have after reboot, added ignore.conf for ignoring rules.

G00dm4n · Sat Jun 20, 2020 3:30 am

Thanks,

I will do some testing when I have time.
In this funny times I have been summoned to join with my team.
After listening that we will start probably at September or so I was requested to join in few days.
Have to put on hold some of my projects for a while.
Will reply ASAP.

MTv · Sat Oct 24, 2020 10:19 pm

Thanks zbe for your script. I wrote a mini instruction for setting up Suricata in conjunction with ROS+Mikrocata on Debian Buster.

pitterbrayn · Tue Jan 12, 2021 3:27 pm

Sorry for being lazy. But does any one have pre-configured image, which supports to install it and change minor configurations like IP address, username and password?

pingpong1428 · Tue May 11, 2021 5:59 pm

Hello Im using instructions from Tom script fast2 and it works nice, i can writte to mikrotik adress list perfectly.
Im using Por sniffer as instructed, but i cannot when y try to trigger some alerts from other computers on the network, nothing happens.
Only the test i made from server.

i dont know if im making a mistake with the sniffer or what.

any help would be apreciate

Regards

pingpong1428 · Tue May 11, 2021 10:04 pm

ok after reading more i use this https://robert.penz.name/849/howto-setu ... ta-as-ids/

and finally been able to install trafr

now if i made trafr -s | tcpdump -r - -n

im seeing lots of info coming from mikrotik packer sniffer

but when i try to run.

trafr -s | suricata -c /etc/suricata/suricata.yaml -r -

i reiceve

root@suricata1:~# trafr -s | suricata -c /etc/suricata/suricata.yaml -r -
11/5/2021 -- 16:03:42 - <Error> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - ERROR: Pcap file does not exist

any help, thanks.

foresthus · Fri Jul 16, 2021 2:38 pm

Are there admins who can share an actual docomentation for this issue. It would be nice if it is in english.

thnx

MTv · Wed Aug 04, 2021 2:56 pm

Made a simple script that processes the generated Suricata eve-log in real time and, based on alerts, adds an ip-address to the MikroTik Address Lists for a specified time for subsequent blocking.

#!/usr/bin/env bash

# Bashcata Variables;
router="" # mikrotik ip;
login="" # user for connect to mikrotik;
privatekey="/root/.ssh/mik_rsa" # private key for ssh;
fw_list="idps_alert" # name firewall list;
fw_timeout="7" # days ban ip;

# - #
script_dir="$(dirname "$(readlink -f "$0")")"
alerts_file="/var/log/suricata/alerts.json"
pid_suricata="$(pidof suricata)"
white_list="${script_dir}/white.list"
mark_ip="${script_dir}/mark.ip"
# - #

# Check files;
if [ ! -e "${white_list}" ]; then touch "${white_list}" ; echo -e "# src_ip\n\n# signature_id" > "${white_list}" ; fi
if [ ! -e "${mark_ip}" ]; then touch "${mark_ip}" ; fi

# Setting the logger utility function;
function logger() {
    find "${script_dir}"/ -maxdepth 1 -name "*.log" -size +100k -exec rm -f {} \;
    echo -e "[$(date "+%d.%m.%Y / %H:%M:%S")]: $1" >> "${script_dir}"/"bash_cata.log"
}

# Tail Conveyor;
tail -q -f "${alerts_file}" --pid="$pid_suricata" -n 500 | while read -r LINE; do

# Parsing Json file via jq;
alerts="$(echo "${LINE}" | jq -c '[.timestamp, .src_ip, .dest_ip, .dest_port, .proto, .alert .signature_id, .alert .signature, .alert .category]' | sed 's/^.//g; s/"//g; s/]//g')"

# White List;
check_list () {
    wl="false"
    if grep -q -E "${src_ip}|${signature_id}" "${white_list}"; then wl="true" ; fi
}

# Mark IP;
check_ip () {
    new_ip="false"
    check_timestamp="$(awk -v t=$(date -d"-${fw_timeout} day" +%Y-%m-%dT%H:%M:%S) '$2<t' "${mark_ip}")"
    for ct in $check_timestamp ; do
        sed -i "/${ct}/d" "${mark_ip}"
    done
    if ! grep -q "${src_ip}" "${mark_ip}"; then new_ip="true" ; echo "${src_ip}, ${timestamp::-12}" >> "${mark_ip}" ; fi
}

# Ban IP;
mik_ban_ip () {
    if [ "$new_ip" = "true" ]; then
        #echo ":: $src_ip :: $dest_ip:$dest_port/$proto :: $signature_id ::"
        comment_mbi=":: $dest_ip:$dest_port/$proto :: [$signature_id] :: $signature :: $category ::"
        cmd_mbi='/ip firewall address-list add list="'${fw_list}'" address="'${src_ip}'" timeout="'${fw_timeout}d'" comment="'$comment_mbi'"'
        if ! else_error_mbi="$(ssh -n -o ConnectTimeout=3 "${login}"@"${router}" -i "${privatekey}" "${cmd_mbi}" 2>&1)"; then
            logger "[!] [@mik_ban_ip] — [:: $src_ip :: $dest_ip:$dest_port/$proto :: $signature_id ::] — Ошибка - ${else_error_mbi}."
            sed -i "/${src_ip}/d" "${mark_ip}"
        fi
    fi
}

for alert in $alerts; do
    IFS=$'\n'
    IFS="," read -r timestamp src_ip dest_ip dest_port proto signature_id signature category <<< "$alert"
    check_list ; if [ "$wl" = "true" ] ; then continue ; fi
    check_ip
    mik_ban_ip
done

done

https://github.com/isMTv/bash_cata

OlofL · Sun Feb 13, 2022 12:58 pm

I have not tested RoS7 and containers since they pulled before hitting stable?

But this seems a bit overkill now that Suricata can be ran inside containers and hopefully containers will be back in RoS7 soon?

OlofL · Tue Jun 21, 2022 11:59 am

Anyone tested this with 7.4beta4 and containers?

fewdenis · Wed Nov 30, 2022 6:17 pm

Maybe this can help u:
https://github.com/angolo40/mikrocata2selks
I created this script (based on https://github.com/zzbe/mikrocata ) to help install TZSP interface on debian for connection between Mikrotik and Suricata.
It uses a debian distro from Selks (Suricata and Elk stack).
Also will send Telegram notification.

ffries · Fri Dec 30, 2022 5:22 pm

Hello,

Thank you for the hard work and happy new year.

I would like to set up SELKS IDS to monitor a Mikrotik CR2004 router (without active response).
I am planning to run a dedicated server for SELKS with KVM.

Before anything, I need to understand:
1) Should I install a complete Debian system with SELKS on KVM vitualisation or should I use Docker?
2) How to integrate Mikrotik sniffer with SELKS using pcap?
https://help.mikrotik.com/docs/display/ ... et+Sniffer
packets are streamed using Tazmen Sniffer Protocol (TZSP) stream receiver.
So I need https://github.com/thefloweringash/tzsp2pcap with SELKS.
Will tzsp2pcap run inside Docker?

Thank you for your kind answer.

An5teifo · Thu Feb 09, 2023 2:41 pm

Hello there,

yesterday I installed SELKS and connected it with my Mikrotik --> viewtopic.php?t=193417

Who is online