Community discussions

MikroTik App
 
faux
just joined
Topic Author
Posts: 6
Joined: Tue Jul 28, 2020 12:55 pm

Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Wed Jun 09, 2021 10:05 pm

Hello

Today I had a slow day so made a script which fetches all the current days Open Threat Exchange (OTX) Indicators of Compromise (IoC) IPv4 addresses and adds them to a firewall address-list (if they don't already exist). This way you can automatically block the naughty IPv4 addresses where malware and exfiltration has been found by others.. I'd call it a free IPS for IPv4 ::)

The addresses are added as AVOTX_IndicatorAddresses so use that in firewall rules.

REQUIREMENTS:
  1. You need to get yourself an AV OTX API key from here: AlienVault OpenThreatExchange (it's free)
HOW TO USE:
  1. Open my "syncOTXIndicators_v3" script in a text editor, change the "PUT_YOUR_API_KEY_HERE" to your actual API Key
  2. Install my "syncOTXIndicators_v3" script which you edited (/import syncOTXIndicators_v3)
  3. Set up a scheduled task to run the script once per day (at 23:55 UTC for example, IoC's resets at 0:00 UTC)
    NOTE! If you run it early in day (UTC time) there may not be any, or any all day! Uncomment line 12 to set custom start date
  4. Add two raw firewall rules to drop & log all src and dst traffic using the "AVOTX_IndicatorAddresses" address-list
.
.
# Fetch IP indicators on OTX from past day to update local blacklist address-list
#
# PUT API KEY HERE:
:local otxApiKey "X-OTX-API-KEY:PUT_YOUR_API_KEY_HERE";
#
:local today [/system clock get date];
:local montonum {"jan"="01";"feb"="02";"mar"="03";"apr"="04";"may"="05";"jun"="06";"jul"="07";"aug"="08";"sep"="09";"oct"="10";"nov"="11";"dec"="12"};
:local otxDate ([:tostr [:pick $today 7 11]]."-".(:$montonum->[:pick $today 0 3])."-".[:tostr [:pick $today 4 6]]."T00:00:00+00:00");
# Uncomment below to set custom start date
#:set otxDate "2021-05-09T00:00:00+00:00"
#
# Ask OTX for all IP indicators
:local otxUrl ("https://otx.alienvault.com/api/v1/indicators/export?types=IPv4&modified_since=".$otxDate);
:log info "AVOTX: Syncing OTX indicators from date $otxDate";
:local otxResponce [:tostr [/tool fetch url=$otxUrl http-header-field=$otxApiKey mode=https output=user as-value]];
#
# Loop for addresses in OTX responce JSON
:local start ([:find $otxResponce "indicator\": "]+13);
:local end ([:find $otxResponce "\"type"]-3);
:local caughtIp [:pick $otxResponce $start $end];
/ip firewall address-list
:while ($end >0) do={
	:if ([:len [find where address=$caughtIp and list="AVOTX_IndicatorAddresses"]] = 0) do={
		add address=$caughtIp list="AVOTX_IndicatorAddresses";
		:log info "AVOTX: Added $caughtIp to blacklist";
	} else={ :log info "AVOTX: $caughtIp already in blacklist" }
	:set start ([:find $otxResponce "indicator\": " $start]+13);
	:set end ([:find $otxResponce "\"type\":" ($end+3)]-3);
	:set caughtIp [:pick $otxResponce $start $end];
}
:log info "AVOTX: Sync completed";
.
.
WARNING
  • The script is CPU intensive, it will max out a single core to 100% until it finishes, this will harm performance on single core or already high load routers for a few minutes.
  • The OTX lists refresh once per day, so use this late at night (UTC time!!) to get the full days list.
  • Over time the IPs will be ceased and possibly be returned to good-guy addresses, after a while if you start getting false-positives this may be why, investigate each log!
NOTES:
  • If you want to run the script with a custom start date, for example, 1 month behind to catch up; uncomment line 11 with your own UTC datetime. I'd run this script once with this line saying 1 month behind, then comment it out again.. it will take a while to parse.
  • Yes you could use the russian MT JSON parser instead but I didn't want that on production units.
You do not have the required permissions to view the files attached to this post.
Last edited by faux on Tue Apr 05, 2022 10:38 am, edited 6 times in total.
 
MaszTo
just joined
Posts: 4
Joined: Sun Jun 02, 2013 1:25 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Mon Jun 14, 2021 11:53 pm

Hello!
I think I followed the instructions, but the script doesn't add addresses to the Address Lists.
There is no execution in the log:
:log info "AVOTX: Added $caughtIp to blacklist";
:log info "AVOTX: $caughtIp already in blacklist"};
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 1:42 am

		:do {
			/ip firewall address-list add list=AVOTX_IndicatorAddresses address=$caughtIp;
			:log info "AVOTX: Added $caughtIp to blacklist";
		} on-error={ :log info "AVOTX: $caughtIp already in blacklist"};
I really do not like this method of coding.


I better prefer to not fill log and do more clean code like the following example, where errors are prevented than wait for errors:
/ip firewall address-list
:if ([:typeof $caughtIp] ~ "ip") do={
    :if ([:len [find where address=$caughtIp and list="AVOTX_IndicatorAddresses"]] = 0) do={
        add address=$caughtIp list="AVOTX_IndicatorAddresses"
    }
}


if logging are required for debug:
/ip firewall address-list
:if ([:typeof $caughtIp] ~ "ip") do={
    :if ([:len [find where address=$caughtIp and list="AVOTX_IndicatorAddresses"]] = 0) do={
        add address=$caughtIp list="AVOTX_IndicatorAddresses"
        :log info "AVOTX: Added $caughtIp to blacklist"
    } else={ :log notice "AVOTX: $caughtIp already in blacklist" }
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 1:57 am

Hello!
I think I followed the instructions, but the script doesn't add addresses to the Address Lists.
There is no execution in the log:
:log info "AVOTX: Added $caughtIp to blacklist";
:log info "AVOTX: $caughtIp already in blacklist"};
probably because forget to remove \ in front of ? (\?) because on terminal must be used \? but on script must be only ?

for example: export?types= must go on script, but for test on terminal must be export\?types=
 
MaszTo
just joined
Posts: 4
Joined: Sun Jun 02, 2013 1:25 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 10:44 am

Hello!
I think I followed the instructions, but the script doesn't add addresses to the Address Lists.
There is no execution in the log:
:log info "AVOTX: Added $caughtIp to blacklist";
:log info "AVOTX: $caughtIp already in blacklist"};
probably because forget to remove \ in front of ? (\?) because on terminal must be used \? but on script must be only ?

for example: export?types= must go on script, but for test on terminal must be export\?types=
I know the character ? in the terminal window calls for help.
In editing script window (WinBox) I have ? in path https://otx.alienvault.com/api/v1/indic ... ".$otxDate
 
MaszTo
just joined
Posts: 4
Joined: Sun Jun 02, 2013 1:25 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 11:09 am

You must add the \ before the ? to the path in the editor.
The script works - it's great because it is not limited to downloading entire lists, which cannot exceed 64k!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 12:45 pm

You must add the \ before the ? to the path in the editor.
The script works - it's great because it is not limited to downloading entire lists, which cannot exceed 64k!
please, you can share one .json file downloaded for see internal structure?
 
xsqrt
just joined
Posts: 1
Joined: Tue Jun 15, 2021 8:30 am

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 1:28 pm

can anyone make some video to you use this script ? pls help
 
faux
just joined
Topic Author
Posts: 6
Joined: Tue Jul 28, 2020 12:55 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 1:52 pm

~~
I really do not like this method of coding.

I better prefer to not fill log and do more clean code like the following example, where errors are prevented than wait for errors:
~~
Hi, thank you very much for you suggestion! You have a cleaner method I agree. I will update my script and upload a newer version when I get home today.

please, you can share one .json file downloaded for see internal structure?

Here is an example of the JSON responce from the API. The IPs are fake examples.

{
    "results": [
        {
            "id": 3078914552,
            "indicator": "1.2.3.4",
            "type": "IPv4",
            "title": "RedLine",
            "description": null,
            "content": ""
        },
        {
            "id": 3081790165,
            "indicator": "4.3.2.1",
            "type": "IPv4",
            "title": null,
            "description": null,
            "content": ""
        }
    ],
    "count": 2,
    "previous": null,
    "next": null
}
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 2:14 pm

thanks!

[:typeof $caughtIp] ~ "ip"
if you do not already have noticed the ~ instead of = is because the accepted value on address list can be ip (1.2.3.4) or ip-prefix (1.2.3.0/24)
no other RouterOS variable type contain "ip" (except v6)

ad search only "ip" is right, alternate, but right, ~ "^(ip|ip-prefix)$" is excessive, if for some reason IPv6 go on list, the regexp for parse IPv4 block it without errors.

I do not know if the returned .json contain or not prefix like 1.2.3.0/24

but if are contained inside, the script must be careful to not write only 1.2.3.0 cutting off /24
 
faux
just joined
Topic Author
Posts: 6
Joined: Tue Jul 28, 2020 12:55 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Tue Jun 15, 2021 7:25 pm

thanks!
[:typeof $caughtIp] ~ "ip"
~~~~

but if are contained inside, the script must be careful to not write only 1.2.3.0 cutting off /24

Hmmmm I will email OTX help and ask if there is ever a 1.2.3.4/24 style address and subnet, becuase if there never is we can cut out a few lines of code! The regex should then only find 1.2.3.4 style addresses and code like this will work nicely?

Edit: They will only ever be normal single IP addresses, never a CIDR range.

:if ($startPos < $endPos) do={
	/ip firewall address-list
	:while ([:len [:pick $arrResults 0]] >0) do={
		:local caughtIp [:toip [:pick $arrResults 0]];
		:if ([:len [find where address=$caughtIp and list="AVOTX_IndicatorAddresses"]] =0) do={
			add list=AVOTX_IndicatorAddresses address=$caughtIp;
			:log info "AVOTX: Added $caughtIp to blacklist";
		} else={ :log info "AVOTX: $caughtIp already in blacklist"};
		:set startPos [:pick $arrResults 2];
		:set arrResults [$findPosix $otxResponce $regex $startPos];
	}
}
I appriciate your help and knowledge ::)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Wed Jun 16, 2021 1:50 am

>useless<
Last edited by rextended on Wed Jun 16, 2021 11:06 am, edited 1 time in total.
 
MaszTo
just joined
Posts: 4
Joined: Sun Jun 02, 2013 1:25 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Wed Jun 16, 2021 10:20 am

The last modification made by Faux works, but the last Rextended suggestion causes a loop if the address already exists (in my excample: AVOTX: xxx.xxx.xxx.xxx alredy in blacklist).
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Wed Jun 16, 2021 11:04 am

The last modification made by Faux works, but the last Rextended suggestion causes a loop if the address already exists (in my excample: AVOTX: xxx.xxx.xxx.xxx alredy in blacklist).
yes, I see the problem
 
faux
just joined
Topic Author
Posts: 6
Joined: Tue Jul 28, 2020 12:55 pm

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Fri Jun 18, 2021 11:24 am

So i made a version which doesn't need Regex, and therefore doesn't need the extra script. Also.. its ~1.5x faster at parsing the results!

Needed it to make a DNS versions for urlhaus/anomali/etc.
# Fetch IP indicators on OTX from past day to update local blacklist address-list
# v3
# PUT API KEY HERE:
:local otxApiKey "X-OTX-API-KEY:PUT_YOUR_API_KEY_HERE";
#
:local today [/system clock get date];
:local montonum {"jan"="01";"feb"="02";"mar"="03";"apr"="04";"may"="05";"jun"="06";"jul"="07";"aug"="08";"sep"="09";"oct"="10";"nov"="11";"dec"="12"};
:local otxDate ([:tostr [:pick $today 7 11]]."-".(:$montonum->[:pick $today 0 3])."-".[:tostr [:pick $today 4 6]]."T00:00:00+00:00");
# Uncomment below to set custom start date
#:set otxDate "2021-05-09T00:00:00+00:00"
#
# Ask OTX for all IP indicators
:local otxUrl ("https://otx.alienvault.com/api/v1/indicators/export?types=IPv4&modified_since=".$otxDate);
:log info "AVOTX: Syncing OTX indicators from date $otxDate";
:local otxResponce [:tostr [/tool fetch url=$otxUrl http-header-field=$otxApiKey mode=https output=user as-value]];
#
# Loop for addresses in OTX responce JSON
:local start ([:find $otxResponce "indicator\": "]+13);
:local end ([:find $otxResponce "\"type"]-3);
:local caughtIp [:pick $otxResponce $start $end];
/ip firewall address-list
:while ($end >0) do={
	:if ([:len [find where address=$caughtIp and list="AVOTX_IndicatorAddresses"]] = 0) do={
		add address=$caughtIp list="AVOTX_IndicatorAddresses";
		:log info "AVOTX: Added $caughtIp to blacklist";
	} else={ :log info "AVOTX: $caughtIp already in blacklist" }
	:set start ([:find $otxResponce "indicator\": " $start]+13);
	:set end ([:find $otxResponce "\"type\":" ($end+3)]-3);
	:set caughtIp [:pick $otxResponce $start $end];
}
:log info "AVOTX: Sync completed";
It parsed a weeks IP list in 67 seconds, down from 99 seconds before. (RB4011iGS+RM)
Will update original post if I don't find any issues with it!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11982
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling

Fri Jun 18, 2021 1:15 pm

if "type" = "IPv4" ...
else { skip and continue }

Who is online

Users browsing this forum: No registered users and 30 guests