Community discussions

 
bbs2web
Member Candidate
Member Candidate
Topic Author
Posts: 201
Joined: Sun Apr 22, 2012 6:25 pm
Location: Johannesburg, South Africa
Contact:

Automating address list maintenance - MANRS compliance

Mon Jan 08, 2018 1:28 pm

Complying with MANRS (https://www.manrs.org/manrs) requires one to filter traffic. Maintaining addresses lists on multiple routers within one's network is however a time consuming task so I wrote the following script which:
  • Creates address list entries from manual entries on a per interface basis
  • Creates address list entries for multiple interfaces when routes match a certain BGP community (ours in this case are 37314:500)
  • Creates address list entries for multiple interfaces when routes match a certain BGP community (customers in this case are 37314:3000)
  • Creates address list entries for certain interfaces when routes match an array of custom per interface BGP communities
  • Removes stale entries which are older than 3.5 days, all entries have their comment updated with the Unix Epoch time stamp each time the script is run.

/system scheduler
  add interval=1d name="Update Interface filter address lists" on-event=\
    "/system script run \"update_interface_address_lists\"" policy=read,write start-date=\
    jan/01/1970 start-time=01:30:00
/system script
  add name=update_interface_address_lists owner=admin policy=read,write source="# {\"<address list>\";\
    \_<1 = prefixes matching 37314:500>; <1 = prefixes matching 37314:3000>; {<manual prefixes>};\
    \_{<custom bgp communities>}};\r\
    \n# Examples:\r\
    \n#            {\"filter-ether1\"; 1; 1; {10.1.0.0/16; 10.2.0.0/16}; {}};\r\
    \n#            {\"filter-ether2\"; 0; 0; {}; {\"37314:2000\"}};\r\
    \n:local ifs { \r\
    \n  {\"filter-external\"; 1; 1; {}; {}};\r\
    \n  {\"filter-external-src\"; 1; 0; {}; {}};\r\
    \n  {\"filter-ixp\"; 1; 1; {10.20.200.0/22}; {}};\r\
    \n  {\"filter-upstream1-vlan43\"; 1; 1; {}; {}};\r\
    \n  {\"filter-upstream2-vlan418\"; 1; 1; {}; {}};\r\
    \n};\r\
    \n\r\
    \n:global fncJD do={\r\
    \n  :local months [:toarray \"jan,feb,mar,apr,may,jun,jul,aug,sep,oct,nov,dec\"];\r\
    \n  :local jd;\r\
    \n  :local M [:pick \$1 0 3];\r\
    \n  :local D [:pick \$1 4 6];\r\
    \n  :local Y [:pick \$1 7 11];\r\
    \n  :for x from=0 to=([:len \$months] - 1) do={\r\
    \n    :if ([:tostr [:pick \$months \$x]] = \$M) do={ :set M (\$x + 1) };\r\
    \n  }\r\
    \n  :if (\$M = 1 || \$M = 2) do={\r\
    \n    :set Y (\$Y-1);\r\
    \n    :set M (\$M+12);\r\
    \n  }\r\
    \n  :local A (\$Y/100);\r\
    \n  :local B (\$A/4);\r\
    \n  :local C (2-\$A+\$B);\r\
    \n  :local E (((\$Y+4716) * 36525)/100);\r\
    \n  :local F ((306001*(\$M+1))/10000);\r\
    \n  :local jd (\$C+\$D+\$E+\$F-1525);\r\
    \n  :return \$jd;\r\
    \n};\r\
    \n\r\
    \n:global timestamp do={\r\
    \n  :global fncJD \$fncJD;\r\
    \n  :local currtime [/system clock get time];\r\
    \n  :local jdnow [\$fncJD [/system clock get date]];\r\
    \n  :local days (\$jdnow - 2440587);\r\
    \n  :local ore [:pick \$currtime 0 2];\r\
    \n  :local minute [:pick \$currtime 3 5];\r\
    \n  :local secunde [:pick \$currtime 6 8];\r\
    \n  :return ((\$days * 86400) + (\$ore * 3600) + (\$minute * 60) + \$secunde - [/system clock get gmt-offset]);\r\
    \n}\r\
    \n:local timenow [\$timestamp];\r\
    \n\r\
    \n:local arrayXpush do={\r\
    \n  :local arrX value=[:toarray \$1];\r\
    \n  :if ([:len \$arrX] = 0) do={ :set \$arrX value=[:toarray \"\"]; };\r\
    \n  :local arrXlen value=[:len \$arrX];\r\
    \n  :local valX value=[:tostr \$2];\r\
    \n  :if (\$valX = \"\") do={ :return value=\$arrX; };\r\
    \n  :local posX value=([:tostr \$3]);\r\
    \n  :if (\$posX = \"\") do={ :set \$posX value=(\$arrXlen + 1); };\r\
    \n  :set \$posX value=([:tonum \$posX] + 0);\r\
    \n  :if (\$posX < 0) do={:set \$posX value=0};\r\
    \n  :if (\$posX > \$arrXlen) do={:set \$posX value=\$arrXlen};\r\
    \n  :if (\$posX = 0) do={:return value=(\$valX,\$arrX)};\r\
    \n  :if (\$posX = \$arrXlen) do={:return value=(\$arrX,\$valX)};\r\
    \n  :return value=([:pick \$arrX 0 (\$posX - 1)],\$valX,[:pick \$arrX (\$posX - 1) \$arrXlen]);\r\
    \n};\r\
    \n\r\
    \n:local addresslist do={\r\
    \n  :local name (\$1);\r\
    \n  :local destination (\$2);\r\
    \n  :local timenow (\$3);\r\
    \n  :if ([:typeof [:find \$destination \"/32\"]] = \"num\") do={:set destination [:pick \$destination 0 [:find \$destination \"/\"]]};\r\
    \n  /ip firewall address-list set [ find list=\"\$name\" and address=\"\$destination\" ] comment=\$timenow;\r\
    \n  :do {/ip firewall address-list add list=\$name address=\$destination comment=\$timenow} on-error={};\r\
    \n}\r\
    \n\r\
    \n:foreach if in=\$ifs do={\r\
    \n  :local name (\$if->0);\r\
    \n  :local prefixes (\$if->3);\t# load manual prefixes\r\
    \n  :local destination;\r\
    \n  :foreach community in=(\$if->4) do={\r\
    \n    :foreach i in=[ /ip route find where active and bgp-communities ~ \$community ] do={\r\
    \n      :set destination [ /ip route get \$i dst-address ];\r\
    \n      :set prefixes [\$arrayXpush \$prefixes \$destination];\r\
    \n    }\r\
    \n  }\r\
    \n  :foreach destination in=\$prefixes do={\$addresslist \$name \$destination \$timenow};\r\
    \n}\r\
    \n\r\
    \n:foreach k,community in={1=\"37314:500\"; 2=\"37314:3000\"} do={\r\
    \n  :set k [:tonum \$k];\r\
    \n  :local prefixes ({});\r\
    \n  :local destination;\r\
    \n  :foreach i in=[ /ip route find where active and bgp-communities ~ \$community ] do={\r\
    \n    :set destination [ /ip route get \$i dst-address ];\r\
    \n    :set prefixes [\$arrayXpush \$prefixes \$destination];\r\
    \n  }\r\
    \n  :foreach if in=\$ifs do={\r\
    \n    :local name (\$if->0);\r\
    \n    if (\$if->\$k = 1) do={\r\
    \n      :foreach destination in=\$prefixes do={\$addresslist \$name \$destination \$timenow};\r\
    \n    }\r\
    \n  }\r\
    \n}\r\
    \n\r\
    \n:local expired ([\$timenow] - 302400);\t# expire after 3.5 days\r\
    \n:foreach if in=\$ifs do={\r\
    \n  :local name (\$if->0);\r\
    \n  :local i;\r\
    \n  :foreach i in=[ /ip firewall address-list find where list=\$name ] do={\r\
    \n    :local comment [ /ip firewall address-list get \$i comment ];\r\
    \n    :local added [:tonum \$comment];\r\
    \n    :if (\$added = \$comment) do={\r\
    \n      :if (\$added < \$expired) do={\r\
    \n        /ip firewall address-list remove \$i;\r\
    \n      }\r\
    \n    }\r\
    \n  }\r\
    \n}\r\
    \n"

Sample firewall rules to:
  • Drop traffic entering interfaces that are members of the 'external' interface list, when not to our or downstream customer prefixes (filter-external)
  • Drop traffic entering interfaces that are members of the 'external' interface list, where from our prefixes. (filter-external-src)
  • Drop traffic leaving external interfaces where the source IPs are not part of those interface lists. (eg filter-upstream1-vlan43)

/interface list
  add name=external
/interface list member
  add interface=ixp list=external
  add interface=upstream1-vlan43 list=external
  add interface=upstream2-vlan418 list=external

/ip firewall raw
  add action=accept chain=prerouting comment="Track - packets destined for this router:" dst-address-type=local
  add action=drop chain=prerouting comment="Drop - In 'external' interface list - not to \"filter-external' address list:" dst-address-list=!filter-external in-interface-list=external
  add action=drop chain=prerouting comment="Drop - In 'external' interface list - from \"filter-external-src' address list:" in-interface-list=external src-address-list=filter-external-src
/ip firewall filter
  add action=drop chain=forward out-interface=ixp src-address-list=!filter-ixp
  add action=drop chain=forward out-interface=upstream1-vlan43 src-address-list=!filter-upstream1-vlan43
  add action=drop chain=forward out-interface=upstream2-vlan418 src-address-list=!filter-upstream2-vlan418
Last edited by bbs2web on Tue Jan 09, 2018 8:23 am, edited 3 times in total.
 
bbs2web
Member Candidate
Member Candidate
Topic Author
Posts: 201
Joined: Sun Apr 22, 2012 6:25 pm
Location: Johannesburg, South Africa
Contact:

Re: Automating address list maintenance - MANRS compliance

Mon Jan 08, 2018 5:50 pm

Herewith the code without 'export' escaping:
# {"<address list>"; <1 = prefixes matching 37314:500>; <1 = prefixes matching 37314:3000>; {<manual prefixes>}; {<custom bgp communities>}};
# Examples:
#            {"filter-ether1"; 1; 1; {10.1.0.0/16; 10.2.0.0/16}; {}};
#            {"filter-ether2"; 0; 0; {}; {"37314:2000"}};
:local ifs { 
  {"filter-external"; 1; 1; {}; {}};
  {"filter-external-src"; 1; 0; {}; {}};
  {"filter-ixp"; 1; 1; {10.20.200.0/22}; {}};
  {"filter-upstream1-vlan43"; 1; 1; {}; {}};
  {"filter-upstream2-vlan418"; 1; 1; {}; {}};
};

:global fncJD do={
  :local months [:toarray "jan,feb,mar,apr,may,jun,jul,aug,sep,oct,nov,dec"];
  :local jd;
  :local M [:pick $1 0 3];
  :local D [:pick $1 4 6];
  :local Y [:pick $1 7 11];
  :for x from=0 to=([:len $months] - 1) do={
    :if ([:tostr [:pick $months $x]] = $M) do={ :set M ($x + 1) };
  }
  :if ($M = 1 || $M = 2) do={
    :set Y ($Y-1);
    :set M ($M+12);
  }
  :local A ($Y/100);
  :local B ($A/4);
  :local C (2-$A+$B);
  :local E ((($Y+4716) * 36525)/100);
  :local F ((306001*($M+1))/10000);
  :local jd ($C+$D+$E+$F-1525);
  :return $jd;
};

:global timestamp do={
  :global fncJD $fncJD;
  :local currtime [/system clock get time];
  :local jdnow [$fncJD [/system clock get date]];
  :local days ($jdnow - 2440587);
  :local ore [:pick $currtime 0 2];
  :local minute [:pick $currtime 3 5];
  :local secunde [:pick $currtime 6 8];
  :return (($days * 86400) + ($ore * 3600) + ($minute * 60) + $secunde - [/system clock get gmt-offset]);
}
:local timenow [$timestamp];

:local arrayXpush do={
  :local arrX value=[:toarray $1];
  :if ([:len $arrX] = 0) do={ :set $arrX value=[:toarray ""]; };
  :local arrXlen value=[:len $arrX];
  :local valX value=[:tostr $2];
  :if ($valX = "") do={ :return value=$arrX; };
  :local posX value=([:tostr $3]);
  :if ($posX = "") do={ :set $posX value=($arrXlen + 1); };
  :set $posX value=([:tonum $posX] + 0);
  :if ($posX < 0) do={:set $posX value=0};
  :if ($posX > $arrXlen) do={:set $posX value=$arrXlen};
  :if ($posX = 0) do={:return value=($valX,$arrX)};
  :if ($posX = $arrXlen) do={:return value=($arrX,$valX)};
  :return value=([:pick $arrX 0 ($posX - 1)],$valX,[:pick $arrX ($posX - 1) $arrXlen]);
};

:local addresslist do={
  :local name ($1);
  :local destination ($2);
  :local timenow ($3);
  :if ([:typeof [:find $destination "/32"]] = "num") do={:set destination [:pick $destination 0 [:find $destination "/"]]};
  /ip firewall address-list set [ find list="$name" and address="$destination" ] comment=$timenow;
  :do {/ip firewall address-list add list=$name address=$destination comment=$timenow} on-error={};
}

:foreach if in=$ifs do={
  :local name ($if->0);
  :local prefixes ($if->3);	# load manual prefixes
  :local destination;
  :foreach community in=($if->4) do={
    :foreach i in=[ /ip route find where active and bgp-communities ~ $community ] do={
      :set destination [ /ip route get $i dst-address ];
      :set prefixes [$arrayXpush $prefixes $destination];
    }
  }
  :foreach destination in=$prefixes do={$addresslist $name $destination $timenow};
}

:foreach k,community in={1="37314:500"; 2="37314:3000"} do={
  :set k [:tonum $k];
  :local prefixes ({});
  :local destination;
  :foreach i in=[ /ip route find where active and bgp-communities ~ $community ] do={
    :set destination [ /ip route get $i dst-address ];
    :set prefixes [$arrayXpush $prefixes $destination];
  }
  :foreach if in=$ifs do={
    :local name ($if->0);
    if ($if->$k = 1) do={
      :foreach destination in=$prefixes do={$addresslist $name $destination $timenow};
    }
  }
}

:local expired ([$timenow] - 302400);	# expire after 3.5 days
:foreach if in=$ifs do={
  :local name ($if->0);
  :local i;
  :foreach i in=[ /ip firewall address-list find where list=$name ] do={
    :local comment [ /ip firewall address-list get $i comment ];
    :local added [:tonum $comment];
    :if ($added = $comment) do={
      :if ($added < $expired) do={
        /ip firewall address-list remove $i;
      }
    }
  }
}
Last edited by bbs2web on Tue Jan 09, 2018 8:23 am, edited 2 times in total.
 
bbs2web
Member Candidate
Member Candidate
Topic Author
Posts: 201
Joined: Sun Apr 22, 2012 6:25 pm
Location: Johannesburg, South Africa
Contact:

Re: Automating address list maintenance - MANRS compliance

Mon Jan 08, 2018 6:42 pm

Credits to 'adeeadee' for the Unix Epoch time functions:
viewtopic.php?t=75555

Credits to 'rextended' for the array push function:
viewtopic.php?t=85992#p434264

And many others for various tips and tricks when working with arrays in RouterOS's scripting language.
Last edited by bbs2web on Mon Jan 08, 2018 9:55 pm, edited 1 time in total.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: Automating address list maintenance - MANRS compliance

Mon Jan 08, 2018 8:08 pm

This is great work! thanks for posting :-)
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
msatter
Forum Guru
Forum Guru
Posts: 1298
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Automating address list maintenance - MANRS compliance

Sun Apr 22, 2018 2:17 pm

Very interesting and also thanks for posting.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)

Who is online

Users browsing this forum: No registered users and 11 guests