Community discussions

 
bpita
just joined
Topic Author
Posts: 10
Joined: Fri Aug 27, 2004 4:56 pm

How to convert a MT box in an Anti Spam server with v2.9

Sat May 12, 2007 6:01 am

Hi there, for months I have the idea how to transform one Mikrotik OS in an Anti Spam server. After more than 20 days applying few scripts, firewall rules and address list, I have achieved reduce from 45.000 mails per day to only 11.000/12.000 without many complaints from my customers.

Before continuing, some details about this:
  • Yes, I know that exists others solutions.
    Yes, I know with Linux can obtain the same results.
    Yes, I know that it seems a crazy solution.
    Yes, I know (in the practice) that this solutions generate a moderate cpu usage. I have a Pentium IV with 75% of cpu usage (this can change with new features from MT... see scripts explanation) and we are a little ISP.
    Yes, yes, yes...
    but
    I use MT from six years ago (when John Tully & Arnis Riekstins answered the company mails themselves) and always I try to resolve any networking necessity whit MT.
    I had the trust that can resolve this whit MT.
    and after thinking about this, I can't never sleep all night from many days, so, for my health and wife I made it !!!
The scripts:

I use 3 scripts:

itwbspam (run every second)
spamtocheck (run every 10 seconds)
tempclear (run every 30 minutes)

Script itwbspam (remove the # lines to work, I comment the most important parts):
:set SCRIPT itwbspam;

# Always I think that MT permit one script running at the same time, for my surprise is not, so I
# use this to run at the same time one script (simulating a threads).
# You can reduce this, if your cpu usage is high or your mail traffic is reduced. 
# In my case, less than 10 permit detect if the incoming SMTP connection is an spam or not
# quickly, and the cpu usage is reasonable.
# If you need run only one script at the same time, set at < 1 because the script it's count the 
# same one when running.

:if ([:len [/system script job find script=$SCRIPT]] < 10) do={ \

# I use a firewall rule to add ips to this list (see below in firewall rules).

   :set ALIST checkspam;
   :foreach IPT in [/ip firewall address-list find list=$ALIST] do={ \
      :set COMM [/ip firewall address-list get $IPT comment];

# For not process the same ip in more than one script, I put a comment to prevent this.
# Maybe in a future we can put a comment in a DYNAMIC list (I don't know why we can do that)
# so that can reduce the cpu usage.

      :if ($COMM = "") do={ \
          /ip firewall address-list set $IPT comment=checking; 


# Now you need to revert the ip for querying a dnsbl server and know if the server that try to
# connect to your SMTP server is a spam or not. 

         :set IP [/ip firewall address-list get $IPT address]; :set IPL $IP;
   
         :set l [:len $IP]; :set n [:find $IP .];
         :set A [:pick $IP 0 $n];
         :set IP [:pick $IP ($n + 1) 15]; :set n [:find $IP .];
         :set B [:pick $IP 0 $n];
         :set l [:len $IP]; :set IP [:pick $IP ($n + 1) 15]; :set n [:find $IP .];
         :set C [:pick $IP 0 $n];
         :set l [:len $IP]; :set IP [:pick $IP ($n + 1) 15];
         :set D $IP;
   
         :set COUNT 0;

# Now ask to black lists.
   
         :set DSBL .dnsbl.sorbs.net;
         :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
         :if ($IP != "0.0.0.0") do={ :set COUNT ($COUNT + 1); }
   
         :set DSBL .blackholes.five-ten-sg.com;
         :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
         :if ($IP != "0.0.0.0") do={ :set COUNT ($COUNT + 1); }
   
         :set DSBL .dnsbl.tqmcube.com;
         :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
         :if ($IP != "0.0.0.0") do={ :set COUNT ($COUNT + 1); }
   
         :set DSBL .zen.spamhaus.org; :set SPAM 127.0.0.4;
         :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
         :if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }
   
         :set DSBL .bl.spamcop.net; :set SPAM 127.0.0.2;
         :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
         :if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }

# You can add so many black lists as you want, only you need set the DSBL and SPAM variable.
# If the ip returned from the dnsbl server coincides whit SPAM, the COUNT variable add
# by one.
# From servers like sorbs.net or five-ten-sg that return more than one value, I don't use the SPAM
# variable and check only the value if different to 0.0.0.0 and I assuming is spam when is
# different.

         :if ($COUNT < 3) do= { \
   
# If the ip server COUNT lees than 3, I assuming maybe need more checks, so you can add
# others black lists to be safe that this ip it's not a spam server !.


            :set DSBL .list.dsbl.org; :set SPAM 127.0.0.2;
            :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
            :if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }
   
            :set DSBL .dnsbl-1.uceprotect.net; :set SPAM 127.0.0.2;
            :set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];
            :if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }
   
         }

# Now according to the value in COUNT you can decide which is the address list that  it
# corresponds to each case. In my case this is the best solution: 
# if the server COUNT 1, is can or not a spam server, so it's fine to me and pass,
# if have 2, I think they are amateurs and they need to sample with other ISPs,
# if have 3 or more... hey, leave me alone !!!
   
         :if ($COUNT > 1) do= { \
            :if ($COUNT = 2) do={ /ip firewall address-list add list=tempspamlight address=$IPL; }
            :if ($COUNT = 3) do={ /ip firewall address-list add list=tempspammedium address=$IPL; }
            :if ($COUNT > 3) do={ /ip firewall address-list add list=tempspamhard address=$IPL; }

# Yes, you need to use STATIC list because we can't add a DYNAMIC list from script or 
# timeout in a STATIC list (I don't know why, if we can in the future, this reduce considerably
# the cpu usage) and maybe, this server it's hacking so you can't assume that this server 
# is a spam server only for a while, so you need to add a timeout value (see below in 
# firewall rules).

         } else= { \
            /ip firewall address-list add list=tempsmtp address=$IPL;
         }

# Once the server is cataloged you need delete the ip from the temp list

         /ip firewall address-list remove $IPT;
   
      }
   }
}
You need this script running all time to check each ip in the list if are spam server or not. After that you have 3 different categories (in this example, you can add more than 3 if you like) so only you need transform this STATIC list in DYNAMIC list whit a timeout value accord to the type of spam that the script it detecting (see below in firewall rules). In my case I have more than 20.000 ip address counting white lists and black lists.

Script spamtocheck: this script purpose it's only transform a DYNAMIC list to STATIC list because in DYNAMIC list you can't put a comment (why !!!???) or counter, or anything that can I use to prevent the script itwbspam take an ip that is checking in another script itwbspam at the same time.
:set ALIST itwbspam; :set BLIST checkspam;
:local IPLIST [/ip firewall address-list find dynamic=yes list=$ALIST];
:foreach IP in $IPLIST do={ \
   :set IPL [/ip firewall address-list get $IP address];
   /ip firewall address-list add list=$BLIST address=$IPL;
   /ip firewall address-list remove $IP;
}   
Script tempclear: thescript purpose it's only clear the address list of servers that never try to contact again our SMTP server. If in the future we can add a timeout to STATIC list, this script is not necessary any more and YES maybe reduce the cpu usage.
:set ALIST tempsmtp;
:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }

:set ALIST tempspamlight;
:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }
chain=smtp protocol=tcp src-port=25 src-address-list=staticwhitelist action=accept 

:set ALIST tempspammedium;
:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }

:set ALIST tempspamhard;
:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }
Now you need to add some firewall rules to make this work.

Firewall rules:

I have an separate server before my SMTP server, so I put this rules in filter rules/XXX chain, maybe you need change this according to your configuration. Obviously I have more than this rules, I mention here only it related whit this scripts.

First you need to be safe that your customers can arrive to the server. In my case I have a relay server configured in different ip for the SMTP in the same server, but if you use the same ip server to receive and send emails you need this rule.
IMPORTANT, if you have customers out side your network whit this scripts they can't send mails from the SMTP server any more unless if your SMTP server are connecting to internet whit the same box where you run this scripts. In this case you need to add a few mangle rules to know which connection it's established for your SMTP server and mark the packet to pass over the spam verification.
;;; Accept network customers !!!
chain=smtp protocol=tcp dst-port=25 src-address-list=customers action=accept 
Now you need to minimize false positive, for that reason I have a bash script (see below in bash script) that collecting the information from two white list and put in STATIC list (whitelist). I cron this bash to run each 3 hours and accept any mail connection that come from this servers. Whit this you can be peaceful about hotmail, yahoo, gmail and many others ISPs, Companies, etc.
On the other hand, I have a static white list address from ISPs or Companies that I know don't make spam or are big ISP and they SMTP servers appears in more than one black list, so I need this servers pass over the spam verification (staticwhitelist). For example, here in Argentina, one of the biggest ISP is Speedy and their SMTP servers appears in more than one black list.
;;; Accept white list !!!
chain=smtp protocol=tcp dst-port=25 src-address-list=whitelist action=accept 
chain=smtp protocol=tcp dst-port=25 src-address-list=staticwhitelist action=accept 
chain=smtp protocol=tcp src-port=25 src-address-list=whitelist action=accept 
chain=smtp protocol=tcp src-port=25 src-address-list=staticwhitelist action=accept 
Whit this rule, you transform all ips that pass the spam verification whit the script itwbspam into DYNAMIC ip, with timeout according to your experience. In my case 12 hours is fine to verify again if are spammer or not.
;;; Pass the temp to smtpok
chain=smtp src-address-list=tempsmtp action=add-src-to-address-list address-list=smtpok address-list-timeout=12h
Now you are sure that the ip server in the smtpok is not a spam server or is a light spammer (COUNT = 1 in my case), so you accept the connection.
;;; Accept tempsmtp !!!
chain=smtp protocol=tcp dst-port=25 src-address-list=smtpok action=accept 
chain=smtp protocol=tcp src-port=25 src-address-list=smtpok action=accept 

;;; NOT accept static spam !!
chain=smtp protocol=tcp dst-port=25 src-address-list=staticspam action=reject reject-with=icmp-network-unreachable 
Now you need to pass from STATIC list to DYNAMIC list all that servers ips that they are spam and put the timeout according to each categories.
;;; Pass the temp to spam
chain=smtp src-address-list=tempspamlight action=add-src-to-address-list address-list=spamlight address-list-timeout=2h 
chain=smtp src-address-list=tempspammedium action=add-src-to-address-list address-list=spammedium address-list-timeout=6h 
chain=smtp src-address-list=tempspamhard action=add-src-to-address-list address-list=spamhard address-list-timeout=2d 
The idea here is reject the packets from ips in the temp address lists that at this time is not checked from our script. Off course the first mail is not delivery to the SMTP server, but if the server that try to connect is not spammer, the most probable thing it's thats try to connect again later (few minutes last in my experience) in contrast with spammers servers that it never come back again. After that, reject all the spam lists.
;;; Pass the temp to spam
chain=smtp src-address-list=tempspamlight action=add-src-to-address-list address-list=spamlight address-list-timeout=2h 
chain=smtp src-address-list=tempspammedium action=add-src-to-address-list address-list=spammedium address-list-timeout=6h 
chain=smtp src-address-list=tempspamhard action=add-src-to-address-list address-list=spamhard address-list-timeout=2d 

;;; Not accept temps
chain=smtp protocol=tcp dst-port=25 src-address-list=itwbspam action=reject reject-with=icmp-host-unreachable 
chain=smtp protocol=tcp dst-port=25 src-address-list=checkspam action=reject reject-with=icmp-host-unreachable 
chain=smtp protocol=tcp dst-port=25 src-address-list=tempsmtp action=reject reject-with=icmp-host-unreachable 

;;; NOT accept spamhard !!
chain=smtp protocol=tcp dst-port=25 src-address-list=spamhard action=reject reject-with=icmp-network-unreachable 

;;; NOT accept spammedium !!
chain=smtp protocol=tcp dst-port=25 src-address-list=spammedium action=reject reject-with=icmp-network-unreachable 

;;; NOT accept spamlight !!
chain=smtp protocol=tcp dst-port=25 src-address-list=spamlight action=reject reject-with=icmp-network-unreachable 
The last thing that you need to do is add a rule to fill the address list spamcheck whit new connections.
;;; Add to check
chain=smtp protocol=tcp dst-port=25 tcp-flags=syn connection-state=new src-address-list=!checkspam action=add-src-to-address-list address-list=itwbspam  address-list-timeout=20m 
chain=smtp protocol=tcp dst-port=25 tcp-flags=syn connection-state=new src-address-list=!checkspam action=reject reject-with=icmp-host-unreachable 
chain=smtp protocol=tcp src-port=25 connection-state=established action=log log-prefix=""
That's all !!!

Basically the process can resume like this:
  • One server try to connect to our SMTP server
    If the ip is in the white list address the packet is delivery to our server
    If the ip server is in the address list smtpok the packet is delivery to our server
    If the packet arrives to the rule that check is a new connection, the ip is added to the address list itwbspam
    The script spamtocheck run every 10 seconds and move the ip address in the DYNAMIC list itwbspam to a STATIC list checkspam because we can't add a comment in DYNAMIC lists !!
    The script itwbspam run every 1 second and check every ip address in the STATIC list checkspam to determine this ip is a spam server or not, if is spam then go to a tempspam list categorie, is not go to tempsmtp
Bash script: inspired by one on this forum.
#!/bin/bash
#
today=`date "+%m%d%y"`;
workdir="/home/mt";
targets=(you ip or ips server);

cd $workdir

echo > wlists.rsc

# whitelist
echo :foreach subnet in [/ip firewall address-list find list=whitelist] do=\{ /ip firewall address-list remove \$subnet \} >> wlists.rsc 
  
wget -t 2 -O dnswl.txt http://www.dnswl.org/data/generic-dnswl 
egrep "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" dnswl.txt | awk '{print $1}' | awk -F";" '{print $1}' | grep -v ";" | sort +1 | awk '{print "/ip firewall address-list add l
ist=whitelist address=" $1}' >> wlists.rsc 
 
wget -t 2 -O rediris.txt http://www.rediris.es/abuses/eswl/data/mtawl.postfix
egrep "[0-9]{1,3}" rediris.txt | awk '$1 > 0 " " $1 {print "/ip firewall address-list add list=whitelist address="$1"/32"}' >> wlists.rsc
 
echo >> wlists.rsc 
 
# Remove duplicate lines
sort wlists.rsc | uniq > wlistsu.rsc 
 
# Reduce to small files to be more manageable for MT
split wlistsu.rsc -a 5 -d -l 500 spwlists
 
# foreach target($targets) 
for X in ${targets[@]}; do 
  echo $X; 
    for I in `find -name "spwlists?????"`; do
    mv $I "$I.rsc";
    echo "put $I.rsc" | ftp $X 21
  done
  for I in `find /home/mt/ -name "spwlists?????.rsc" -printf "%f\n" | sort`; do
    echo "procesando $I";
    ssh -l black -i /home/mt/black.dsa $X "/import $I" >> /home/mt/wlists.output 
  done 
  for I in `find /home/mt/ -name "spwlists?????.rsc" -printf "%f\n" | sort`; do
    echo "delete $I" | ftp $X 21 
    rm -f $I;
  done 
done
TODO
  • When we have more attributes to manipulate from script a STATIC address list it will be reduce the cpu usage a lot.
    If MT adds a counter to the address list, then we can controlling how many times the ip in the smtpok address list is hitting, so, we can assuming that if there are more than 100.000 hits in a particular ip then this server is not spammer and we can move to a white list whit a timeout of 1 month.
    When the dnsbl server returns an ip like 10.0.0.7 or 10.0.0.10 is definitively a BIG SPAMMER. To know this it need to fix the itwbspam script to accept more than one ip from a dnsbl server.
P.S.: if you add modifications in my scripts, please share it whit us !!! :lol:
P.S.: if you add more black lists, please share it whit us !!! :lol:
P.S.: if you add more white lists, please share it whit us !!! :lol:

Enjoy,
Bernardo Pita
Argentina
 
Ghassan
Member Candidate
Member Candidate
Posts: 213
Joined: Mon May 29, 2006 11:08 pm
Location: Lebanon
Contact:

Thu May 17, 2007 11:37 am

did you means that you has converted MT ROuter to ant spam , it is a good choice . great job .. great idea .


Kind Regards,

Ghassan
 
Ghassan
Member Candidate
Member Candidate
Posts: 213
Joined: Mon May 29, 2006 11:08 pm
Location: Lebanon
Contact:

Thu May 17, 2007 11:40 am

can you send me this configuration to my email please ...

but without comments .

ghassan.gharabli@lebol.net


Thank you .
 
User avatar
ashish
Long time Member
Long time Member
Posts: 550
Joined: Mon Feb 12, 2007 5:50 am
Location: Virginia, USA.

Thu May 17, 2007 12:48 pm

Great JOB man,

Please forward the Script / configuration at pashish.patel@hotmail.com

PLEASE....
- AP - ASHISH PATEL - 757 309 0236 -
 
Sauron
just joined
Posts: 2
Joined: Wed Jul 21, 2004 2:31 am
Location: Estonia

Fri May 18, 2007 11:23 am

Please send to me too this configuration.
My e-mail is sauron@tsl.ee

Thank you in advance!
 
User avatar
kapulan
Member Candidate
Member Candidate
Posts: 110
Joined: Tue Feb 07, 2006 7:48 pm
Location: Hungary

re

Fri May 18, 2007 11:55 am

Please send my mail address this configuration!
krisztian.modos@kapulan.hu
Thank you!
Great Job!
:)
 
User avatar
tgrand
Long time Member
Long time Member
Posts: 665
Joined: Mon Aug 21, 2006 2:57 am
Location: Winnipeg, Manitoba, Canada

Fri May 18, 2007 6:09 pm

Why not just post it into the WIKI?
 
onowojemma
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Sep 11, 2005 5:27 pm
Location: Nigeria

Fri May 18, 2007 7:09 pm

Pls can i also have the config an new to scripting my email address is onowojemma@yahoo.com
Mikrotik! making networking easy
 
trottolino1970
Member Candidate
Member Candidate
Posts: 192
Joined: Thu May 17, 2007 4:25 pm
Contact:

Re:

Wed Sep 19, 2007 10:44 am

Pls can i also have the config an new to scripting my email address is onowojemma@yahoo.com
pls send me the configuration without comment at fmeduri@email.it
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6267
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: How to convert a MT box in an Anti Spam server with v2.9

Wed Sep 19, 2007 3:14 pm

can you put it into wiki?

it is good idea, and nice implementation, it could give other people thoughts on how to create some other "applications" of RouterOS
 
trottolino1970
Member Candidate
Member Candidate
Posts: 192
Joined: Thu May 17, 2007 4:25 pm
Contact:

Re: How to convert a MT box in an Anti Spam server with v2.9

Tue Mar 18, 2008 12:05 pm

and if don't take a script?
 
atheel hassan
just joined
Posts: 1
Joined: Mon Mar 31, 2008 11:47 am

Re: How to convert a MT box in an Anti Spam server with v2.9

Mon Mar 31, 2008 11:41 pm

Great Jop, if send the configation to me you will be the best :D :D at atheel.hassan@hotmail.com
 
glodif
just joined
Posts: 12
Joined: Thu Mar 06, 2008 10:13 am

Re: How to convert a MT box in an Anti Spam server with v2.9

Tue Apr 01, 2008 8:38 pm

pls i need the scripts, can u send it to my email: glodif_direct@yahoo.com.
 
abab_rafiq
Member Candidate
Member Candidate
Posts: 120
Joined: Thu Aug 24, 2006 12:47 pm
Location: Dhaka

Re: How to convert a MT box in an Anti Spam server with v2.9

Tue Apr 01, 2008 8:59 pm

I think it will better for post on wiki. email goes for personal, doesn't it ?? :shock:
To bpita, wish a good health 4 u and ur wife  :D 
 
camari
just joined
Posts: 8
Joined: Thu Aug 17, 2006 8:40 pm
Location: San Pedro Sula, Honduras CA
Contact:

Re: How to convert a MT box in an Anti Spam server with v2.9

Wed Apr 02, 2008 7:24 pm

Great!
can you send me this configuration to my email? scripts....

chaman@mayanet.hn

Regards,
Carlos
 
xezen
Long time Member
Long time Member
Posts: 628
Joined: Fri May 30, 2008 10:23 am
Location: South Africa

Re: How to convert a MT box in an Anti Spam server with v2.9

Wed Apr 28, 2010 3:12 pm

WAS THIS ADDED TO WIKI?
If i dont No Ask someone That Does!
 
dipson
newbie
Posts: 41
Joined: Fri Nov 07, 2008 7:05 pm

Re: How to convert a MT box in an Anti Spam server with v2.9

Wed May 12, 2010 1:46 pm

please send me this script without the comment. diposalako@yahoo.com
Best Regards,
Oladipupo Salako
Double D Communications Engineering Services,
Lagos, Nigeria.
 
hansin
just joined
Posts: 8
Joined: Tue Feb 01, 2011 2:38 pm

Re: How to convert a MT box in an Anti Spam server with v2.9

Mon Jul 25, 2011 7:52 am

Dear,

Please Send me complete script without any comment at lovely@globalpanipat.com, so that i put firewall directly to my MT firewall rule.

Thanks & Regards,

Lovely.
 
ajccomputer
just joined
Posts: 1
Joined: Sun Oct 16, 2011 3:53 pm

Re: How to convert a MT box in an Anti Spam server with v2.9

Sun Oct 16, 2011 4:12 pm

Please send our email from you excelent of idea like this topic. Please send to completed. so please send on our email: ajccomputer@gmail.com
Before and the first time, we glad thank you very much....
 
User avatar
Bongo
newbie
Posts: 34
Joined: Thu Aug 11, 2011 4:28 am

Re: How to convert a MT box in an Anti Spam server with v2.9

Sun Oct 16, 2011 11:42 pm

Hi can you mail me the script please!!! johan1786@gmail.com

Thanks...!
 
User avatar
darkking
just joined
Posts: 10
Joined: Sun Dec 04, 2011 7:27 pm
Location: Jõgeva, Estonia
Contact:

Re: How to convert a MT box in an Anti Spam server with v2.9

Wed Jun 27, 2012 6:12 pm

Hi, very good idea. Can You please send me the script also to my email ahtometting@live.com .
Thanx.

Who is online

Users browsing this forum: No registered users and 59 guests