Community discussions

 
hakhunamathata
just joined
Topic Author
Posts: 2
Joined: Sat Dec 10, 2016 7:25 am

How to Save Browsing history of clients through Mikrotik

Tue Dec 13, 2016 5:04 am

Hello everybody, I hope you all doing well. I have a question to ask. I am using Mikro Tik Model - RB1100AHx2 , OS Level is 6 . How do I save or monitor my client's browsing history by a remote computer? I have already downloaded MT_Syslog software on my pc. Waiting for a suitable answer. Gimme the easiest way please because I am just beginner, Thanks in advance.
 
 
miq
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Fri Nov 06, 2009 3:18 am

Re: How to Save Browsing history of clients through Mikrotik

Wed Dec 14, 2016 9:49 am

You can log SYN packets:
/ip firewall filter add chain=forward action=log tcp-flags=syn protocol=tcp src-address=192.168.1.0/24 log=yes log-prefix="SYN-FORWARD"
and send log to remote machine:
 /system logging set 1 action=remote
where 1 it's a info level.

Define remote action
/system logging action add name="remote" target=remote remote=192.168.1.1 remote-port=514 src-address=192.168.1.2 bsd-syslog=no syslog-time-format=bsd-syslog syslog-facility=daemon syslog-severity=auto 
where remote it's a syslog server src-address - local interface

Then define rule on your syslog server:
# cat /etc/syslog-ng/conf.d/02-mikrotik.conf

source s_net { udp (); };
 
filter f_mikrotik_syn { host( "192.168.1.2" ); message("SYN-FORWARD");};

destination df_mikrotik_syn {
 file("/home/mikrotik/syn/${YEAR}.${MONTH}.${DAY}.log");
};
log { source ( s_net ); filter( f_mikrotik_syn ); destination ( df_mikrotik_syn ); };
You can also rotate this logs:
#!/bin/bash

set -e
renice  19 $$ > /dev/null


BASE=/home/mikrotik/
DATA=`date -d "yesterday 13:00 " '+%Y.%m.%d'`
DIRS=(syn)

for i in ${DIRS[*]}; do
        if [ \! -f ${BASE}${i}/${DATA}.log ]; then
                echo "No logfile ${BASE}${i}/${DATA}.log"
                exit 1;
        fi
        gzip -c -9 ${BASE}${i}/${DATA}.log > ${BASE}${i}/${DATA}.log.gz
        chown mtlog:mtlog ${BASE}${i}/${DATA}.log.gz
        rm ${BASE}${i}/${DATA}.log
done

/etc/init.d/syslog-ng reload | grep -v 'Reload system logging: syslog-ng.'

 
katit
newbie
Posts: 32
Joined: Wed May 13, 2015 6:01 am

Re: How to Save Browsing history of clients through Mikrotik

Wed Jan 18, 2017 10:25 pm

Great, thank you for the info!

I would like to point that this firewall rule need to be moved to the top in order for this info to be captured.

I have couple of simple questions:

1. What is "syn" exactly? Does it capture ANY activity? Basically any TCP packets sent through router will be captured, right? HTTPS/VPN will be captured as well? I want to be able to tell if someone connects and uses encrypted VPN
2. I get logs like this. From here I can tell how packets were flowing.
SYN-FORWARD forward: in:ether1-gateway out:bridge-local, src-mac a4:7a:a4:2e:af:50, proto TCP (SYN,ACK), 157.240.2.20:443->192.168.33.106:50785, NAT 157.240.2.20:443->(23.114.87.129:50785->192.168.33.106:50785), len 60

3. I got all this info in SQL database. Now I can parse out all pieces. I know local IPs and MACs so it's easy to detect local clients. For external ones, I need to do reverse DNS lookups, is that something easily done?

My logic to get history report is to collect data for a day, for example.
1. Parse out every record, detect from/to
2. Eliminate "duplicates" - for example by timestamp if there is multiple packets within a minute, I can just leave 1 row
3. Get unique IPs and do reverse DNS lookups to see remote host names.


Then, for each individual internal host I will be able to display history.

Does this sound correct?
 
miq
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Fri Nov 06, 2009 3:18 am

Re: How to Save Browsing history of clients through Mikrotik

Wed Jan 18, 2017 11:19 pm

Great, thank you for the info!

I would like to point that this firewall rule need to be moved to the top in order for this info to be captured.
Not in all cases. Imagine an DOS from one of your clients.
I have couple of simple questions:

1. What is "syn" exactly? Does it capture ANY activity? Basically any TCP packets sent through router will be captured, right? HTTPS/VPN will be captured as well? I want to be able to tell if someone connects and uses encrypted VPN
In short - SYN it's a init package, please look at three way hanhdshake http://www.inetdaemon.com/tutorials/int ... hake.shtml. Any activity before two hosts starting from SYN package.


2. I get logs like this. From here I can tell how packets were flowing.
SYN-FORWARD forward: in:ether1-gateway out:bridge-local, src-mac a4:7a:a4:2e:af:50, proto TCP (SYN,ACK), 157.240.2.20:443->192.168.33.106:50785, NAT 157.240.2.20:443->(23.114.87.129:50785->192.168.33.106:50785), len 60
And it's enough - all information about this connection are logged.
3. I got all this info in SQL database. Now I can parse out all pieces. I know local IPs and MACs so it's easy to detect local clients. For external ones, I need to do reverse DNS lookups, is that something easily done?

My logic to get history report is to collect data for a day, for example.
1. Parse out every record, detect from/to
2. Eliminate "duplicates" - for example by timestamp if there is multiple packets within a minute, I can just leave 1 row
3. Get unique IPs and do reverse DNS lookups to see remote host names.


Then, for each individual internal host I will be able to display history.

Does this sound correct?
First - there is no duplicated entries. If you have two lines with the same content one by one - there was two connections. Second I think it's a bad idea to make it on the fly. There is a really lot of information, in my case 18095665 lines, 3 GB log file (~ 20 hours, ~ 500 users). Third - revdns It's useless in this case. For example in my case google.com point to:
$ host google.com
google.com has address 216.58.209.78

$ host 216.58.209.78
78.209.58.216.in-addr.arpa domain name pointer waw02s06-in-f14.1e100.net.
78.209.58.216.in-addr.arpa domain name pointer waw02s06-in-f78.1e100.net.
78.209.58.216.in-addr.arpa domain name pointer waw02s06-in-f14.1e100.net.
78.209.58.216.in-addr.arpa domain name pointer waw02s06-in-f78.1e100.net.
If you have an ip address just use whois. Store logs in file, and find needed informations when it's necessary.
 
katit
newbie
Posts: 32
Joined: Wed May 13, 2015 6:01 am

Re: How to Save Browsing history of clients through Mikrotik

Wed Jan 18, 2017 11:28 pm

Not in all cases. Imagine an DOS from one of your clients.
Where should I put it? At the end (where script put's it) - there is nothing in log
First - there is no duplicated entries. If you have two lines with the same content one by one - there was two connections.
Good to know. But even if it's 2 connections, why should I care? Browser can open multiple connections to download pictures, etc.
I think I should clean it up a little, tell me where I might be wrong?
Second I think it's a bad idea to make it on the fly. There is a really lot of information, in my case 18095665 lines, 3 GB log file (~ 20 hours, ~ 500 users). Third - revdns It's useless in this case. For If you have an ip address just use whois. Store logs in file, and find needed informations when it's necessary.
Oh no, I'm not making it on a fly. I'm only parsing out pieces from a string and store them into DB. This part is done and yes, there is a lot!

Then, at the end of day I envision running some job to do post-processing. SQL script to get rid of duplicates. Then get DISTINCT external IPs and then do revdns on those. And I can also cache those in DB so I don't have to do it all over every day.

What about whois? Is there any programmatic way to pull this data?

ALSO! I guess it's important to make it even "better". If HTTP traffic - it would be nice to collect actual URLs which is not visible from TCP packets logging. I get it with secured protocols we can't but with regular HTTP traffic it would be nice what is being done.
 
miq
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Fri Nov 06, 2009 3:18 am

Re: How to Save Browsing history of clients through Mikrotik

Thu Jan 19, 2017 12:01 am

Not in all cases. Imagine an DOS from one of your clients.
Where should I put it? At the end (where script put's it) - there is nothing in log
In my rules right behind antiddos/limits/connlimit rules.
First - there is no duplicated entries. If you have two lines with the same content one by one - there was two connections.
Good to know. But even if it's 2 connections, why should I care? Browser can open multiple connections to download pictures, etc.
I think I should clean it up a little, tell me where I might be wrong?
If you want to log all conenctions - yes you are wrong. If you only want to know about connection with some host - you can filter logs.
Second I think it's a bad idea to make it on the fly. There is a really lot of information, in my case 18095665 lines, 3 GB log file (~ 20 hours, ~ 500 users). Third - revdns It's useless in this case. For If you have an ip address just use whois. Store logs in file, and find needed informations when it's necessary.
Oh no, I'm not making it on a fly. I'm only parsing out pieces from a string and store them into DB. This part is done and yes, there is a lot!

Then, at the end of day I envision running some job to do post-processing. SQL script to get rid of duplicates. Then get DISTINCT external IPs and then do revdns on those. And I can also cache those in DB so I don't have to do it all over every day.

What about whois? Is there any programmatic way to pull this data?
http://bfy.tw/9YYi ;)

For example in linux systems you can do this:
$ whois 216.58.209.78

#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#


#
# The following results may also be obtained via:
# https://whois.arin.net/rest/nets;q=216.58.209.78?showDetails=true&showARIN=false&showNonArinTopLevelNet=false&ext=netref2
#

NetRange:       216.58.192.0 - 216.58.223.255
CIDR:           216.58.192.0/19
NetName:        GOOGLE
NetHandle:      NET-216-58-192-0-1
Parent:         NET216 (NET-216-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS15169
Organization:   Google Inc. (GOGL)
RegDate:        2012-01-27
Updated:        2012-01-27
Ref:            https://whois.arin.net/rest/net/NET-216-58-192-0-1



OrgName:        Google Inc.
OrgId:          GOGL
Address:        1600 Amphitheatre Parkway
City:           Mountain View
StateProv:      CA
PostalCode:     94043
Country:        US
RegDate:        2000-03-30
Updated:        2015-11-06
Ref:            https://whois.arin.net/rest/org/GOGL


OrgTechHandle: ZG39-ARIN
OrgTechName:   Google Inc
OrgTechPhone:  +1-650-253-0000 
OrgTechEmail:  arin-contact@google.com
OrgTechRef:    https://whois.arin.net/rest/poc/ZG39-ARIN

OrgAbuseHandle: ABUSE5250-ARIN
OrgAbuseName:   Abuse
OrgAbusePhone:  +1-650-253-0000 
OrgAbuseEmail:  network-abuse@google.com
OrgAbuseRef:    https://whois.arin.net/rest/poc/ABUSE5250-ARIN


#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#

If you just want to know about connections between your clients and some ip you can parse this logs. If you want log information about all connections - you must store this logs.

Who is online

Users browsing this forum: MSN [Bot] and 43 guests