Community discussions

MUM Europe 2020
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Logging web traffic to remote webserver

Sat May 10, 2014 8:26 pm

I want to send web traffic log to a remote server listening on UDP 514 from a CPE and then save it with a properly software.
CPE config:
ip firewall filter rules
 5   ;;; default configuration
     chain=forward action=accept connection-state=established 

 6   ;;; default configuration
     chain=forward action=accept connection-state=related 

 7   chain=forward action=log tcp-flags=syn protocol=tcp log-prefix="clientlog>
system logging rules
 4    topics=web-proxy prefix="clientlog" action=remote 
system logging action
 3 * name="remote" target=remote remote-port=514 src-address=0.0.0.0 
     bsd-syslog=no syslog-facility=daemon syslog-severity=auto 

 4   name="remotelog" target=remote remote=MY PUBLIC IP ADDRESS remote-port=514 
     src-address=0.0.0.0 bsd-syslog=no syslog-facility=daemon 
     syslog-severity=auto
port 514 on webserver firewall open
port 514 on modem firewall open

still no data received!
Something wrong with CPE config?
Arpanet.
There's no distance in the world.
 
jaykay2342
Member
Member
Posts: 335
Joined: Tue Dec 04, 2012 2:49 pm
Location: /Vigor/LocalGroup/Milky Way/Earth/Europe/Germany

Re: Logging web traffic to remote webserver

Sat May 10, 2014 9:06 pm

Are you using the buildin proxy-server? because topics=web-proxy selects the log from the buildin web-server not logs from the firewall
9-5 Job: Securityanalyst at a major MSSP.
Free time volunteer: Networkadmin and founder at a small non-profit WISP.
Certifications: ITILv3, GCIA
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sat May 10, 2014 9:22 pm

now I set topic to firewall, but no data yet :(
Arpanet.
There's no distance in the world.
 
lordzar
Frequent Visitor
Frequent Visitor
Posts: 67
Joined: Sat May 29, 2004 7:47 pm

Re: Logging web traffic to remote webserver

Sat May 10, 2014 10:37 pm

First thing I see is that your logging rule says "remote" and not "remotelog"
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sat May 10, 2014 10:50 pm

You're right . I changed to "remote" now but still no data.
Maybe it's my network since is natt'd.
Should I open port 514 for each NAT node?
Arpanet.
There's no distance in the world.
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 12:51 am

ok with topic=firewall now it works.
which is the different with web-proxy topic? Which is better to use?
Arpanet.
There's no distance in the world.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:11 am

the question is not what is better, but what is working with your actual configuration


remember:
for Italian law, as ISP, you must log:

all tcp and udp new connection with source and destionation ip and port and the exact time on GMT 00:00 and the username used.
and all of the other type of new connection (like ICMP, GRE, etc.)

and you MUST NOT LOG site and page visited for privacy LAW...
I'm Italian, not English. Sorry for my imperfect grammar.
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 1:41 pm

ok, sorry for my incompetence.
so what method do you think it's better for logging customer traffic respecting italian privacy law?
I thought to log to a remote server (e.g syslog watcher) listening on udp 514 and saving all firewall topics.
so data captured are like this:
Image
Arpanet.
There's no distance in the world.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:20 pm

I use (and I still to use) Syslog Watcer Pro v2!!!

Put those rules at the end of firewall filter drop on forward rules (if you have any):

lista_ip_clienti = automatic list made by pppoe-server profile on "address list" field
this log only the client traffic, and not all the network traffic.

The first rule log TCP, UDP, ICMP, IGMP etc. new connection,
the second the RELATED connection for the first,
the other rule log the other protocols.
add action=log chain=forward comment="Invio Log connessioni in uscita al Server" connection-state=new src-address-list=lista_ip_clienti
add action=log chain=forward connection-state=related log-prefix=RELATED src-address-list=lista_ip_clienti
add action=log chain=forward comment="Tunnel IPv6 dentro IPv4" protocol=ipv6 src-address-list=lista_ip_clienti
add action=log chain=forward protocol=gre src-address-list=lista_ip_clienti
add action=log chain=forward protocol=ipsec-esp src-address-list=lista_ip_clienti
And configure accordingly system / logging

/system logging action
add bsd-syslog=yes name=LogServerNotice remote=1.1.1.1 syslog-facility=local1 syslog-severity=notice target=remote
add bsd-syslog=yes name=LogServerInfo remote=1.1.1.1 syslog-facility=local1 syslog-severity=info target=remote

/system logging
add action=LogServerNotice topics=pppoe,info,!debug
add action=LogServerInfo topics=firewall,info,!debug
Those rules not only log the firewall, but also the pppoe autentication.

ALL THIS RULE ARE VALID ONLY FOR "PPP" CLIENTS, for HotSpot the method is different...

For HotSpot I suggest you to use "Cloud4Wi"...
I'm Italian, not English. Sorry for my imperfect grammar.
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:29 pm

if users do not log in via ppoe, should I manually configure each client's cpe for remote logging?
Arpanet.
There's no distance in the world.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:39 pm

if users do not log in via ppoe, should I manually configure each client's cpe for remote logging?
No, if you assign static public IP to client, you not need to log anything, each IP is exactly that customer :)
You must only track each time you change the "owner" of IP address :)

What method you use for login the CPE to Gateway?

any "ppp" method can be logged with that described on previous post, simply change pppoe in the log, with what you want.
I'm Italian, not English. Sorry for my imperfect grammar.
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:01 pm

by now I access cpe's via routing.
the problem is cpe addresses are assigned by Access Point dhcp server.
so manual config I think is necessary..
Arpanet.
There's no distance in the world.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:05 pm

by now I access cpe's via routing.
the problem is cpe addresses are assigned by Access Point dhcp server.
so manual config I think is necessary..
You decide that yourself or other "ISP starter" company as suggested that to you?

It's very hard log all easy with this method,
at this point the rule for logging must be put on access point, not on cpe.
I'm Italian, not English. Sorry for my imperfect grammar.
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:11 pm

why on the access point?
I configured a customer cpe to remote log on udp 514 to a pc into my lan and it seems to work as picture I posted in previous post.
Last edited by Arpanet on Sun May 11, 2014 6:20 pm, edited 2 times in total.
Arpanet.
There's no distance in the world.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:17 pm

why on the access point?
I configured a customer cpe to remote log on udp 514 to a pc into my lan and it seems to work as picture I posted in previous post.
1) Because you use user traffic for log the user, and you also log the logging of the user... ;)
2) If you want change any, you change only the AP, not all the CPE...
I'm Italian, not English. Sorry for my imperfect grammar.
 
User avatar
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:21 pm

multiple cpe's logging is allowed?
lf I set user cpe for logging this will be heavy on user connection speed?
Last edited by Arpanet on Sun May 11, 2014 6:27 pm, edited 1 time in total.
Arpanet.
There's no distance in the world.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:22 pm

Yes.

Syslog use less space than "packet capture"....
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Wed Jul 09, 2014 6:28 pm


Those rules not only log the firewall, but also the pppoe autentication.

ALL THIS RULE ARE VALID ONLY FOR "PPP" CLIENTS, for HotSpot the method is different...

For HotSpot I suggest you to use "Cloud4Wi"...
Hi rextended, thanks for the conf.
What about hotspot loging configuration?

Thanks in advance, Simone.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Wed Jul 09, 2014 9:47 pm

after this rule:
/ip firewall nat
add action=passthrough chain=unused-hs-chain comment="place hotspot rules here" disabled=yes

immediately after add this:
/ip firewall nat
add action=log chain=pre-hotspot

and this on system logging:
/system logging action
add bsd-syslog=yes name=LogServerNotice2 remote=1.1.1.1 syslog-facility=local2 syslog-severity=notice target=remote
add bsd-syslog=yes name=LogServerInfo2 remote=1.1.1.1 syslog-facility=local2 syslog-severity=info target=remote

/system logging
set 0 prefix=INFO topics=info,!firewall
add action=LogServerInfo2 topics=firewall,info,!debug
add action=LogServerNotice2 topics=hotspot,info,!debug
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Wed Jul 09, 2014 10:43 pm

Hi thanks for the post. I'll try it tomorrow.

What are the differences between this method?

[admin@Test-HotSpot] > ip firewall filter add chain=forward action=log connection-state=new
[admin@Test-HotSpot] > system logging add topics=firewall action=remote
[admin@Test-HotSpot] > system logging action set remote remote=[remote server ip]
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Thu Jul 10, 2014 10:39 am

Hi thanks for the post. I'll try it tomorrow.

What are the differences between this method?

[admin@Test-HotSpot] > ip firewall filter add chain=forward action=log connection-state=new
[admin@Test-HotSpot] > system logging add topics=firewall action=remote
[admin@Test-HotSpot] > system logging action set remote remote=[remote server ip]
"mine" is better and complete... ;)
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 11:10 am

Hi thanks for the post. I'll try it tomorrow.

What are the differences between this method?

[admin@Test-HotSpot] > ip firewall filter add chain=forward action=log connection-state=new
[admin@Test-HotSpot] > system logging add topics=firewall action=remote
[admin@Test-HotSpot] > system logging action set remote remote=[remote server ip]
"mine" is better and complete... ;)
ok :D

Some questions:
- I can't see logs in log shell of rb. How i can do it? (without storing them)
- here a simple log (from my syslog)
Image
Any way to show not just calling ip but radius user too?
- How I can send radius / captive portal logs to syslog? (eg. user logins / failure, ip assigned, etc.)

Thanks, Simone.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 3:59 pm

ci sono due regole apposta, la prima logga le connessioni, la seconda (add action=LogServerNotice2 topics=hotspot,info,!debug) logga gli utenti, così rimane la corrispondenza tra utente e mac address/IP


>>>- I can't see logs in log shell of rb. How i can do it? (without storing them)
/system logging
add action=memory topics=firewall,info,!debug
add action=memory topics=hotspot,info,!debug
Remember to disable it after you have used...
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:29 pm

ci sono due regole apposta, la prima logga le connessioni, la seconda (add action=LogServerNotice2 topics=hotspot,info,!debug) logga gli utenti, così rimane la corrispondenza tra utente e mac address/IP


>>>- I can't see logs in log shell of rb. How i can do it? (without storing them)
/system logging
add action=memory topics=firewall,info,!debug
add action=memory topics=hotspot,info,!debug
Remember to disable it after you have used...

I'm sorry but with your conf I can't see hotspot infos.
/ip firewall nat add action=log chain=pre-hotspot
/system logging action
add bsd-syslog=yes name=LogServerNotice2 remote=192.168.2.2 syslog-facility=local2 syslog-severity=notice target=remote
add bsd-syslog=yes name=LogServerInfo2 remote=192.168.2.2 syslog-facility=local2 syslog-severity=info target=remote

/system logging
set 0 prefix=INFO topics=info,!firewall
add action=LogServerInfo2 topics=firewall,info,!debug
add action=LogServerNotice2 topics=hotspot,info,!debug
About log shell: why I have to remember to disable it after use?

PS: skype?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:40 pm

***NO SKYPE***

Because if the users make traffic, you can not reach to read other logs...
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:47 pm

Ok, I don't uderstand :D

Any tips about hotspot logs that not working?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:51 pm

really for me is working, is my production config.....

for see any log on hotspot side you must wait user login/logoff...
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:04 pm

RouterOS version? Mine is 6.15
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:13 pm

RouterOS version? Mine is 6.15
RB1200 RouterOS 6.15
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:18 pm

Mine is RB450. Damn, I can't find where is the problem...
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:36 pm

I think I found the problem:
/system logging
set 0 prefix=INFO topics=info,!firewall
Set 0 not working:
[admin@MikroTik] /system logging> set action=
LogServerInfo2  LogServerNotice2  disk  echo  memory  remote
Which one I have to use?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2950
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:41 pm

I think I found the problem:
/system logging
set 0 prefix=INFO topics=info,!firewall
Set 0 not working:
[admin@MikroTik] /system logging> set action=
LogServerInfo2  LogServerNotice2  disk  echo  memory  remote
Which one I have to use?
memory, the default.

If you syslog is not bsd capable, remove that flag on logging action
I'm Italian, not English. Sorry for my imperfect grammar.
 
simogere
newbie
Posts: 40
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:44 pm

With this one works fine:
/system logging
set LogServerNotice2 prefix=INFO topics=info,!firewall
My syslog is a pc with The Dude 3.6
 
User avatar
niamul
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Tue Dec 04, 2007 9:33 am
Location: Dhaka, Bangladesh
Contact:

Re: Logging web traffic to remote webserver

Tue May 02, 2017 12:26 pm

Dear rextended,
thank you for the help. it has really saved me.
cheers
Niamul
- waiting for that last laugh

Who is online

Users browsing this forum: PandaArea and 117 guests