Community discussions

MikroTik App
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Possible connection tracking problem??

Sat Jul 12, 2008 12:51 am

Greetings,

When I enable logging for packets that are dropped in the input chain, I get flooded with entries that look like this:

input: in:Public DSL out:(none), src-mac 00:0f:cc:89:18:24, proto TCP (SYN,ACK), 72.246.103.18:80->75.54.4.45:2897, len 48
input: in:Public DSL out:(none), src-mac 00:0f:cc:89:18:24, proto TCP (RST), 66.135.217.231:80->75.54.4.45:3521, len 40
input: in:Public DSL out:(none), src-mac 00:0f:cc:89:18:24, proto TCP (SYN,ACK), 68.180.207.182:80->75.54.4.45:3641, len 44
input: in:Public DSL out:(none), src-mac 00:0f:cc:89:18:24, proto TCP (SYN,ACK), 68.180.207.182:80->75.54.4.45:3643, len 44
input: in:Public DSL out:(none), src-mac 00:0f:cc:89:18:24, proto TCP (SYN,ACK), 72.247.242.178:80->75.54.4.45:1378, len 48
input: in:Public DSL out:(none), src-mac 00:0f:cc:89:18:24, proto TCP (SYN,ACK), 68.180.195.221:80->75.54.4.45:3639, len 48

These appear to be responses from external servers to HTTP requests made by people on our back network.

As far as I understand, these responses should be processed by the FORWARD and not the INPUT chain. It almost seems to me like our RouterBoard is not realizing that these responses are related to a SNAT'd HTTP request.

Any suggestions for troubleshooting this?
 
SurferTim
Forum Guru
Forum Guru
Posts: 4637
Joined: Mon Jan 07, 2008 10:31 pm
Location: Miramar Beach, Florida

Re: Possible connection tracking problem??

Sat Jul 12, 2008 3:01 am

I don't see a problem. Most of those are probably looking for an open relay email server to spam through. There are some among those that are really bad people, looking only to make your life very uncomfortable. Those are hackers and spammers. They should be dropped. I would not waste the log storage tho and remove the logging.
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Re: Possible connection tracking problem??

Sat Jul 12, 2008 5:28 am

I don't see a problem. Most of those are probably looking for an open relay email server to spam through. There are some among those that are really bad people, looking only to make your life very uncomfortable. Those are hackers and spammers. They should be dropped. I would not waste the log storage tho and remove the logging.
SurferTim,

I don't interpret the logs that way. Those are not people scanning for open SMTP relays. That would be traffic destined to port 25. This is response traffic coming back from legitimate web servers (i.e. google, yahoo, akamai, etc) that is hitting the INPUT instead of FORWARD chain.
 
SurferTim
Forum Guru
Forum Guru
Posts: 4637
Joined: Mon Jan 07, 2008 10:31 pm
Location: Miramar Beach, Florida

Re: Possible connection tracking problem??

Sat Jul 12, 2008 5:57 am

I should have looked closer. How do you have the log entry setup? Did you put a log action just before the "chain=input action=drop"?
Is anyone on your net reporting weird "page cannot be displayed" errors or the like?
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Re: Possible connection tracking problem??

Sat Jul 12, 2008 6:05 am

I should have looked closer. How do you have the log entry setup? Did you put a log action just before the "chain=input action=drop"?
Is anyone on your net reporting weird "page cannot be displayed" errors or the like?
It does seem like HTTP connections are occasionally being dropped (i.e. sometimes I have to hit reload to get a web page up). I'm not 100% sure that this is due to the FW and not "life on the net".

The log action that generated those was the second to last line of the input chain. The last line is the drop rule.

BTW, I also tried expanding out the number of IPs used for SNAT and it didn't seem to make a difference.
 
SurferTim
Forum Guru
Forum Guru
Posts: 4637
Joined: Mon Jan 07, 2008 10:31 pm
Location: Miramar Beach, Florida

Re: Possible connection tracking problem??

Sat Jul 12, 2008 6:11 am

BTW, SYN/ACK is what my tarpit setting returns to a potential intruder. Maybe someone is going somewhere they are not supposed to be? Maybe they are getting tarpitted? But why to the input chain?

ADD: Check this out.
http://en.wikipedia.org/wiki/Portscanning
Look a few paragraphs down under the heading "SYN scanning".
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Re: Possible connection tracking problem??

Sat Jul 12, 2008 11:23 am

BTW, SYN/ACK is what my tarpit setting returns to a potential intruder. Maybe someone is going somewhere they are not supposed to be? Maybe they are getting tarpitted? But why to the input chain?

ADD: Check this out.
http://en.wikipedia.org/wiki/Portscanning
Look a few paragraphs down under the heading "SYN scanning".
I should try and do some after-hours testing when people aren't on the network, but I'm pretty sure that I've even seen my own HTTP traffic appear in these rules before. It really seems like they are HTTP response packets...
 
SurferTim
Forum Guru
Forum Guru
Posts: 4637
Joined: Mon Jan 07, 2008 10:31 pm
Location: Miramar Beach, Florida

Re: Possible connection tracking problem??

Sat Jul 12, 2008 11:46 am

The only consolation is they all are "wait a second" packets from what I can see. And why to the input? I can't see why they would go there. Do you have any firewall rules that would put them there?
 
alternativi_boy
Frequent Visitor
Frequent Visitor
Posts: 53
Joined: Tue Apr 01, 2008 8:39 pm

Re: Possible connection tracking problem??

Sat Jul 12, 2008 3:43 pm

you should try filter stoping this, because its very danger for you're MikrOTik, and you should be more carefuly when you work in public interface..:D:D;
 
SurferTim
Forum Guru
Forum Guru
Posts: 4637
Joined: Mon Jan 07, 2008 10:31 pm
Location: Miramar Beach, Florida

Re: Possible connection tracking problem??

Sat Jul 12, 2008 3:58 pm

Hi alternativi-boy

They are being dropped. He is concerned that maybe they should NOT be dropped.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8389
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Possible connection tracking problem??

Sun Jul 13, 2008 1:42 am

did you try to increase timeouts in connection tracking?
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
changeip
Forum Guru
Forum Guru
Posts: 3818
Joined: Fri May 28, 2004 5:22 pm

Re: Possible connection tracking problem??

Mon Jul 14, 2008 9:07 pm

they are being dropped because the NAT table no longer knows them. Thats why they hit the input chain because they are no longer in NAT. Most of the time this is normal if they are FIN or RST type.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Re: Possible connection tracking problem??

Tue Jul 15, 2008 12:31 am

they are being dropped because the NAT table no longer knows them. Thats why they hit the input chain because they are no longer in NAT. Most of the time this is normal if they are FIN or RST type.
ChangeIP -- thank you for expressing more clearly than I was able to exactly what I believe the problem is.

As these are SYN,ACK packets, I believe that these should be going through and forwarded and that the MT is dropping them from the NAT table.

My connection tracking parameters are:

TCP syn/received: 00:00:05
TCP established: 01:00:00
UDP stream timeout: 00:03:00
Generic timeout: 00:10:00
everything else: 00:00:010

TCP syncookies are disabled. Should I be changing any of these to higher values??
 
changeip
Forum Guru
Forum Guru
Posts: 3818
Joined: Fri May 28, 2004 5:22 pm

Re: Possible connection tracking problem??

Tue Jul 15, 2008 12:34 am

here is mine, can't remember if these are defaults or not:

[xxx@xxx] ip firewall connection tracking> print
enabled: yes
tcp-syn-sent-timeout: 10s
tcp-syn-received-timeout: 5s
tcp-established-timeout: 4h <- modified for sure.
tcp-fin-wait-timeout: 10s
tcp-close-wait-timeout: 10s
tcp-last-ack-timeout: 10s
tcp-time-wait-timeout: 10s
tcp-close-timeout: 10s
udp-timeout: 10s
udp-stream-timeout: 1m
icmp-timeout: 10s
generic-timeout: 10m
tcp-syncookie: yes
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Re: Possible connection tracking problem??

Tue Jul 15, 2008 9:18 pm

I was poking around some of the MT documentation and they wrote:

tcp-syn-received-timeout (time; default: 1m) - maximal amount of time connection tracking entry will survive after having seen a matching connection request (SYN)

tcp-syn-sent-timeout (time; default: 1m) - maximal amount of time connection tracking entry will survive after having seen a connection request (SYN) from connection initiator

I upped these timeouts to 1M from my 5S and this seems to have cut down the amount of SYN,ACK's that I have received.

I'm still receiving a fair number of ACK,FIN RST and ACK,FIN,PSH responses from web servers. My guess is that this is also timeout related. I found this information about ACK,FIN,PSH
Here's what's happening:
TCP 3 packet handshake takes place
Client issues a data request
Client issues a FIN/ACK since its done transmitting info
Netfilter drops the state time out to 60 seconds
Server starts transmitting data back to the client
More than 60 seconds goes by
Netfilter removes the state entry
Server can never complete the data transfer and continually tries to
issue a FIN/ACK to close the connection
Netfilter drops all FIN/ACK's because the state table entry is gone

I reported this problem back in 2000 and the time out was increased to
120 seconds. At some point a few years back the time out was dropped
back down again causing the problem you are seeing.
I was doing some more digging around and found this:
Timeouts

Something to note is that timeouts are reset to the maximum each time a connection sees traffic. Timeouts are set in /usr/src/linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c at compile time. Here is the relevant section of code:

static unsigned long tcp_timeouts[]
= { 30 MINS, /* TCP_CONNTRACK_NONE, */
5 DAYS, /* TCP_CONNTRACK_ESTABLISHED, */
2 MINS, /* TCP_CONNTRACK_SYN_SENT, */
60 SECS, /* TCP_CONNTRACK_SYN_RECV, */
2 MINS, /* TCP_CONNTRACK_FIN_WAIT, */
2 MINS, /* TCP_CONNTRACK_TIME_WAIT, */
10 SECS, /* TCP_CONNTRACK_CLOSE, */
60 SECS, /* TCP_CONNTRACK_CLOSE_WAIT, */
30 SECS, /* TCP_CONNTRACK_LAST_ACK, */
2 MINS, /* TCP_CONNTRACK_LISTEN, */
};

There is no absolute timeout for a connection.
I wonder if the MT default timeouts are too aggressive (i.e. short)?
 
andreacoppini
Trainer
Trainer
Posts: 489
Joined: Wed Apr 13, 2005 11:51 pm
Location: Malta, Europe

Re: Possible connection tracking problem??

Wed Jul 16, 2008 2:50 am

Are you using web-proxy?

May sound basic, but it's usually the obvious which is overlooked....
- No strings attached -

<< Please give good Karma if this post helped you. Press the + button above the Location entry
 
invader zog
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Wed Jan 03, 2007 9:04 pm

Re: Possible connection tracking problem??

Wed Jul 16, 2008 2:58 am

Are you using web-proxy?

May sound basic, but it's usually the obvious which is overlooked....
We aren't doing any web proxying (i.e. the MT neither serves nor makes use of any proxy servers)...
 
changeip
Forum Guru
Forum Guru
Posts: 3818
Joined: Fri May 28, 2004 5:22 pm

Re: Possible connection tracking problem??

Wed Sep 17, 2008 11:19 pm

I just upgraded from 2.9.51 to 3.14rc1. I noticed now RST packets are dropped where they were not before. Is there any reason a single RST packet is dropped and not accepted as closing the connection on 3.x ?
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
pekr
Member Candidate
Member Candidate
Posts: 138
Joined: Tue Feb 22, 2005 9:05 pm
Location: Czech Republic
Contact:

Re: Possible connection tracking problem??

Fri Sep 26, 2008 6:22 pm

Hi,

last two months or so we started to see strange problem on our network. Some sites as youtube were sporadically not fully loaded, especially in the case where the site used off-site .CSS file storage. We ended up with content loaded, but without CSS it was not fully functional.

Apart from MT unreliable DNS (and yes, we still consider it being unreliable, so we don't point our subnodes to our main router anymore), we are recently still observing:

1) occassional "can't display the page". When you hit reload, it is loaded ...
2) rather high "invalid connection" traffic. Those are mainly of type (ACK, FIN), (ACK, RST, FIN), (RST), (ACK, FIN, PSH)

Our main router conntrack setting is:

enabled: yes
tcp-syn-sent-timeout: 5s
tcp-syn-received-timeout: 5s
tcp-established-timeout: 1d
tcp-fin-wait-timeout: 10s
tcp-close-wait-timeout: 10s
tcp-last-ack-timeout: 10s
tcp-time-wait-timeout: 10s
tcp-close-timeout: 10s
udp-timeout: 10s
udp-stream-timeout: 3m
icmp-timeout: 10s
generic-timeout: 10m
tcp-syncookie: no
max-entries: 524288
total-entries: 7445


-pekr-
 
changeip
Forum Guru
Forum Guru
Posts: 3818
Joined: Fri May 28, 2004 5:22 pm

Re: Possible connection tracking problem??

Fri Sep 26, 2008 7:32 pm

I am seeing this as well. Exact same settings between 2.9 and 3.x started dropping TONs more packets. I had to increase the initial timeouts to almost 20-30s to get them to slow down. I have a feeling some packets aren't making it into the connection tracking so the other end has to retransmit (after 10-20s). I will run some packet sniffs to confirm.
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: Possible connection tracking problem??

Fri Sep 26, 2008 7:50 pm

I am seeing this as well. Exact same settings between 2.9 and 3.x started dropping TONs more packets. I had to increase the initial timeouts to almost 20-30s to get them to slow down. I have a feeling some packets aren't making it into the connection tracking so the other end has to retransmit (after 10-20s). I will run some packet sniffs to confirm.
Did your configs solve the problem ? if yes , can you share it with us please ?
 
changeip
Forum Guru
Forum Guru
Posts: 3818
Joined: Fri May 28, 2004 5:22 pm

Re: Possible connection tracking problem??

Fri Sep 26, 2008 7:56 pm

here is what I am currently using, although I am still seeing more than usual drops:

/ip firewall connection tracking> print
enabled: yes
tcp-syn-sent-timeout: 1m
tcp-syn-received-timeout: 1m
tcp-established-timeout: 4h
tcp-fin-wait-timeout: 30s
tcp-close-wait-timeout: 30s
tcp-last-ack-timeout: 20s
tcp-time-wait-timeout: 20s
tcp-close-timeout: 20s
udp-timeout: 10s
udp-stream-timeout: 1m
icmp-timeout: 10s
generic-timeout: 10m
tcp-syncookie: no
max-entries: 524288
total-entries: 1910
Colo and Wholesale Bandwidth Available! Sales at SanDiegoBroadband dot com
 
User avatar
omidkosari
Trainer
Trainer
Posts: 634
Joined: Fri Sep 01, 2006 4:18 pm
Location: Iran , Karaj
Contact:

Re: Possible connection tracking problem??

Fri Sep 26, 2008 8:13 pm

i changed my config like yours except tcp-syncookie: yes . i will test and say the result here
 
bugtoo
just joined
Posts: 9
Joined: Mon Sep 29, 2008 12:04 pm

Re: Possible connection tracking problem??

Mon Sep 29, 2008 12:10 pm

I'm seeing the exact same problem since upgrading from 2.9.49 to 3.11 in July. It seems like 3.14 improves somehow the situation a little bit, but we still see some pages not fully loaded and a lot of timeouts.

Who is online

Users browsing this forum: Google [Bot] and 87 guests