I have a couple of static IP addresses and I just changed over the NAT gateway to a Mikrotik box. Here is the strange part. When users connect to our mail provider via POP3 sometimes the connection fails and they have to wait a few seconds to a minute for it to start working again. I did some packet sniffing and found the following. Normally the connection is established with the client sending a SYN, then the server replies with a SYN/ACK, and finally the client replies with a ACK and the connection is established. Well when the connection starts failing the client sends a SYN, the server replies with an ACK only, and the client sends a RST. I originally thought the problem was on the server end, but I changed back to the old gateway and stopped having these problems. What is happening?
do you have ConnectionTracking enabled? do you use Tarpit rules?
Yes connection tracking is enabled and I do not use any tarpit rules.
I have tcp checksum offloading so that’s why you see the TCP CHECKSUM INCORRECT messages.
|Time | 10.0.0.19 | 208.xxx.xxx.xxx |
|0.000 | 16269 > pop3 [SYN] |TCP: 16269 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(16269) ------------------> (110) |
|0.031 | [TCP ACKed lost seg |TCP: [TCP ACKed lost segment] pop3 > 16269 [ACK] Seq=1 Ack=191164058 Win=5840 Len=0
| |(16269) <------------------ (110) |
|0.031 | 16269 > pop3 [RST] |TCP: 16269 > pop3 [RST] Seq=191164058 Win=0 Len=0
| |(16269) ------------------> (110) |
|2.955 | 16269 > pop3 [SYN] |TCP: 16269 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(16269) ------------------> (110) |
|2.985 | [TCP Dup ACK 2#1] p |TCP: [TCP Dup ACK 2#1] pop3 > 16269 [ACK] Seq=1 Ack=191164058 Win=5840 Len=0
| |(16269) <------------------ (110) |
|2.985 | 16269 > pop3 [RST] |TCP: 16269 > pop3 [RST] Seq=191164058 Win=0 Len=0
| |(16269) ------------------> (110) |
|8.970 | 16269 > pop3 [SYN] |TCP: 16269 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(16269) ------------------> (110) |
|8.998 | [TCP Dup ACK 2#2] p |TCP: [TCP Dup ACK 2#2] pop3 > 16269 [ACK] Seq=1 Ack=191164058 Win=5840 Len=0
| |(16269) <------------------ (110) |
|8.998 | 16269 > pop3 [RST] |TCP: 16269 > pop3 [RST] Seq=191164058 Win=0 Len=0
| |(16269) ------------------> (110) |
|79.516 | 16291 > pop3 [SYN] |TCP: 16291 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(16291) ------------------> (110) |
|79.556 | pop3 > 16291 [SYN, |TCP: pop3 > 16291 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460
| |(16291) <------------------ (110) |
|79.556 | 16291 > pop3 [ACK] |TCP: 16291 > pop3 [ACK] Seq=1 Ack=1 Win=65535 [TCP CHECKSUM INCORRECT] Len=0
| |(16291) ------------------> (110) |
|79.591 | Response: +OK Hello |POP: Response: +OK Hello fuse46 MailAnyone POP3 v2.2.0 re
| |(16291) <------------------ (110) |
|79.732 | 16291 > pop3 [ACK] |TCP: 16291 > pop3 [ACK] Seq=1 Ack=49 Win=65487 [TCP CHECKSUM INCORRECT] Len=0
| |(16291) ------------------> (110) |
|80.464 | Request: q| |POP: Request: q
| |(16291) ------------------> (110) |
|80.502 | pop3 > 16291 [ACK] |TCP: pop3 > 16291 [ACK] Seq=49 Ack=2 Win=5840 Len=0
| |(16291) <------------------ (110) |
|80.573 | Request: u| |POP: Request: u
| |(16291) ------------------> (110) |
|80.609 | pop3 > 16291 [ACK] |TCP: pop3 > 16291 [ACK] Seq=49 Ack=3 Win=5840 Len=0
| |(16291) <------------------ (110) |
|80.659 | Request: i| |POP: Request: i
| |(16291) ------------------> (110) |
|80.692 | pop3 > 16291 [ACK] |TCP: pop3 > 16291 [ACK] Seq=49 Ack=4 Win=5840 Len=0
| |(16291) <------------------ (110) |
|80.747 | Request: t| |POP: Request: t
| |(16291) ------------------> (110) |
|80.777 | pop3 > 16291 [ACK] |TCP: pop3 > 16291 [ACK] Seq=49 Ack=5 Win=5840 Len=0
| |(16291) <------------------ (110) |
|80.880 | Request: | |POP: Request:
| |(16291) ------------------> (110) |
|80.920 | pop3 > 16291 [ACK] |TCP: pop3 > 16291 [ACK] Seq=49 Ack=7 Win=5840 Len=0
| |(16291) <------------------ (110) |
|80.920 | Response: +OK Bette |POP: Response: +OK Better luck next time.
| |(16291) <------------------ (110) |
|80.920 | pop3 > 16291 [FIN, |TCP: pop3 > 16291 [FIN, ACK] Seq=77 Ack=7 Win=5840 Len=0
| |(16291) <------------------ (110) |
|80.920 | 16291 > pop3 [ACK] |TCP: 16291 > pop3 [ACK] Seq=7 Ack=78 Win=65459 [TCP CHECKSUM INCORRECT] Len=0
| |(16291) ------------------> (110) |
|80.920 | 16291 > pop3 [FIN, |TCP: 16291 > pop3 [FIN, ACK] Seq=7 Ack=78 Win=65459 [TCP CHECKSUM INCORRECT] Len=0
| |(16291) ------------------> (110) |
|80.952 | pop3 > 16291 [ACK] |TCP: pop3 > 16291 [ACK] Seq=78 Ack=8 Win=5840 Len=0
| |(16291) <------------------ (110) |
I did some more sniffing I think I found the problem. I believe the Mikrotik router is reusing the outgoing port too quickly. I’m using src-nat and a outgoing port range of 30000-60000. Is there a way to set the outgoing port reuse time to say 20 seconds or so?
You can try changing Connection Tracking timeouts. You could also add a range of IP’s in the srcnat rule to cause the router to use ports on multiple addresses thus decreasing probability of collision.
There isn’t collisions per-say, it’s not like two hosts are trying to use the same to-port on src-nat. Before all the TCP timeouts were set to 10secs, so for example if I made a POP3 connection and src-nat used port 30117 for the to-port and wait 10-11secs to reopen the connection it’ll reuse that port 30117. When it reuses a port I start getting the ACK and RST issues. I changed the timeouts to 20secs and now it does the same thing, but I just have to wait 20-21 seconds. I’m so confused…
I just changed the timeout to 30secs and it’s the same thing. Source nat reuses the to-port after 30-31 secs and if I make an outgoing to connection the same host and port right after I get TCP ACKed lost segments errors.
|Time | 75.xxx.xxx.xxx | 208.xxx.xxx.xxx |
|0.000 | 31176 > pop3 [SYN] |TCP: 31176 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(31176) ------------------> (110) |
|0.029 | pop3 > 31176 [SYN, |TCP: pop3 > 31176 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460
| |(31176) <------------------ (110) |
|0.030 | 31176 > pop3 [ACK] |TCP: 31176 > pop3 [ACK] Seq=1 Ack=1 Win=65535 Len=0
| |(31176) ------------------> (110) |
|0.058 | Response: +OK Hello |POP: Response: +OK Hello fuse46 MailAnyone POP3 v2.2.0 ready.
| |(31176) <------------------ (110) |
|0.242 | 31176 > pop3 [ACK] |TCP: 31176 > pop3 [ACK] Seq=1 Ack=49 Win=65487 Len=0
| |(31176) ------------------> (110) |
|1.742 | Request: q| |POP: Request: q
| |(31176) ------------------> (110) |
|1.771 | pop3 > 31176 [ACK] |TCP: pop3 > 31176 [ACK] Seq=49 Ack=2 Win=5840 Len=0
| |(31176) <------------------ (110) |
|1.772 | Request: uit |POP: Request: uit
| |(31176) ------------------> (110) |
|1.800 | pop3 > 31176 [ACK] |TCP: pop3 > 31176 [ACK] Seq=49 Ack=5 Win=5840 Len=0
| |(31176) <------------------ (110) |
|10.342 | Request: | |POP: Request:
| |(31176) ------------------> (110) |
|10.373 | pop3 > 31176 [ACK] |TCP: pop3 > 31176 [ACK] Seq=49 Ack=7 Win=5840 Len=0
| |(31176) <------------------ (110) |
|10.374 | Response: +OK Bette |POP: Response: +OK Better luck next time.
| |(31176) <------------------ (110) |
|10.374 | pop3 > 31176 [FIN, |TCP: pop3 > 31176 [FIN, ACK] Seq=77 Ack=7 Win=5840 Len=0
| |(31176) <------------------ (110) |
|10.379 | 31176 > pop3 [ACK] |TCP: 31176 > pop3 [ACK] Seq=7 Ack=78 Win=65459 Len=0
| |(31176) ------------------> (110) |
|10.379 | 31176 > pop3 [FIN, |TCP: 31176 > pop3 [FIN, ACK] Seq=7 Ack=78 Win=65459 Len=0
| |(31176) ------------------> (110) |
|10.411 | pop3 > 31176 [ACK] |TCP: pop3 > 31176 [ACK] Seq=78 Ack=8 Win=5840 Len=0
| |(31176) <------------------ (110) |
|40.757 | [TCP Port numbers r |TCP: [TCP Port numbers reused] 31176 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(31176) ------------------> (110) |
|40.790 | [TCP ACKed lost seg |TCP: [TCP ACKed lost segment] pop3 > 31176 [ACK] Seq=1 Ack=1869814713 Win=5840 Len=0
| |(31176) <------------------ (110) |
|40.792 | 31176 > pop3 [RST] |TCP: 31176 > pop3 [RST] Seq=1869814713 Win=0 Len=0
| |(31176) ------------------> (110) |
|43.775 | 31176 > pop3 [SYN] |TCP: 31176 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(31176) ------------------> (110) |
|43.808 | [TCP Dup ACK 18#1] |TCP: [TCP Dup ACK 18#1] pop3 > 31176 [ACK] Seq=1 Ack=1869814713 Win=5840 Len=0
| |(31176) <------------------ (110) |
|43.811 | 31176 > pop3 [RST] |TCP: 31176 > pop3 [RST] Seq=1869814713 Win=0 Len=0
| |(31176) ------------------> (110) |
|49.790 | 31176 > pop3 [SYN] |TCP: 31176 > pop3 [SYN] Seq=0 Win=65535 Len=0 MSS=1280
| |(31176) ------------------> (110) |
|49.819 | [TCP Dup ACK 18#2] |TCP: [TCP Dup ACK 18#2] pop3 > 31176 [ACK] Seq=1 Ack=1869814713 Win=5840 Len=0
| |(31176) <------------------ (110) |
|49.822 | 31176 > pop3 [RST] |TCP: 31176 > pop3 [RST] Seq=1869814713 Win=0 Len=0
| |(31176) ------------------> (110) |
Okay I’ve changed the timeout to 60secs and so far no problems…
what timeout did you change to 30s ? If tcp-established is 30s then you will have all kinds of problems.
All the tcp timeouts that were previously 10s were changed to 60s.
/ip firewall connection tracking
set enabled=yes generic-timeout=10m icmp-timeout=10s tcp-close-timeout=1m
tcp-close-wait-timeout=1m tcp-established-timeout=1d
tcp-fin-wait-timeout=1m tcp-last-ack-timeout=1m tcp-syn-received-timeout=
5s tcp-syn-sent-timeout=5s tcp-syncookie=no tcp-time-wait-timeout=1m
udp-stream-timeout=3m udp-timeout=10s