Sporadic connection refused on services behind nat

danypd69 · September 6, 2016, 11:18pm

Hello,
I have a simple routeros setup that I’m using to forward some services to an internal host; everything works fine but some times per day the connection to the services are refused.
I’m using an RB 1100Hx2 with routeros 6.36.2 and I am currently testing the setup with an ftp server.
The only other firewall rule is a drop on UDP 53.

The firewall nat rules are the following.

 0    chain=srcnat action=netmap to-addresses=1.1.1.1 src-address=2.2.2.2
 1    chain=dstnat action=dst-nat to-addresses=2.2.2.2 to-ports=21 protocol=tcp dst-address=1.1.1.1 dst-port=21
 2   ;;; Default masquerade for internal network
      chain=srcnat action=masquerade out-interface=Wan-1

I have done a packet capture both on the router and the host and it seems that when the connection is refused the router does not forward the packets correctly to the host.
The times are not syncronized so ignore them.

Source IP (zabbix server): 5.5.5.5
Router IP: 1.1.1.1
Host IP (on the internal network): 2.2.2.2

Dump of a failed connection on the router

23015	2016-09-07 00:19:31.617002	5.5.5.5		1.1.1.1	TCP	74	39022 → 21 [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=117085500 TSecr=0 WS=64
23016	2016-09-07 00:19:31.617059	1.1.1.1		5.5.5.5	TCP	74	21 → 39022 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=30545209 TSecr=117085500 WS=128
23017	2016-09-07 00:19:31.623477	5.5.5.5		1.1.1.1	TCP	66	39022 → 21 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=117085507 TSecr=30545209
23018	2016-09-07 00:19:31.623502	5.5.5.5		2.2.2.2	TCP	66	39022 → 21 [ACK] Seq=1 Ack=1 Win=229 Len=0 TSval=117085507 TSecr=30545209
23019	2016-09-07 00:19:31.623854	2.2.2.2		5.5.5.5	TCP	60	21 → 39022 [RST] Seq=1 Win=0 Len=0
23020	2016-09-07 00:19:31.623875	1.1.1.1		5.5.5.5	TCP	54	21 → 39022 [RST] Seq=1 Win=0 Len=0
23021	2016-09-07 00:19:32.614496	1.1.1.1		5.5.5.5	TCP	74	[TCP Spurious Retransmission] 21 → 39022 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=30545309 TSecr=117085500 WS=128
23022	2016-09-07 00:19:34.814494	1.1.1.1		5.5.5.5	TCP	74	[TCP Spurious Retransmission] 21 → 39022 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=30545529 TSecr=117085500 WS=128
23023	2016-09-07 00:19:39.014495	1.1.1.1		5.5.5.5	TCP	74	[TCP Spurious Retransmission] 21 → 39022 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=30545949 TSecr=117085500 WS=128
23024	2016-09-07 00:19:47.014497	1.1.1.1		5.5.5.5	TCP	74	[TCP Spurious Retransmission] 21 → 39022 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=30546749 TSecr=117085500 WS=128
23025	2016-09-07 00:20:03.014495	1.1.1.1		5.5.5.5	TCP	74	[TCP Spurious Retransmission] 21 → 39022 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=30548349 TSecr=117085500 WS=128

Dump of failed connection on the host

4816	2016-09-07 00:19:28.951912	5.5.5.5		2.2.2.2	TCP	66	39022 → 21 [ACK] Seq=1 Ack=1 Win=229 Len=0 TSval=117085507 TSecr=30545209
4817	2016-09-07 00:19:28.951950	2.2.2.2		5.5.5.5	TCP	54	21 → 39022 [RST] Seq=1 Win=0 Len=0

Dump of succesful connection on the router

22993 2016-09-07 00:18:31.587422    5.5.5.5     1.1.1.1  TCP 74     38994 → 21 [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=117025471 TSecr=0 WS=64
22994 2016-09-07 00:18:31.587473    5.5.5.5     2.2.2.2  TCP 74     38994 → 21 [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=117025471 TSecr=0 WS=64
22995 2016-09-07 00:18:31.587677    2.2.2.2     5.5.5.5  TCP 74     21 → 38994 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=383727304 TSecr=117025471 WS=128
22996 2016-09-07 00:18:31.587703    1.1.1.1     5.5.5.5  TCP 74     21 → 38994 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=383727304 TSecr=117025471 WS=128
22997 2016-09-07 00:18:31.593977    5.5.5.5     1.1.1.1  TCP 66     38994 → 21 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=117025479 TSecr=383727304
22998 2016-09-07 00:18:31.593996    5.5.5.5     2.2.2.2  TCP 66     38994 → 21 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=117025479 TSecr=383727304
22999 2016-09-07 00:18:31.596050    2.2.2.2     5.5.5.5  FTP 386    Response: 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
23000 2016-09-07 00:18:31.596075    1.1.1.1     5.5.5.5  FTP 386    Response: 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------

Dump of succesful connection on the host

4805 2016-09-07 00:18:28.915787     5.5.5.5      2.2.2.2  TCP 74     38994 → 21 [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=117025471 TSecr=0 WS=64
4806 2016-09-07 00:18:28.915824     2.2.2.2      5.5.5.5  TCP 74     21 → 38994 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=383727304 TSecr=117025471 WS=128
4807 2016-09-07 00:18:28.922307     5.5.5.5      2.2.2.2  TCP 66     38994 → 21 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=117025479 TSecr=383727304
4808 2016-09-07 00:18:28.924237     2.2.2.2      5.5.5.5  FTP 386    Response: 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------

Is someone able to give me some advice on how to fix it ?

Thanks a lot
Daniele

jarda · September 7, 2016, 6:00am

ip firewall filter add action=accept chain=forward connection-nat-state=dstnat

danypd69 · September 7, 2016, 7:29am

Rule added, I will report what happens.
Thanks
Daniele

danypd69 · September 7, 2016, 9:34am

It happened again; the capture on the router shows a SYN from my client to the router and an immediate RST,ACK from the router to the client.

37697	2016-09-07 10:32:32.631282	5.5.5.5	1.1.1.1	TCP	74	54818 → 21 [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=153865795 TSecr=0 WS=64
37698	2016-09-07 10:32:32.631318	1.1.1.1	5.5.5.5	TCP	54	21 → 54818 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

The capture on the host does not show anything related to the connection.

pukkita · September 7, 2016, 9:43am

FTP is an prehistoric protocol conceived even before TCP existed ('71), in an era where firewalls weren’t necessary; from nowadays point of view, its design is a complete mess both in implementation, security, etc…

Unless you set your FTP server to use passive connections, you’ll run into problems, as 21 isn’t the only port used.

Even if you set your FTP server to only allow passive connections, you’ll run into problems with remote clients depending on their firewall and nat settings.

Solution: don’t use it. FTP is fine inside a LAN for limited management purposes; use SFTP (SSH) or HTTP for anything else.

danypd69 · September 7, 2016, 10:13am

pukkita you are right but I’m using ftp just as an example,
The problem happens also with other protocols and is related to the connection setup phase not to the ftp protocol.

pukkita · September 7, 2016, 10:19am

Are those all the rules on ip > firewall??? (I mean, including filter, mangle…)

A drop all invalid rule for all forwarded traffic not only belong to best practices, but you could use it along with logging to further diagnose this issue…

Have you tried 6.36.3? Maybe this is related to

*) arp - fixed crash that caused Ethernet frames to go out via wrong interface;

danypd69 · September 7, 2016, 10:39am

This is my firewall config, I have other nat rules but they refers to other hosts so they should not cause problems.

/ip firewall filter
add action=drop chain=input dst-port=53 in-interface=Wan-1 protocol=tcp
add action=drop chain=input dst-port=53 in-interface=Wan-1 protocol=udp
add action=reject chain=input dst-port=23 in-interface=Wan-1 log=yes protocol=tcp reject-with=tcp-reset
add action=accept chain=forward connection-nat-state=dstnat

/ip firewall nat
add action=netmap chain=srcnat log-prefix=HOSTING src-address=2.2.2.2 to-addresses=1.1.1.1
add action=dst-nat chain=dstnat dst-address=1.1.1.1 dst-port=21 log=yes log-prefix=FTP-21 protocol=tcp to-addresses=2.2.2.2 to-ports=21
add action=masquerade chain=srcnat comment="Default masquerade for internal network" out-interface=Wan-1

I have not tried 6.36.3 yet, I will upgrade the router tonight.

jarda · September 7, 2016, 6:50pm

you are missing this rule:
/ip firewall filter add action=accept chain=forward connection-state=established,related

Should be placed first.

Dropping port 53 is useless. Delete those rules and add last general dropping rule instead:
/ip firewall filter add action=drop chain=forward

Should be placed last.

Everything what is not explicitely allowed will be dropped. The same two rules should be in input chain also. What you want to allow, place between those two rules.

Your forward chain should look something like this:

add action=accept chain=forward connection-state=established,related
add action=drop chain=forward connection-state=invalid
add action=accept chain=forward connection-state=new out-interface=Wan-1
add action=accept chain=forward connection-nat-state=dstnat
add action=drop chain=forward

1 allows bidirectional communication that belongs to already approved connection.
2 drops invalid packets
3 allows to establish trafic out from local network to wan
4 allows to pass dstnatted traffic (in firewall nat rules)
5 drops everything else

1,2,5 should be for input chain used similarly.

At least.

danypd69 · September 7, 2016, 7:32pm

Ok, here are my new rules.

 0    chain=forward action=accept connection-state=established,related 
 1    chain=forward action=drop connection-state=invalid 
 2    chain=forward action=accept connection-state=new out-interface=Wan-1 
 3    chain=forward action=accept connection-nat-state=dstnat 
 4    chain=forward action=drop 
 5    chain=input action=accept protocol=tcp dst-address=ROUTER_PUBLIC_ADDRESS dst-port=1194
 6    chain=input action=accept protocol=tcp dst-address=ROUTER_PRIVATE_ADDRESS  dst-port=8291
 8    chain=input action=accept connection-state=established,related 
 9    chain=input action=drop 
10    chain=input action=drop connection-state=invalid

I have also upgraded to 6.36.3, I will report if it works ok or not.
thank you for your help
Daniele

jarda · September 7, 2016, 7:42pm

wrong order in input chain.
reorder them this way: 8, 10, 5, 6, 9.

instead dst address use input interface or interface list.

danypd69 · September 7, 2016, 8:16pm

ok done, now I have

 5    chain=input action=accept connection-state=established,related 
 6    chain=input action=drop connection-state=invalid 
 7    chain=input action=accept protocol=tcp in-interface=Wan-1 dst-port=1194 
 8    chain=input action=accept protocol=tcp in-interface=LAN (ether6) dst-port=8291
 9    chain=input action=accept protocol=tcp in-interface=all-ppp dst-port=8291 
10    chain=input action=drop

the rule 9 is used to allow winbox through an openvpn connection.

danypd69 · September 8, 2016, 9:18am

Hello,
I tested it but the problem persists.
I got two connection attempts that failed, from the packet dump it seems that the router ignores the incoming connection.
I will enable log on all the drop rules to see if something is logged when it happens.

141 2016-09-08 09:34:29.667713    5.5.5.5    1.1.1.1  TCP      74     39040 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=856770 TSecr=0 WS=64
142 2016-09-08 09:34:30.667249    5.5.5.5    1.1.1.1  TCP      74     [TCP Retransmission] 39040 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=857770 TSecr=0 WS=64
143 2016-09-08 09:34:32.667039    5.5.5.5    1.1.1.1  TCP      74     [TCP Retransmission] 39040 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=859770 TSecr=0 WS=64
144 2016-09-08 09:34:36.667195    5.5.5.5    1.1.1.1  TCP      74     [TCP Retransmission] 39040 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=863770 TSecr=0 WS=64
145 2016-09-08 09:34:44.667271    5.5.5.5    1.1.1.1  TCP      74     [TCP Retransmission] 39040 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=871770 TSecr=0 WS=64

danypd69 · September 8, 2016, 9:46am

Ok some other informations, the packets are dropped by the firewall as invalid packets.
The question now is why are they marked as invalid?

Firewall log

  
Sep  8 11:32:28 192.168.100.254 firewall,info INPUT-INVALID input: in:Wan-1 out:(none), src-mac 00:21:a0:50:50:80, proto TCP (SYN), 5.5.5.5:36006->1.1.1.1:21, len 60
Sep  8 11:32:29 192.168.100.254 firewall,info INPUT-INVALID input: in:Wan-1 out:(none), src-mac 00:21:a0:50:50:80, proto TCP (SYN), 5.5.5.5:36006->1.1.1.1:21, len 60
Sep  8 11:32:31 192.168.100.254 firewall,info INPUT-INVALID input: in:Wan-1 out:(none), src-mac 00:21:a0:50:50:80, proto TCP (SYN), 5.5.5.5:36006->1.1.1.1:21, len 60
Sep  8 11:32:35 192.168.100.254 firewall,info INPUT-INVALID input: in:Wan-1 out:(none), src-mac 00:21:a0:50:50:80, proto TCP (SYN), 5.5.5.5:36006->1.1.1.1:21, len 60
Sep  8 11:32:43 192.168.100.254 firewall,info INPUT-INVALID input: in:Wan-1 out:(none), src-mac 00:21:a0:50:50:80, proto TCP (SYN), 5.5.5.5:36006->1.1.1.1:21, len 60

Router packets

3503 2016-09-08 11:32:29.752940  5.5.5.5  1.1.1.1  TCP  74  36006 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=7936716 TSecr=0 WS=64
3504 2016-09-08 11:32:30.751770  5.5.5.5  1.1.1.1  TCP  74  [TCP Retransmission] 36006 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=7937716 TSecr=0 WS=64
3505 2016-09-08 11:32:32.751949  5.5.5.5  1.1.1.1  TCP  74  [TCP Retransmission] 36006 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=7939716 TSecr=0 WS=64
3506 2016-09-08 11:32:36.751927  5.5.5.5  1.1.1.1  TCP  74  [TCP Retransmission] 36006 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=7943716 TSecr=0 WS=64
3507 2016-09-08 11:32:44.752011  5.5.5.5  1.1.1.1  TCP  74  [TCP Retransmission] 36006 → ftp [SYN] Seq=0 Win=14600 Len=0 MSS=1402 SACK_PERM=1 TSval=7951716 TSecr=0 WS=64