Firewall forward established and related, not catching established connections

shunkica · Thu Jun 09, 2022 1:19 pm

There is a very weird thing happening where some traffic between two computers in a local network was being dropped, and I can't figure it out.

I set a rule just before the drop statement which explicitly allows all traffic from this client, and logs the packet, to see what exactly is going on.
Here is a example of a packet that would be dropped without this explicit rule:

forward: in:br_local out:br_local, src-mac c4:54:44:2f:72:d7, proto TCP (ACK), 192.168.1.21:53966->10.20.10.1:3001, len 40

The firewall is a accept related/established firewall, then explicitly allow new, reject invalid from local interfaces, and finally drop everything else.

/ip firewall filter
add action=accept chain=forward comment="Accept established/related forward from all interfaces" connection-state=established,related
add action=accept chain=forward comment="Accept new forward from local bridge" connection-state=new in-interface=br_local
... some other stuff ...
add action=accept chain=forward log=yes src-address=192.168.1.21  <--- this is the explicit rule i put in just before the drop
add action=reject chain=forward comment="Reject invalid tcp forward from local bridge" connection-state=invalid in-interface=br_local protocol=tcp reject-with=tcp-reset tcp-flags=!rst
add action=drop chain=forward comment="Drop all forward"

Now if this is a established connection, i think it should get caught by the first filter rule. So I checked the connections, and there it was:

the_connection.jpg

And the interesting thing is that all other traffic works correctly, the only traffic that was logged by the eplicit rule I put in was the traffic from this pc at 192.168.1.21 to the server at 10.20.10.1:3001
The port on the client changes, but I checked all of them and they all have established connections in the connection tracker.

connections.jpg

Why is this happening?

sindy · Thu Jun 09, 2022 1:34 pm

I'd say add connection-state=invalid to the dedicated rule that saves these packets from getting dropped and see whether it keeps counting. Because the only explanation to come to my mind is that the connection tracking doesn't like something about their TCP header and considers them invalid, hence they do not match the "established or related" condition.

Capturing one such TCP session into a file and analysing it using Wireshark could explain why the connection tracking tags them as invalid.

shunkica · Thu Jun 09, 2022 1:55 pm

The problem was not just limited to these two computers. There were other problems with connection tracking between these two subnets.

For example 10.20.10.1 could no longer connect to MySQL on 192.168.1.201

I added a rule that explicitly allows invalid forward from the local bridge now and logs these packets.

It might have something to do with this rule:

add action=reject chain=forward comment="Reject invalid tcp forward from local bridge" connection-state=invalid in-interface=br_local protocol=tcp reject-with=tcp-reset tcp-flags=!rst

I have disabled it for now to see if it will fix the problem.

shunkica · Thu Jun 09, 2022 2:50 pm

Disabling the reject rule does not seem to fix the problem.

Could it have something to do with hardware offloading?
All of these computers are connected to the same ethernet port.
I think regardless of that hw offloading should be impossible if they are in different subnets?

EDIT: Looks like I was wrong? It seems disabling hw offloading on this port may have eliminated the problem. But I don't understand why some packets would be hw offloaded and some not.

sindy · Thu Jun 09, 2022 3:10 pm

Hardware offloaded bridging indeed should not work if different subnets are used; hardware accelerated routing can work (some models do support it in ROS7).

But I've recently noticed that in a topology like yours (two distinct subnets on same interface) RouterOS sends ICMP redirects even though the destination is in a different subnet than the source, and some hosts even obey them, sending ARP requests for addresses outside their own subnet as a consequence. So if this happens and some packet bypasses the router, the connection tracking may tag the ACK packet for an unseen sequence number as an invalid one. That's why I recommend sniffing.

shunkica · Thu Jun 09, 2022 3:19 pm

I was too quick to judge that disabling hw offloading solved the problem...

more-invalid.jpg

Also I can confirm that sometimes ICMP redirect is sent. This is a ping from 10.20.10.1

$ ping 192.168.1.201
PING 192.168.1.201 (192.168.1.201) 56(84) bytes of data.
64 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.285 ms
From 10.20.10.254 icmp_seq=2 Redirect Host(New nexthop: 192.168.1.201)

And the resulting log line:

accept-invalid forward: in:br_local out:br_local, src-mac <hidden>, proto ICMP (type 0, code 0), 192.168.1.201->10.20.10.1, len 84

There is one more thing I don't understand. Since loose-tcp-tracking is enabled, shouldnt these ACK packets just be treated as a "new" connection?

sindy · Thu Jun 09, 2022 3:27 pm

I'm not that deep into connection tracking details, but I'd say there is a difference in treatment of a packet whose src and dst socket addresses do not match any existing connection vs. a packet which does match an existing connection by sockets but the TCP state is not the expected one. In the former case, it will definitely be treated as new and create the connection in the tracking list (by practical experience, the reason why the loose-tcp-tracking element has appeared in the configuration is that I've come across that and filed a support ticket on it).

shunkica · Thu Jun 09, 2022 3:43 pm

The fact that ROS sends a ICMP redirect for a client on a different subnet seems like a bug to me, regardless of the fact if this is the thing causing my problem or not.

sindy · Thu Jun 09, 2022 3:48 pm

I fully agree, just had other priorities than filing a ticket back then (two weeks ago or so) and then forgot about it.

Thu Jun 09, 2022 4:14 pm

Maybe I misunderstood the problem, but since when redirects should not be sent if you want to reach different network? That is exactly redirects are for.


dst-net
   |
   |
 gw
|   |
R1--R2

gw,R1,R2 is in one broadcast network.

If you are trying to route from R2 to dst-net via R1, then of course redirect will be sent, to reach dst-net over gw directly.

shunkica · Thu Jun 09, 2022 4:47 pm

Maybe I misunderstood the problem, but since when redirects should not be sent if you want to reach different network? That is exactly redirects are for.

gw,R1,R2 is in one broadcast network.

If you are trying to route from R2 to dst-net via R1, then of course redirect will be sent, to reach dst-net over gw directly.

10.20.10.1 and 192.168.1.201 are two hosts in different subnets. They need a l3 device to communicate.
Why then would the router send 192.168.1.201 as the "new nexthop" to the 10.20.10.1 host?

mkx · Thu Jun 09, 2022 4:56 pm

Maybe I misunderstood the problem, but since when redirects should not be sent if you want to reach different network? That is exactly redirects are for.

gw,R1,R2 is in one broadcast network.

If you are trying to route from R2 to dst-net via R1, then of course redirect will be sent, to reach dst-net over gw directly.
10.20.10.1 and 192.168.1.201 are two hosts in different subnets. They need a l3 device to communicate.
Why then would the router send 192.168.1.201 as the "new nexthop" to the 10.20.10.1 host?

If 10.20.10.0/24 and 192.168.1.0/24 share same broadcast domain (read: same ethernet port), then what @mrz wrote makes sense. However, this behaviour doesn't make sense to clients from different L3 subnets which rightfully expect any advertised (or statically configured) gateway to belong to same L3 subnet as the host itself. And it makes even less sense to stateful firewalls.

But then it might be conceptual error to depend on proper firewall operation for communication between hosts in same broadcast domain which means that it would require minimum effort from an attacker to circumvent the firewall in the first place.

sindy · Thu Jun 09, 2022 5:39 pm

gw,R1,R2 is in one broadcast network.

If you are trying to route from R2 to dst-net via R1, then of course redirect will be sent, to reach dst-net over gw directly.

But it's not the case where there is another router in the same subnet, so R1 uses ICMP redirect to tell R2 that gw is a better gateway to the destination network than R1 itself. R1, R2 and gw are indeed in the same broadcast domain but in different subnets. In fact, nothing actually prevents the hosts with addresses from different subnets from sending packets directly to each other if they can respond each other's ARP requests, it just doesn't feel "correct". I was looking for some RFC mandating how it should be, but only found some derivative work stating that not only same interface but also same subnet are mandatory to send the redirect.

sindy · Thu Jun 09, 2022 6:25 pm

OK, so it seems to be RFC 792 itself that states when a redirect message is to be sent:

The gateway sends a redirect message to a host in the following situation. A gateway, G1, receives an internet datagram from a host on a network to which the gateway is attached. The gateway, G1, checks its routing table and obtains the address of the next gateway, G2, on the route to the datagram's internet destination network, X. If G2 and the host identified by the internet source address of the datagram are on the same network, a redirect message is sent to the host.

Although the whole procedure only makes sense on P-MP interfaces, a broadcast domain is not explicitly mentioned there, so I'd personally prefer the conservative interpretation of this paragraph, i. e. to understand "network" as "subnet".

RFC1009 doesn't seem to be much helpful at first glance:
The ICMP Redirect message is sent by a gateway to a host on the same network, in order to change the gateway used by the host for routing certain datagrams.
But if we take it strictly enough, it mentions a gateway, not a destination host, on the same network as the source, which suggests that a broadcast domain shared by multiple subnets was not a target case for ICMP redirect.

RFC1122 gives a hint:
A Redirect message SHOULD be silently discarded if the new gateway address it specifies is not on the same connected (sub-)net through which the Redirect arrived
Although RFC1122 describes requirements on a host rather than on a gateway, this statement suggests that a redirect message indicating a gateway outside the source subnet is an anomaly and should be disregarded.

@mrz, is it worth a support ticket?

Firewall forward established and related, not catching established connections

Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Re: Firewall forward established and related, not catching established connections

Who is online