MT-A is able to ping anyone in 10.0.200.0/24 (VPN clients)
SERVER is able to ping anyone in 10.0.200.0/24 (albeit with ICMP REDIRECT being sent but this makes sense)
CLIENT is able to ping MT-B and MT-A
Here’s where the problem starts
Attempting to ping SERVER from CLIENT or, even better, initiate a TCP connection from CLIENT to SERVER makes MT-A scream with “connection-state=invalid” packets:
FWD drop invalid! forward: in:bridge-trusted-intranet out:bridge-trusted-intranet, src-mac e8:39:35:0e:60:66, proto TCP (SYN,ACK), 10.0.30.40:80->10.0.200.2:55347, len 60
Here’s what I think happens:
Packets originating from CLIENT arrive in MT-B, then they’re sent via default gateway to MT-A. As the SYN packet “dst-address” is in 10.0.30.0/24 it gets switch-ed directly to SERVER. Then the SYN-ACK from SERVER is sent to MT-A. MT-A has to actually route the packet, but the contrack has no idea about that connection so it becomes “connection-state=invalid” and packet is dropped.
What am I doing wrong here?
If I don’t drop these invalid packets but accept them everything is working. However, it feels like what I created is attempting to put a square peg in a round hole
Your analysis is mostly correct. What you’re doing wrong? You’re passing traffic between 10.0.30.0 and 10.0.200.0 subnet through statefull firewall on MT-A without ensuring both directions pass L3 layer of MT-A.
So either configure firewall on MT-A not to treat these packets in any way (accept invalid is one way of doing it).
Or configure MT-B with IP address in different subnet than server which will force packets from client towards server to be routed between MT-B and MT-A (which will allow firewall on MT-A to see whole connection stream) instead of being switched between MT-B and server. Address configuration on MT-A should include appropriate changes as well.
This got me thinking… how you do it in real life when you have 10 routers? Will they all be forced to deal with L3 and to perform well the routing has to happen in ASIC? Will all these routers need to exchange routes via OSPF/BGP in such case? (which of course with 2 routers doesn’t make sense)
on MT-A will solve the problem, as the packet will arrive at the interface, routing decision will be made, and the packet will be just forwarded by the CPU to the MT-B.
Is there any advantage to this if MT-A & MT-B are both within the trusted intranet and the separate class of addresses is only to enforce rules on VPN clients? The only advantage I see here is that the filtering decisions will be made centrally on MT-A, while if marked as “notrack” all filtering for VPN clients will have to be done on MT-B.
I’d say that when using multiple routers, one doesn’t run stateful firewall on all of them. One runs firewall (possibly dedicated hardware) only affecting traffic on interfaces where it can process both directions. Which means: firewall on MT-A has to be configured to only affect internet traffic. If some firewall needs to inspect traffic between client and server, then one needs another firewall on MT-B … or one needs routing subnet which forces traffic between different LAN subnets flow symetrically through involved routers.
But as you indicated in the last part of your last post, there are certainly performance benefits of not introducing additional routing subnet, specially so if routers involved emit ICMP redirect messages (not sure if ROS does) meaning subsequent packets will bypass default gateway in both directions. So yes, if traffic between client and server is not supposed to be firewalled, the only correct setting is to configure firewall on MT-A to ignore that traffic.
