Invalid forward packets between bridge VLAN

[mickey]

Hello, I have a problem with my VLAN setup. I have two devices connected to Eth8 (Vlan99) (trunk port) and Eth9 (vlan10, vlan99) (hybrid port). Ping between (vlan99<->vlan10) is working fine in both ways, but when I do HTTP request from Eth8 (vlan99) to Eth9(vlan10) then connection hangs for a few seconds and counters on FW rule for invalid forward is increasing, after that lag finally I get response . Reversed request from Eth9(vlan10) to Eth8 (vlan99) have no problems.

My config:
config.rsc (5.16 KB)
Diagram with my problem

Drop invalid forward FW rule:

 9    ;;;  drop invalid
      chain=forward action=drop connection-state=invalid log=no log-prefix="invalid"

Log:

17:56:28 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK), 192.168.99.243:65375->10.0.10.3:8080, len 52
17:56:28 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:28 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:29 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:29 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:29 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:29 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK), 192.168.99.243:65375->10.0.10.3:8080, len 52
17:56:30 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:30 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131
17:56:31 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK), 192.168.99.243:65375->10.0.10.3:8080, len 52
17:56:32 firewall,info invalid forward: in:BASE_VLAN out:HOME_LAB_VLAN, src-mac 30:xx:xx:xx:xx:xx, proto TCP (ACK,PSH), 192.168.99.243:65375->10.0.10.3:8080, len 131

[Zacharias]

I don’t understand what is the point of PVID 10 on your Trunk port ether9… Is there any type of untagged traffic generated from the Switch ( on the left ) where ether8 port belongs that you want to make member of Vlan10 on the right switch ?
Even if there is a reason for doing that ( there might be cases ), port ether8 on left switch does not accept untagged traffic. So what would egress from ether9 on right switch that belonged to Vlan10, would at the end be dropped from left switch since it does not accept any untagged traffic… There are extra parameters you should take into account when doing that…

Also, on what ports of the switch (left) and router(right) your devices are connected ? You only mention the Trunk ports ether8 and ether9…

[mkx]

My (educated) guess: routing triangle.

I’ll call left PC “ether8” and right PC ether9.

So when ether9 connects ether8, it’ll use 192.168.99.246 as source address (unless forced otherwise) because it’ll figure out that destination address is directly connected (same IP subbet) and should thus be really cheaply accessed using this source address. And ether8 replies happily while router acts as a switch and firewall doesn’t apply at all (router’s L3 stack doesn’t see those packets at all).

Now when ether8 connects ether9 at 10.0.10.3 (TCP SYN), it uses router as gateway and connection tracking machinery notices start of new connection. Router then uses vlan 10 to push packet towards ether9. When packet arrives at ether9, PC constructs reply packet (TCP SYNACK) and figures that it can send it back via vlan99 because return address is directly connected there. And ether8 receives reply, constructs third step of handshake (TCP ACK) and sends it via router (because of dst address). Connection tracking machinery on router now sees packet that doesn’t conform to connection state (which is completed step 1 of standard 3 step handshake because connection tracking machinery didn’t see the second one - SYNACK) and marks packet as invalid, firewall then drops it.

Solution? Reconsider need for ether9 to be member of multiple vlans. Or avoid using vlan10 address when connecting from another vlan99 host.

[mickey]

My (educated) guess: routing triangle.

I’ll call left PC “ether8” and right PC ether9.

So when ether9 connects ether8, it’ll use 192.168.99.246 as source address (unless forced otherwise) because it’ll figure out that destination address is directly connected (same IP subbet) and should thus be really cheaply accessed using this source address. And ether8 replies happily while router acts as a switch and firewall doesn’t apply at all (router’s L3 stack doesn’t see those packets at all).

Now when ether8 connects ether9 at 10.0.10.3 (TCP SYN), it uses router as gateway and connection tracking machinery notices start of new connection. Router then uses vlan 10 to push packet towards ether9. When packet arrives at ether9, PC constructs reply packet (TCP SYNACK) and figures that it can send it back via vlan99 because return address is directly connected there. And ether8 receives reply, constructs third step of handshake (TCP ACK) and sends it via router (because of dst address). Connection tracking machinery on router now sees packet that doesn’t conform to connection state (which is completed step 1 of standard 3 step handshake because connection tracking machinery didn’t see the second one - SYNACK) and marks packet as invalid, firewall then drops it.

Solution? Reconsider need for ether9 to be member of multiple vlans. Or avoid using vlan10 address when connecting from another vlan99 host.

Thank you for your detailed answer. I didn’t know that the response will go with a different interface, which after your explanation makes sense (shortest route to src subnet). I’ve managed to trick this behavior with ip tables but I’dont want to maintain such a configuration in my home network. I will reconsider usage of multiple vlans on ether9. Thanks again

For future reference, iptables config which overrides “main” table
based on this solution: https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming

~ $  echo 10 vlan10 >> /etc/iproute2/rt_tables
255	local
254	main
253	default
10 vlan10

~ $  ip rule add from 10.0.10.3 table vlan10 prio 10
0:	from all lookup local
1:	from 10.0.10.3 lookup vlan10
32766:	from all lookup main
32767:	from all lookup default

~ $  ip route add default via 10.0.10.1 dev eth0 table vlan10
default via 10.0.10.1 dev eth

~$ ip route show table main
default via 10.0.10.1 dev eth0
10.0.10.0/24 dev eth0 proto kernel scope link src 10.0.10.3
192.168.99.0/24 dev eth0.99 proto kernel scope link src 192.168.99.246

Request stays in L3 (same subnet)

Request routing (different subnet)
Before iptables

After iptables

[Zacharias]

@mkx, but isn’t that an example of inter vlan routing ?
If a device on e.g. vlan10 tries to reach a device on vlan20, and there is a Router that can route those vlans in between , then vlan10 device will successfully communicate with vlan20 device and vice versa .
The way you explained it, i understand that the router should drop the connection as invalid because the router will send a reply to vlan10 using the vlan10 interface and not vlan20…
So, what do i miss here? What differs from the inter vlan routing here ?

PC constructs reply packet (TCP SYNACK) and figures that it can send it back via vlan99

I could only understand that if the pc connected to ether9 listens to tagged traffic on vlan99 itself… and knows that it can reach the other host through Layer2.

[mkx]

@mkx, but isn’t that an example of inter vlan routing ?
If a device on e.g. vlan10 tries to reach a device on vlan20, and there is a Router that can route those vlans in between , then vlan10 device will successfully communicate with vlan20 device and vice versa .

Routing is fine. The problem is that ether9 in use case by @mickey is routing between vlan10 and vlan99 on its own (even if only own traffic). Which is not wrong per se, the problem is that stateful firewall (such as the one in ROS) needs to see packets in both directions to function properly. With routing triangles it can miss one direction and thus connection state is not correct any more and that upsets the firewall.
One possibility is to simply add a firewall raw rule matching the “offending” packets (seen only in one direction) with action=notrack … which will have two benefits in this particular case: a) it’ll save some CPU cycles for connection tracking machinery (it won’t have to classify those packets) and b) if firewall filter rule set includes the rule with chain=forward connection-state=untracked action=accept (default rule set does include rule which covers this) firewall will no longer drop these packets. But it does effectively remove firewall from the path between the two VLANs which might not be what network admin wants, in which case one has to get rid of routing triangles.

So it all boils down to understand both own network topology (to identify possible routing triangles) and packet flow inside router (to construct appropriate firewall and routing rules).

[mkx]

PC constructs reply packet (TCP SYNACK) and figures that it can send it back via vlan99

I could only understand that if the pc connected to ether9 listens to tagged traffic on vlan99 itself… and knows that it can reach the other host through Layer2.

If you observe diagram from original post, you’ll see that ether9 host indeed listens to vlan 10 untagged and vlan 99 tagged (that’s the eth0**.99** part of config; that’s linux-way of creating vlan interface).

[Zacharias]

Thanks @mkx, now i do understand…