I have a firewall issue that I haven't been able to figure out. I've been going back and forth with support via email on it, but I don't think they are understanding my question. Perhaps the community has some insight.

The question is, why are packets failing to match a connection state=established rule for an active flow?

In the below screenshot, I have highlighted rules 73, 74, and 79. Rule 79 (logging with prefix !!!) matches on source port 80 and 443, so this would typically match the return traffic of an active session. I expect no packets to match rule 79. I expect all return traffic with source port 80 and 443 to match rule 73 (and 74 if traffic is "related"). As you can see in the log however, a lot of traffic is matching rule 79. Curiously, almost all packets have a length of 52. This indicates to me that the packets are probably ACKs with no payload. This may be a curiosity, or it may be indicative that ACKs with no payload do not match the established=yes rule at 73.
The below screenshot shows that there is an established session where a packet did not match rule 73 and was logged. Please share your thoughts and ideas on this behavior.

Thanks

From a logical point of view, this behavior is absolutely reasonable as a connection needs an ACK for getting established state.
So it is not yet established when you log the packet as it still has to be forwarded from routers point of view.

As I never used outgoing firewall before, I am not sure, how the implementation works.
To be sure you can allow TCP Flag ACK (advanced) to local subnet.
But I do not know what happens if you drop the packet in rule 79. Two possibilities:
1. Connection never will get established because Router knows that the ACK will never reach the local computer
2. Although the first ACK is being dropped, Router will mark is as established (because of the ack). Local Client does not receive it and resend request. External server will reply again with the ACK and this time is already established. So connection will work but with a delay.

Thanks for your comment. I can assure you that these ACKs are packets from established sessions. They are not SYN-ACKs for sessions in the TCP handshake phase, i.e.: in the process of being established.

Since the previous screenshot showing established sessions may not have been clear, I just took another screenshot to unambiguously demonstrate that these ACK packets are part of an established flow and should match rule 73 instead of rule 79.

The screenshot was captured at 10:41:28. This screenshot shows 2 packets matched rule 79 which are timestamped 10:41:26 in the log. The session timeouts are 23:59:53 and 23:59:54, indicating the sessions have been established for at least 6 seconds, or since 10:41:22). The orig/reply byte counters also indicate that these are established traffic flows.

Theese two log entries have different ports, also connections show different ports.
Am I right assuming that each dst/src ip/port combination occurs only once?

Theese two log entries have different ports, also connections show different ports.
Am I right assuming that each dst/src ip/port combination occurs only once?

The 2 log entries showing the ACK packets correspond to the 2 established flows shown on the connections tab above. They are 2 distinct established sessions. They are unrelated to each other except that it is 2 unique TCP sessions between the same client and server.

Each source IP + source port and dest IP + dest port appear in the connections view once.

Did you try if they match connection-state=invalid? It could be some duplicate packets, but I'm not sure how exactly it works with them.

Did you try if they match connection-state=invalid? It could be some duplicate packets, but I'm not sure how exactly it works with them.

That sir is it. Thanks! The packets are considered invalid for some reason. I don't know why that never occurred to me.

Now to run a packet capture to see if I'm actually getting duplicates or not. I find it hard to believe I'm getting this many duplicate packets passing through the network.

And now we need some good soul who can explain all hows and whys, so that I don't have to research it myself. Because I'm interested, but lazy.

Have I got the perfect deal for you!

As an update to this thread, the packets in question don't seem to be duplicates. My packet capture consistently shows only one packet, which is being flagged by RouterOS as invalid.

This issue came up some time ago with our clients due to poor throughput. At the time it was discovered that packets were being dropped by the invalid rule that shouldn't have been, although I didn't have time to dig into it then. Recent circumstances have required me to dig into it further, so here I am. I'll reach out to support again and see if I can't catch some traction this time.

Support replied explaining invalids as follows:

The "INVALID" state packets are not only duplicate packets, it means that the packet can't be identified or it does not have determined state in connection tracking (usually - severe out-of-order packets, packets with wrong sequence/ack number, or in case of resource over usage on router).

That makes sense of course, but none of the conditions seem to apply in my case. Incidentally, I've been seeing this behavior (improper packet classification as invalid) for some time. It typically results in poor throughput and other various symptoms associated with packet loss. To work around the issue, my invalid rule drops only if the TCP FIN flag is set to work around the issue. However, I'm now at a point where I either need to figure out what is going on, or work around the issue permanently. I don't like workarounds, so here I am. I would love to hear from anyone else that is either dropping all packets flagged as invalid without issue, or anyone that is allowing invalids under certain conditions due to the behavior I'm reporting.

The router is a CCR1016 running at 1-3% CPU, so overutilization seems unlikely.

I ran another packet capture while monitoring the log for invalids. Attached is a packet capture of a very interactive TLS session where the outbound and inbound packets are exchanged one-for-one with no invalid SYN or ACK values.

For those playing along at home, here's some additional detail if you would like to do some analysis:

The attached pcap contains the flow for a session using TCP port 52505. The below screenshot shows the log events (prefixed with ***) indicating the packet is flagged as invalid by the router.

Here are the rules generating the log events:

Hi,
did you figure out what was going wrong?
I've run into a similar situation: I've exported my firewall rules on router A and imported them on router B. While everything seems to be fine on router A, on router B no packages get the established/related state. Package counters stay at 0. If I add a rule for invalid packages the counters for it increase further and further. The log show that these are ACK packages.
Also my fallback rule, which I've set to accept for now, shows a lot of ACK packages, that actually should have been catched before.
Router reboot had no effect at all.