packets "escaping" marking in mangle

RouterOS General
1

Using ROS 6.27.

I’ve set up mangle to classify traffic in order to shape bandwidth in queue tree.

I see that some traffic (packets) that should be marked, actually aren’t, and cannot figure out why, so unless there are some “special” conditions under ros under which it doesn’t mark traffic otherwise specified to do so, there should be either a ROS, or L8 bug (most probably) :slight_smile:

these are the mangle rules:

/ip firewall mangle
# HTTP/S SSTP
add action=mark-connection chain=forward comment=HTTP/S connection-state=new dst-port=80,443,5228 new-connection-mark=HTTP protocol=tcp
add action=mark-packet chain=forward comment=HTTP-Ficheros connection-bytes=5000000-0 connection-mark=HTTP new-packet-mark=HTTP_PD passthrough=no protocol=tcp
add action=mark-packet chain=forward comment=HTTP-Trafico connection-mark=HTTP new-packet-mark=HTTP_P passthrough=no
add action=log chain=forward comment=HTTP/S connection-mark=no-mark log=yes log-prefix=ESCAPANDO port=80,443,5228 protocol=tcp

# Email related
add action=mark-connection chain=forward comment=CORREO connection-state=new dst-port=25,57,109,110,993,995,143,465,587 new-connection-mark=CORREO protocol=tcp
add action=mark-packet chain=forward connection-mark=CORREO new-packet-mark=CORREO_P passthrough=no

# Management
add action=mark-connection chain=forward comment=MGMT connection-state=new dst-port=22,7490,3389,5900,8291 new-connection-mark=MGMT protocol=tcp
add action=mark-packet chain=forward connection-mark=MGMT new-packet-mark=MGMT_P passthrough=no
add action=mark-connection chain=forward comment=Resto connection-state=new new-connection-mark=RESTO

# GRE
add action=mark-connection chain=forward comment=GRE connection-state=new new-connection-mark=VPN protocol=gre
add action=mark-packet chain=forward connection-mark=VPN new-packet-mark=VPN_P passthrough=no

# All the rest
add action=mark-packet chain=forward connection-mark=RESTO new-packet-mark=Resto_P passthrough=no

# Direct marking DNS and ICMP
add action=mark-packet chain=forward comment="Marcado directo" dst-port=53 new-packet-mark=DNSU passthrough=no protocol=udp
add action=mark-packet chain=forward new-packet-mark=DNSD passthrough=no protocol=udp src-port=53
add action=mark-packet chain=forward icmp-options=8 new-packet-mark=ICMP_U passthrough=no protocol=icmp
add action=mark-packet chain=forward icmp-options=0 new-packet-mark=ICMP_D passthrough=no protocol=icmp

# Check unmarked
add action=log chain=forward log=yes log-prefix=SINMARCA packet-mark=no-mark

However, while in firewall > connections all appear as marked, I’m seeing lots of unmarked packets that otherwise should according to mangle conditions, i.e. that match HTTP_P, HTTP_PD, CORREO_P, etc but get logged with SINMARCA.

using 1 wan inteface for access winbox
2

hello
Where do you see the packages are not market?
Usually in the filter to drop invalid packets do you have?

Show log after the addition of such a rule
add action=jump chain=forward connection-mark=HTTP jump-target=C1 packet-mark=!HTTP_P
add action=log chain=C1 log-prefix=NOT_MARKET_HTTP_CONNETCT packet-mark=!HTTP_PD

3

thanks for your reply vasilevkirill

I’m seeing it here (I setup just one for HTTP to check, but all kind of traffic has unmarked packets), in the last HTTP/S rule:

add action=log chain=forward comment=HTTP/S connection-mark=no-mark log=yes log-prefix=ESCAPANDO port=80,443,5228 protocol=tcp



firewall,info ESCAPANDO forward: in:WAN out:bridge1, src-mac aa:bb:cc:dd:ee:ff, proto TCP (ACK,PSH), 173.194.78.189:443->178.bbb.ccc.ddd:64690, len 109

and by the last mangle rule:

# Check unmarked
add action=log chain=forward log=yes log-prefix=SINMARCA packet-mark=no-mark



firewall,info SINMARCA forward: in:bridge1 out:WAN, src-mac bb:aa:cc:dd:ee:ff, proto TCP (RST), zzz.yyy.www.zzz:35842->80.239.137.106:443, len 40

There are unmarked packets that should on both directions, and all specified protocol/ports

4

Your problem is probably from using the forward chain.
Use prerouting - at least for the packet mark actions. This happens pretty much before everything except hotspot chains.

As for connection marking, prerouting catches all traffic to be forwarded, and all traffic TO the mikrotik.
Output catches those packets from the Mikrotik CPU going anywhere. (output traffic never goes through prerouting, IIRC)

I’ve seen some “clever dan” tricks where packets were marked in pre-routing, queued, and then re-marked and re-queued again with postrouting… but I think this was for ROS v5 and the same effects can be done without these great lengths being taken…

Look at some examples in the v6 packet flow diagram. You might see where the leaks happen.

5

I know the flow diagram, that’s whats puzzles me. Traffic not being marked was strictly flowing through the router; these were public IPs behind the router (PPPoE), going to the internet.

I know by marking in Prerouting I’ll be including input + forward, and marking in postrouting = output + forward.

But what I want to understand is why traffic that is specifically forwarded, doesn’t get connection-marked in forward, but gets marked in prerouting.

If marking in postrouting, it won`t show in forward, that’s to be expected as the packet already went through the forward chain.

But to confuse me even more, even being connection-marked in prerouting, and seemingly mostly later packet-marked in forward, why still doesn’t appear as packet-marked later in forward, for example:

The last rule, doing the logging, has all “email ports” set in “any port” field; BUHO = WAN interface, bridge1= LAN interface

firewall,info ESCAPANDO forward: in:BUHO out:bridge1, src-mac xx:xx:xx:xx:xx:xx, proto TCP (SYN), 93.93.69.53:56800->178.156.aa.bbb:25, len 60

178.156.aa.bbb is a public IP on the LAN side, so AFAIK this is strictly forwading; the firewall lets this traffic pass through.

What am I missing?

6

Well, you have lots and lots of criteria on the rule, every single one must be true in order for the action to be taken.
The strangest is the fact that it only matches SYN. The src-mac might be the problem as well… I haven’t looked at the flow diagram yet, but this rule could possibly be getting checked after the Mikrotik’s bridge interface’s MAC address is put on the packet for egress onto the LAN. . .

The source port is also very likely not to match unless the smtp client on that host is configured to always use port 56800 when connecting to a remote SMTP server. . .

7

the src-mac is edited out, sorry, in the original log the mac is the right mac.

What do you mean with the source port being 56800? In matching any port for the log I only specify the usual email destination related ports, because there are servers “inside” and outside this router.

8

I knew it was edited out, but I misunderstood the post. I was thinking of that information as if it were a sumarized output of the rules, not a log entry. Thus, I thought your rule was only matching if the source was from the IP shown, from the port shown, etc.

Your screenshot of the mangle table doesn’t show everything, so if you could, run this command on the terminal, and post the results here:
/ip firewall export compact

Thanks!

9

ok - I’ve had a chance to look through your rules in my own copy of winbox, and now your screenshot makes more sense - rule 7 was just there to catch un-marked SMTP connections as they’re forwarded, and log the info…

The thing is that when you consider inbound / outbound / src-port / dst-port, you actually have 4 possible combinations, but only two rules… this is basically why some packets are being missed (in a nutshell)

I assume that interface BUHO is the wan interface, and bridge1 is the LAN…

So consider the packet you posted the log sample for:
in:BUHO
out:bridge1
93.93.69.53:56800->178.156.aa.bbb:25

Now think about this packet in the prerouting chain.
It’s not going to match rule 4 because rule 4 examines the SOURCE port. (which is 56800 on this packet)
Then the packet will go along and hit the forward chain… still having not yet been connection marked.
Thus rule 7 matches your packet and it is logged.

Consider what happens AFTER the log, though…
rule 18 marks all remaining unmarked connections as RESTO

Then the packet will go through post-routing chain…
Rule 22 will replace the connection mark RESTO with the connection mark CORREO
(because you’re using the connection state and not marking state, this rule has a chance to re-do the mark)
This first SYN packet will exit the Mikrotik having not been PACKET marked, but the CONNECTION is now marked CORRECO. The log rule you used to test this gave you a false impression because it was checking for the mark before the mark was taking place.

Now if 178.156.aa.bbb was to originate an SMTP connection to some outside host, you’re still going to see that with your log rule #7 because the packet will not get caught by rule 27 in prerouting chaing, because the destination port is 25, not the source port. (most services don’t use the same port when they ORIGINATE the connection). It follows the same steps as my first example above.

In fact - rule 27 is absolutely useless (as written) and will probably never match anything. All of the listed services sit there and listen for connections on those ports - they don’t START their own new connections. The only exception is SMTP, but when SMTP starts a connection, it uses a non-privileged port as the source (not port 25)… so the very first packet in an email connection is NEVER going to be FROM one of those ports listed. The very first reply packet to these connections is indeed going to be TO one of those ports, but then the connection state won’t be NEW anymore, it’s going to be ESTABLISHED, so the rule won’t match then either.

If you change rule 27 to look at the DST port, and not the SRC port, it will match as you want.
(The same holds for rule 0, by the way)

In general, when marking new connections, you’re going to want to look at the destination port only. The source port really doesn’t matter. When replies come from the servers, the connections will have already been marked, and those packets are going to get packet-marked based on their connection marks (not port numbers) anyway.

10

Lots of food for thought to chew on for me :slight_smile:

I started using only the forward chain as being for QoS I wanted to have separate control over input/output traffic, but started tinkering later with prerouting/postrouting after seeing packets went by without being marked.

At first I didn’t specify input interface, used forward chain, connection=new and destination ports, and still some packets weren’t marked. Then changed to prerouting/postrouting with different dst-port/src-port permutations, still unmarked packets.

I have changed now the connection marking rules, still using the prerouting chain, but instead of new connections the criteria now is unmarked packets, and destination ports. Still, seeing unmarked packets.

You mentioned these could be forged or stale connection packets; in fact I’ve seen these come from connections being correctly marked but in Unreplied state, HTTP mostly, initiated both from the outside and inside. There are a very small percentage of unmarked HTTP packets, like 50Kib per each 150MiB; on email the percentage is way smaller, like 60KiB per each 5GiB, so your reasoning matches closely…

Apart from forged packets from the outside I have confirmed, regarding “legit” connections, could it be a device in my network is inducing these anomalies due to a flaw?