Dear guru,

1. can anyone explain when we should use mark-connection + mark-packet and mark-packet only?
Is there any performance different between the two method?

Note: All of the marking will use to do shaping with Queue tree or Simple Queue.

2. Another thing, if we do shaping with mangle + simple queue and mangle + queue tree.
Is there any performance different between the two method?

Note: we don't do priority.

Thanks for reading,

1. There are performance gains wen using connection mark+packet mark.

If only packet mark is used, every packet that goes through firewall is processed by each rule until matched. If connection mark is used, then firewall compares assigned connection marks and only if they match packet is processed by firewall rule.

2. Queue trees are faster.

Is connection-mark is unidirectional or bi-directional?

example: /ip firewall mangle add chain=forward src-address=1.1.1.1 new-connection-mark=conn-mark
if 1.1.1.1 downloads a file from outside, the download connection(outside -> 1.1.1.1) is mark as conn-mark.
if 1.1.1.1 ping to outside, the icmp-request connection (1.1.1.1 -> outside) is as conn-mark too.

is it correct?

If you mar connection then you have to mark connection only once (with connection-type=new).
Connection is bidirectional.

if so, download and upload traffic need to mark one time only?

example: /ip firewall mangle add chain=forward src-address=1.1.1.1 new-connection-mark=conn-mark?

conn-mark will caught both download and upload of 1.1.1.1.

Does it work for all protocol like TCP, UDP, ICMP, etc?

Yes, the connection mark is present on all packets of a connection - and connections are bidirectional and include upstream and downstream traffic.

Connection marks work with all connections that are recognized as stateful. That includes all TCP and the vast majority of UDP and ICMP (basically everything that is bidirectional, and within the timing parameters of the limits set in "/ip firewall connection-tracking").

For performance, to me it breaks down like this: if you only need to do one thing to a packet and can recognize it very easily by one comparison (e.g. just the source address, or the in-interface) you might as well just apply the marks to the packet directly and not mark the connection. Whether I'm comparing the incoming interface or the connection mark for each packet is irrelevant.

When you're comparing more than just one property or need to change several things on a packet (e.g. apply a packet mark, a routing mark, change DSCP and TTL) it's better to just go off the connection mark. Also, ensure you limit the rule that sets the connection mark via "connection-mark=no-mark"- otherwise you're remarking the connection with EVERY packet of the connection, which kills all the performance gains you might otherwise see.

Below some code.

This applies a connection mark based on just one parameter and then sets a packet mark based on the connection mark: That is wasteful to me - the below does the same thing in one fewer rule, and compares just as many parameters: The below works without connection marks and has to compare four packet parameters for each packet: That's relatively expensive to do. The adjusted below version applies a connection mark based on several parameters that have to be compared, and only does so for connections that don't have a mark yet (so one packet per connection), and then all subsequent packets can be marked based on just one comparison. However, each packet still has to first be determined to not have a connection mark yet and will be checked against the first rule until it doesn't match: Here's where I have a question for the devs: is the 'connection-mark=no-mark' condition always evaluated first among checks for a packet, and does the comparison process then immediately jump out of that rule by short circuit logic?
Or does each packet get evaluated against ALL of the the comparisons, or some of them because they get evaluated by the code before the connection mark is? I'd imagine it's short circuit logic, but would like to know if connection marks always get checked first. If they aren't and, for example, are evaluated after source and destination addresses but before protocol and port then the above would still check every single packet of a flow against three comparisons (source address, destination address, then connection state that doesn't match and the rule doesn't fire) before going to the second rule that then marks the packet. That would lead to a total of four comparisons before a packet mark is applied (source and destination address and connection mark in the first rule, connection mark in the second rule), which isn't more efficient to do than the original version without connection marks.

Though I guess you could ensure to optimize processing speed like below where packets that have a connection mark are immediately marked and then bail out via passthrough=no, and then mark connections on remaining packets (first packets of the connections), and then apply the same rule again to finally mark that first packet:
That way packets in a connection that have been marked require always just one comparison before the packets themselves can be marked. Or is there a more optimal way to write that ruleset?

Thanks for confirmation.

If we do as below:
1. Above config will limit download and upload together meaning upload and download will share 256k together. is it correct ?
2. If I want to separate download and upload to different queue, how do we write the code with connection-mark ?
3. If I use PCQ to limit, will PCQ shape download and upload separately as PCQ has classifier to identify upload and download ?

Note: I am using web proxy(redirect 80 to 8080) and SRC-NAT on this box.

1. Above config will limit download and upload together meaning upload and download will share 256k together. is it correct ?
2. If I want to separate download and upload to different queue, how do we write the code with connection-mark ?
3. If I use PCQ to limit, will PCQ shape download and upload separately as PCQ has classifier to identify upload and download ?

Note: I am using web proxy(redirect 80 to 8080) and SRC-NAT on this box.

1. Yes.
2. and 3. see http://wiki.mikrotik.com/wiki/Manual:Qu ... r_of_Users - though that only uses packet marks in the example it could of course be based on connection marks triggering packet marks.

The web proxy and NAT don't necessarily interfere with this - global-in happens after destination NAT (after source NAT is undone), global-out happens before source NAT. Usually that makes things work because the source and destination IP addresses are what you would expect them to be.

Thanks for the link. It look simple to do but It make me confuse now

Anyway, Can we go more detail about the mangle and shaping?
Based on IP flow below:
Both upload and download traffic will pass through

1. Mangle at Prerouting and shaping at Global-in
or
2. Mangle at Postrouting and shaping at Global-out

So, upload or download can be shaped in Global-in or Global-out??
Base on http://wiki.mikrotik.com/wiki/Manual:Qu ... r_of_Users, why we need to shape download at Global-In and upload at Global-Out?

In RouterOS, these hierarchical structures can be attached at 4 different places:

global-in: represents all the input interfaces in general (INGRESS queue). Queues attached to global-in apply to traffic that is received by the router before the packet filtering
global-out: represents all the output interfaces in general (EGRESS queue).
global-total: represents all input and output interfaces together (in other words it is aggregation of global-in and global-out). Used in case when customers have single limit for both, upload and download.
<interface name>: - represents one particular outgoing interface. Only traffic that is designated to go out via this interface will pass this HTB queue.

http://wiki.mikrotik.com/wiki/Manual:Queue

If you use global-in for both upload and download, you cannot include traffic from the router to the client (proxied traffic, whatever) because global-in happens in prerouting, which traffic generated by the router never sees (that goes straight in the output chain, and there is no ingress interface). You can rate limit client upload in total-in because that will contain all traffic from the client, and it makes sense to do so because it's the earliest point at which you can throw the traffic above the rate away. Why process traffic above the rate at all? Again, with downloads you usually don't want global-in firing on the WAN interface because it might not be all traffic to the client.
More importantly, the only marking facility before global-in is prerouting mangle. The packet flow is prerouting mangle, prerouting filter, destination NAT, global-in/global-total. Destination NAT isn't only used for port forwarding, it is also where source NAT is undone for return traffic. Client contacts a server, gets source NAT'd on the WAN interface to the public IP of the router. The server responds to that public IP and puts it into the destination IP header field. This is rewritten to the clients actual IP address in destination NAT (and is how the router decides the packet will go into the forward chain, and not the input chain). Prerouting mangle is before that, so you do not yet have the real destination IP address of the packet, and cannot make any marking decisions based on destination address. For download traffic (destined to the client) usually the destination IP address is important, so prerouting mangle is a bad fit and the packet is marked in forward or postrouting instead and rate limited in global-out. You can't even mark based on out-interface, because no routing decision has been made yet (you're still in prerouting, after all). However, global-in happens after destination NAT so the dst-address classifier of PCQ could make the right decision.
So in summary: if you don't need to look at the destination address or out-interface to make a packet marking decision, and if you don't care to include download traffic from the router to the client, you can use global-in to rate limit download to the client. This is true either for very, very simple deployment scenarios (everyone gets the same rate limit, there are no user classes, and you don't care if you rate limit management traffic to the router), or for transit routers where NAT has already been undone and the router provides no services other than transit (forwarding traffic). Because the majority of use cases doesn't fit those situations the wiki will show download rate limited in global-out.

I'm trying to do outbound mangle/queue QoS. Based on what is stated above, would this be the proper settings?

For http/https traffic, I want everything outbound under 512k to go to a higher queue tree priority, and everything over that to go to a lower "download" priority. I'm only trying to queue outbound (lan-wan) traffic...

I've set connection marking to mark connections that have no marks and then rise above 512k... mangle rules below:
The "http" queue is higher than the "download" queue, naturally. Using queue trees. Can anyone make it clear if this is correct? It seems to be working, but I wonder if there's any side-effects I may be missing...

1. There are performance gains wen using connection mark+packet mark.

If only packet mark is used, every packet that goes through firewall is processed by each rule until matched. If connection mark is used, then firewall compares assigned connection marks and only if they match packet is processed by firewall rule.

Efficacy of connection mark varies by application. Based on my tests (ROSv4.5) PCC marking packet directly - instead of marking connection than packet based on connection - is faster, about 50% faster.

Based on my tests (ROSv4.5) PCC marking packet directly - instead of marking connection than packet based on connection - is faster, about 50% faster.

are you sure you marked each connection only once, not for each packet? =)

If you use SNAT in the same router with shaper, then you may - It works fine.

Based on my tests (ROSv4.5) PCC marking packet directly - instead of marking connection than packet based on connection - is faster, about 50% faster.

are you sure you marked each connection only once, not for each packet? =)

yes, i only send a packet to pcc when connection-mark is no mark. all packets are it is packed marked based on conn mark. all action mark... are passthrough no


if you think about it, it does make sense that conn marking is more expensive for web browsing traffic- where new connections are plenty and are short lived.