PPPoE dynamic mangle rules are broken

RouterOS General
1

This is discussed somewhat in the thread “MTU problems”.

To summarize:

If in a PPPoE Profile, one has Change TCP MSS set, then two dynamic mangle rules are created:

One, which changes all outgoing syn packets from the PPPoE interface explicitly to the PPPoE max permissible value.

And one, which changes all incoming syn packets to the PPPoE max permissible value.

The problem is that these rules do this irregardless of what the previous values were.

This breaks communication with any site that has/needs a lower value.

The rules need to be altered to either use clamp to pmtu and/or add a TCP MSS statement (in winbox, this is shown in the advanced tab of the specific mangle rule) to only match packets which have an MSS larger than what the rule is going to change it to.

This problem specifically observed in 2.9.44

I discovered this while trying to prove that a remote site was stripping ICMP packet-to-big messages, and discovered in a packet capture:

Packet entering mikrotik:
https > 2463 [SYN, ACK] Seq=0 Ack=1 Win=4140 Len=0 MSS=1380
Same packet leaving mikrotik via PPPoE interface:
https > 2463 [SYN, ACK] Seq=0 Ack=1 Win=4140 Len=0 MSS=1452

2

An update: clamp to pmtu can’t be used reliably. It may not have enough information at the time this rule is applied to set the value correctly. Therefore use the tcp-mss match option.

Dynamic rule as currently generated by RouterOS when MTU is 1492:

chain=forward action=change-mss new-mss=1452 tcp-flags=syn in-interface= protocol=tcp

What it should be:

chain=forward action=change-mss new-mss=1452 tcp-flags=syn,!rst in-interface= protocol=tcp tcp-mss=1453-65535

And the same change needs to be made for the rule generated for the out-interface.

For now as a workaround, I’ve disabled changing MSS in the PPPoE profile, and have just applied this rule manually:

chain=forward action=change-mss new-mss=1452 tcp-flags=syn,!rst protocol=tcp tcp-mss=1453-65535

which has the side-effect of affecting all traffic going through the router in question.

3

To add to the above, here is:

An RFC: http://www.ietf.org/rfc/rfc879.txt

And a (fairly lengthly) white paper on the topic: http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

In particular, note the phrase (emphasis mine):

The sending host is > required to limit > the size of data in a single TCP segment to a value > less than or equal > to the MSS reported by the receiving host.

Which is the part that the RouterOS dynamic PPPoE mangle rule violates (by setting it to a value which is potentially larger than the remote host specifies).

The required behavior is specified/discussed in at least the following RFCs: 1191, 1063, 791, 793, and in particular, RFC 879.

4

This is much more easer to do with just simple reducing MTU and MRU on the server to the proper value.

Default change-mss rules are usable in 99% of situations, other 1% is for complex setup, where combination of VPN is used you need something different, so in this case you have to create setup manually.

I am not ready to suffer increase of the CPU load just because every rule will have to lookup one more condition.

5

I disagree, doing something wrong, really fast, is worse than doing it the right way.
The dynamic mss adjustment rules are wrong and breaks things.

There are plenty of hosts out on the internet that has very low mtu’s.
The problem is that mss is set unconditionally and not just lowered when it needs to, thereby breaking tcp connectivity between those hosts.
As an example; If mikrotik.com had mtu of 1400 and your connectivity was set to 1420, you wouldn’t be able to surf this site.

I would also like to add that clamp-to-pmtu option will in a typical case only work one way;
packets entering ethernet interface (mtu 1500) and exitting pppoe interface (mtu 1492), then mss will be 1452
but packets entering pppoe interface (mtu 1492) and exitting ethernet interface (mtu 1500), then mss will remain at whatever it was before (typically 1460).
Additionally clamp-to-pmtu works by doing an additional route lookup, which also slows the routers alot.
However this is often faster than doing hundreds of dynamic mss adjustment rules (as enabled in ppp profile).

ppp implementations on other platforms often implement mss adjustment as part of the ppp/ip driver and none of the above mentioned drawbacks.

6

This issue has been taken under considiration.

Dynamic PPPoE was designed for basic customers, if your PPP server have more than 50 clients at the same time we strongly advice you to disable dynamic mangle rules and create static rules.

This way you will be able to introduce the necesary mangle configuration and increase performance, by reducing number of mangle rules, at the same time.

7

Any ppp server with any number of clients can replace all dynamic mangle rules by this static rule setup:

  1. lets create jump to the new chain for all tcp syn packages
/ip firewall mangle add action=jump chain=forward jump-target=change-mss protocol=tcp tcp-flags=syn,!rst
  1. exclude all tcp syn packages that don’t need to be changed (basically packages that comes from all other interfaces except ppp interfaces). I have 4 interfaces (ether1,ether2,ether3,wlan1) and pppoe server is on ether3.
/ip firewall mangle add action=return chain=change-mss in-interface=ether1 
/ip firewall mangle add action=return chain=change-mss in-interface=ether2 
/ip firewall mangle add action=return chain=change-mss in-interface=wlan1
  1. change-mss to all ppp interfaces with only one rule
/ip firewall mangle add action=change-mss chain=change-mss new-mss=1452 protocol=tcp tcp-flags=syn tcp-mss=1453-65535
8

Your example only adjusts mss one way. Do you adjust mss on the client end as well?

9

In my case clients are end-customer PCs - only one way mss adjustment is necessary
When client will create TCP syn packet it will assign mss automatically from the directly connected ppp interface.

If you use VPN as a tunnel (between 2 networks), then you need to apply these rules on the client end also.

10

In that case, are you sure you need to adjust mss?

11

sten - It is just an example that might be useful to those why have this problem!

In my case both setups worked correctly, only one had ~300 dynamic mangle rules, second 5 mangle rules (obviously faster). And I had no problems applying suggested changes to this second setup, even if i don’t need them.