VPLS over GRE and MTU Help Needed

RouterOS Forwarding Protocols
1

Hello all,

So here’s the deal. I’m trying to setup a downstream VPLS link to provide to my sites. I’ve got this setup however, my problem regards a failover configuration. Currently I have the following topology.

ISP 1 - L2 Interface ISP 2 - DIA L3 Interface (Cable Modem)
\ /
\ /
-------- MikroTik CCR1009 ------
|
VPLS
|
Windows PC

Everything runs over the primary ISP 1 L2 Interface, unless that interface fails, in which case everything is supposed to fail over to the L3 cable modem. This works in theory. OSPF, MPLS, VPLS interface, etc all failover to the GRE tunnel properly.

The problem comes in when the fail over occurs. When this happens the hosts that are downstream from the VPLS interfaces are having a very hard time communicating with each other. Obviously the GRE tunnel does not have the same MTU as the L2 interface. Therefore, I believe fragmentation is occurring when this happens.

I have attempted to resolve this by forcing the MTU down on the VPLS pseudo-wires to be below the GRE MTU. I have also modified the MPLS MTU to below this threshold to attempt to resolve this. This does seem to help a bit, but I’m still running into a problem where the workstations downstream from the VPLS link are still attempting to send at an 1500 MTU. Transfers using Windows file explorer work fine in this configuration, however, I cannot send data via an FTP server between hosts on the VPLS mesh.

I’ve also been playing around with the tcp-mss clamping features and have that set to change the MSS to 1360 (which I suppose would force an mtu of 1400). I don’t know if this is low enough or not.

I’m also doing transport-mode IPSEC encryption on the GRE tunnels and I’m not sure what kind of additional overhead this may be causing to occur as well.

Here are the pertinent pieces of the configurations:

MPLS Interface:
mpls interface
set [ find default=yes ] mpls-mtu=1416

GRE Interface:
/interface gre
add comment=“GRE Tunnel to p1-edge-a.” !keepalive local-address=5.5.5.5 mtu=1426 name=
Tunnel0 remote-address=.1.1.1.1
add comment=“GRE Tunnel to p1-edge-b.” !keepalive local-address=5.5.5.5 mtu=1426 name=
Tunnel1 remote-address=1.1.1.2

VPLS:
/interface vpls cisco-bgp-vpls
add bridge=VPLS-1102 bridge-horizon=1 export-route-targets=2:1 import-route-targets=2:1
l2router-id=12.12.12.12 name=VPLS-1102 pw-mtu=1408 route-distinguisher=2:1 vpls-id=2:1

/interface bridge
add name=VPLS-1102 protocol-mode=none
/interface bridge port
add bridge=VPLS-1102 interface=ether3


All routers in this configuration are CCR1009-8G-1S-1S+ running code version 6.27

Ideally I would want the customers downstream from the links to not have to do any reconfiguration on their routers to support this link (essentially we’ll be taking over the CPE duties for them).

Any help with this problem would be very greatly appreciated. I will continue to work on the solution myself and post if I find any additional information. I’m hoping someone out there has seen something like this and can help me with resolving this problem.

Thank you all in advance!

2

So I have a little more information… Or at least I think this is what the problem is… It appears that the VPLS interface is not responding with a ICMP code 3:4 (DF bit set but fragmentation needed). Can anyone think of a way to enable the interface to respond with this if the MTU has been over-provisioned from the client.

Thanks!

3

So I think I figured this one out…

Here’s the important pieces of the configuration:

First make sure the bridge is configured to use the ip-firewall:

/interface bridge settings
set use-ip-firewall=yes

Next, make sure your MPLS and VPLS MTUs are set properly (Leave VPLS at default 1500):

/mpls interface
set [ find default=yes ] mpls-mtu=1526

Configure this firewall mangle rule to force the clients to use an MSS at 1360 (MTU of 1400 effectively):

add action=change-mss chain=postrouting comment="Clamp MSS to 1360." new-mss=1360 protocol=\
    tcp tcp-flags=syn

Once I did all of this the configuration seems to work properly. More testing is going to be needed of course.


For all you MPLS/VPLS Purists out there…
Of course I am aware that this is not an ideal configuration. In fact it will require a rewrite of my QoS and Queue trees to ensure that the proper max-limits are set on my GRE Tunnel interfaces. And this configuration will also introduce a large amount of fragmentation on the network so I wouldn’t recommend it for full production and for running on it permanently. However, my whole point in doing this is to use these fail over links only in the event of an emergency situation in which the primary layer 2 link has failed for some reason. Therefore providing a diverse carrier configuration for my customers. So, given that this is a fail over line and we are specifically not giving the customers any type of SLA beyond making sure that it’s available in the event of an emergency, there is no need to ensure that my configuration can fully support both paths.

In other words, this is VPLS with a redundant “VPN” seamless fail over.