Kinda in a bind... having an mpls issue where when we are turning up on portion of our network the bandwidth goes from 300Mb to 2Mb instantly when a router in path has mpls on. MTU sizes all set and tested with ping and DF flag across paths.

Any ideas? Thanks for your time...

Bit more info:
ROS 6.6 // involves CCR and i386 routers
MTU set 2000 all interfaces
L2MTU set 2020 all interfaces
Radios and such between routers all support jumbo frames (licensed SAF and airfiber links)

Scott

What size frames are you actually running through those links?
Are the frames actually 2k, because it sounds like you dont have MTUs configured right.

Whats your MTU, L2MTU, and MPLS MTUs at?
Are you running VPLS?

What size frames are you really sending through links?

Hello Tomaskir.

We've got the interface MTU set to 2000, The L2MTU set to 2020. And the MPLS MTU set to 2000 on all devices. On the x86 routers which don't show the L2MTU we've just set the interface MTU. And the MPLS MTU.

The actual frames going over the links should be ~1508 (we set 2000 to allow VPLS in the future) as the interfaces feeding into the MPLS network are 1500. And this is just running on our core. I've attempted a packetsniff however MPLS switched traffic is not displayed. Nor does Torch show the traffic. Before we turn up each link, We verify we can ping 2000byte requests over the link in question with DF set. And they do indeed work. I've also tried lowering the MPLS MTU to 1508 (Default) and we have the same issue.

Here's a test we ran.

Router 1 > Licensed Wireless Link > Router 2 > Rest of network.

Enabling MPLS between router 1 and router 2 took an immediate traffic hit. The traffic on the interface between router 1 and 2 dropped >50%. And winbox also started reporting slowly.

Router 1 is a Core i5 maxwave router.
Router 2 is a Atom Based Maxwave Router. Tested with both 6.2 and 6.6.

Here is the relevant configs on both. Note, The LDP Interfaces are disabled right now since it causes issues.

Router 1. (5.26
/mpls
set dynamic-label-range=16-1048575 propagate-ttl=yes
/mpls interface
set [ find default=yes ] disabled=no interface=all mpls-mtu=2000
/mpls ldp
set distribute-for-default-route=no enabled=yes hop-limit=255 loop-detect=no lsr-id=10.255.255.7 path-vector-limit=255 transport-address=10.255.255.7 use-explicit-null=no
/mpls ldp interface
add accept-dynamic-neighbors=yes disabled=yes hello-interval=5s hold-time=15s interface=sfp4 transport-address=0.0.0.0
/interface
set 11 arp=enabled auto-negotiation=yes cable-settings=default comment="XXXX" disable-running-check=yes disabled=no full-duplex=yes mac-address=00:12:C0:XX:XX:XX mtu=\
2000 name=sfp4 speed=1Gbps

ROUTER 2 (6.2 so exports are more limited.)
/mpls interface
set [ find default=yes ] mpls-mtu=2000
/mpls ldp
set enabled=yes lsr-id=10.255.255.6 transport-address=10.255.255.6
/mpls ldp interface
add disabled=yes interface=ether2
add disabled=yes interface=ether1
/interface ethernet
set 0 mtu=2000

Your configs are good, the problem is the x86 machines that do not report L2MTU.

If L2MTU is not reported by an interface, RouterOS assumes that L2MTU=MTU. But since you are adding MPLS tags on top, you need L2MTU bigger then MTU.
But you cant configure L2MTU since the card doesnt report it, hence you run into an MTU problem.
Its a known problem with cheap NICs and RouterOS on x86.

In your particular case however, due to how MPLS works, and MPLS PHP, this should not show up yet, (on your single hop test). It will only be a problem for MPLS frames traversing multiple MPLS switched hops.
But still, you should not touch interface L3 MTU (should be 1500 over the whole network), and only configure L2MTU and MPLS MTU. Your problems most likely stem from actually changing the L3 MTU (just MTU for Mikrotik) to 2000, and being ununiform in your network.
You probably have something somewhere that breaks PMTUD.

Look at pages 18-19 and 45-49 of my MPLS presentation on this year's US MUM
http://mum.mikrotik.com/presentations/US13/kirnak.pdf

Ok. So Reading the slides, I would understand that the interfaces which do not support an L2MTU would need to be replaced? What about X86 interfaces that DO show an L2MTU but just do not allow you to change it (Grayed out, says 9014). If it were the case that it was dropping packets >1500 wouldn't this also drop my ping packets at 2000?

In testing. In another segment of our network. I turned up MPLS between two cloud cores. MTU 2000 L2 2020 MPLS MTU 2000 and it didn't seem to cause the same issues. So I think you're onto something here..

If L2 MTU is reported, then that interface is fine. It can be greyed out and not-changeble, as long as its there, its OK.
As for interfaces that do not report L2MTU, you have to change them if you want MPLS on those interfaces.

As I mentioned before, ideally, your L3 MTU would be uniform at 1500 across the whole network, unless you really are using jumbo packets and really routing them around.
Then increase MPLS MTU and L2 MTU according to what you need.

Example: full size frame (1500) inside a PPPoE session carried by a VPLS tunnel inside of a vlan; the MTUs are covered on the page 19.
In that case:
ethernet MTU 1500, L2 MTU 1534
vlan interface MTU 1500, L2 MTU 1530
MPLS MTU 1530
VPLS interface MTU 1500, L2MTU 1508

Notice how L3 MTU is uniform at 1500, since a routed packet will never be larger then 1500.
L2 MTUs and MPLS MTUs are however large enough to account for all needed headers and tags.

Hi,

most probably Tomaskir is right, and it is an MTU issue on your x86.
Anyway, I thought I'd throw in another possibility:
In earlier versions, setting the MPLS-MTU for "all" interfaces did not work.
Since then, I always declare each one of the interfaces.
Instead of use Hope this helps.

saludos
Bernardo

Thanks for those who have taken the time to help out - we appreciate it.

Ok, so we have made lots of progress. Most of which involves setting up lots of equipment in the lab to replicate out network and we were able to reproduce the problem and find the solution in the lab, which is excellent - love it when that happens.

Boils down to x86 router does not work taking IP traffic into the MPLS network with version 6.6 any way - maybe all v6 ROS. Swap out x86 with routerboard (any model) and MPLS works. Or, leave x86 in the mix and downgrade to ROS 5.26 and it also works. We also had issues with some long-standing EoIP tunnels between BGP routers that ROS 6.6 made unstable. I was hoping we could roll forward to ROS 6.6 but its not happening except on CCR hardware we don't have a choice with. Any way - the couple of x86 boxes were the problem and only one traffic direction and only with ROS 6.6.

Will provide more info once all is said and done.

Scott

Now I know why everyone was saying - its probably an MTU problem no matter what doesn't work in MPLS. Because it probably is

Take note: Don't buy Tripplite fiber media converters - they only support 1504 MTU. And x86 routers just don't play well with MPLS. Two lessons learned so far...

Thanks

Hey man, thats just how it is with MPLS, and thats why I started asking questions about MTU right away.

Good to hear you got everything working

I have an issue on an x86 mikrotik router that DID report an L2MTU of 9014 but still didn't work. It's an intel pci-e card. It transmitted the larger frames but doesn't receive them. I have bought the rb44ge card to see if that works but haven't installed it yet. Hoping a mikrotik supported card works.

I have an issue on an x86 mikrotik router that DID report an L2MTU of 9014 but still didn't work. It's an intel pci-e card. It transmitted the larger frames but doesn't receive them. I have bought the rb44ge card to see if that works but haven't installed it yet. Hoping a mikrotik supported card works.

Remember that its not only about the card. Was there a VLAN on top of the ehternet interface? Was there a LACP trunk, or any other kind of bonding interface? VRRP?

All of these have to report L2MTU correctly, for example, currently, bonding interfaces are unusable in RouterOS with MPLS, since the bonding interface doesnt report a L2MTU, even if the underlying ethernet cards do. (page 49 of my presentation I linked earlier)

I have an issue on an x86 mikrotik router that DID report an L2MTU of 9014 but still didn't work. It's an intel pci-e card. It transmitted the larger frames but doesn't receive them. I have bought the rb44ge card to see if that works but haven't installed it yet. Hoping a mikrotik supported card works.

Remember that its not only about the card. Was there a VLAN on top of the ehternet interface? Was there a LACP trunk, or any other kind of bonding interface? VRRP?

All of these have to report L2MTU correctly, for example, currently, bonding interfaces are unusable in RouterOS with MPLS, since the bonding interface doesnt report a L2MTU, even if the underlying ethernet cards do. (page 49 of my presentation I linked earlier)

There is a vlan on the ethernet interface but it also reported L2MTU 9014-4=9010 vlan L2MTU.
Nothing else fancy.

There is a vlan on the ethernet interface but it also reported L2MTU 9014-4=9010 vlan L2MTU.
Nothing else fancy.

Then it should work no problem. Feel free to start a new topic with your issues, I will try to help.

There is a vlan on the ethernet interface but it also reported L2MTU 9014-4=9010 vlan L2MTU.
Nothing else fancy.

Then it should work no problem. Feel free to start a new topic with your issues, I will try to help.

This is an age old problem for me, but see the last few posts near the end of :

http://forum.mikrotik.com/viewtopic.php?f=14&t=77787