mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

RouterOS General
1

We’d use to have a simple routed network. (And bridged or switches in some places.) Central dhcp-servers assigned clients of IP’s and each AP had its own network and was routed towards from the gateway.

To upgrade our service level and improve the quality of the network (redundancy!) we started to work with OSPF on parts of the network. Use mpls/vpls from gateway towards remote traffic concentrators (router serving several AP’s) and for client authentication we have several PPPoE servers (one at each VPLS interface).
At some parts of the network we also have vlan’s to make local fixed line clients connect to same VPLS as building’s AP clients.
Apart from that, in several locations of our AP’s we have 2,4Ghz hotspot APs’ that all connected to a dedicated vlan network running over the same physical network as the rest.

Actually since we started to implement the pppoe to the clients which should improve things (we sorted some switches and separated traffic in routers) we are getting more and more complaints from clients saying their internet is sooo slow or they only can get half the speeds of what they are believed to have signed for.
Each and any time we can run MT bandwidth tests from client’s CPE (SXT’s) and always get the full assigned (simple queue by pppoe server) capacity. But tests done from clients premises reveals indeed only half of that is getting into the clients network… how come?

Recently we found an issue might be the mtu setting of the Mikrotik routers.
Now I can find several technical explanations with a lot of tech talk which is all hard to understand by me, but the bottom line is, how do I set, and which, Mikrotik routers? Is there not one setting that arranges all?
And where? Only in some major routers? Or in any router/switch settings needs to be set?

[Apart from 4 Netonix Switches the whole network consists of modern MT devices (some rb411’s still around as CPE).
Off course, client’s wifi routers are several brands, usually TP-Link and our gateway CCR connects to the Cisco of our bandwidth provider.]

2

Was going to ask you before if you were using netonix… where are those switches located? Are they connected to a CCR?

3

Yes. Why?
We have one WS-8-250-AC that has a CCR1009-8G-1S-1S+ directly behind it. We actually use it as a gigabit power inserter for 3 antenna links.

4

Hook a spare CCR (:smiley: or 2011 at least) to the already deployed CCR, and measure speeds from / to each RB to have a reference or “calibration test”.

Then repeat the same test, this time across the Netonix (CCR → Netonix ← CCR/2011)

Do you get similar results?

5

ehh, you suspect the Netonix mtu setting might be an issue?


This is what we have:
/=SXT-HG==Wifi==>SXT-HG==
Cisco==>CCR-A= =Netonix=CCR-B=Netonix==Netmetal==> deeper in network.
=NetMetal==Wifi==>NetMetal==/

The dual link is OSPF semi duplex. SXT (20Mhz) has the upload, the NetMetals (80Mhz) do the download.
The both radios on the other end get their power from the Netonix but since we use is as power inserter traffic for both radios have their own ethernet into the CCR.
Since this same CCR after routing decisions has 3 new backhauls leaving 2 of these get their power from the same Netonix again.

So, packages from CCR-A go over the NetMetal link to a port on the Netonix, than out another port to CCR-B.
CCR-B no make decision to where to send the package and that might well be it goes back to the same Netonix (but yet again another cable) where the netonix puts 24V on the port feeding a new Netmal that now goes deeper into our network.
So the package enters twice and leave twice this Netonix.
M7 Netonix Vlan.JPG
M7 Netonix Vlanb.JPG

6

ehh, you suspect the Netonix mtu setting might be an issue?

I couldn’t know, but for a clean start, you should do that test first: measure bandwidth between two directly connected routers for a reference point, then repeat the test with each router connected to the netonix, so that traffic crosses it.

7

If you’re running tagged VPLS, I wouldn’t have an L2 MTU lower than 1530 anywhere in the network for simplicity. In the WISP world, I’ve been burned a number of times with “Jumbo” frames set on PtP radios (from more than one manufacturer) where we actually weren’t passing an L2 MTU that would support 1530 or higher. Typically this is a bug and i’ve seen it most frequently on Ubiquity AirFiber, but it’s never a bad idea to actually audit the MTU you can pass with pings and packet captures.

Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/US16/presentation_3327_1462279781.pdf

8

Ok, its a bit hard since we don’t have a CCR spare (well, nor really) and the netonix we speak about has live traffic on all ports and is in a remote location.
But; I took a closer look in both our and the netonix forum on the subject on ‘Flow Control’.
First I tried to find out why in MT ‘off’ is the default and would the use of it somewhere help in a very mixed network setup. My conclusion so far are negative. It ruins tcp and Voip.
Secondly, since Netonix have in ‘on’ by default, and the forum is full with discussion about issues related to ubnt’s 24af units all to do with the flow control. They actually put new firmware out to fight the issue but actually nodded it was more an ubnt issue since these ‘fibre’s’ use flow control.
Anyway, took a closer look on the data of our netonix switch and found on some port huge amount of tx ‘pause’ packages. And mainly on the heavy used links… even although these ports connect by giga ethernet to the CCR or a NetMetal on the other end.
So, testing would cost us too much time and hazzle (there is more, other important stuff to do) we just decided to give it a try… so we disabled all flow controls on the 3 netonix units we have. Well, it brought all traffic to a standstill for a second or so but after that all seems to work fine again.. (And why not, the infrastructure on both ends of these switches is wide enough to transport the eventual max of traffic that would pass.

Next stop is we have to look closer to the mtu settings in our network.
(Then QoS, on the moment no QoS at all, on advice of my new assitant. His arguments made sense initially but I’m not sure if that still holds ground…)

9

Ok, you guys seems to know what you talk about .
(Send me a pm, maybe I’ve got some business for you?)

Anyway, in regard of the setting in our network of mtu, and L2 mtu. I don’t think we have anywhere any old stuff left that can’t adjust mtu to high figures.
So, in your presentation the biggest mtu I see is 1530. In ROS I can set MTU and L2 MTU. I see in most of my routers MTU is set to 1500 where L2 MTu is sometimes set to 1590, sometimes to 1600

So, the question from this dummy; Would setting higher than actually needed (set L2 MTU to 1600) OK in all routers?
And what should be the setting of the MTU field? Just 1500 or higher? Also 1600?

Maybe too simple the question for a techno like you but I can’t afford to read-up a lot on mtu now since I need to sort issues soon…

[The issue is we have a rather complicated network; Xmas tree a-like network with at the base some 30 Ap’s and all backhauls wireless in a heavy used spectrum (interference!) All links are NV2 and 95% is duo chain ‘n’ where we now have some ‘ac’ links. I’m more a radio operator with absolutely no IT background (used to be a sailor for haven sake, hence the radio background. I am a licensed radio telecommunication operator). I think the wireless part is pretty much under control. But in networking I’m running short. We have flow control, queuing, QoS, routing protocols, gigabit and fast Ethernet mixed with wireless links. Over subscribed AP’s and/or under performing AP’s. Like somebody wrote somewhere else, the network has become a ‘live beast’ that needs to be controlled. But its getting more and more complicated since the beast is growing and always hungry to more data…
I need to go step by step in each issue. Luckily I have this very smart assistant from university that can assist me at time but he, like me, is still in the learning curve… (his one goes faster than mine though.. :wink: ) and not always available…
I spend hours reading and writing in this forum but sometimes don’t see the trees through the forest any more.
I need to go forward to be honest…

10

Just to another question about MTU if min size for above routing is 1530 how about routers that Max-L2MTU of 4074 (etc) should they also be set to L2=1530, in other words should all routers on a network be set to the same L2 size ?

11

If all your gear will support L2 MTU 1530 and higher, then you should be ok. If you use the values on Page 13 of the MUM presentation for tagged VPLS throughout your network, you should be ok. Be sure to set the L2, L3 and MPLS values to what i have listed and see if that makes a difference. Good luck!

We build fairly large and complex MPLS networks all over the world - If you need some in-depth professional help, shoot us an e-mail to: consulting [at] iparchitechs.com

Note: I noticed you are in Spain, and we have Spanish speaking network engineers if required.

12

In an ideal world, L2 MTU should match on either side of a link. However, this isn’t always practical as different models of routers define “Jumbo” as different frame sizes. For example, most Cisco routers top out at 9216 Bytes, while a CCR1072 has a max L2 MTU of 10226.

L2 MTU serves as the base to support L3 MTUs, so the smaller of two different L2 MTU sizes will be the effective MTU and normally works without issue if the L3 packet is less than or equal to the L2 MTU (minus the frame headers).

So if we establish a 10 gig link between a Cisco ASR1006-X and a CCR1072, the smaller CIsco MTU of 9216 would be the maximum frame size we could pass on that link.

13

Put another way could a router which has say 1598 L2MTU start transmitting data at that rate (?) and then this router has to reduce the L2MTU to 1530 to match the Max L2MTU on another router ?

Or If the lowest max L2MTU is 1520 (apart from replacing the unit with a higher L2MTU ) should all other routers on that segment lower their max L2MTU to 1520 ?

14

@WirelessRudy: just wanted you to conduct the test in order to realize you could be having issues with the netonix switches.

In my tests, didn’t matter with or without flow control, duplex, MTU settings or whatever, pure gigabit throughput across the netonix switch was cut in more than half.

CCR directly connected to a RB2011 measured about 140Mbps (maxed CPU), whereas the same devices across the netonix had their throughput cut to 80Mbps. That is through gigabit ports on all of them.

After contacting Netonix support, and by their answers on the forum (the problem was always on the other vendors side) I simply advised the WISP I was auditing (where this switch was located in) to ditch it. Hardware specs looked really promising and just like the answer to WISP prayers, but software… was a different beast, I found it not mature enough to trust it into production, where pristine L2 performance is required, so it was replaced with a POE panel and a switch.

To tame a beast you need time and constant/systematic tender, testing, tender, testing… analyzing each part of the network in an isolated manner, then progressively including bigger areas in the tender/test cycle.

Didn’t realize it was you (Kevin) the one writing in the forum, really liked the Slovenia MUM BGP presentation (and the Mikrotik tacs! :smiley:)

Really enjoyed this one, even with its density, I really digged how you drawn the “bigger picture” and always provided a dual technical/business perspective to all matters. Top Notch!

It won’t “autoadjust” L2MTU, that’s the problem… datagrams will have to be fragmented, causing overhead un sub-par performance.

Max L2MTU != L2MTU. Max L2MTU is a hardware capability. L2MTU is the actual set value on the interface.
Instead of lowering them, you’ll need to make sure datagrams flowing to the L2MTU “handicapped” router either actually “fit” in its Max L2MTU (1520), or set L2MTU on the router interface facing it to match.

15

:open_mouth: >

[quote=“pukkita”]
@WirelessRudy: just wanted you to conduct the test in order to realize you could be having issues with the netonix switches.

In my tests, didn’t matter with or without flow control, duplex, MTU settings or whatever, pure gigabit throughput across the netonix switch was cut in more than half.

CCR directly connected to a RB2011 measured about 140Mbps (maxed CPU), whereas the same devices across the netonix had their throughput cut to 80Mbps. That is through gigabit ports on all of them.
[/quote]

:open_mouth: >
If that is true? To test the switches in the field is going to be hard. But we still have a mini here in the box. We could make some test over him? Do you thing the issue is models wide? Meaning it would effect all their devices?
Would it just cut in half ‘any’ data steam? Regardless if its 1Mb or 100Mb? If the 50% for every stream would be true this would explain why a lot people roughtly only get half as what we tell them we deliver. (But the bandwidth tests are usually better…) This would really be a bummer if this is true. Because there are not so many multi port PoE-out gigabit devices that can also do battery feed out in the field… So we have to start thinking of another way to PoE-out feed muttiple devices with gigabit… pfff.

After contacting Netonix support, and by their answers on the forum (the problem was always on the other vendors side) I simply advised the WISP I was auditing (where this switch was located in) to ditch it. Hardware specs looked really promising and just like the answer to WISP prayers, but software… was a different beast, I found it not mature enough to trust it into production, where pristine L2 performance is required, so it was replaced with a POE panel and a switch.

Well, if the mini has the same issue, and you would be right in your statement, you’ll bet you’ll see me on their forum about it!

To tame a beast you need time and constant/systematic tender, testing, tender, testing… analysing each part of the network in an isolated manner, then progressively including bigger areas in the tender/test cycle.

Fully agree and totally true. You only forgot one item…; the audience that want to see you tamed it today… Although we have the feeling our network improves (we see average usage go up) we are getting more and more people complaining ‘the internet is soo slow’. Then we do a bandwidth test and if’s really fast… But saturation during peak hours must be worse due the heavier usage… And if you now tell out Netonix could be an issue… it carries some 85% of our network, in fact all the clients we are indeed getting complaints from. Never from the 15% that doesn’t see a netonix in its path…

16

Thanks!! Glad you enjoyed it and the Tik Tacs :slight_smile: The dual business and technical requirements are always something I try to include because many companies we deal with will not accept a technical design with a business justification. So we spend a lot of time trying to weigh the business impact of the network engineering decisions we make especially when designing or revamping large networks. The presentations are always my favorite part of the MUM. Not only do we get to share things we have been working on, but we also get to see great ideas from all the other presentations. I always come away from the MUM having learned something new which gives me fresh ideas. :slight_smile:

This year was our first time to exhibit and attend the MUM in Europe and it was great!

17

Agree with you 100%. I’ve been very hesitant to use WISP focused POE switches in the designs we do because they don’t seem to perform well at scale. They seem to be ok in smaller WISPs, but we get a lot of customers that want to scale to tens of thousands of subscribers or more and it’s tough to do that with limited performance and feature sets in L2 switching.

Flow control seems to be used frequently in several WISP switch vendors as a work around patch for lack of proper buffer depth in the switch ports to account for a mix of different speed links. Also, if QoS and traffic shaping is done properly, you shouldn’t need or want flow control. Many (not all) flow control implementations break QoS becasue they slow the rate of traffic even for ports that aren’t congesting the network so prioritized traffic gets dropped or rate limited. In a modern network, the only area I’ve seen a need for flow control is inside the data center for converged storage networks like FCoE that cannot drop packets ever and have to slow down as needed to be able to reliably write data. We manage some extremely large global networks that don’t use flow control at all and have excellent performance across links of all different speeds because of proper buffer depth and proper QoS and shaping at the source of traffic.

It’s also important to remember that TCP has it’s own flow control mechanisms that typically work much better (and are way more advanced) than layer 2 flow control for congestion an drop detection. There are some newer Ethernet flow control mechanisms that work much better than legacy flow control, but they are still focused primarily at storage and data center networking and not run-of-the mill IP traffic over an ISP network.

Outside of cost, the big argument to use WISP focused POE switches instead of enterprise or carrier switches stems around not wanting to separate switch function and PoE into an injector panel. And to some extent, I understand the desire to simplify the physical implementation where space and power are a big concern, but it’s difficult to grow into tens of thousands of subscribers or more if you don’t have the proper layer 2 switching with advanced feature sets.

Depending on the environment and network requirements, we usually put enterprise or metro ethernet switches in and PoE injector panels to keep the install as clean as possible and get much better performance and fewer issues when putting MPLS into the network with MTU.

I don’t want to give the impression these are bad for everyone, because i’m sure there are use cases where they work well and code development will continue to make them better…so as always YMMV.

18

With regard to 433AH that has the lowest max L2MTU of 1526 on ether1 on the network, is the setting of 1526 for MPLS interfaces for this and all other routers running MPLS on the network + VPLS also set to 1526, correct?

Another question is on this routed network using VLAN’s should the L2MTU for the VLAN’s setting be reduced to the lowest max L2MTU of 1526?

19

I have now almost everywhere L2 MTU set to 1600 and some port here and there still read 1580
The Netonix switches and one IgniNet Metrolinq 60Ghz link are also set to 1600.

I only see that vlan’s get automatically 14 bytes lower to 1586

The VPLS interfaces are still 1500 since I am not sure if they can be changed like that (and the change brings this whole VPLS network down for almost 2 mins)
I only changed one VPLS start and end point into 1600 to see if that would make any difference from a certain client. It didn’t.
In fact, from that client I did a mtu test ping and the biggest package I could send (‘Don’t Fragment’ ticked) was 1480…

Ok, clients are authenticated by pppoe and traffic thus runs through the pppoe tunnel. (From concentrator CCR = Gateway, to CPE of client)

 /interface pppoe-client add ac-name="" add-default-route=yes allow=pap,chap,mschap1,mschap2 default-route-distance=0 dial-on-demand=no disabled=no interface=wlan1     keepalive-timeout=60 max-mru=auto max-mtu=auto mrru=disabled name=pppoe-out1 password="" profile=default service-name=internet use-peer-dns=yes user="xxxxxxxxxxxxxxxxxxxxxxx"

Also automatically this is add in the mangle:

0  D chain=forward action=change-mss new-mss=1440 passthrough=yes tcp-flags=syn protocol=tcp out-interface=all-ppp tcp-mss=1441-65535 log=no log-prefix="" 

 1  D chain=forward action=change-mss new-mss=1440 passthrough=yes tcp-flags=syn protocol=tcp in-interface=all-ppp tcp-mss=1441-65535 log=no log-prefix=""

Now, as far as we can see these mss change rules are creates automatically when the pppoe-client connects to the server. But why would be the new-mss set to 1440 where the mtu of the server is 1500??

We want to have these rules automatically set to a mss that relates to the mtu of 1500.

On the server;

interface pppoe-server server add authentication=pap,chap,mschap1,mschap2 default-profile=defaultMTU1500noTCP_MSS disabled=no interface=ether3 keepalive-timeout=10 max-mru=1500 max-mtu=1500 max-sessions=unlimited mrru=disabled one-session-per-host=yes pado-delay=0 service-name=internet

According my tech (that is not here today, university days…) he tried same in virtual network setup and client was set to higher mss and could send mtu of 1500. But in real this does not work…

Any suggestions?

20

In regard to the Netonix switch with PoE inserter;

I did some tests between the two CCR1016-12G’s I have. And in fact there is hardly any difference in the traffic just over a cat6 cable between the two or with a Netonix Mini in between as switch.

What struck me though is that max speed one way is only some 470Mbps where the other CCR downloads with only 225Mbps
Capture CCR 2 CCR via Netonix tcp.JPG
As you can see both bandwidth tests are receiving and CPU’s are not really under load. The left device shows higher cpu since it is our live gateway handling some 200Mbps traffic at the time with the pppoe servers and simple queues it creates…

We change cables, we change ‘receiving’ into ‘sendin’ but the difference in performance between the two directions stays merely the same…

This is the same test, now direct cat6 cable connected. The same as with the Netonix in between…
Capture CCR 2 CCR direct cable tcp.JPG
1st remark; Unless a Netonix AC unit behaves differently (that’s the one in the field I cannot test) I don’t think a netonix is keeping traffic down.

2nd remark; Why should we not have almost 1Gb of traffic between CCR’s when there is only a cable? (traffic was tcp but I also ran a udp test and max speed was just the same.)
Or at least 500Mb one way, 500Mb the other way? Now we don’t even have 1Gb aggregated…

3rd remark,; The netonix have flow control ‘on’ be default. We disabled it everywhere but I didn’t check this specific one used in the test. So probably fc is still ‘on’. I’ll come back to that in a later post…

4rd remark; Maybe the issue with the Netonix as you guys prescribed is not ‘visible’ yet since we don’t get near to the speeds two CCR’s should be able to send over an gigabit connection…?