Community discussions

MUM Europe 2020
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 10, 2016 12:17 pm

We'd use to have a simple routed network. (And bridged or switches in some places.) Central dhcp-servers assigned clients of IP's and each AP had its own network and was routed towards from the gateway.

To upgrade our service level and improve the quality of the network (redundancy!) we started to work with OSPF on parts of the network. Use mpls/vpls from gateway towards remote traffic concentrators (router serving several AP's) and for client authentication we have several PPPoE servers (one at each VPLS interface).
At some parts of the network we also have vlan's to make local fixed line clients connect to same VPLS as building's AP clients.
Apart from that, in several locations of our AP's we have 2,4Ghz hotspot APs' that all connected to a dedicated vlan network running over the same physical network as the rest.

Actually since we started to implement the pppoe to the clients which should improve things (we sorted some switches and separated traffic in routers) we are getting more and more complaints from clients saying their internet is sooo slow or they only can get half the speeds of what they are believed to have signed for.
Each and any time we can run MT bandwidth tests from client's CPE (SXT's) and always get the full assigned (simple queue by pppoe server) capacity. But tests done from clients premises reveals indeed only half of that is getting into the clients network....... how come?

Recently we found an issue might be the mtu setting of the Mikrotik routers.
Now I can find several technical explanations with a lot of tech talk which is all hard to understand by me, but the bottom line is, how do I set, and which, Mikrotik routers? Is there not one setting that arranges all?
And where? Only in some major routers? Or in any router/switch settings needs to be set?

[Apart from 4 Netonix Switches the whole network consists of modern MT devices (some rb411's still around as CPE).
Off course, client's wifi routers are several brands, usually TP-Link and our gateway CCR connects to the Cisco of our bandwidth provider.]
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 10, 2016 1:02 pm

Was going to ask you before if you were using netonix... where are those switches located? Are they connected to a CCR?
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 10, 2016 2:25 pm

Was going to ask you before if you were using netonix... where are those switches located? Are they connected to a CCR?
Yes. Why?
We have one WS-8-250-AC that has a CCR1009-8G-1S-1S+ directly behind it. We actually use it as a gigabit power inserter for 3 antenna links.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 10, 2016 3:27 pm

Hook a spare CCR (:D or 2011 at least) to the already deployed CCR, and measure speeds from / to each RB to have a reference or "calibration test".

Then repeat the same test, this time across the Netonix (CCR --> Netonix <-- CCR/2011)

Do you get similar results?
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 10, 2016 5:49 pm

Hook a spare CCR (:D or 2011 at least) to the already deployed CCR, and measure speeds from / to each RB to have a reference or "calibration test".

Then repeat the same test, this time across the Netonix (CCR --> Netonix <-- CCR/2011)

Do you get similar results?
ehh, you suspect the Netonix mtu setting might be an issue?


This is what we have:
/=SXT-HG==Wifi==>SXT-HG==\
Cisco==>CCR-A= =Netonix=CCR-B=Netonix==Netmetal==> deeper in network.
\=NetMetal==Wifi==>NetMetal==/

The dual link is OSPF semi duplex. SXT (20Mhz) has the upload, the NetMetals (80Mhz) do the download.
The both radios on the other end get their power from the Netonix but since we use is as power inserter traffic for both radios have their own ethernet into the CCR.
Since this same CCR after routing decisions has 3 new backhauls leaving 2 of these get their power from the same Netonix again.

So, packages from CCR-A go over the NetMetal link to a port on the Netonix, than out another port to CCR-B.
CCR-B no make decision to where to send the package and that might well be it goes back to the same Netonix (but yet again another cable) where the netonix puts 24V on the port feeding a new Netmal that now goes deeper into our network.
So the package enters twice and leave twice this Netonix.
M7 Netonix Vlan.JPG
M7 Netonix Vlanb.JPG
You do not have the required permissions to view the files attached to this post.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Wed May 11, 2016 2:06 pm

ehh, you suspect the Netonix mtu setting might be an issue?
I couldn't know, but for a clean start, you should do that test first: measure bandwidth between two directly connected routers for a reference point, then repeat the test with each router connected to the netonix, so that traffic crosses it.
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Wed May 11, 2016 7:15 pm

If you're running tagged VPLS, I wouldn't have an L2 MTU lower than 1530 anywhere in the network for simplicity. In the WISP world, I've been burned a number of times with "Jumbo" frames set on PtP radios (from more than one manufacturer) where we actually weren't passing an L2 MTU that would support 1530 or higher. Typically this is a bug and i've seen it most frequently on Ubiquity AirFiber, but it's never a bad idea to actually audit the MTU you can pass with pings and packet captures.

Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/U ... 279781.pdf
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 12, 2016 12:38 am

ehh, you suspect the Netonix mtu setting might be an issue?
I couldn't know, but for a clean start, you should do that test first: measure bandwidth between two directly connected routers for a reference point, then repeat the test with each router connected to the netonix, so that traffic crosses it.
Ok, its a bit hard since we don't have a CCR spare (well, nor really) and the netonix we speak about has live traffic on all ports and is in a remote location.
But; I took a closer look in both our and the netonix forum on the subject on 'Flow Control'.
First I tried to find out why in MT 'off' is the default and would the use of it somewhere help in a very mixed network setup. My conclusion so far are negative. It ruins tcp and Voip.
Secondly, since Netonix have in 'on' by default, and the forum is full with discussion about issues related to ubnt's 24af units all to do with the flow control. They actually put new firmware out to fight the issue but actually nodded it was more an ubnt issue since these 'fibre's' use flow control.
Anyway, took a closer look on the data of our netonix switch and found on some port huge amount of tx 'pause' packages. And mainly on the heavy used links... even although these ports connect by giga ethernet to the CCR or a NetMetal on the other end.
So, testing would cost us too much time and hazzle (there is more, other important stuff to do) we just decided to give it a try....... so we disabled all flow controls on the 3 netonix units we have. Well, it brought all traffic to a standstill for a second or so but after that all seems to work fine again.. (And why not, the infrastructure on both ends of these switches is wide enough to transport the eventual max of traffic that would pass.

Next stop is we have to look closer to the mtu settings in our network.
(Then QoS, on the moment no QoS at all, on advice of my new assitant. His arguments made sense initially but I'm not sure if that still holds ground.......)
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 12, 2016 1:02 am

If you're running tagged VPLS, I wouldn't have an L2 MTU lower than 1530 anywhere in the network for simplicity. In the WISP world, I've been burned a number of times with "Jumbo" frames set on PtP radios (from more than one manufacturer) where we actually weren't passing an L2 MTU that would support 1530 or higher. Typically this is a bug and i've seen it most frequently on Ubiquity AirFiber, but it's never a bad idea to actually audit the MTU you can pass with pings and packet captures.

Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/U ... 279781.pdf
Ok, you guys seems to know what you talk about .
(Send me a pm, maybe I've got some business for you?)

Anyway, in regard of the setting in our network of mtu, and L2 mtu. I don't think we have anywhere any old stuff left that can't adjust mtu to high figures.
So, in your presentation the biggest mtu I see is 1530. In ROS I can set MTU and L2 MTU. I see in most of my routers MTU is set to 1500 where L2 MTu is sometimes set to 1590, sometimes to 1600

So, the question from this dummy; Would setting higher than actually needed (set L2 MTU to 1600) OK in all routers?
And what should be the setting of the MTU field? Just 1500 or higher? Also 1600?

Maybe too simple the question for a techno like you but I can't afford to read-up a lot on mtu now since I need to sort issues soon.....

[The issue is we have a rather complicated network; Xmas tree a-like network with at the base some 30 Ap's and all backhauls wireless in a heavy used spectrum (interference!) All links are NV2 and 95% is duo chain 'n' where we now have some 'ac' links. I'm more a radio operator with absolutely no IT background (used to be a sailor for haven sake, hence the radio background. I am a licensed radio telecommunication operator). I think the wireless part is pretty much under control. But in networking I'm running short. We have flow control, queuing, QoS, routing protocols, gigabit and fast Ethernet mixed with wireless links. Over subscribed AP's and/or under performing AP's. Like somebody wrote somewhere else, the network has become a 'live beast' that needs to be controlled. But its getting more and more complicated since the beast is growing and always hungry to more data......
I need to go step by step in each issue. Luckily I have this very smart assistant from university that can assist me at time but he, like me, is still in the learning curve.... (his one goes faster than mine though.. :wink: ) and not always available....
I spend hours reading and writing in this forum but sometimes don't see the trees through the forest any more.
I need to go forward to be honest.....
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
n21roadie
Forum Guru
Forum Guru
Posts: 1892
Joined: Fri Aug 07, 2009 10:36 pm
Location: Limerick,Ireland

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 12, 2016 4:19 pm

If you're running tagged VPLS, I wouldn't have an L2 MTU lower than 1530 anywhere in the network for simplicity. In the WISP world, I've been burned a number of times with "Jumbo" frames set on PtP radios (from more than one manufacturer) where we actually weren't passing an L2 MTU that would support 1530 or higher. Typically this is a bug and i've seen it most frequently on Ubiquity AirFiber, but it's never a bad idea to actually audit the MTU you can pass with pings and packet captures.

Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/U ... 279781.pdf
Just to another question about MTU if min size for above routing is 1530 how about routers that Max-L2MTU of 4074 (etc) should they also be set to L2=1530, in other words should all routers on a network be set to the same L2 size ?
N21roadie,
Network 100% MT for Now?
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 12, 2016 7:58 pm

If you're running tagged VPLS, I wouldn't have an L2 MTU lower than 1530 anywhere in the network for simplicity. In the WISP world, I've been burned a number of times with "Jumbo" frames set on PtP radios (from more than one manufacturer) where we actually weren't passing an L2 MTU that would support 1530 or higher. Typically this is a bug and i've seen it most frequently on Ubiquity AirFiber, but it's never a bad idea to actually audit the MTU you can pass with pings and packet captures.

Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/U ... 279781.pdf
Ok, you guys seems to know what you talk about .
(Send me a pm, maybe I've got some business for you?)

Anyway, in regard of the setting in our network of mtu, and L2 mtu. I don't think we have anywhere any old stuff left that can't adjust mtu to high figures.
So, in your presentation the biggest mtu I see is 1530. In ROS I can set MTU and L2 MTU. I see in most of my routers MTU is set to 1500 where L2 MTu is sometimes set to 1590, sometimes to 1600

So, the question from this dummy; Would setting higher than actually needed (set L2 MTU to 1600) OK in all routers?
And what should be the setting of the MTU field? Just 1500 or higher? Also 1600?

Maybe too simple the question for a techno like you but I can't afford to read-up a lot on mtu now since I need to sort issues soon.....

[The issue is we have a rather complicated network; Xmas tree a-like network with at the base some 30 Ap's and all backhauls wireless in a heavy used spectrum (interference!) All links are NV2 and 95% is duo chain 'n' where we now have some 'ac' links. I'm more a radio operator with absolutely no IT background (used to be a sailor for haven sake, hence the radio background. I am a licensed radio telecommunication operator). I think the wireless part is pretty much under control. But in networking I'm running short. We have flow control, queuing, QoS, routing protocols, gigabit and fast Ethernet mixed with wireless links. Over subscribed AP's and/or under performing AP's. Like somebody wrote somewhere else, the network has become a 'live beast' that needs to be controlled. But its getting more and more complicated since the beast is growing and always hungry to more data......
I need to go step by step in each issue. Luckily I have this very smart assistant from university that can assist me at time but he, like me, is still in the learning curve.... (his one goes faster than mine though.. :wink: ) and not always available....
I spend hours reading and writing in this forum but sometimes don't see the trees through the forest any more.
I need to go forward to be honest.....
If all your gear will support L2 MTU 1530 and higher, then you should be ok. If you use the values on Page 13 of the MUM presentation for tagged VPLS throughout your network, you should be ok. Be sure to set the L2, L3 and MPLS values to what i have listed and see if that makes a difference. Good luck!

We build fairly large and complex MPLS networks all over the world - If you need some in-depth professional help, shoot us an e-mail to: consulting [at] iparchitechs.com

Note: I noticed you are in Spain, and we have Spanish speaking network engineers if required.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 12, 2016 8:08 pm

If you're running tagged VPLS, I wouldn't have an L2 MTU lower than 1530 anywhere in the network for simplicity. In the WISP world, I've been burned a number of times with "Jumbo" frames set on PtP radios (from more than one manufacturer) where we actually weren't passing an L2 MTU that would support 1530 or higher. Typically this is a bug and i've seen it most frequently on Ubiquity AirFiber, but it's never a bad idea to actually audit the MTU you can pass with pings and packet captures.

Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/U ... 279781.pdf
Just to another question about MTU if min size for above routing is 1530 how about routers that Max-L2MTU of 4074 (etc) should they also be set to L2=1530, in other words should all routers on a network be set to the same L2 size ?
In an ideal world, L2 MTU should match on either side of a link. However, this isn't always practical as different models of routers define "Jumbo" as different frame sizes. For example, most Cisco routers top out at 9216 Bytes, while a CCR1072 has a max L2 MTU of 10226.

L2 MTU serves as the base to support L3 MTUs, so the smaller of two different L2 MTU sizes will be the effective MTU and normally works without issue if the L3 packet is less than or equal to the L2 MTU (minus the frame headers).

So if we establish a 10 gig link between a Cisco ASR1006-X and a CCR1072, the smaller CIsco MTU of 9216 would be the maximum frame size we could pass on that link.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
n21roadie
Forum Guru
Forum Guru
Posts: 1892
Joined: Fri Aug 07, 2009 10:36 pm
Location: Limerick,Ireland

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 12, 2016 9:05 pm


In an ideal world, L2 MTU should match on either side of a link. However, this isn't always practical as different models of routers define "Jumbo" as different frame sizes. For example, most Cisco routers top out at 9216 Bytes, while a CCR1072 has a max L2 MTU of 10226.

L2 MTU serves as the base to support L3 MTUs, so the smaller of two different L2 MTU sizes will be the effective MTU and normally works without issue if the L3 packet is less than or equal to the L2 MTU (minus the frame headers).

So if we establish a 10 gig link between a Cisco ASR1006-X and a CCR1072, the smaller CIsco MTU of 9216 would be the maximum frame size we could pass on that link.
Put another way could a router which has say 1598 L2MTU start transmitting data at that rate (?) and then this router has to reduce the L2MTU to 1530 to match the Max L2MTU on another router ?

Or If the lowest max L2MTU is 1520 (apart from replacing the unit with a higher L2MTU ) should all other routers on that segment lower their max L2MTU to 1520 ?
N21roadie,
Network 100% MT for Now?
 
User avatar
pukkita
Trainer
Trainer
Posts: 2986
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Fri May 13, 2016 4:43 pm

@WirelessRudy: just wanted you to conduct the test in order to realize you could be having issues with the netonix switches.

In my tests, didn't matter with or without flow control, duplex, MTU settings or whatever, pure gigabit throughput across the netonix switch was cut in more than half.

CCR directly connected to a RB2011 measured about 140Mbps (maxed CPU), whereas the same devices across the netonix had their throughput cut to 80Mbps. That is through gigabit ports on all of them.

After contacting Netonix support, and by their answers on the forum (the problem was always on the other vendors side) I simply advised the WISP I was auditing (where this switch was located in) to ditch it. Hardware specs looked really promising and just like the answer to WISP prayers, but software... was a different beast, I found it not mature enough to trust it into production, where pristine L2 performance is required, so it was replaced with a POE panel and a switch.

To tame a beast you need time and constant/systematic tender, testing, tender, testing... analyzing each part of the network in an isolated manner, then progressively including bigger areas in the tender/test cycle.
Here is a presentation I did on MPLS for WISPs a couple of weeks ago at the US MUM in Dallas,TX. Page 13 has all the MPLS MTU requirements laid out.

http://mum.mikrotik.com/presentations/U ... 279781.pdf
Didn't realize it was you (Kevin) the one writing in the forum, really liked the Slovenia MUM BGP presentation (and the Mikrotik tacs! :D)

Really enjoyed this one, even with its density, I really digged how you drawn the "bigger picture" and always provided a dual technical/business perspective to all matters. Top Notch!
Put another way could a router which has say 1598 L2MTU start transmitting data at that rate (?) and then this router has to reduce the L2MTU to 1530 to match the Max L2MTU on another router ?
It won't "autoadjust" L2MTU, that's the problem... datagrams will have to be fragmented, causing overhead un sub-par performance.
Or If the lowest max L2MTU is 1520 (apart from replacing the unit with a higher L2MTU) should all other routers on that segment lower their max L2MTU to 1520 ?
Max L2MTU != L2MTU. Max L2MTU is a hardware capability. L2MTU is the actual set value on the interface.
Instead of lowering them, you'll need to make sure datagrams flowing to the L2MTU "handicapped" router either actually "fit" in its Max L2MTU (1520), or set L2MTU on the router interface facing it to match.
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Fri May 13, 2016 8:25 pm

:shock:
@WirelessRudy: just wanted you to conduct the test in order to realize you could be having issues with the netonix switches.

In my tests, didn't matter with or without flow control, duplex, MTU settings or whatever, pure gigabit throughput across the netonix switch was cut in more than half.

CCR directly connected to a RB2011 measured about 140Mbps (maxed CPU), whereas the same devices across the netonix had their throughput cut to 80Mbps. That is through gigabit ports on all of them.
:shock:
If that is true? To test the switches in the field is going to be hard. But we still have a mini here in the box. We could make some test over him? Do you thing the issue is models wide? Meaning it would effect all their devices?
Would it just cut in half 'any' data steam? Regardless if its 1Mb or 100Mb? If the 50% for every stream would be true this would explain why a lot people roughtly only get half as what we tell them we deliver. (But the bandwidth tests are usually better....) This would really be a bummer if this is true. Because there are not so many multi port PoE-out gigabit devices that can also do battery feed out in the field..... So we have to start thinking of another way to PoE-out feed muttiple devices with gigabit..... pfff.
After contacting Netonix support, and by their answers on the forum (the problem was always on the other vendors side) I simply advised the WISP I was auditing (where this switch was located in) to ditch it. Hardware specs looked really promising and just like the answer to WISP prayers, but software... was a different beast, I found it not mature enough to trust it into production, where pristine L2 performance is required, so it was replaced with a POE panel and a switch.
Well, if the mini has the same issue, and you would be right in your statement, you'll bet you'll see me on their forum about it!
To tame a beast you need time and constant/systematic tender, testing, tender, testing... analysing each part of the network in an isolated manner, then progressively including bigger areas in the tender/test cycle.
Fully agree and totally true. You only forgot one item...; the audience that want to see you tamed it today.... Although we have the feeling our network improves (we see average usage go up) we are getting more and more people complaining 'the internet is soo slow'. Then we do a bandwidth test and if's really fast...... But saturation during peak hours must be worse due the heavier usage.... And if you now tell out Netonix could be an issue... it carries some 85% of our network, in fact all the clients we are indeed getting complaints from. Never from the 15% that doesn't see a netonix in its path....
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Fri May 13, 2016 8:59 pm

Didn't realize it was you (Kevin) the one writing in the forum, really liked the Slovenia MUM BGP presentation (and the Mikrotik tacs! :D)

Really enjoyed this one, even with its density, I really digged how you drawn the "bigger picture" and always provided a dual technical/business perspective to all matters. Top Notch!
Thanks!! Glad you enjoyed it and the Tik Tacs :) The dual business and technical requirements are always something I try to include because many companies we deal with will not accept a technical design with a business justification. So we spend a lot of time trying to weigh the business impact of the network engineering decisions we make especially when designing or revamping large networks. The presentations are always my favorite part of the MUM. Not only do we get to share things we have been working on, but we also get to see great ideas from all the other presentations. I always come away from the MUM having learned something new which gives me fresh ideas. :-)

This year was our first time to exhibit and attend the MUM in Europe and it was great!
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Fri May 13, 2016 9:47 pm

After contacting Netonix support, and by their answers on the forum (the problem was always on the other vendors side) I simply advised the WISP I was auditing (where this switch was located in) to ditch it. Hardware specs looked really promising and just like the answer to WISP prayers, but software... was a different beast, I found it not mature enough to trust it into production, where pristine L2 performance is required, so it was replaced with a POE panel and a switch.
Agree with you 100%. I've been very hesitant to use WISP focused POE switches in the designs we do because they don't seem to perform well at scale. They seem to be ok in smaller WISPs, but we get a lot of customers that want to scale to tens of thousands of subscribers or more and it's tough to do that with limited performance and feature sets in L2 switching.

Flow control seems to be used frequently in several WISP switch vendors as a work around patch for lack of proper buffer depth in the switch ports to account for a mix of different speed links. Also, if QoS and traffic shaping is done properly, you shouldn't need or want flow control. Many (not all) flow control implementations break QoS becasue they slow the rate of traffic even for ports that aren't congesting the network so prioritized traffic gets dropped or rate limited. In a modern network, the only area I've seen a need for flow control is inside the data center for converged storage networks like FCoE that cannot drop packets ever and have to slow down as needed to be able to reliably write data. We manage some extremely large global networks that don't use flow control at all and have excellent performance across links of all different speeds because of proper buffer depth and proper QoS and shaping at the source of traffic.

It's also important to remember that TCP has it's own flow control mechanisms that typically work much better (and are way more advanced) than layer 2 flow control for congestion an drop detection. There are some newer Ethernet flow control mechanisms that work much better than legacy flow control, but they are still focused primarily at storage and data center networking and not run-of-the mill IP traffic over an ISP network.

Outside of cost, the big argument to use WISP focused POE switches instead of enterprise or carrier switches stems around not wanting to separate switch function and PoE into an injector panel. And to some extent, I understand the desire to simplify the physical implementation where space and power are a big concern, but it's difficult to grow into tens of thousands of subscribers or more if you don't have the proper layer 2 switching with advanced feature sets.

Depending on the environment and network requirements, we usually put enterprise or metro ethernet switches in and PoE injector panels to keep the install as clean as possible and get much better performance and fewer issues when putting MPLS into the network with MTU.

I don't want to give the impression these are bad for everyone, because i'm sure there are use cases where they work well and code development will continue to make them better...so as always YMMV.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
n21roadie
Forum Guru
Forum Guru
Posts: 1892
Joined: Fri Aug 07, 2009 10:36 pm
Location: Limerick,Ireland

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 17, 2016 6:09 pm

With regard to 433AH that has the lowest max L2MTU of 1526 on ether1 on the network, is the setting of 1526 for MPLS interfaces for this and all other routers running MPLS on the network + VPLS also set to 1526, correct?

Another question is on this routed network using VLAN's should the L2MTU for the VLAN's setting be reduced to the lowest max L2MTU of 1526?
N21roadie,
Network 100% MT for Now?
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 17, 2016 7:38 pm

I have now almost everywhere L2 MTU set to 1600 and some port here and there still read 1580
The Netonix switches and one IgniNet Metrolinq 60Ghz link are also set to 1600.

I only see that vlan's get automatically 14 bytes lower to 1586

The VPLS interfaces are still 1500 since I am not sure if they can be changed like that (and the change brings this whole VPLS network down for almost 2 mins)
I only changed one VPLS start and end point into 1600 to see if that would make any difference from a certain client. It didn't.
In fact, from that client I did a mtu test ping and the biggest package I could send ('Don't Fragment' ticked) was 1480...

Ok, clients are authenticated by pppoe and traffic thus runs through the pppoe tunnel. (From concentrator CCR = Gateway, to CPE of client)
 /interface pppoe-client add ac-name="" add-default-route=yes allow=pap,chap,mschap1,mschap2 default-route-distance=0 dial-on-demand=no disabled=no interface=wlan1     keepalive-timeout=60 max-mru=auto max-mtu=auto mrru=disabled name=pppoe-out1 password="" profile=default service-name=internet use-peer-dns=yes user="xxxxxxxxxxxxxxxxxxxxxxx"
Also automatically this is add in the mangle:
0  D chain=forward action=change-mss new-mss=1440 passthrough=yes tcp-flags=syn protocol=tcp out-interface=all-ppp tcp-mss=1441-65535 log=no log-prefix="" 

 1  D chain=forward action=change-mss new-mss=1440 passthrough=yes tcp-flags=syn protocol=tcp in-interface=all-ppp tcp-mss=1441-65535 log=no log-prefix=""
Now, as far as we can see these mss change rules are creates automatically when the pppoe-client connects to the server. But why would be the new-mss set to 1440 where the mtu of the server is 1500??

We want to have these rules automatically set to a mss that relates to the mtu of 1500.

On the server;
interface pppoe-server server add authentication=pap,chap,mschap1,mschap2 default-profile=defaultMTU1500noTCP_MSS disabled=no interface=ether3 keepalive-timeout=10 max-mru=1500 max-mtu=1500 max-sessions=unlimited mrru=disabled one-session-per-host=yes pado-delay=0 service-name=internet
According my tech (that is not here today, university days...) he tried same in virtual network setup and client was set to higher mss and could send mtu of 1500. But in real this does not work......

Any suggestions?
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Tue May 17, 2016 7:52 pm

In regard to the Netonix switch with PoE inserter;

I did some tests between the two CCR1016-12G's I have. And in fact there is hardly any difference in the traffic just over a cat6 cable between the two or with a Netonix Mini in between as switch.

What struck me though is that max speed one way is only some 470Mbps where the other CCR downloads with only 225Mbps
Capture CCR 2 CCR via Netonix tcp.JPG
As you can see both bandwidth tests are receiving and CPU's are not really under load. The left device shows higher cpu since it is our live gateway handling some 200Mbps traffic at the time with the pppoe servers and simple queues it creates.....

We change cables, we change 'receiving' into 'sendin' but the difference in performance between the two directions stays merely the same...

This is the same test, now direct cat6 cable connected. The same as with the Netonix in between....
Capture CCR 2 CCR direct cable tcp.JPG
1st remark; Unless a Netonix AC unit behaves differently (that's the one in the field I cannot test) I don't think a netonix is keeping traffic down.

2nd remark; Why should we not have almost 1Gb of traffic between CCR's when there is only a cable? (traffic was tcp but I also ran a udp test and max speed was just the same.)
Or at least 500Mb one way, 500Mb the other way? Now we don't even have 1Gb aggregated....

3rd remark,; The netonix have flow control 'on' be default. We disabled it everywhere but I didn't check this specific one used in the test. So probably fc is still 'on'. I'll come back to that in a later post....

4rd remark; Maybe the issue with the Netonix as you guys prescribed is not 'visible' yet since we don't get near to the speeds two CCR's should be able to send over an gigabit connection......?
You do not have the required permissions to view the files attached to this post.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Thu May 19, 2016 12:23 am

No more post from you guys?
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
n21roadie
Forum Guru
Forum Guru
Posts: 1892
Joined: Fri Aug 07, 2009 10:36 pm
Location: Limerick,Ireland

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Fri May 20, 2016 9:55 pm

@WirelessRudy

Have you noticed any improvements ?

For me I am spending some time on
(1) Flow control - OFF
(2) Set to 100Mb sections of the network that have a mix of gigabit and 100Mb.. ( and test )
(3) Queue buffers
(4) Correctly set Ethernet/MPLS/VPLS/ VLAN MTU and L2MTU
(5) The effects of bridging interfaces ( example CPE in station bridge ) and how this impacts on max L2MTU
N21roadie,
Network 100% MT for Now?
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Sat May 21, 2016 3:06 am

@WirelessRudy

Have you noticed any improvements ?

For me I am spending some time on
(1) Flow control - OFF
(2) Set to 100Mb sections of the network that have a mix of gigabit and 100Mb.. ( and test )
(3) Queue buffers
(4) Correctly set Ethernet/MPLS/VPLS/ VLAN MTU and L2MTU
(5) The effects of bridging interfaces ( example CPE in station bridge ) and how this impacts on max L2MTU
Good some replies! :D I was getting afraid you guys found me a bit too much demanding.... :(

Anyway:
1) Wherever we could find, we switched flow control off.
2) In fact we don't have nowhere cable links with on one end gig and the other end fast. If we go deeper in the network it just changes to fast.
3) You mean the hardware buffer of the interfaces? Basically we have 'hardware only' everywhere. But is there a tool/way to see it these buffers are utilizes and to what extend? I don't seem to be able to find some info. Only the setting...
4) That's ongoing. My just graduated engineer spend the last week to figure out what we exactly need to set. We are for 80% done in setting the proper mtu everywhere. See if it will indeed improve things...
5) I am not exactly sure what you mean with this? Obviously we have many bridges. Most backhaul radios have their ether net and wireless in a bridge. The bridge actually mirrors the mtu's set on the interfaces.....

I have some discussion with my engineer about the use of queues.
Apart from the simple queues made by the pppoe server we don't have any in the network. Basically because we never fill our pipe from provider and the first 2 or 3 pipes to the clients are also never filled.
I have the 'feeling' we need to do some priority queueing on the AP or CPE level.
My engineer tells me NV2 default settings take care of that but in fact nobody on the forum or in the wiki can tell on what system this is based? "Priority based on package type and size"? What exactly is being prioritized....

My reasoning is that download wise seen in the network the first bottleneck client traffic enters is the simple queue. After that each client's 'pipe' (pppoe tunnel) 'fits' easy in the first 2 or 3 pipes down into our network. The first 2 or 3 pipes have actually bigger capacity than what the provider can deliver us in its 300/300 pipe.... So QoS on the download end at the AC wouldn't make any sense imho.
But after that some AP's can have the need to serve so many clients with so much traffic at the same time these combined clients flow can't all be handled by the AP's PtMP network. Imho we need to do some better QoS at the AP (or just before it with a router) than the standard embedded NV2 priority system can do?

On the upload; Clients can basically send unlimited amount of udp traffic. The bottleneck will we the client's simple queue in the AC router but before that it could already have ruined the PtMP network this client is part off....
Imho we should do some simple limiting and/or prioritization to udp (and tcp per client level) traffic that leaves the CPE in the upload.
I have the feeling that some of the over subscribed AP's have problems with uploads which ruins the download.
In the 'old' days we started to put as little processing stuff on the CPE's as we could because of the lack of cpu. But now the most CPE are 400Mhz so it should not be such an issue any-more?

I need some good suggestions on this issues. Most examples in the forum or wiki etc are based on simple situations where one AP connected to a big backbone serves some clients.
We have a big Xmas tree some 7 levels deep at places and some AP need to serve 35 (!) clients where others only serve 3. Some are in absolutely full spectrum, other can be set to 40Mhz to get stuff going...
And some AP's we can manage to get all its CPE's within a range of -50 to -60dB where others just have to deal with -70 up to -78dB clients too! Not ideal but hé, not serving some of these clients were accepted in the days we took everybody on board. We really can't tell these now; "Sorry, but with all the new advantages of the technology you hear us bragging about we have to disconnect you because for you is doesn't count! " :o


By the way, for some info. We have our first 60Ghz IgniteNet's PTP running. Although a lot of firmware issues it gives us a very, very stable link with high throughputs on some smaller distance link. (600Meter and already I've pumped 600Mb aggregated tcp over it... But according the manufacturer we should be able to do better with new firmwares coming up.....)
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
n21roadie
Forum Guru
Forum Guru
Posts: 1892
Joined: Fri Aug 07, 2009 10:36 pm
Location: Limerick,Ireland

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Sat May 21, 2016 11:46 am


2) In fact we don't have nowhere cable links with on one end gig and the other end fast. If we go deeper in the network it just changes to fast.
What for example on one port of CCR is GB to start with but along the way it changes to fast - well my opinion is that section maybe needs to be set fast from the port of the CCR?


3) You mean the hardware buffer of the interfaces? Basically we have 'hardware only' everywhere. But is there a tool/way to see it these buffers are utilizes and to what extend? I don't seem to be able to find some info. Only the setting...
Hardware Yes but for our centralized PPPoE server we have set a custom buffer
 /queue> type print detail
 5  name="PPPoE-Customers" kind=pfifo pfifo-limit=10 
4) That's ongoing. My just graduated engineer spend the last week to figure out what we exactly need to set. We are for 80% done in setting the proper mtu everywhere. See if it will indeed improve things...
5) I am not exactly sure what you mean with this? Obviously we have many bridges. Most backhaul radios have their ether net and wireless in a bridge. The bridge actually mirrors the mtu's set on the interfaces.....
My question what happens to CPE Ether max L2MTU of 1526 and then is "Station bridge" to Wlan which has L2MTU of 1600 ?
N21roadie,
Network 100% MT for Now?
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Sat May 21, 2016 12:00 pm


2) In fact we don't have nowhere cable links with on one end gig and the other end fast. If we go deeper in the network it just changes to fast.
What for example on one port of CCR is GB to start with but along the way it changes to fast - well my opinion is that section maybe needs to be set fast from the port of the CCR?
I don't think that would be the approach. Every link ends with fast Ethernet. 95% of the clients router and his PC/Laptop have not (yet) gigabit. Imho (I mean to have red that somewhere too) we only have to look at the ports on both sides of the link. Some said in download direction (most traffic) a router with a fast Ethernet port connected to a cat5e of better cable could be connected to a router with a gigabit port. The other way around would not be recommended since the gig port can send data faster then the fast Ether can handle. tcp will adjust for that but udp just keep hitting the bottle neck and many package will get lost. This is what 'flow control' could be helpful for. Downside is that flow control kills the tcp acknowledgement system and any QoS in place....


3) You mean the hardware buffer of the interfaces? Basically we have 'hardware only' everywhere. But is there a tool/way to see it these buffers are utilizes and to what extend? I don't seem to be able to find some info. Only the setting...
Hardware Yes but for our centralized PPPoE server we have set a custom buffer
 /queue> type print detail
 5  name="PPPoE-Customers" kind=pfifo pfifo-limit=10 
Ok, I'm going to try that tonight. See if we see some changes....
4) That's ongoing. My just graduated engineer spend the last week to figure out what we exactly need to set. We are for 80% done in setting the proper mtu everywhere. See if it will indeed improve things...
5) I am not exactly sure what you mean with this? Obviously we have many bridges. Most backhaul radios have their ether net and wireless in a bridge. The bridge actually mirrors the mtu's set on the interfaces.....
My question what happens to CPE Ether max L2MTU of 1526 and then is "Station bridge" to Wlan which has L2MTU of 1600 ?
I'm going to look into this later, got to do some tower work now.. :)
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.
 
WirelessRudy
Forum Guru
Forum Guru
Topic Author
Posts: 3089
Joined: Tue Aug 08, 2006 5:54 pm
Location: Spain

Re: mtu in a mpls/vpls ospf vlan and routed mixed MikroTik network?

Sat May 21, 2016 7:13 pm

[quote]
Hardware Yes but for our centralized PPPoE server we have set a custom buffer
 /queue> type print detail
 5  name="PPPoE-Customers" kind=pfifo pfifo-limit=10 
These are the queues we'd use;
CCR queues.JPG
These are the queue type as used by the simple queues;
CCR simple queues used queue.JPG
And these are the pcq queues used in the simple queues;
pcq queues CCR.JPG
To be honest, I am confused with all these queues. Which, when and where to be used and in what sequence....
any advice is appreciated because its a bit dazzling to me......
You do not have the required permissions to view the files attached to this post.
Show your appreciation of this post by giving me Karma! Thanks.

Rudy R. Puister

WISP operator based on MT routerboard & ROS.

Who is online

Users browsing this forum: No registered users and 101 guests