Community discussions

MikroTik App
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

PPPoE over EOIP - better switch to VPLS?

Fri Nov 14, 2014 10:28 am

Guys,
I need your advice.

Situation:
Small WISP with about 150 customers on wireless.
Fully routed network on 5.x GHz
Every AP has EOIP tunnel to central router. EOIP tunnes bridged together and PPPoE server running on bridge.
At APs the interfaces connecting customers are bridged with the EOIP tunnels.
Central router/PPPoE server is CCR 1036, heaps of CPU available.

Problem:
Trouble achieving desired speed at peak times.
Wireless links are tested and perform well over actual aggregated speed to customers.

So what is the problem?
I have been wondering whether this is about fragmentation.
Ping from CPE to PPPoE server with "Don't fragment" stops at packet size above 1480.
What would be the optimal MTU/MRU/MRRU settings for EOIP and PPPoE?
I control every device in the network so I can do whatever I need to.

Another theory is that the concept of EOIP is slowing me down and that it is worth moving to VPLS.
According to Janis: http://mum.mikrotik.com/presentations/U ... s_mpls.pdf , at certain conditions VPLS is 60% faster than EOIP. Is that really true?

Considering employing VPLS, will I have to create LDP on every single router interface to adjacent router, or will bridged routers be transparent to LDP?
For instance, at many sites the APs for connecting downstream devices is typically a Sextant with ether1 and wlan1 bridged. Can i omit these when applying LDP, defining LDP only on downstream and upstream routers?

What is your experience guys with VPLS in such a scenario?
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Mon Nov 17, 2014 2:18 pm

*BUMP*

Nobody out there with experience with VPLS??
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: PPPoE over EOIP - better switch to VPLS?

Mon Nov 17, 2014 3:11 pm

Guys,
I need your advice.

Situation:
Small WISP with about 150 customers on wireless.
Fully routed network on 5.x GHz
Every AP has EOIP tunnel to central router. EOIP tunnes bridged together and PPPoE server running on bridge.
At APs the interfaces connecting customers are bridged with the EOIP tunnels.
Central router/PPPoE server is CCR 1036, heaps of CPU available.

Problem:
Trouble achieving desired speed at peak times.
Wireless links are tested and perform well over actual aggregated speed to customers.

So what is the problem?
I have been wondering whether this is about fragmentation.
Ping from CPE to PPPoE server with "Don't fragment" stops at packet size above 1480.
What would be the optimal MTU/MRU/MRRU settings for EOIP and PPPoE?
I control every device in the network so I can do whatever I need to.

Another theory is that the concept of EOIP is slowing me down and that it is worth moving to VPLS.
According to Janis: http://mum.mikrotik.com/presentations/U ... s_mpls.pdf , at certain conditions VPLS is 60% faster than EOIP. Is that really true?

Considering employing VPLS, will I have to create LDP on every single router interface to adjacent router, or will bridged routers be transparent to LDP?
For instance, at many sites the APs for connecting downstream devices is typically a Sextant with ether1 and wlan1 bridged. Can i omit these when applying LDP, defining LDP only on downstream and upstream routers?

What is your experience guys with VPLS in such a scenario?
MPLS/VPLS is the way to go as it gives you a 1500byte Network without fragmentation. Pushing 1500 byte over tunnels which are smaller doubles the needed packet rate of the routers/switches in between and needs the tunnel routers to cut/paste packets.

If your routers are all fast enough to process the current bandwidth and assemble/disassemble packets you will not see a difference.
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Mon Nov 17, 2014 3:20 pm

Thank you for your reply ste.

Yes there is something in there that is slowing me down.
First I thought it was my central router that could not cope with it all, so I upgraded from RB1100AH to CCR-1036
Then I suspected the load of simple queues not being distributed to all cores, so I went for queue trees with PCQ.
Then I pointed at my mainlink and changed it to ac which is capable of more than I need.

So now I am on the "fragmentation theory" which you partly confirm as far as I understand.

What about my LDP question?
Also, will the LDP loopback addresses have to be different all the way through my network?
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: PPPoE over EOIP - better switch to VPLS?

Mon Nov 17, 2014 3:58 pm

Thank you for your reply ste.

Yes there is something in there that is slowing me down.
First I thought it was my central router that could not cope with it all, so I upgraded from RB1100AH to CCR-1036
Then I suspected the load of simple queues not being distributed to all cores, so I went for queue trees with PCQ.
Then I pointed at my mainlink and changed it to ac which is capable of more than I need.

So now I am on the "fragmentation theory" which you partly confirm as far as I understand.

What about my LDP question?
Also, will the LDP loopback addresses have to be different all the way through my network?
A bridge is transparent to MPLS/VPLS as long as it's L2MTU is high enough to carry the packets. So a Wireless bridge in between dont need MPLS/VPLS configured. Each MPLS/VPLS Router needs a loopback address and all Interfaces which are within the MPLS/VPLS Network have to be included.
Be aware that troubleshooting routing will be more difficult as traceroute does no longer work as expected. Do some reading in the Mikrotik-Wiki to understand what happens.
 
User avatar
tomaskir
Trainer
Trainer
Posts: 1162
Joined: Sat Sep 24, 2011 2:32 pm
Location: Slovakia

Re: PPPoE over EOIP - better switch to VPLS?

Mon Nov 17, 2014 8:48 pm

Watch the presentation in my sig.
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 11:09 am

Thank you for your help so far ste and tomaskir.

I am struggling trying to get LDP to work. Appears like it's arbitrary which routers appear as dynamic neighbors.

To sort out anything that might disturb I threw up 3x RB750GL and made a simple setup.
ROS 6.22, started with /system reset-configuration before applying settings.
I just used Janis' presentation http://mum.mikrotik.com/presentations/CZ09/MPLS.pdf as a basis

Router1 and Router2 are connected ether2-ether2 while Router2 and Router3 are connected ether3-ether3.
Router1 sees Router2 as dynamic neighbor and Router2 sees Router3.
However, Router2 does not see Router1 and Router3 does not see Router2.

Any idea what's wrong??

This is the config:

Router1:
/interface bridge
add name=bridge-loopback

/ip address
add address=9.9.9.1/32 interface=bridge-loopback 
add address=10.0.0.1/24 interface=ether2

/ip route
add dst-address=10.0.1.0/24 gateway=10.0.0.2

/mpls ldp
set enabled=yes lsr-id=9.9.9.1 transport-address=9.9.9.1

/mpls ldp interface
add interface=ether2

/system identity
set name=Router1
Router2:
/interface bridge
add name=bridge-loopback

/ip address
add address=9.9.9.2/32 interface=bridge-loopback 
add address=10.0.0.2/24 interface=ether2
add address=10.0.1.1/24 interface=ether3

/mpls ldp
set enabled=yes lsr-id=9.9.9.2 transport-address=9.9.9.2

/mpls ldp interface
add interface=ether2
add interface=ether3

/system identity
set name=Router2
Router3:
/interface bridge
add name=bridge-loopback

/ip address
add address=9.9.9.3/32 interface=bridge-loopback 
add address=10.0.1.2/24 interface=ether3 

/ip route
add dst-address=10.0.0.0/24 gateway=10.0.1.1

/mpls ldp
set enabled=yes lsr-id=9.9.9.3 transport-address=9.9.9.3

/mpls ldp interface
add interface=ether3

/system identity
set name=Router3
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 11:28 am

I am missing the routes for the loopback addresses.
Routing/ping have to work before MPLS jumps in.
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 12:07 pm

Ok so routes for the loopback interfaces must be present as well? I didn't know that.

Then how can some of the dynamic neighbors show anyway?
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 4:33 pm

Actually, adding the routes for the loopback IPs did the trick, thank you.

For testing purposes I set up a PPPoE server on the VPLS interface on Router1, and a PPPoE client on the other end at Router3, and it establishes like a charm.
But I am still confused about the settings for MRRU, MTU and MRU.

I have set the following for the VPLS tunnel:
MTU: 1500
MRU: 1500
Advertised L2MTU: 1500

This should at least restrict the packet sizes running unfragmented through the tunnel to 1500 or what?

The PPPoE server defaults to an MRRU of 1600. I can see from terminal that Max MTU and Max MRU defaults to 1480.

But in PPPoE client status tab MRU is 1600 and MTU is 1596.
How can it possibly be when the VPLS tunnel through which it runs is 1480???

I need advise on what to configure for the VPLS tunnel and for the PPPoE in order to achieve maximum throughput and prevent fragmentation.
Ideally I would like to pass full ethernet frames all the way through.
In my "real world" network I control every network component all the way through, if switches should be a critical point.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 4:41 pm

The minimal MPLS MTU to support VPLS is 1526. If you want to carry vlans you've to add the tagging bytes.
Make sure every MPLS device has set this in MPLS Interface and be sure every link/switch supports this big packets.

Then your vpls tunnel are able to carry 1500 Byte packages without fragmentation.

Subtract your pppoe header information and you get the MTU you can carry inside of your PPPOE-Tunnel.
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 4:51 pm

Thank you ste,

You are talking about MPLS and VPLS together so I'm still a little confused.
The MTU I shall set on the VPLS interface, right? What has this got to do with MPLS?

Will the PPPoE be able to carry 1500 bytes?

Sorry for my ignorance, but as you understand this is new stuff to me....
Can you please guide me on all the values I need to set both for my VPLS tunnel and for the PPPoE?
There are MRRU, MRU, MTU, L2MTU, Advertised L2MTU and I am unable to see the whole picture.

And regarding my previous post, how can the PPPoE tunnel carry larger packets than the tunnel through which it runs?
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: PPPoE over EOIP - better switch to VPLS?

Wed Nov 26, 2014 5:23 pm

Thank you ste,
You are talking about MPLS and VPLS together so I'm still a little confused.
The MTU I shall set on the VPLS interface, right? What has this got to do with MPLS?
No. The MPLS MTU (winbox->MPLS->MPLS Interface) Interface=all.
This is the packet size MPLS uses.
Will the PPPoE be able to carry 1500 bytes?
Then MPLS MTU has to be at least 1546.
Sorry for my ignorance, but as you understand this is new stuff to me....
Can you please guide me on all the values I need to set both for my VPLS tunnel and for the PPPoE?
There are MRRU, MRU, MTU, L2MTU, Advertised L2MTU and I am unable to see the whole picture.

And regarding my previous post, how can the PPPoE tunnel carry larger packets than the tunnel through which it runs?
You've to draw a picture how packets look like to understand what is happening.
Read this: http://mum.mikrotik.com/presentations/CZ09/MPLS.pdf
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Thu Nov 27, 2014 10:00 am

Thank you once again ste for spending time on this.

I have now tried to implement the setup according to the presentations made by Janis and Kirnak.

Setup:

Router1:
/interface bridge
add name=bridge-loopback

/interface vpls
add advertised-l2mtu=1508 name=R1toR3 disabled=no\
    remote-peer=9.9.9.3 vpls-id=10:0

/ip pool
add name=PPPoE-pool ranges=192.168.1.100-192.168.1.199

/ppp profile
add local-address=192.168.1.1 name=PPPoE-profile remote-address=PPPoE-pool

/interface pppoe-server server 
add default-profile=PPPoE-profile interface=R1toR3 max-mru=1500 \
    max-mtu=1500 service-name=PPPoE-server1 disabled=no 

/ip address
add address=9.9.9.1/32 interface=bridge-loopback 
add address=10.0.0.1/24 interface=ether2

/ip route
add dst-address=9.9.9.2/32 gateway=10.0.0.2
add dst-address=9.9.9.3/32 gateway=10.0.0.2
add dst-address=10.0.1.0/24 gateway=10.0.0.2

/mpls interface
set [ find default=yes ] mpls-mtu=1550

/mpls ldp
set enabled=yes lsr-id=9.9.9.1 transport-address=9.9.9.1

/mpls ldp interface
add interface=ether2

/ppp secret
add name=test password=test profile=PPPoE-profile service=pppoe

/system identity
set name=Router1
Router2:
/interface bridge
add name=bridge-loopback

/ip address
add address=9.9.9.2/32 interface=bridge-loopback 
add address=10.0.0.2/24 interface=ether2
add address=10.0.1.1/24 interface=ether3

/ip route
add dst-address=9.9.9.1/32 gateway=10.0.0.1
add dst-address=9.9.9.3/32 gateway=10.0.1.2

/mpls ldp
set enabled=yes lsr-id=9.9.9.2 transport-address=9.9.9.2

/mpls ldp interface
add interface=ether2
add interface=ether3

/system identity
set name=Router2
Router3:
/interface bridge
add name=bridge-loopback

/interface vpls
add advertised-l2mtu=1508 name=R3toR1 disabled=no \
    remote-peer=9.9.9.1 vpls-id=10:0

/interface pppoe-client
add add-default-route=yes disabled=no\
    interface=R3toR1 \
    max-mru=1500 max-mtu=1500 name=pppoe-out1 \
    password=test use-peer-dns=yes user=test

/ip address
add address=9.9.9.3/32 interface=bridge-loopback 
add address=10.0.1.2/24 interface=ether3 

/ip route
add dst-address=9.9.9.1/32 gateway=10.0.1.1
add dst-address=9.9.9.2/32 gateway=10.0.1.1
add dst-address=10.0.0.0/24 gateway=10.0.1.1

/mpls interface
set [ find default=yes ] mpls-mtu=1550

/mpls ldp
set enabled=yes lsr-id=9.9.9.3 transport-address=9.9.9.3

/mpls ldp interface
add interface=ether3

/system identity
set name=Router3
Now maximum packet size over PPPoE tunnel is 1478.
How can I make it 1500?

Funny thing is that with my previous setup, without touching MPLS interface MTU, I was able to ping with 1596 bytes packet size, appearently unfragged.
As soon as I touched this setting (Raised it!) the ping failed.

Can you see what's restricting me now?
 
User avatar
tomaskir
Trainer
Trainer
Posts: 1162
Joined: Sat Sep 24, 2011 2:32 pm
Location: Slovakia

Re: PPPoE over EOIP - better switch to VPLS?

Thu Nov 27, 2014 11:10 am

Watch the presentation in my sig, as I mentioned earlier.

It goes into heavy detail on MTU with MPLS/VPLS and especially PPPoE over VPLS.
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Thu Nov 27, 2014 10:24 pm

Yes Tomas, I've watched it and read through the pdf several times, and I've tried to implement my setup according to what's mentioned there and in Janis' presentation.
But as you understand I still have some problems getting the whole picture, and I cannot just point on which setting has to be changed.
So it would be very kind if you experienced guys in here would take your time to look through my config above and tell me what has to be changed to obtain a full 1500 bytes MTU over PPPoE.

Or maybe Janis could chime in?

Thanks in advance!
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Mon Dec 01, 2014 1:22 pm

In my test setup I'm playing with MTUs, but I never get more than 1478 bytes packet size over PPPoE.

I have raised MTU of ethernet interfaces to 1584 and they seem to let me through all fine with 1584 bytes.
Next is MPLS interface MTU which is currently set to 1550.
On VPLS interface I now tried Advertised L2MTU of 1550 and MTU of 1542.
Tried to set statis IPs on both end of the VPLS tunnel, however I can't get more than 1486 bytes through it....this is weird? :shock:

On PPPoE server and client I am unable to raise above 1500 MTU.

So what is the trick to achieve what I want, i.e. 1500 bytes MTU over PPPoE?
I am afraid reposting of the links to the mentioned presentations won't get me moving, I need an advise from you gurus out there on which values to use on the various interfaces.

Thanks in advance!
 
Bomber67
Member
Member
Topic Author
Posts: 385
Joined: Wed Nov 08, 2006 10:36 am

Re: PPPoE over EOIP - better switch to VPLS?

Tue Dec 02, 2014 8:49 am

Some progress is made, this is my latest config, employing OSPF instead of static routes:

Router1:
/interface ethernet
set [ find default-name=ether2 ] mtu=1584

/interface bridge
add name=bridge-loopback

/interface vpls
add advertised-l2mtu=1508 name=R1toR3 disabled=no\
    remote-peer=9.9.9.3 vpls-id=10:0

/ip pool
add name=PPPoE-pool ranges=192.168.1.100-192.168.1.199

/ppp profile
add local-address=192.168.1.1 name=PPPoE-profile remote-address=PPPoE-pool

/interface pppoe-server server 
add default-profile=PPPoE-profile interface=R1toR3 max-mru=1500 \
    max-mtu=1500 service-name=PPPoE-server1 disabled=no 

/ip address
add address=9.9.9.1/32 interface=bridge-loopback 
add address=10.0.0.1/24 interface=ether2

/routing ospf instance
set [ find default=yes ] router-id=9.9.9.1

/routing ospf network
add area=backbone network=9.9.9.0/24
add area=backbone network=10.0.0.0/24

/mpls interface
set [ find default=yes ] mpls-mtu=1550

/mpls ldp
set enabled=yes lsr-id=9.9.9.1 transport-address=9.9.9.1

/mpls ldp interface
add interface=ether2

/ppp secret
add name=test password=test profile=PPPoE-profile service=pppoe

/system identity
set name=Router1
Router2:
/interface ethernet
set [ find default-name=ether2 ] mtu=1584
set [ find default-name=ether3 ] mtu=1584

/interface bridge
add name=bridge-loopback

/ip address
add address=9.9.9.2/32 interface=bridge-loopback 
add address=10.0.0.2/24 interface=ether2
add address=10.0.1.1/24 interface=ether3

/routing ospf instance
set [ find default=yes ] router-id=9.9.9.2

/routing ospf network
add area=backbone network=9.9.9.0/24
add area=backbone network=10.0.0.0/24
add area=backbone network=10.0.1.0/24

/mpls interface
set [ find default=yes ] mpls-mtu=1550

/mpls ldp
set enabled=yes lsr-id=9.9.9.2 transport-address=9.9.9.2

/mpls ldp interface
add interface=ether2
add interface=ether3

/system identity
set name=Router2
Router3:
/interface ethernet
set [ find default-name=ether3 ] mtu=1584

/interface bridge
add name=bridge-loopback

/interface vpls
add advertised-l2mtu=1508 name=R3toR1 disabled=no \
    remote-peer=9.9.9.1 vpls-id=10:0

/interface pppoe-client
add add-default-route=yes disabled=no\
    interface=R3toR1 \
    max-mru=1500 max-mtu=1500 name=pppoe-out1 \
    password=test use-peer-dns=yes user=test

/ip address
add address=9.9.9.3/32 interface=bridge-loopback 
add address=10.0.1.2/24 interface=ether3 

/routing ospf instance
set [ find default=yes ] router-id=9.9.9.3

/routing ospf network
add area=backbone network=9.9.9.0/24
add area=backbone network=10.0.1.0/24

/mpls interface
set [ find default=yes ] mpls-mtu=1550

/mpls ldp
set enabled=yes lsr-id=9.9.9.3 transport-address=9.9.9.3

/mpls ldp interface
add interface=ether3

/system identity
set name=Router3
I have set Ethernet interface MTUs to 1584 just to be sure. This has to be 14 bytes below L2MTU, right?
With MPLS interface MTU of 1550 everything seems fine.

I can see by sniffing that with MPLS interface MTU below 1534 the packets were fragged on the ethernet interface, so I set it to 1534. Then on router 3 packets go unfragged at tx 1548 bytes and rx 1544 on the Ethernet interface (why not same size?).
When sniffing on the VPLS interface I see that traffic goes unfragged at a packet size of 1522.
Sniffing on the PPPoE interface packets are 1500 bytes.

I probably also want to run VLANs on my VPLS tunnels,so for a test (not included in code above) I raised MPLS interface MTU to 1538, VPLS L2MTU/MTU to 1512/1504 and set VLAN MTU to 1500. This seems to work, as packets then goes unfragged on Ethernet at 1552/1548.

Does it seem that I finally found a formula, or do I miss any important considerations?

In Kirnak’s presentation there is a drawing on page 19 showing all the headers adding up, and the required MTU sizes.
However, I am unable to see where my numbers fit in. in this sketch MPLS interface MTU is 1530, how can that work, as I need 1534 (without VLANS)?
 
ozairakhlaq
just joined
Posts: 24
Joined: Fri Mar 16, 2018 11:34 am

Re: PPPoE over EOIP - better switch to VPLS?

Thu Dec 12, 2019 8:34 am

Hi @bomber76

I also have a huge network running with PPPoE over EOIP, but it feels that my network performance is not upto the mark.
Just wondering if you did make VPLS run properly on your network and if it did enhance your network performance.
 
marekm
Member
Member
Posts: 391
Joined: Tue Feb 01, 2011 11:27 pm

Re: PPPoE over EOIP - better switch to VPLS?

Thu Dec 12, 2019 2:56 pm

Just a suggestion since the thread is a few years old and RFC4638 was implemented in the meantime: +8 to all these MTUs, so that PPPoE can pass MTU 1500 without fragmentation (if supported by both ends of PPPoE tunnel). I'm also interested in converting a big bridge passing PPPoE into something more manageable (EoIP vs VPLS not decided yet).
 
User avatar
Anumrak
Forum Guru
Forum Guru
Posts: 1174
Joined: Fri Jul 28, 2017 2:53 pm

Re: PPPoE over EOIP - better switch to VPLS?

Thu Dec 12, 2019 3:45 pm

VPLS is always better.

Who is online

Users browsing this forum: No registered users and 55 guests