Community discussions

MikroTik App
 
troy
Member
Member
Topic Author
Posts: 320
Joined: Thu Jun 30, 2005 6:47 pm

Troy's Adventures in MPLS, VPLS, and BGP

Wed Aug 03, 2011 7:56 pm

Title says it all. I got the BGP up and running as per the docs in the wiki, the dynamic vpls tunnels are created and added to the bridge, but they never get to a running state.

Here's the gory details:

RB1200 > RB435 > RB435 > RB1200 (all running ROS 5.5 and all connections are GigE)

IP addresses on a loopback bridge for all devices are 172.17.85.1-4. All routers are running OSPF, all routes in place and are advertising connected routes as type-1.

At first I set up BGP from R1/R2, R2-R3, R3-R4. Nothing worked... even with R1 and R4 as route reflectors. Back to the example, I took BGP out of R2 & R3, set up R1 as a reflector and have BGP session between R1 and R4:
[admin@R1] > /routing bgp peer print status 
Flags: X - disabled, E - established 
 0 E name="BMRS" instance=default remote-address=172.17.85.4 remote-as=65530 tcp-md5-key="" 
     nexthop-choice=default multihop=no route-reflect=yes hold-time=3m ttl=default 
     in-filter="" out-filter="" address-families=l2vpn update-source=lo0 
     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 
     remote-id=172.17.85.4 local-address=172.17.85.1 uptime=15m9s prefix-count=0 
     updates-sent=1 updates-received=1 withdrawn-sent=0 withdrawn-received=0 
     remote-hold-time=3m used-hold-time=3m used-keepalive-time=1m refresh-capability=yes 
     as4-capability=yes state=established 

[admin@R4] > /routing bgp peer print status 
Flags: X - disabled, E - established 
 0 E name="GFT" instance=default remote-address=172.17.85.1 remote-as=65530 tcp-md5-key="" 
     nexthop-choice=default multihop=no route-reflect=no hold-time=3m ttl=default 
     in-filter="" out-filter="" address-families=ip,l2vpn update-source=lo0 
     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 
     remote-id=172.17.85.1 local-address=172.17.85.4 uptime=16m1s prefix-count=0 
     updates-sent=1 updates-received=1 withdrawn-sent=0 withdrawn-received=0 
     remote-hold-time=3m used-hold-time=3m used-keepalive-time=1m refresh-capability=yes 
     as4-capability=yes state=established 
Next, I added the entries for the bgp vpls:
[admin@R1] > /interface vpls bgp-vpls export 
/interface vpls bgp-vpls
add bridge=ppp_bridge bridge-cost=0 bridge-horizon=1 disabled=no export-route-targets=1:4 \
    import-route-targets=1:4 name=ppp_vpls pw-mtu=1500 route-distinguisher=1:4 site-id=1 \
    use-control-word=yes

[admin@R4] > /interface vpls bgp-vpls export 
/interface vpls bgp-vpls
add bridge=ppp_bridge bridge-cost=50 bridge-horizon=1 disabled=no export-route-targets=1:4 \
    import-route-targets=1:4 name=ppp_vpls pw-mtu=1500 route-distinguisher=1:4 site-id=4 \
    use-control-word=yes
At this point, the VPLS interfaces are created and added to the bridge, but they won't enter into a run state:
[admin@R1] > /interface vpls print 
Flags: X - disabled, R - running, D - dynamic, B - bgp-signaled, C - cisco-bgp-signaled 
 0  DB name="vpls1" mtu=1500 l2mtu=1500 mac-address=02:73:BF:5E:F2:14 arp=enabled 
       disable-running-check=no remote-peer=172.17.85.4 cisco-style=no cisco-style-id=0 
       advertised-l2mtu=1500 pw-type=raw-ethernet vpls=ppp_vpls 

[admin@R4] > /interface vpls print 
Flags: X - disabled, R - running, D - dynamic, B - bgp-signaled, C - cisco-bgp-signaled 
 0  DB name="vpls1" mtu=1500 l2mtu=1500 mac-address=02:80:6B:9B:78:80 arp=enabled 
       disable-running-check=no remote-peer=172.17.85.1 cisco-style=no cisco-style-id=0 
       advertised-l2mtu=1500 pw-type=raw-ethernet vpls=ppp_vpls 
So, why won't the VPLS tunnels run? I tried logging mpls, but no useful information came through.

Is there something I need to do with the routers in the middle? The wiki article describes setting up R1, R4, and R5, but does not otherwise mention any special configurations on R2 or R3, which are in the middle.

Thanks,

-Troy
Last edited by troy on Sat Aug 13, 2011 11:24 pm, edited 1 time in total.
 
User avatar
Eising
Member Candidate
Member Candidate
Posts: 272
Joined: Mon Oct 27, 2008 10:21 am
Location: Copenhagen, Denmark

Re: BGP VPLS - tunnels not running

Thu Aug 04, 2011 9:50 am

How about MPLS and LDP? Do you have full MPLS connectivity and label exchange between your routers?
 
troy
Member
Member
Topic Author
Posts: 320
Joined: Thu Jun 30, 2005 6:47 pm

Re: BGP VPLS - tunnels not running

Thu Aug 11, 2011 9:26 pm

How about MPLS and LDP? Do you have full MPLS connectivity and label exchange between your routers?
I'm not sure what you mean by "full MPLS connectivity." Every router on my network has MPLS installed.

As for the LDP, I don't want LDP, I'm using BGP (see the name of the thread).

The BGP VPLS tunnels are created. I can sit here all day long and create VPLS tunnels, but not a one of them will run.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7042
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: BGP VPLS - tunnels not running

Fri Aug 12, 2011 8:36 am

VPLS are not running because BGP signals only labels used for VPLS tunnels. MPLS switching must be configured by setting up LDP on all MPLS routers. Read carefully:
http://wiki.mikrotik.com/wiki/BGP_based ... le_network
 
troy
Member
Member
Topic Author
Posts: 320
Joined: Thu Jun 30, 2005 6:47 pm

Re: BGP VPLS - tunnels not running

Fri Aug 12, 2011 4:03 pm

A couple questions on this...

If you still have to enable LDP on every interface on every router, then what's the advantage of BGP? I can add LDP tunnels as easily as BGP tunnels, and LDP tunnels eliminate the 3rd party interaction with the BGP routing process. So, what's the advantage of BGP again?

Also, if this is the case, then the WIKI is a little unclear:
  • No mention of enabling LDP on any interface on any router in the example network
  • No mention of ANY configuration whatsoever on R2 and R3
If VPLS between R5, R4, and R1 requires that LDP be enabled, the wiki article needs to clearly state that.

As I press my way through, I may take the notes necessary to update the article myself, but I don't know if I'll become comfortable enough with the protocols to make a positive change to the article.

EDIT1: Now, an issue I'm running into: add ether1 as an LDP interface, and it's comes up as invalid, but no hint as to WHY it's invalid. Can you shed some light on this? What are the possible causes for the LDP interface to be invalid?

EDIT2: On another router, when I add LDP interfaces, winbox (and the web server) no longer responds. I can telnet in, remove or disable the interface, and things start working again. EH? What's up with this?
[admin@TCAP_AP2_AP3] /mpls ldp> /system resource pr
                   uptime: 1h37m42s
                  version: 5.6
              free-memory: 113964KiB
             total-memory: 127172KiB
                      cpu: MIPS 24Kc V7.4
                cpu-count: 1
            cpu-frequency: 680MHz
                 cpu-load: 1%
           free-hdd-space: 36592KiB
          total-hdd-space: 61440KiB
  write-sect-since-reboot: 3242
         write-sect-total: 1098616
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: RB433AH
                 platform: MikroTik
[admin@TCAP_AP2_AP3] /mpls ldp> /system routerboard print 
       routerboard: yes
             model: 433AH
     serial-number: 1A7701DAB076
  current-firmware: 2.29
  upgrade-firmware: 2.29
blah.PNG
To be sure... I did a /system reset-configuration, then reconfigured from scratch:
/int br add name=lo0
/ip address add address=172.17.86.27/29 interface=ether1
/ip address add address=171.17.84.62/32 interface=lo0
/routing ospf instance set 0 router-id=172.17.84.62 redistribute-connected=as-type-1
/routing ospf network add network=172.17.86.24/29 area=backbone

... Verify routing comes up ...
... connect via winbox ...

/mpls ldp set lsr-id=172.17.84.62 transport-address=172.17.84.62 enabled=yes
/mpls ldp interface add interface=ether1

... wait a minute... winbox disconnects ...
... remove the ldp interface, scratch head, eat lunch ...
... repeat adding ldp interface, get screenshot, go home ...
For fun, I was able to repeat this on a neighboring RB411AH... add an LDP interface, and same thing...
You do not have the required permissions to view the files attached to this post.
 
troy
Member
Member
Topic Author
Posts: 320
Joined: Thu Jun 30, 2005 6:47 pm

Sat Aug 13, 2011 11:23 pm

Ok, so I'm just going to keep talking to myself.

The problem, it seems, is that the system mtu on the switch was set to 1500. I raised it to 1600, and we're good to go. I'm not sure I understand why this makes a difference, as I'm not using MPLS, VPLS, or any of that crud to communicate with the routers themselves, but it seems that if MPLS/LDP is enabled on 2 routers, then the overhead is incurred. Odd way of doing things, but whatever...

R1 <wireless> R2 <ether> SW1 <ether> AP1 & AP2

So, now that I have overcome that hurdle, I set up the BGP VPLS on 3 routers (R1, AP1, AP2), and the tunnels came up. Unfortunately, they didn't pass 1500 byte packets, which stumped me at first, because I read somewhere (I think), that if a packet is too large, MPLS will fragment and re-assemble it. This isn't happening.

I got to looking closer (by going to the command line), and saw an item called 'pw-mtu,' but I can't seem to find ANY reference to it here in the forum, or on the WIKI (at least Google couldn't find anything). [EDIT: even after posting this, the forum search still comes up empty... search broken?]

So, I figured... VLAN is 4 bytes, so let's see what happens when I set pw-mtu to 1504. Lovely stuff, I can now pass 1500 byte packets through the tunnel.

Question: Why no documentation on pw-mtu? Why is this option hidden away?

Anyways, tunnels are up and running, but not quite as expected... when I brought up R1/AP1, the tunnel was up, in the proper bridge, and traffic passing. When I brought up AP2, it created a tunnel to R1 (desired effect), but it also brought up a tunnel to AP1 (undesired effect). When/if I add other APs to the mix, am I going to be looking at tunnels from every AP to every other AP? I don't think I want this... in fact, I know I don't want this. I need to be able to control/limit any client-to-client communications, there is no free lunch here folks.

I'm starting to lose sight of why I wanted to use MPLS, VPLS, BGP, etc... Seems that the end product isn't really any better than just bridging everything everywhere. Hopefully some kind soul will read this and help me understand what's going on.

Oh, another question... reading about the horizon thingie... with this set, I don't have to worry about (r)stp, right? Did I at least understand that part of the docs correctly?
 
troy
Member
Member
Topic Author
Posts: 320
Joined: Thu Jun 30, 2005 6:47 pm

Re: Troy's Adventures in MPLS, VPLS, and BGP

Sun Aug 14, 2011 1:26 am

Well, I tried to do a little more work on this before calling it a day, but when I tried to add ether1 to the LDP interface on 2 different routers, it kept coming up as invalid. I'm seriously at a loss as to what's going on here, and I can't exactly wipe out the config on these units, as they both have active customers on them. Maybe if I get up early enough some morning, I can give it a try.

Anyone at all know what would cause an LDP interface to be invalid? I'm struggling with this one.
[admin@E19_AP2_AP3] > /mpls exp
# aug/13/2011 18:29:30 by RouterOS 5.6
#
/mpls
set dynamic-label-range=16-1048575 propagate-ttl=yes
/mpls interface
add disabled=no interface=all mpls-mtu=1508
/mpls ldp
set distribute-for-default-route=no enabled=yes hop-limit=255 loop-detect=no \
    lsr-id=172.17.84.53 path-vector-limit=255 transport-address=172.17.84.53 \
    use-explicit-null=no
/mpls ldp interface
add accept-dynamic-neighbors=yes disabled=no hello-interval=5s hold-time=15s \
    interface=ether1 transport-address=0.0.0.0

[admin@E19_AP2_AP3] > /mpls ldp interface print
Flags: X - disabled, I - invalid
 #   INTERFACE                        HELLO-INTERVAL       HOLD-TIME
 0 I ether1                           5s                   15s
The config is IDENTICAL (save for IP addresses) as another router on the same switch. No idea what's going on.
[admin@E19_P2P-TCAP] > /mpls exp
# aug/13/2011 18:32:22 by RouterOS 5.6
# software id = V7FK-L5RC
#
/mpls
set dynamic-label-range=16-1048575 propagate-ttl=yes
/mpls interface
add disabled=no interface=all mpls-mtu=1508
/mpls ldp
set distribute-for-default-route=no enabled=yes hop-limit=255 loop-detect=no \
    lsr-id=172.17.84.60 path-vector-limit=255 transport-address=172.17.84.60 \
    use-explicit-null=no
/mpls ldp interface
add accept-dynamic-neighbors=yes disabled=no hello-interval=5s hold-time=15s \
    interface=ether1 transport-address=0.0.0.0
add accept-dynamic-neighbors=yes disabled=no hello-interval=5s hold-time=15s \
    interface=wlan1 transport-address=0.0.0.0

[admin@E19_P2P-TCAP] > /mpls ldp interface print
Flags: X - disabled, I - invalid
 #   INTERFACE                        HELLO-INTERVAL       HOLD-TIME
 0   ether1                           5s                   15s
 1   wlan1                            5s                   15s
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7042
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: BGP VPLS - tunnels not running

Mon Aug 15, 2011 9:26 am

If you still have to enable LDP on every interface on every router, then what's the advantage of BGP? I can add LDP tunnels as easily as BGP tunnels, and LDP tunnels eliminate the 3rd party interaction with the BGP routing process. So, what's the advantage of BGP again?
Since BGP VPLS deals with autodiscovery and signaling, you don't need to configure VPLS tunnels manually from each to every router. You don't need to maintain full mesh of VPLS tunnels and no need for configuration adjustments for every router if new router is added.
  • No mention of enabling LDP on any interface on any router in the example network
  • No mention of ANY configuration whatsoever on R2 and R3
Consider that MPLS switching is configured and running, as discussed in MPLSVPLS, but no any VPLS configuration has been applied yet. the rest of this document deals with specifics that are introduced by use of BGP for VPLS signaling.
EDIT2: On another router, when I add LDP interfaces, winbox (and the web server) no longer responds. I can telnet in, remove or disable the interface, and things start working again. EH? What's up with this?
When you enable LDP, label switching is performed for all destinations listed in "/mpls forwarding-table" menu. And if if MTUs are not set up properly, packets will be silently discarded and winbox will not work.
If you need label switching only to loopback addresses (for VPLS tunnels) set up ldp filters as discussed in MPLSVPLS article.
Question: Why no documentation on pw-mtu? Why is this option hidden away?
Thank you for pointing it out. This option was added recently and documentation was not updated yet.
 
troy
Member
Member
Topic Author
Posts: 320
Joined: Thu Jun 30, 2005 6:47 pm

Re: BGP VPLS - tunnels not running

Mon Aug 15, 2011 5:53 pm

If you still have to enable LDP on every interface on every router, then what's the advantage of BGP? I can add LDP tunnels as easily as BGP tunnels, and LDP tunnels eliminate the 3rd party interaction with the BGP routing process. So, what's the advantage of BGP again?
Since BGP VPLS deals with autodiscovery and signaling, you don't need to configure VPLS tunnels manually from each to every router. You don't need to maintain full mesh of VPLS tunnels and no need for configuration adjustments for every router if new router is added.
Yeah, I'm starting to get that. Unfortunately, while BPG automates the formation of a full mesh of VPLS tunnels, it seems that a full mesh is the ONLY way BGP signaled VPLS will happen. This is pretty cool for a customer with >2 sites, but not ideal for tunneling individual connections back to a PPPoE concentrator or VLAN termination point. In the end, I think I will stick with BGP signaling, but will manually configure the tunnel to each AP (which kind of sucks, because it will make it difficult to drop in a replacement for the VPLS concentrator when/if needed).

  • No mention of enabling LDP on any interface on any router in the example network
  • No mention of ANY configuration whatsoever on R2 and R3
Consider that MPLS switching is configured and running, as discussed in MPLSVPLS, but no any VPLS configuration has been applied yet. the rest of this document deals with specifics that are introduced by use of BGP for VPLS signaling.
EDIT2: On another router, when I add LDP interfaces, winbox (and the web server) no longer responds. I can telnet in, remove or disable the interface, and things start working again. EH? What's up with this?
When you enable LDP, label switching is performed for all destinations listed in "/mpls forwarding-table" menu. And if if MTUs are not set up properly, packets will be silently discarded and winbox will not work.
If you need label switching only to loopback addresses (for VPLS tunnels) set up ldp filters as discussed in MPLSVPLS article.
Eeeks... that's going to require some work or some very extensive filtering (I'm using the same /24 for PTP and loopbacks). Think I'll tackle this project another day, but keep filters in mind when/if I have to connect through a switch that I can't alter the MTU on.
Question: Why no documentation on pw-mtu? Why is this option hidden away?
Thank you for pointing it out. This option was added recently and documentation was not updated yet.
While were talking about undocumented features, what are the chances that we can specify the name of the VPLS tunnel as well, or at least derive the name from the BGP VPLS entry? It sure would make it easier to see what's going on.

Also, after some more playing around, I think I found out why my LDP interfaces were coming up as invalid. If you have ether1 in a bridge, you can't add it to the LDP interface. Hrrm... is there a way around this if I only need VPLS/LDP/whatever between loopback interfaces?

I'm going to eventually get ether1 out of the bridge, but not until I can turn up the VPLS interface. I'd rather not risk an extended service interruption while implementing VPLS. I'm trying to think my way through this, but to tell the truth, it's starting to hurt my head a little as I add more and more layers on my network.

L1 - Easy
L2 - Implied, trying to get rid of the bridges to everywhere
L3 - Done
L2vpn over L3 over L2 over L1 - Working on it
L3 over L2vpn over L3 over L2 over L1 = Headache!

I feel sorry for the poor sap that has to set up l2vpn over l3 over l2 over l3 over l2 over l1.

The only really cool thing, is that if I can get this done and get rid of all the legacy config options, I'll have something that should be quite beautiful (except for the whole VLAN-per-customer thing that I can't change (yet)).

Thanks for giving me a place to talk to myself, and even more thanks for giving me some feedback.

Today, I get to bring up yet another RB435 with a pair of DBii cards and a RD30 dish. We'll be shooting a 10km link, I can't wait to see this thing humming along, I'll be very happy if actual throughput tops 200Mbit/s.

Who is online

Users browsing this forum: No registered users and 20 guests