MPLS/VPLS seriously, confused...

savage · October 20, 2016, 10:19pm

Ok,

Let’s start with a basic quick question… Can a VPLS tunnel be established between a LER ↔ LER (directly adjacent), or MUST a LSR be in between?

Provided a LER can connect directly to another LER with VPLS…

Router 1: CCR

/interface ethernet
set [ find default-name=ether8 ] l2mtu=10226 mtu=1600 name="ether8"
/mpls interface
set [ find default=yes ] mpls-mtu=1522
/mpls ldp
set enabled=yes loop-detect=yes lsr-id=10.0.1.17 transport-address=10.0.1.17
/mpls ldp interface
add interface="ether8"
/ip address
add address=10.0.1.1/28 interface="ether1" network=10.0.1.0
add address=10.0.1.17 interface=Loopback network=10.0.1.17
add address=10.255.255.254/30 interface="ether8" network=10.255.255.252

Router 2: 750GL

/interface ethernet
set [ find default-name=ether5 ] l2mtu=4074 mtu=1600 name="ether5"
/mpls interface
set [ find default=yes ] mpls-mtu=1522
/mpls ldp
set enabled=yes loop-detect=yes lsr-id=198.19.255.192 transport-address=198.19.255.192
/mpls ldp interface
add interface="ether5"
/ip address
add address=10.255.255.253/30 interface="ether5" network=10.255.255.252
add address=198.19.255.192 interface=Loopback network=198.19.255.192

Both routers are accessible with correct routing, and support large packets:
CCR:

/ping 198.19.255.192 do-not-fragment size=1600
  SEQ HOST                                     SIZE TTL TIME  STATUS             
    0 198.19.255.192                           1600  64 0ms  
    1 198.19.255.192                           1600  64 0ms  
    sent=2 received=2 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

750GL:

/ping 10.0.1.17 do-not-fragment size=1600
  SEQ HOST                                     SIZE TTL TIME  STATUS             
    0 10.0.1.17                                1600  64 0ms  
    1 10.0.1.17                                1600  64 0ms  
    sent=2 received=2 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

VPLS Interface on CCR:

/interface vpls pr      
Flags: X - disabled, R - running, D - dynamic, 
B - bgp-signaled, C - cisco-bgp-signaled 
 0 R   name="VPLS - RB750" mtu=1500 l2mtu=1500 
       mac-address=02:0A:84:6E:D0:58 arp=enabled disable-running-check=no 
       remote-peer=198.19.255.192 vpls-id=1:1 cisco-style=no cisco-style-id=0 
       advertised-l2mtu=1500 pw-type=raw-ethernet use-control-word=default 
/interface vpls monitor 0
       remote-label: 109
        local-label: 282
      remote-status: 
          transport: 0.0.0.0/0
  transport-nexthop: 10.255.255.253
     imposed-labels: 109

On the RB750 however, the interface is down (yet, labels and other information IS present)

/interface vpls print 
Flags: X - disabled, R - running, D - dynamic, 
B - bgp-signaled, C - cisco-bgp-signaled 
 0     name="VPLS - CCR" mtu=1500 l2mtu=1500 
       mac-address=02:33:19:0A:B8:71 arp=enabled arp-timeout=auto 
       disable-running-check=no remote-peer=10.0.1.17 vpls-id=1:1 
       cisco-style=no cisco-style-id=0 advertised-l2mtu=1500 
       pw-type=raw-ethernet use-control-word=default 
/interface vpls monitor 0
   remote-label: 282
    local-label: 109
  remote-status:

The two devices are directly connected, MTU’s are fine, jumbo packets is fine, LDP is fine… I’m pulling my hair out here. The VPLS tunnel on the 750GL just doesn’t want to go up

mducharme · October 20, 2016, 10:38pm

This is not necessarily the cause of your problem, but why do you have MTU set to 1600 for the interfaces connecting the two routers? I see no good reason here to change it from the default 1500.

savage · October 20, 2016, 10:41pm

Just trying sh!t really to get it to work. Have no idea why it’s not working Still need to look at MTU sizes and what not (which is dependent on the type of services I’ll be running).

mducharme · October 20, 2016, 10:44pm

Try lowering the MTU for the interface to 1500, leave the MPLS MTU at 1522. MPLS MTU should generally be larger than MTU for the interface, so with an interface MTU of 1600 you might need an MPLS MTU of 1622 for proper operation with VPLS tunnels.

savage · October 20, 2016, 10:49pm

Nope, same story. Tunnel up on the CCR, down on the 750GL

mducharme · October 20, 2016, 10:50pm

OSPF configuration?
LDP neighbor relationships formed properly?

savage · October 20, 2016, 10:58pm

I believe so yes:
OSPF/LDP CCR:

/mpls ldp neighbor print detail 
Flags: X - disabled, D - dynamic, O - operational, T - sending-targeted-hello, 
V - vpls 
 0 DOTV transport=198.19.255.192 send-targeted=no peer=198.19.255.192:0 
        local-transport=10.0.1.17 
        addresses=10.255.255.253,169.0.253.185,185.159.48.64,198.19.255.1,
          198.19.255.33,198.19.255.65,198.19.255.97,198.19.255.192,
          198.19.255.254 
/ip route print 
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADo  0.0.0.0/0                          10.255.255.253          110
 1 ADo  10.0.0.0/8                         10.255.255.253          110
 2 ADoU 10.0.1.0/27                                                109
 3 ADC  10.0.1.0/28        10.0.1.1        ether1 - WBTG-HS01        0
 4 ADC  10.0.1.17/32       10.0.1.17       Loopback                  0
 5 ADo  10.0.1.24/32                       10.0.1.8                110
 6 ADC  10.255.255.252/30  10.255.255.254  ether8 - tmp Up...        0

OOOH - no route to 198.19.255.192/32… mmmmmmmmm Let me check that (OSPF filters more than likely). I recall a post from wee wee in the past that a default route wasn’t sufficient. It needed a more precise route. That’s something to look at, definitely. Always helps when an extra pair of eyes are available

OSPF/LDP 750GL:

/mpls ldp neighbor print detail 
Flags: X - disabled, D - dynamic, O - operational, T - sending-targeted-hello, 
V - vpls 
 0 DOTV transport=10.0.1.17 send-targeted=yes peer=10.0.1.17:0 
        local-transport=198.19.255.192 
        addresses=10.0.1.1,10.0.1.17,10.255.255.254,185.159.48.17 
/ip route print 
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 1 ADS  0.0.0.0/0                          x.x.x.129             1
 4 ADo  10.0.1.0/27                        10.255.255.254          110
 5 ADC  10.255.255.252/30  10.255.255.253  ether5 - High Site        0
<snip>
23 ADC  198.19.255.192/32  198.19.255.192  Loopback                  0

Summary route on the 750GL better be OK… I’d HATE to populate my tables with hundreds of /32s.

Lemme sort out that missing route, and see what happens then

mducharme · October 20, 2016, 11:02pm

You do need to add your loopbacks. You can’t summary in one area with OSPF. Yes, you will have a bunch of /32’s but it shouldn’t be a huge deal.

savage · October 20, 2016, 11:05pm

Ok, so route to my RB750GL’s loopback was missing, and after fixing that, the transport address on the VPLS tunnel switched to the local interface address instead of the loopback address (which is what I would expect).

On the RB750GL (this is a bug IMHO), the 10.0.1.0/27 OSPF Summary Route, IS NOT ENOUGH. There MUST be a specific /32 route. That’s a bigy for me…

Mikrotik guys?

VPLS is now up on both sides. I don’t however agree with the fact that a /32 is REQUIRED to bring the tunnel up, a summary route SHOULD be sufficient. ROS6.36.6 on the affected RB750GL.

savage · October 20, 2016, 11:06pm

ARGH Seriously?!?!?!

Ok, well that’s my issue then. Will need to look at what I can do there. The entire network is summarized for reasons.

mducharme · October 20, 2016, 11:08pm

OSPF only allows route summarization going to different areas.. if you have routers in area 0 that have /32’s, those have to be individual routes in area 0, but those routes can be summarized going to area 1. This is by design with OSPF.

savage · October 20, 2016, 11:13pm

Yes.

I have 100s of /27 areas. /28 used for a LAN, /28 used for loopbacks. The entire /27 is summarized back to the backbone.

So I can’t summarize at all anymore now. Will have to advertise the /28 LAN, and 16 x /32s loopbacks

mducharme · October 20, 2016, 11:17pm

Also factor in if you want to use MPLS traffic engineering - typically then you will want a flatter OSPF design without lots of extra areas. MPLS TE can only function within a single area per router.

The good news is that MPLS and VPLS cut down greatly on the processing load of the router because it is so efficient, so the impact of the larger routing tables is minimized for any tunneled traffic.

I have a fairly large single area OSPF network - about 40 routers in the one area. No real problems with it, unless routers are going up and down all the time. MPLS/MPLS-TE/VPLS work very well, MPLS traffic is very fast. I have a few other OSPF areas but they are for smaller remote islands.

The only place I would probably summarize is end user routes, where if they are supplied a router, it might go up or down (ex. /32’s for PPPoE).

StubArea51 · October 21, 2016, 2:11pm

Might want to look at raising your MTU slightly as well. Here is a presentation I did at the MUM in MPLS for WISPs (applies to Fiber/Copper ISPs as well) and it has a slide on page 13 that lists best practices for MTUs that we have used in many production MPLS networks.

http://mum.mikrotik.com/presentations/US16/presentation_3327_1462279781.pdf

savage · October 21, 2016, 2:14pm

Thnx

ZeroByte · October 21, 2016, 2:33pm

I’ve seen several statements that it’s bad practice to summarize loopback IPs anyway (and this was all in Cisco forums), so I wouldn’t consider that to be a limitation in Mikrotik land.

It kind of makes sense when you stop and ponder how inter-area routing works. Summaries have metrics attached to them that come in two basic flavors:
Cost = the worst cost of any sub-prefix seen by the ABR + path cost to ABR
Cost = path cost to ABR itself (this one I don’t completely remember, but I think it’s either just the router itself, or else the router + cost of only first hop into area)

Whatever the case, the routers outside the summarized area might choose sub-optimal routes. The distance to the ABR itself is a factor as well. So some router might have a short path to reach some ABR but the ABR is in fact not the best choice to reach some particular destination inside the area itself. (Another ABR would’ve been a better choice, but since the particulars of the summarized route hide the final details)

When using protocols that tunnel directly between a given pair of routers, it’s good for the routers to be able to see explicit details about how to reach each other, so that the best choice can be made.

I don’t have experience with TE tunnels specifically, though, so definitely take some of these other posters’ opinions above mine.

savage · October 21, 2016, 2:42pm

Can’t see why I would deploy TE, so I’m not too phased about it at this stage. It may change after I sit for a few hours and apply my mind to all of this though. The main reasoning at this stage for MPLS is VRF’s, to ‘hide’ internal parts of the networks running on RFC1918, and fragmentation.

But yes, as others suggested I would definitely need to look at stopping to summarize. It’s not such a big deal, just a bit of a pain. I really wasn’t aware that the actual /32 (loopback) must be in the routing table, and excuse my ignorance, but I don’t recall anywhere that this is documented either, the documentation merely states “properly routed” and “reachable” in terms of loopbacks - a tad bit misleading I would say (I recall seeing a post from I think Normis a few years ago, mentioning something to this affect in some random forum post - and that is all). So just a tad bit surprised really I guess.

Either way, lesson learned

mrz · October 21, 2016, 2:51pm

You need actual loopback route in the table because LDP uses that routing information to assign labels and distribute to the neighbors. If such route does not present or is summarized then LDP cannot do that.