MPLS - Forwarding table incorrect

RouterOS Forwarding Protocols
1

I’m new to MPLS on RouterOS and don’t understand why the forwarding table on CCR1, in the example below, contains an outbound label. Is there perhaps something wrong with my OSPF configuration?

Overview:

CCR1                   lo = 41.79.21.1
    seacom-vlan21-vlan210 = 198.19.12.53/30
       |
    seacom-vlan21-vlan210 = 198.19.12.54/30 
CCR2                   lo = 41.79.23.1
  neotel-vlan1323-vlan210 = 198.19.16.5/30
       |
    customer

Problem:

CCR1 is unable to reach 41.79.23.48/29 subnet via CCR2. I assume MPLS forwarding table should not have ‘out-labels’ associated for the route to 41.79.23.48/29, as the MPLS label should be popped before forwarding it on to CCR2 (last MPLS router before customer CPE). I would have expected ‘mpls forwarding-table print detail where destination in 41.79.23.48/29’ to show the following:
0 L in-label=2590 out-labels=“” interface=seacom-vlan21-vlan210 nexthop=198.19.12.54 destination=41.79.23.48/29

PS: Loading ‘/mpls ldp accept-filter add accept=no prefix=41.79.23.48/29’ restores connectivity but obviously disables MPLS for the subnet.



CCR1:

[admin@Syrex - CCR1] > ip route print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 ADo  dst-address=41.79.23.48/29 gateway=198.19.12.54 
        gateway-status=198.19.12.54 reachable via  seacom-vlan21-vlan210 
        distance=110 scope=20 target-scope=10 ospf-metric=30 
        ospf-type=external-type-1

[admin@Syrex - CCR1] > mpls forwarding-table print detail where destination in 41.79.23.48/29
Flags: L - ldp, V - vpls, T - traffic-eng
 0 L in-label=2590 out-labels=1155 interface=seacom-vlan21-vlan210 nexthop=198.19.12.54 destination=41.79.23.48/29

[admin@Syrex - CCR1] > mpls local-bindings print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - advertised, D - dynamic, L - local-route, G - gateway-route, e - egress
 0 ADG  dst-address=41.79.23.48/29 label=2590 adv-path="empty" peers=41.79.23.1:0

[admin@Syrex - CCR1] > mpls remote-bindings print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - active, D - dynamic
 0 AD dst-address=41.79.23.48/29 label=1155 nexthop=198.19.12.54 peer=41.79.23.1:0 path="empty"

CCR2:

[admin@Syrex - CCR2] > ip route print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=41.79.23.48/29 gateway=198.19.16.6 
        gateway-status=198.19.16.6 reachable via  neotel-vlan1323-vlan210 
        distance=1 scope=30 target-scope=10 

[admin@Syrex - CCR2] > mpls forwarding-table print detail where destination in 41.79.23.48/29
Flags: L - ldp, V - vpls, T - traffic-eng
 0 L in-label=1155 out-labels="" interface=*0 nexthop=198.19.16.6 destination=41.79.23.48/29

[admin@Syrex - CCR2] > mpls local-bindings print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - advertised, D - dynamic, L - local-route, G - gateway-route, e - egress 
 0 ADG  dst-address=41.79.23.48/29 label=1155 adv-path="empty" peers=41.79.21.1:0,154.119.64.1:0,154.119.64.2:0 

[admin@Syrex - CCR2] > mpls remote-bindings print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - active, D - dynamic
 0  D dst-address=41.79.23.48/29 label=2590 peer=41.79.21.1:0 path="empty" 
 1  D dst-address=41.79.23.48/29 label=46 peer=154.119.64.1:0 path="empty" 
 2  D dst-address=41.79.23.48/29 label=48 peer=154.119.64.2:0 path="empty"

Configurations:

CCR1:

/ip address
  add address=41.79.21.1 interface=lo network=41.79.21.1
  add address=198.19.12.53/30 interface=seacom-vlan21-vlan210 network=198.19.12.52
/routing ospf instance
  set [ find default=yes ] redistribute-connected=as-type-1 redistribute-static=as-type-1 router-id=41.79.21.1
/routing ospf interface
  add authentication=md5 authentication-key=notrealkey dead-interval=2m hello-interval=1s interface=seacom-vlan21-vlan210 network-type=point-to-point
/routing ospf network
  add area=backbone network=198.19.12.52/30
/mpls interface
  set [ find default=yes ] mpls-mtu=4400
/mpls ldp
  set enabled=yes lsr-id=41.79.21.1 transport-address=41.79.21.1
/mpls ldp interface
  add interface=seacom-vlan21-vlan210

CCR2:

/ip address
  add address=41.79.23.1 interface=lo network=41.79.23.1
  add address=198.19.12.54/30 interface=seacom-vlan21-vlan210 network=198.19.12.52
/routing ospf instance
  set [ find default=yes ] redistribute-connected=as-type-1 redistribute-static=as-type-1 router-id=41.79.23.1
/routing ospf interface
  add authentication=md5 authentication-key=notrealkey dead-interval=2m hello-interval=1s interface=seacom-vlan21-vlan210 network-type=point-to-point
/routing ospf network
  add area=backbone network=198.19.12.52/30
/mpls interface
  set [ find default=yes ] mpls-mtu=4400
/mpls ldp
  set enabled=yes lsr-id=41.79.23.1 transport-address=41.79.23.1
/mpls ldp interface
  add interface=seacom-vlan21-vlan210
2

What RouterOS version are you using?

Are there other MPLS enabled routers in this path that aren’t MikroTIk?

I would expect your forwarding table entry for a non MPLS subnet to look more like this (without an out label)

[admin@CCR1] > mpls forwarding-table print detail where destination in 41.79.23.48/29
Flags: L - ldp, V - vpls, T - traffic-eng
 0 L in-label=17 out-labels="" interface=seacom-vlan21-vlan210 nexthop=198.19.12.54
     destination=41.79.23.48/29
3

Both routers are running 6.33.5 and there are no other routers in the path between CCR1 and CCR2, so my understanding appears to match yours.

I’m sure the folks at Mikrotik get loads of false positives so I wanted to confirm this with someone who had MPLS experience before wasting their time.


Thanks for your help!

4

If you post your full config and version, we may be able to help more. We do a lot of MPLS and the scenario you described normally works without issue on just about all the 6.x code versions. I don’t think you are running into a bug. Did you edit the label range because the label numbering typically starts at 16,17,etc by default. Not that it should matter, but it seems like you may have some kind of a hidden config issue.

5

We sometimes face the situation that mpls forwarding differs from routing information learned by ospf. Disabling/enabling LDP helps in this situation. It is triggered by a reboot. Sent supout to MT months ago but they can’t figure it out. This happens only at one site so seems configuration dependent. Overall we see slow mpls adaption. Without mpls our ospf network adapts fast to link failures. Quite annoying sometimes as everything is ok but mpls forwarding still goes in the wrong direction for minutes. LDP down and up and it runs again.

6

The two routers contain fairly extensive configurations and additionally handle BGP sessions to 9 peers between them.

We have redundant links between CCR1 and CCR2 with OSPF weighting to make them work as active/passive.The problem appears to have disappeared when disabling OSPF and LDP on the backup interfaces.

MPLS still has an out label on CCR1 which I believe shouldn’t be there but at least now it’s working:

ip route print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 0 ADo  dst-address=41.79.23.48/29 gateway=198.19.12.54 gateway-status=198.19.12.54 reachable via  seacom-vlan21-vlan210 distance=110 scope=20 target-scope=10 ospf-metric=30 ospf-type=external-type-1

mpls forwarding-table print detail where destination in 41.79.23.48/29
Flags: L - ldp, V - vpls, T - traffic-eng
 0 L in-label=141 out-labels=299 interface=seacom-vlan21-vlan210 nexthop=198.19.12.54 destination=41.79.23.48/29

mpls local-bindings print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - advertised, D - dynamic, L - local-route, G - gateway-route, e - egress
 0 ADG  dst-address=41.79.23.48/29 label=141 adv-path="empty" peers=41.79.23.1:0

mpls remote-bindings print detail where dst-address in 41.79.23.48/29
Flags: X - disabled, A - active, D - dynamic
 0 AD dst-address=41.79.23.48/29 label=299 nexthop=198.19.12.54
      peer=41.79.23.1:0 path="empty"
7

Out of curiosity, I tried the original config you posted in our virtual lab and it worked as I expected with no out label. This means that most likely,something else in your config is influencing the issue. Try setting up a test either in VMs or actual hardware and strip out all the other unnecessary config to see if you still are getting an out label.

This is what it looked like when I labbed the config you posted (which is different than what you’re getting)

[admin@CCR1] > mpls forwarding-table print detail where destination in 41.79.23.48/29
Flags: L - ldp, V - vpls, T - traffic-eng
 0 L in-label=17 out-labels="" interface=seacom-vlan21-vlan210 nexthop=198.19.12.54
     destination=41.79.23.48/29
8

I’ve been able to replicate this in a lab environment. I used two hEX (RouterBOARD 750G r2) units where I connected together ether1 and ether2. I accessed the devices using ether5 and created a VLAN on ccr2’s ether5 to simulate the routing issue I’m seeing.

PS: I reduced the L2MTU as these devices don’t support L2MTUs larger than 4074 bytes.

ccr1 config:

/interface bridge
  add name=lo
/interface ethernet
  set [ find default-name=ether1 ] l2mtu=4074
  set [ find default-name=ether2 ] l2mtu=4074
/ip address
  add interface=lo address=41.79.21.1
  add interface=ether1 address=198.19.12.53/30
  add interface=ether2 address=198.19.12.33/30
/routing ospf instance
  set [ find default=yes ] redistribute-connected=as-type-1 redistribute-static=as-type-1 router-id=41.79.21.1
/routing ospf interface
  add authentication=md5 authentication-key=NRGEBnv5aO11qOOT dead-interval=2m hello-interval=1s interface=ether1 network-type=point-to-point
  add authentication=md5 authentication-key=NRGEBnv5aO11qOOT dead-interval=2m hello-interval=1s interface=ether2 network-type=point-to-point cost=100
/routing ospf network
  add area=backbone network=198.19.12.32/30
  add area=backbone network=198.19.12.52/30
/mpls interface
  set [ find default=yes ] mpls-mtu=4000
/mpls ldp
  set enabled=yes lsr-id=41.79.21.1 transport-address=41.79.21.1
/mpls ldp interface
  add interface=ether1
  add interface=ether2
/system identity
  set name=CCR1

ccr2 config:

/interface bridge
  add name=lo
/interface ethernet
  set [ find default-name=ether1 ] l2mtu=4074
  set [ find default-name=ether2 ] l2mtu=4074
/ip address
  add interface=lo address=41.79.23.1
  add interface=ether1 address=198.19.12.54/30
  add interface=ether2 address=198.19.12.34/30
/routing ospf instance
  set [ find default=yes ] redistribute-connected=as-type-1 redistribute-static=as-type-1 router-id=41.79.23.1
/routing ospf interface
  add authentication=md5 authentication-key=NRGEBnv5aO11qOOT dead-interval=2m hello-interval=1s interface=ether1 network-type=point-to-point
  add authentication=md5 authentication-key=NRGEBnv5aO11qOOT dead-interval=2m hello-interval=1s interface=ether2 network-type=point-to-point cost=100
/routing ospf network
  add area=backbone network=198.19.12.32/30
  add area=backbone network=198.19.12.52/30
/mpls interface
  set [ find default=yes ] mpls-mtu=4000
/mpls ldp
  set enabled=yes lsr-id=41.79.23.1 transport-address=41.79.23.1
/mpls ldp interface
  add interface=ether1
  add interface=ether2
/system identity
  set name=CCR2
/interface vlan
  add name=ether5-vlan122 interface=ether5 vlan-id=122
/ip address
  add interface=ether5-vlan122 address=1.1.1.1/30
/ip route
  add dst-address=2.2.2.0/29 gateway=1.1.1.2

The MPLS forwarding table on ccr1 subsequently associates an outbound label for the 2.2.2.0/29 route:

[admin@CCR1] > mpls forwarding-table print
Flags: L - ldp, V - vpls, T - traffic-eng
 #   IN-LABEL      OUT-LABELS   DESTINATION                    IN NEXTHOP
 0   expl-null
 1 L 16                         41.79.23.1/32                  et 198.19.12.54
 2 L 18                         1.1.1.0/30                     et 198.19.12.54
 3 L 19            17           2.2.2.0/29                     et 198.19.12.54

Local and Remote MPLS binding tables:

[admin@CCR1] > mpls local-bindings print
Flags: X - disabled, A - advertised, D - dynamic, L - local-route, G - gateway-route, e - egress
 #      DST-ADDRESS        LABEL                       PEERS
 0 ADLe 41.79.21.1/32      impl-null                   41.79.23.1:0
 1 ADLe 198.19.12.52/30    impl-null                   41.79.23.1:0
 2 ADLe 198.19.12.32/30    impl-null                   41.79.23.1:0
 3 ADG  41.79.23.1/32      16                          41.79.23.1:0
 4 ADG  1.1.1.0/30         18                          41.79.23.1:0
 5 ADG  2.2.2.0/29         19                          41.79.23.1:0

[admin@CCR1] > mpls remote-bindings print
Flags: X - disabled, A - active, D - dynamic
 #    DST-ADDRESS        NEXTHOP         LABEL         PEER
 0  D 41.79.21.1/32                      16            41.79.23.1:0
 1  D 198.19.12.52/30                    impl-null     41.79.23.1:0
 2  D 198.19.12.32/30                    impl-null     41.79.23.1:0
 3 AD 41.79.23.1/32      198.19.12.54    impl-null     41.79.23.1:0
 4 AD 1.1.1.0/30         198.19.12.54    impl-null     41.79.23.1:0
 5 AD 2.2.2.0/29         198.19.12.54    17            41.79.23.1:0

MPLS forwarding table from CCR2:

[admin@CCR2] > mpls forwarding-table print
Flags: L - ldp, V - vpls, T - traffic-eng
 #   IN-LABEL      OUT-LABELS   DESTINATION                    IN NEXTHOP
 0   expl-null
 1 L 16                         41.79.21.1/32                  et 198.19.12.53
 2 L 17                         2.2.2.0/29                     et 1.1.1.2

Local and Remote MPLS binding tables:

[admin@CCR2] > mpls local-bindings print
Flags: X - disabled, A - advertised, D - dynamic, L - local-route, G - gateway-route, e - egress
 #      DST-ADDRESS        LABEL                       PEERS
 0 ADG  41.79.21.1/32      16                          41.79.21.1:0
 1 ADLe 198.19.12.52/30    impl-null                   41.79.21.1:0
 2 ADLe 198.19.12.32/30    impl-null                   41.79.21.1:0
 3 ADLe 41.79.23.1/32      impl-null                   41.79.21.1:0
 4 ADLe 1.1.1.0/30         impl-null                   41.79.21.1:0
 5 ADG  2.2.2.0/29         17                          41.79.21.1:0

[admin@CCR2] > mpls remote-bindings print
Flags: X - disabled, A - active, D - dynamic
 #    DST-ADDRESS        NEXTHOP         LABEL         PEER
 0 AD 41.79.21.1/32      198.19.12.53    impl-null     41.79.21.1:0
 1  D 198.19.12.52/30                    impl-null     41.79.21.1:0
 2  D 198.19.12.32/30                    impl-null     41.79.21.1:0
 3  D 41.79.23.1/32                      16            41.79.21.1:0
 4  D 1.1.1.0/30                         18            41.79.21.1:0
 5  D 2.2.2.0/29                         19            41.79.21.1:0
9

The second OSPF route, with cost of 100, does not appear to influence the lab setup. Perhaps the outbound label on the route is correct like this?

10

As far as I understand, labels are correct. Pay attention to ‘G’ in appropriate local binding on CCR2 - it says that this is binding for “gateway route”:

 5 ADG  2.2.2.0/29         19                          41.79.23.1:0

Route to 2.2.2.0/29 has gateway:

/ip route
  add dst-address=2.2.2.0/29 gateway=1.1.1.2

So CCR2 can actually label switch (fast) data destined to 2.2.2.0/29 by popping last label and forwarding to 1.1.1.2 right away, instead of routing it (slow). Compare this to connected route 1.1.1.0/30 - this needs to be routed on CCR2, so it advertises implicit null label for this (and CCR1 does “penultimate hop popping”).

As to original report of some connectivity problems - I think more info on setup is needed - everything seems to be correct in label distribution.

11

Thank you for your assistance, I’ll try reproduce the problem in a simplified lab environment…