Community discussions

 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 111
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

OSPF "Flapping" Issues

Wed Jan 22, 2014 9:35 pm

All -

For months now we have been struggling with OSPF "flapping" where OSPF will simply lose adjacency with a neighbor and then come back up.

I opened an issue with Mikrotik about it back in November and was told to upgrade to RouterOS v6.x. We have thirty+ routers, but I upgraded where I could. The issue was not resolved.

Basically in a nutshell, we use OSPFv2 and run NBMA with MD5 authentication. We run across wireless links and in almost all cases that is UBNT RocketM5's on v5.5.4 or v5.5.6. Trust me when I say that we have tweaked OSPF settings, used broadcast, PtP etc... Nothing would help. Same issues.

After months of troubleshooting I think I have finally pinpointed the root cause of the problem: EoIP Tunnels!

The common denominator at all the locations which are affected is the use of EoIP on the same links that run OSPF.

The common denominator at the sites that are not affected are links with just OSPF and no EoIP.

Yesterday I setup a new EoIP tunnel between two of the NON-AFFECTED sites and within 24 hours the flapping started. These two sites never had an OSPF issue until that EoIP tunnel was setup. Now I'm receiving alerts as frequent as all my other affected sites.

To Mikrotik Support

I opened Ticket#2013110566000895 on 11/6/2013 about this. I sent you very detailed configuration information for my three routers in our backbone area. As stated above, I was told to upgrade to RouterOS v6.x - which we did.

I truly believe that something else is happening here that needs attention.

Please let me know what else I can provide.

Thanks,
Scott
 
CelticComms
Forum Guru
Forum Guru
Posts: 1766
Joined: Wed May 02, 2012 5:48 am

Re: OSPF "Flapping" Issues

Wed Jan 22, 2014 10:05 pm

What traffic levels do you typically see on the relevant interfaces? Can you share any more configuration details? It would be interesting to see if this can be replicated in a lab setup.
Interlynx | Networking and Information Security Consultants & Trainers | Email: routerlynx@gmail.com
BGP | EIGRP | OSPF | MPLS | Firewall | VPN | IPsec | Multicast | QOS | IPv4/6 | STP | VLAN | PON | AE | M2M | and more!

 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 111
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

Re: OSPF "Flapping" Issues

Wed Jan 22, 2014 10:49 pm

That's a hard question to answer because of the diversity of sites we have and user levels.

First off, we use EoIP to keep an old legacy layer 2 network running as we attrition people off to UBNT and onto our routed L3 network. So basically at most of our sites we have legacy gear + new gear and a router. We are tearing down legacy backhauls which are all part of this old layer 2 network and using the EoIP to tunnel the layer 2 traffic where it needs to go. The layer 2 traffic is riding on new UBNT links we are putting up.

So in a single sentence, we have bits and pieces of this massive layer 2 network sitting behind our routed sites and we are using the new UBNT links to move the layer 2 traffic via EoIP.

As for traffic. It varies as I stated. When you read this, remember that I have many EOIP tunnels and many backhaul links running OSPF. During peak hours (6p - 2a) I can see EoIP tunnel traffic anywhere from 6mbps to 20mbps. This traffic then hops onto the link running OSPF which is moving all our routed L3 traffic from whichever site. My UBNT links can see on average up to 40-50mbps during peak hours. As we approach our CORE location, two things happen:

1. All this legacy layer 2 traffic rides a dedicated RocketM5 link to our CORE router.
2. The routed traffic rides a dedicated AirFiber link to our CORE router.

Now, two more things to consider:

1. When the OSPF flaps it only affects our routed network customers (the majority).
2. The EoIP is flawless and never drops. EoIP only goes down if I lose a link or power at a site.

And to throw a curve ball into this whole mix, sometimes the OSPF flaps during off peak hours when hardly any traffic is moving across a link.

As for configuration. Here's a basic example with some items masked:

Router A
/interface bridge
add l2mtu=1600 name=EOIP-ZYOGNI
add name=loopback

/interface eoip
add local-address=10.10.0.233 mac-address= name=\
    ZYO-MTB-EOIP_ROOT remote-address=10.10.0.238 tunnel-id=10
add local-address=10.10.0.241 mac-address= name=\
    ZYO-POM-EOIP_ROOT remote-address=10.10.0.246 tunnel-id=11
	
/routing ospf instance
set [ find default=yes ] distribute-default=always-as-type-1 \
    redistribute-connected=as-type-1 redistribute-rip=as-type-1 router-id=\
    10.0.254.128

/interface bridge port
add bridge=EOIP-ZYOGNI interface=ZYO-MTB-EOIP_ROOT
add bridge=EOIP-ZYOGNI interface=ether1
add bridge=EOIP-ZYOGNI interface=ZYO-POM-EOIP_ROOT

/ip address
add address=172.16.252.247/21 comment="GNI Equipment Switch" interface=ether1 \
    network=172.16.248.0
add address=10.0.254.128/32 comment=OSPF_Loopback interface=loopback network=\
    10.0.254.128
add address=10.10.0.233/29 comment=ZYO>MTB interface=ether6 network=\
    10.10.0.232
add address=10.10.0.241/29 comment=ZYO>POM interface=ether7 network=\
    10.10.0.240

/routing filter
add action=discard chain=ospf-out prefix=172.16.248.0/21
add action=discard chain=rip-out prefix=172.16.248.0/21
/routing ospf interface
add authentication=md5 authentication-key= \
    authentication-key-id=13 dead-interval=2m hello-interval=30s interface=\
    ether6 network-type=nbma priority=2
add authentication=md5 authentication-key= \
    authentication-key-id=13 dead-interval=2m hello-interval=30s interface=\
    ether7 network-type=nbma priority=2
/routing ospf nbma-neighbor
add address=10.10.0.238 poll-interval=1m30s
add address=10.10.0.246 poll-interval=1m30s
/routing ospf network
add area=backbone network=10.0.254.128/32
add area=backbone network=10.10.0.232/29
add area=backbone network=10.10.0.240/29
Router B
/interface bridge
add l2mtu=1600 name=EOIP-ZYOPOM
add name=loopback

/interface eoip
add local-address=10.10.0.246 mac-address= name=\
    ZYO-POM-EOIP_RMT remote-address=10.10.0.241 tunnel-id=11

/routing ospf instance
set [ find default=yes ] redistribute-connected=as-type-2 router-id=\
    10.0.254.140

/interface bridge port
add bridge=EOIP-ZYOPOM interface=ZYO-POM-EOIP_RMT
add bridge=EOIP-ZYOPOM interface="ether1 - 248.1"

/ip address
add address=172.16.252.197/21 comment="Uplink to GNI Switch" interface=\
    "ether1 - 248.1" network=172.16.248.0
add address=10.0.254.140/32 comment=OSPF_Loopback interface=loopback network=\
    10.0.254.140
add address=10.10.0.246/29 comment=ZYO>POM interface=ether4 network=\
    10.10.0.240
add address=10.10.0.254/29 comment=MTB>POM interface=ether5 network=\
    10.10.0.248

/routing filter
add action=discard chain=ospf-out prefix=172.16.248.0/21
/routing ospf interface
add authentication=md5 authentication-key= \
    authentication-key-id=13 dead-interval=2m hello-interval=30s interface=\
    ether4 network-type=nbma
add authentication=md5 authentication-key= \
    authentication-key-id=13 dead-interval=2m hello-interval=30s interface=\
    ether5 network-type=nbma
/routing ospf nbma-neighbor
add address=10.10.0.241 poll-interval=1m30s
add address=10.10.0.249 poll-interval=1m30s
/routing ospf network
add area=backbone network=10.0.254.140/32
add area=backbone network=10.10.0.240/29
add area=backbone network=10.10.0.248/29
 
WirelessDSL
newbie
Posts: 29
Joined: Thu Nov 24, 2011 12:43 pm
Location: Germany
Contact:

Re: OSPF "Flapping" Issues

Thu Jan 23, 2014 10:40 am

Maybe it´s not an answer at all.

But do you tought about using MPLS/VPLS instead of EoIP? It´s more efficient and the overhead is smaller.
MTCNA MTCRE MTCTCE MTCUME MTCWE MTCINE
 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 111
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

Re: OSPF "Flapping" Issues

Thu Jan 23, 2014 10:33 pm

I have, but I have not tried it. I do not have any experience with that.

Do you mind maybe sharing your configuration? Just a snippet would be helpful.

Thanks,
Scott
 
dperowne
just joined
Posts: 7
Joined: Mon Dec 16, 2013 4:54 pm

Re: OSPF "Flapping" Issues

Mon Feb 10, 2014 3:04 pm

You're not alone. I'm doing some work for a firm on some WiFi stuff and an issue in the core is causing issues for the service dropping clients. I've found myself looking at OSPF issues over these MikroTik routers. The same issue is happening with a router losing adjacency daily. Until reading this I hadn't considered the EoIP thing. I've looked and EoIP is also terminating on the OSPF interface of this particular device. I need to find out if this is the only device with the issue. They're not running any wireless in this network, or using MD5 or NBMA, so this is a core OSPF issue somewhere.

When you say OSPF and no EoIP, do you mean no EoIP at all on the router? Or just no EoIP tunnels terminating onto the OSPF physical interface (or do you mean IP address if there are VLANs?) ?

I've looked through the configuration and we only appear to have OSPF enabled on the internal/core interface (which can include sub-interfaces). The majority of these have public IPs on the Internet where some tunnels terminate on other devices, but obviously OSPF isn't enabled on that physical interface. They don't have loopbacks set up so I'm going to recommend that is implemented as well for obvious reasons too.
 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 111
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

Re: OSPF "Flapping" Issues

Tue Feb 11, 2014 5:08 pm

When I say OSPF and no EoIP I basically mean a wireless link that does not have an EoIP tunnel configured between the two routers the link spans.

That doesn't mean that either router doesn't have EoIP configured on another link.

We do not use VLANs.

The issue is prevalent across any links stemming from router(s) that have EoIP configured. Routers that do not have any EoIP configured do not experience the issue.

I can almost certainly assure Mikrotik that there is an issue here related to EoIP and OSPF. About a week ago I enabled an EoIP tunnel between two routers that never had EoIP configured. Up until that point my redundant ring of links (3 links) never had an OSPF hiccup. Almost within hours of configuring a single EoIP link I began to receive OSPF flaps. We continue to get them day in and day out.

It is extremely frustrating that Mikrotik has not chimed in on this thread. We sink thousands of dollars into their equipment throughout our network.

If they are going to offer EoIP functionality in their software then they need to support it. I understand that it is not the most solid option but in my particular case I do not have many other choices.
 
User avatar
bysard
Member Candidate
Member Candidate
Posts: 295
Joined: Thu Apr 22, 2010 2:53 pm

Re: OSPF "Flapping" Issues

Wed Feb 12, 2014 8:39 am

+1. Same problem here. We were also told to move to RoS v6.x RCdunnowhat when it came out. Same problem still on v6.9. Eventually I got tired of trying to find an error and moved to RIP.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: OSPF "Flapping" Issues

Wed Feb 12, 2014 5:11 pm

OSPF is very MTU sensitive. If the MTUs don't match or the MTU you are sending is larger than the MTU of the transport segment, OSPF will flap.

Have you identified and calculated all of your L2 and L3 MTUs in the path to see if there is a mismatch? If you connect two test routers with the same config via an ethernet cable and build EOIP tunnels, does the issue persist?
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 111
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

Re: OSPF "Flapping" Issues

Mon Feb 17, 2014 6:44 pm

Bysard -

I hear ya' about RIP! Trust me, I'd use it exclusively if I could but I need the instantaneous fail over that OSPF provides.

IPAN -

In regards to MTU, that makes the most sense and we have already addressed that at a few locations but the problem persists. I'm assuming you are making reference to ensuring that switch ports, router ports and EOIP bridge interfaces are all consistent with the MTU settings (1500)?
 
kevinpez
just joined
Posts: 2
Joined: Sat Apr 07, 2012 10:46 pm

Re: OSPF "Flapping" Issues

Fri Feb 21, 2014 7:45 am

I'm having a similar problem, is there anyone from Mikrotik looking into this issue? This seem to be a major bug, any advice from those that have solved it would be greatly appreciated.
 
User avatar
ScottReed
Member Candidate
Member Candidate
Topic Author
Posts: 111
Joined: Thu Sep 24, 2009 9:47 pm
Location: Montana / Western Massachusetts

Re: OSPF "Flapping" Issues

Fri Feb 21, 2014 6:14 pm

Mikrotik, are you investigating this?

If not then please provide me assistance with MPLS/VPLS configuration.

We continue to sink thousands of dollars into your gear and it's the least you could do.

Thanks,
Scott
 
AlexS
Member Candidate
Member Candidate
Posts: 258
Joined: Thu Oct 10, 2013 7:21 am

Re: OSPF "Flapping" Issues

Sat Feb 22, 2014 1:56 am

Bysard -

I hear ya' about RIP! Trust me, I'd use it exclusively if I could but I need the instantaneous fail over that OSPF provides.

IPAN -

In regards to MTU, that makes the most sense and we have already addressed that at a few locations but the problem persists. I'm assuming you are making reference to ensuring that switch ports, router ports and EOIP bridge interfaces are all consistent with the MTU settings (1500)?
I'm guessing eoip is l2 bridging over IP. So the MTU of eoip is not going to be 1500...

Miss configured MTU in ospf network is a pain. Can you not turn on log debug for ospf .. She why its flapping ?
 
WirelessDSL
newbie
Posts: 29
Joined: Thu Nov 24, 2011 12:43 pm
Location: Germany
Contact:

Re: OSPF "Flapping" Issues

Wed Mar 12, 2014 2:37 pm

Mikrotik, are you investigating this?

If not then please provide me assistance with MPLS/VPLS configuration.

We continue to sink thousands of dollars into your gear and it's the least you could do.

Thanks,
Scott
I can give you assistance for MPLS/VPLS if you still need it.

Please send me an email to sebastian.kirschner(at)drahtlos-dsl.de
MTCNA MTCRE MTCTCE MTCUME MTCWE MTCINE
 
mprokopov
just joined
Posts: 2
Joined: Wed Apr 17, 2013 10:18 am

Re: OSPF "Flapping" Issues

Fri Aug 05, 2016 3:29 pm

You should just uncheck BFP mark in OSPF link and issue goes away.

Who is online

Users browsing this forum: No registered users and 6 guests