OSPF loses routes after days

iam8up · Tue Dec 04, 2018 5:54 pm

I have a CCR1009 that feeds my customers. It has an ethernet port with OSPF doing broadcast. From here there are some wireless backhauls going north and other backhauls going west.

Through the west backhauls there is an RB750. I've tried various versions of 6.42 and 6.43 (only the RB750) but all three routers are now 6.43.7 with the included firmware. This RB750 is doing OSPF to the CCR1009. This router would have an issue where every 24-48 hours or so it would lose all the routes outside of anything adjacent to it. It will stay broken for hours, at least 24. I simply disable the OSPF instance and re-enable fixes it for the time being. We thought it was the wireless backhauls but couldn't find anything. After a week of this, I pointed it to a Powercode BMU (I believe it's some open source OSPF project) and it's been flawless for 3+ weeks. This tells me it can't be the backhauls (unless additional backhauls correct it somehow) and it's an issue specifically between Mikrotiks.

The router to the north is a Powerbox Pro. It will do the same thing (OSPF works for several days, then loses routes until I flap the OSPF instance). It shares nothing in common network wise with the RB750 except it talks to the CCR1009 as well.

The CCR and Powercode BMU have no problems between each other.

The log will repeat this when it breaks until I flap the instance:

08:28:37 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:29:18 route,ospf,info Database Description packet has different master status flag 
08:29:18 route,ospf,info     new master flag=false 
08:29:18 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:29:59 route,ospf,info Database Description packet has different master status flag 
08:29:59 route,ospf,info     new master flag=false 
08:29:59 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:30:40 route,ospf,info Database Description packet has different master status flag 
08:30:40 route,ospf,info     new master flag=false 
08:30:40 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:31:19 system,info,account user admin logged in from 69.133.101.6 via web 
08:31:22 route,ospf,info Database Description packet has different master status flag 
08:31:22 route,ospf,info     new master flag=false 
08:31:22 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:31:36 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to Down 
08:31:36 system,info OSPFv2 instance changed by admin 
08:31:39 system,info OSPFv2 instance changed by admin

OSPF is very simple config:

/routing ospf instance
set [ find default=yes ] redistribute-connected=as-type-1 redistribute-static=as-type-1 router-id=10.255.0.16
/routing ospf network
add area=backbone network=1.1.1.1/30

(there are several ospf network on the CCR - including one that goes to the Powercode BMU)

If anyone has any ideas or things to check I would greatly appreciate it. This has been a struggle for weeks and we can't find any resolution.

mducharme · Tue Dec 04, 2018 6:15 pm

Why are you redistributing everything? Why not add everything as OSPF networks? That may not be the cause of your problem, but OSPF is quite stable for us, and we do very little redistribution. Redistributing things in general is not recommended if there is a better way. Everything that is redistributed becomes a type 5 LSA.

Note that there were stability fixes recently for type 5 LSA's in 6.43.7, this could resolve your issue (although it is still a bad idea to do the redistribution):

*) ospf - improved stability while handling type-5 LSAs;

We also prioritize OSPF over wireless links with the built in QoS features on such links.

iam8up · Tue Dec 04, 2018 6:26 pm

The redistribution is the customer subnets, it's the entire reason I'm doing OSPF. I previously did it with static routes which I'm trying to get away from.

I don't see how OSPF networks would do anything for me on the customer facing interfaces of my routers. I'm not talking OSPF to them, they're subnets on my routers (RB750 & Powerbox Pro).

mducharme · Wed Dec 05, 2018 4:22 am

Normally you would add network statements for your customer subnets, not redistribute them. This will advertise them to the other routers as a regular LSA instead of an external LSA. The way you are doing it is really not recommended.

bbs2web · Wed Dec 05, 2018 6:25 am

We exclusively use OSPF to distribute router's loopback IPs and necessary point to point or broadcast IPs to reach the loopbacks. All customer, peer or IXP routes are distributed using BGP.

I've detailed the structure in another post which you can reference here:
viewtopic.php?f=14&t=42268&p=688104#p688104

maznu · Wed Dec 05, 2018 9:06 am

The log will repeat this when it breaks until I flap the instance:

08:28:37 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:29:18 route,ospf,info Database Description packet has different master status flag 
08:29:18 route,ospf,info     new master flag=false 
08:29:18 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:29:59 route,ospf,info Database Description packet has different master status flag 
08:29:59 route,ospf,info     new master flag=false 
08:29:59 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:30:40 route,ospf,info Database Description packet has different master status flag 
08:30:40 route,ospf,info     new master flag=false 
08:30:40 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:31:19 system,info,account user admin logged in from 69.133.101.6 via web 
08:31:22 route,ospf,info Database Description packet has different master status flag 
08:31:22 route,ospf,info     new master flag=false 
08:31:22 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to 2-Way 
08:31:36 route,ospf,info OSPFv2 neighbor 10.255.0.3: state change from Full to Down 
08:31:36 system,info OSPFv2 instance changed by admin 
08:31:39 system,info OSPFv2 instance changed by admin

OSPF is very simple config:

/routing ospf instance
set [ find default=yes ] redistribute-connected=as-type-1 redistribute-static=as-type-1 router-id=10.255.0.16
/routing ospf network
add area=backbone network=1.1.1.1/30

(there are several ospf network on the CCR - including one that goes to the Powercode BMU)

If anyone has any ideas or things to check I would greatly appreciate it. This has been a struggle for weeks and we can't find any resolution.

I've seen this behaviour, and transitioned away from having lots of routes in OSPF (i.e. customer routes) to having a few core ones instead. I had wanted to keep customer routes in OSPF because our BGP converge time is slow on CCR (we're carrying full tables)… but I'd rather have slow converge times on what is essentially a fairly stable topology than have OSPF flap all over the place after a few days.

I'm going to be talking about this in my presentation to the UKWISPA conference in January: https://twitter.com/UKWISPA/status/1070022770619240454

mducharme · Wed Dec 05, 2018 1:32 pm

How many routes do you have? We have not experienced this issue at all with OSPFv2 with 450 OSPFv2 routes and 130 routers. It's been completely stable.

amt · Wed Dec 05, 2018 2:20 pm

I encountered the same problem, some times ospf router's lost routes and after reboot it start to work again, I was using the stub area an i thought i put many devices in one stub are and divided them into different stub areas.. now problem does not repeat again I'm waiting if problem continue... but now I have another problem that I cant summarize Ip ranges on ABR router

and routing table growing on backbone routers

maznu · Wed Dec 05, 2018 3:54 pm

How many routes do you have? We have not experienced this issue at all with OSPFv2 with 450 OSPFv2 routes and 130 routers. It's been completely stable.

760000 in BGPv4, 60000 in BGPv6

And was several hundred in OSPFv2 (/32 per customer device), and several hundred in OSPFv3 (/48, /56, /64 per customer/device) now just tens.

iam8up · Wed Dec 05, 2018 4:28 pm

So the suggestion here is to do it more like this?

/routing ospf instance
set [ find default=yes ] router-id=10.255.0.16
/routing ospf network
add area=backbone network=2.2.2.2/30
add area=backbone network=10.16.0.0/24
add area=backbone network=10.255.0.0/24
add area=backbone network=2.2.2.44/30

jprietove · Wed Dec 05, 2018 4:46 pm

Check if there are several routers with same Id. Check also that router up address in loopback interface is correct, with /32 and network equal to address.
Sometimes if configuration is copied from one router and pasted in another, and then the loopback IP is changed, the network remains.
For example, if router loopback IP is 10.255.255.1/32 and it is copied to another and the IP address is changed in winbox to 10.255.255.2/32, the resultant configuration is address=10.255.255.2 network=10.255.255.1

Enviado desde mi Redmi 3 mediante Tapatalk

iam8up · Wed Dec 05, 2018 6:28 pm

Check if there are several routers with same Id. Check also that router up address in loopback interface is correct, with /32 and network equal to address.
Sometimes if configuration is copied from one router and pasted in another, and then the loopback IP is changed, the network remains.
For example, if router loopback IP is 10.255.255.1/32 and it is copied to another and the IP address is changed in winbox to 10.255.255.2/32, the resultant configuration is address=10.255.255.2 network=10.255.255.1

Enviado desde mi Redmi 3 mediante Tapatalk

I just checked all my routers. All of the ip, network (same being /32), and router IDs are unique to each router.

mducharme · Wed Dec 05, 2018 6:36 pm

So the suggestion here is to do it more like this?

/routing ospf instance
set [ find default=yes ] router-id=10.255.0.16
/routing ospf network
add area=backbone network=2.2.2.2/30
add area=backbone network=10.16.0.0/24
add area=backbone network=10.255.0.0/24
add area=backbone network=2.2.2.44/30

Yes, exactly. That is the recommended way instead of doing redistribution. You can also set your customer interfaces “passive” to prevent forming OSPF neighbor with your customer on the off chance they have OSPF.

jprietove · Wed Dec 05, 2018 7:10 pm

I just checked all my routers. All of the ip, network (same being /32), and router IDs are unique to each router.

Try also to change interface network-type. Instead of broadcast, it seems your configuration could be PTMP because it (appears to) has a central Router. This way, the routers will not look for a DR, only your main router will create the topology.

As an example:

MAIN_ROUTER   ether1    ----- ether1     CUSTOMER 1
                         +--- ether1     CUSTOMER 2
                         +--- ether1     CUSTOMER 3

Main Router: /routing ospf interface add interface=ether1 network-type=ptmp
Customer 1 Router: /routing ospf interface add interface=ether1 network-type=point-to-point
Customer 2 Router: /routing ospf interface add interface=ether1 network-type=point-to-point

iam8up · Wed Dec 05, 2018 8:41 pm

OK I'm going to see if my previous mentioned changes fix the problem for a week or two.

I feel like changing from broadcast to ptmp would be better based on what I read in the Wiki, but time will tell.

mducharme · Wed Dec 05, 2018 11:44 pm

For GRE/EoIP tunnels, we use point to point network type for all such tunnels. In most cases however we use broadcast for wireless backhaul /30's and haven't had any problems, so we haven't bothered to change them to point-to-point (even though point-to-point is considered to be a bit better in that case as well). However, we also prioritize OSPFv2 over the backhaul by having the router set the OSPF traffic with a VLAN priority tag with CoS 7 that the backhaul radios read and prioritize over all other traffic (and the traffic over the backhaul receives a VLAN tag so the backhaul radios can read it), so that there are never any drops of the routing traffic. I would recommend considering such priority settings if the radios support only VLAN priority for QoS.

amt · Thu Dec 06, 2018 10:46 am

How many routes do you have? We have not experienced this issue at all with OSPFv2 with 450 OSPFv2 routes and 130 routers. It's been completely stable.

hi mducharme,
I have some problems on ospf, you have experience on ospf and can you help me with this topic
viewtopic.php?f=14&t=142180

thanks

iam8up · Sat Dec 08, 2018 12:17 am

I just checked all my routers. All of the ip, network (same being /32), and router IDs are unique to each router.
Try also to change interface network-type. Instead of broadcast, it seems your configuration could be PTMP because it (appears to) has a central Router. This way, the routers will not look for a DR, only your main router will create the topology.

As an example:
Code: Select all
MAIN_ROUTER   ether1    ----- ether1     CUSTOMER 1
                         +--- ether1     CUSTOMER 2
                         +--- ether1     CUSTOMER 3
Code: Select all
Main Router: /routing ospf interface add interface=ether1 network-type=ptmp
Customer 1 Router: /routing ospf interface add interface=ether1 network-type=point-to-point
Customer 2 Router: /routing ospf interface add interface=ether1 network-type=point-to-point

As expected I am still having issues. All I've changed is not redistributing connected routes/static routes as type 1 - instead I added the subnets to /routing ospf networks. After a few hours, maybe a day or two, the adjacency is broken between my routers.

I've never changed OSPF interface type - do I simply change all of the interfaces from broadcast to ptmp? Is there any additional configuration I need to do? The wiki just says there's no manual configuration (relating to nbma).

mducharme · Sat Dec 08, 2018 1:34 am

If they are on different interfaces it would be point-to-point on both sides. ptmp is only if there is a shared subnet (as opposed to a different interface or VLAN) that connects to multiple things in a hub-and-spoke setup. If everything is individual /30's then just use point to point on both sides. All you need to do is go into the interfaces tab and copy the dynamic interface and change the network type, do that on the far end first. This will of course cause the adjacency to be broken and reform.

iam8up · Mon Dec 10, 2018 4:58 pm

If they are on different interfaces it would be point-to-point on both sides. ptmp is only if there is a shared subnet (as opposed to a different interface or VLAN) that connects to multiple things in a hub-and-spoke setup. If everything is individual /30's then just use point to point on both sides. All you need to do is go into the interfaces tab and copy the dynamic interface and change the network type, do that on the far end first. This will of course cause the adjacency to be broken and reform.

So you're saying if I have two routers that go into ether4 and they're two different /30 IPs I should use ptp

But if I use say 172.16.0.0/24 and there are two routers - maybe 172.16.0.5 and 172.16.0.10 - for this need to do ptmp?

What if I have both examples?

mducharme · Mon Dec 10, 2018 7:51 pm

What if I have both examples?

Wait - you are saying you have two IPs on one interface? ex. two different IPs in two different /30's at the same time on one interface and OSPF neighbor with both?

If so, I think that is a bad idea, and it may be responsible for your instability. Whenever I have a single interface going to multiple routers, I create VLANs on that interface, then the VLAN gets the IP. OSPF sometimes behaves a bit strangely when you have more than one IP on an interface that you are trying to form a neighbor relationship over.

iam8up · Mon Dec 10, 2018 8:16 pm

What if I have both examples?
Wait - you are saying you have two IPs on one interface? ex. two different IPs in two different /30's at the same time on one interface and OSPF neighbor with both?

If so, I think that is a bad idea, and it may be responsible for your instability. Whenever I have a single interface going to multiple routers, I create VLANs on that interface, then the VLAN gets the IP. OSPF sometimes behaves a bit strangely when you have more than one IP on an interface that you are trying to form a neighbor relationship over.

I have many IPs on that interface. Multiple of them are doing OSPF. A couple are not.

Keep in mind the issue with one router is every 1-3 days. The other router is after maybe a week or two. The first router has 0 problems talking to a Powercode BMU in the same situation - it only happens when I move it back to a Mikrotik (CCR).

mducharme · Mon Dec 10, 2018 11:22 pm

I have many IPs on that interface. Multiple of them are doing OSPF. A couple are not.

Keep in mind the issue with one router is every 1-3 days. The other router is after maybe a week or two. The first router has 0 problems talking to a Powercode BMU in the same situation - it only happens when I move it back to a Mikrotik (CCR).

Yes I understand, but the way you have things set up is very uncommon, and so you are highly likely to run into some corner case bugs than if your setup was more typical. It also makes it harder to manage your OSPF interfaces because they are all tied together. I would strongly advise moving to a VLAN-based design rather than having multiple IPs on the same interface, at least for links that you are forming OSPF neighbor relationships over.

I know Cisco routers do not allow you to form OSPF adjacency on a secondary IP address, so if you had Cisco your current setup would not be possible (that in itself should demonstrate how unusual this setup is). MikroTik may allow you to configure this, but I do not know if it will work correctly. You could run into issues with DR/BDR reachability that would cause routes to disappear, which seems to be what you are actually experiencing.

iam8up · Wed Feb 06, 2019 4:46 am

Hopefully it's acceptable to update to this thread. I'm still having issues.

This morning I created a vlan 118 interface on the router with multiple IPs and then on the new tower router I did the same. I did a /30 between them (on this vlan 118).

It worked right away up until about 9 hours later, at which point the same problem came back - the remote router would only have routes in OSPF from itself.

Any other suggestions or ideas at this point?

OSPF loses routes after days

OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Re: OSPF loses routes after days

Who is online