One potential issue that may affect OSPF in noisy point-to-multipoint environments is the way multicasts are transmitted (OSPF hellos in this case) - when AP transmits multicast (or broadcast), it is transmitted as radio multicast without retries - if that single transmission attempt is lost, frame is lost. When Station transmits multicast it is transmitted as radio unicast (to AP), therefore its delivery to AP is guaranteed. You should be able to figure out if this is the case by enabling OSPF packet logs on both sides and checking if OSPF hellos are lost.
If this is the case there are several things you can do:
- configure OSPF neighbors manually (so that unicast hellos are sent)
- try to avoid/reduce noise/interference on your backhaul link (e.g. by moving it to different channel and having only PtP link for your backhaul)
- by setting up WDS link (either AP-to-AP or AP-to-stationwds) between devices forming backhaul - traffic over WDS links is always sent as radio unicast and therefore is subject to retry algorithm.