md5 authentication appears to be broken in 3.0.0b5 (or in 2.9.38, although we have adjacencies with cisco routers just fine in 2.9.38 )
We upgraded one of our routerboards to 3.0.0b5, and ospf will rarely come up now. This is connected via an IPIP tunnel to another routerboard with 2.9.38. When both routerboards were 2.9.x, OSPF worked fine.
Changing the key type from md5 to simple is allowing it to work again.
I did a packet sniff while they were set to md5, and the 3.0.0b5 router kept sending hellos to the 2.9.38 router, which appeared to ignore them, and sent no packets of its own at all. This is a stub area.
meanwhile, the 2.9.38 maintained adjacencies with a 3rd 2.9.38 router through a second ipip tunnel. I have tried setting the tunnel interface going to the 3.0.0b5 router to both point-to-point, and to broadcast, it seems to make no difference.