I'm working on adding routers to my ISP network and ran into an issue with a new router being unable to BGP peer with a route reflector. Turns out it can't ping the route reflector using the loopback address as the source, although using the interface address is fine. Here's a simplified layout:
R12 --- R1 ---- (R3 or R7 or R9) --- R5. The route reflector hangs off of R5. R12 also can't ping any of the loopback or interface addresses on R5. There are 3 potential routes between R1 and R5. It doesn't appear to matter which route the ping takes. Yesterday R12 was having trouble reaching a device attached to R9. Downgrading the firmware on R9 and R12 from 6.48 to 6.47.9 seemed to solve that issue. R3 is still on 6.48, however R5 is way back on 6.46.6. R9 and R12 are CCR2004, R1, R3 and R5 are 1072 and R7 is 1016.
On R5 in the mangle prerouting chain, I can see the pings getting that far. However, they do not show up in either mangle forward or mangle input. As far as I can tell from the wiki packet flow diagrams, the only thing between mangle prerouting and mangle input would be dst-nat and the routing decision. There is no nat on R5. The pinged addresses show up in the routing table as connected, so it should go to input, but even if it goes to forward, I would expect it show up in mangle forward filter as the TTL is fine according to packet sniffer. There's no IPSEC or MPLS or PPP on R5 other than the default stuff I can't delete. Connection tracking is not enabled on R5.
If I add another address to the R12 bridge - x.x.x.13 - it can ping everything. I've looked through the verbose export.rsc from R5 and don't see anything that blocks the 12 address. There are no other rules in mangle other than those to troubleshoot this issue. I'm tempted to permanently use 13 and call it a day, but I'm afraid the issue will pop up again later after I've forgotten everything.
Any suggestions on where to look next? What else could be between mangle prerouting and mangle input?