I'm working on adding routers to my ISP network and ran into an issue with a new router being unable to BGP peer with a route reflector. Turns out it can't ping the route reflector using the loopback address as the source, although using the interface address is fine. Here's a simplified layout:

R12 --- R1 ---- (R3 or R7 or R9) --- R5. The route reflector hangs off of R5. R12 also can't ping any of the loopback or interface addresses on R5. There are 3 potential routes between R1 and R5. It doesn't appear to matter which route the ping takes. Yesterday R12 was having trouble reaching a device attached to R9. Downgrading the firmware on R9 and R12 from 6.48 to 6.47.9 seemed to solve that issue. R3 is still on 6.48, however R5 is way back on 6.46.6. R9 and R12 are CCR2004, R1, R3 and R5 are 1072 and R7 is 1016.

On R5 in the mangle prerouting chain, I can see the pings getting that far. However, they do not show up in either mangle forward or mangle input. As far as I can tell from the wiki packet flow diagrams, the only thing between mangle prerouting and mangle input would be dst-nat and the routing decision. There is no nat on R5. The pinged addresses show up in the routing table as connected, so it should go to input, but even if it goes to forward, I would expect it show up in mangle forward filter as the TTL is fine according to packet sniffer. There's no IPSEC or MPLS or PPP on R5 other than the default stuff I can't delete. Connection tracking is not enabled on R5.

If I add another address to the R12 bridge - x.x.x.13 - it can ping everything. I've looked through the verbose export.rsc from R5 and don't see anything that blocks the 12 address. There are no other rules in mangle other than those to troubleshoot this issue. I'm tempted to permanently use 13 and call it a day, but I'm afraid the issue will pop up again later after I've forgotten everything.

Any suggestions on where to look next? What else could be between mangle prerouting and mangle input?

thanks
jim

Currently rp-filter is no, but changing it to loose or strict didn't help.

thanks
jim

ip settings print
ip-forward: yes
send-redirects: yes
accept-source-route: no
accept-redirects: no
secure-redirects: yes
rp-filter: no
tcp-syncookies: no
max-neighbor-entries: 8192
arp-timeout: 30s
icmp-rate-limit: 10
icmp-rate-mask: 0x1818
route-cache: yes
allow-fast-path: yes
ipv4-fast-path-active: no
ipv4-fast-path-packets: 0
ipv4-fast-path-bytes: 0
ipv4-fasttrack-active: no
ipv4-fasttrack-packets: 0
ipv4-fasttrack-bytes: 0

The idea was that the packet was being dropped in the routing phase (between prerouting and input or foward) because the backward route didn't match the in-interface of that packet. But if rp-filter is no, it cannot be the case.

OSPF routes from R1 to R5 shows all 3 options, but from R5 to R1 only shows going through R7. I think I'll tinker with OSPF costs just to see what happens even though RP-filter is off.

equalized path costs for the return so all the return path options were in the routing table and it didn't help.