Ok, it's pretty clear what's going on now. Your routing works as expected. It is not your outgoing ICMP echo-request packets (pings) that are being mis-routed and/or discarded, but rather incoming ICMP echo-reply packets get rejected by your rp-filter.
The rp-filter=loose works by checking if the source address from incoming packet can be reached via the interface where that packet came from. It takes into account your routing rules and routing tables (not sure about mangle rules, but I think those are unlikely to be considered). It won't do lookup in your custom routing table unless instructed by routing rules. Whatever you specified in your ping command is never considered by rp-filter, and that's as expected.
Thanks for this breakdown. I do understand much better now. At least the mysticism is gone, we have pinpointed the cause and have something to work with now
However I suppose that my question still stands though, about why adding a bogus default gateway to main routing table, corrects the timeouts? And then I guess I would also like to know, what would be the preferred way to address this besides for adding such a route?
It's obviously not ideal to add a virtual interface and set up a default gateway via that interface...surely? On the one hand, yes I am glad/fortunate that it works, as I know it's a dodgy workaround that for all intents and purposes, probably shouldn't be working... but on the other hand I'm just wondering what the best approach for dealing with such a scenario would be, as it seems I'm torn between two less than ideal alternatives - namely disabling rp-filter completely (bad for security of course), versus needing to add a seemingly pointless/"irrelevant" route that would never actually be used for anything, for no other reason than to ensure that everything else works as expected? :/
I'm sure there's an urge to suggest "Why would you not want to have any default gateway on the main routing table anyway?", but I should stress that I think that is besides the point. I find this behavior somewhat counter-intuitive and baffling, in spite of the explanations thus far, so hoping we can add further conclusive clarity and put this to rest.
I hope this makes sense... Any ideas welcome:)
Oh and I do also have to make a point of saying that the level of support on these forums in terms of number of volunteers and their skillset, and TTL for responses on these forums has left me in awe. You guys are amazing. Cheers for that