NATing for server with multiple live IPs

Here is the situation:

[Microtik] ↔ [server]

server has 2 live IPs in it in a /27 and /29. The networks extent to the MKT, and one of both the /27 and /29 is the gateway ip assigned to the MKT. ie

[server]-----------[Mikrotik]
41.XX.XX.2/29------41.XX.XX.1/29
51.XX.XX.3/27------51.XX.XX.1/27
GW: 41.XX.XX.1

So the source IP of the server will be 41.XX.XX.2, because the GW is in the same range etc.

Both live ips on both ranges are reachable when using something like Ping, HTTP etc, because it doesn’t matter that when requesting on the 51. ip the traffic actually goes out via the 41. gateway… i think this is asynchronous routing or something to that effect.

However this does not work with UDP - well it doesn’t work with SIP, (im not sure about other UPD based systems). You can clearly see unreplied packets in the connection tab, from 41. when you tried to reach in on 51.

So my question is - isn’t there same clever NAT that i can apply to the 51. ip, that would nat it to the 41. network - or something? Maybe there would be nat in both directions… not sure - please help.

Btw, you may be thinking - whats the point, just give customers the 41. ip… but these networks are from completely different service providers… and if a customer connect to us from ISP A, they must route back to them via ISP A, same for ISP B.

Thanks