Internet accessibility, 2 gateways,

I have set up like this: 2 sets of local ip addresses, 2 interenet connections with 2 public IPs. Adresss A are nated on 1st internet connection, Address B are natted on 2nd internet connection. All that works fine. Default gateway is gateway of 1st net connection and it makes my 2nd public ip inaccessible from the internet. Is there a way around it, so i can for example get L2tp connections on both public IPs?