I have two BGP routers, with one upstream BGP peer each. Since it is possible for connections to go out one router, and the return packets come in the other router, should I be disableing connection tracking on the routers?

yes.

that's what I was thought, thanks for confirming.

Why is this necessary?

If the SYN packet comes in one router, gets put into the connection tracking on one router, and then leaves the other router, the first router won't see a complete connection and will start blocking things on the next inbound packets. Now you also have an outbound connection (reply) on the second router that looks new, but didn't start with a SYN packet. It's possible you could work around it by putting some rules in that just allowed everything, but that's not ideal and could get messy or complex when you need something else to work.