We have a rather complex situation which I would like some advice on.

At customer sites we place mostly hAP ac lite devices running latest bugfix version 6.39.3 on most recent RouterBoard firmware 3.41.
We configure an OpenVPN client on them, dialing in to our CCR1008 in the datacenter.

Behind those customer RB's, we have several SIP phones. We configured routes towards our infrastructure to go over that VPN tunnel, otherwise just 'out' through the regular internet connection of the customer.
The phones running on TLS are no issue at all.
The ones running UDP however run fine as long as the VPN tunnel is up. When it goes down and back up, e.g. because of internet interruption at the customer site, the UDP devices fail to register again.
if we look at our logs we see that after the VPN goes down, the UDP devices go through the regular internet and that fails because we don't do NAT over the VPN etc etc; however when the VPN is back up, the devices keep going through the regular internet and are not going back through the VPN tunnel. So they are in fact not following the route table.
Rebooting devices etc doesn't help, but what does is at that point change the private IP or local SIP port on the device to something else.

So it seems like this gets stuck in some table, memory or cache.

Any guidance on what we could do to get this resolved?

We have solved similar problem with UDP packets going through VPN over 3G. If VPN was disconnected, UDP packets went through public internet but as soon as VPN was up, UDP did not return to VPN. It behaved the same way as you describe. We have solved it by setting UDP stream timeout to 1s in connection tracking settings. I suppose this setting assured, that UDP connection was deleted from firewall connections list and created again. Maybe you could try to delete appropriate connection in connections table.
Maybe it can be some bug in UDP connection processing in RouterOS, which arises only in specific situations.

Thank you for your reply. I will try that out.

What would be the impact if we disabled connection tracking overall?

Thank you for your reply. I will try that out.

What would be the impact if we disabled connection tracking overall?

One of them is your firewall being no longer statefull.