Weird perf issue with CHR on Proxmox

All,

As I have ben using CHR quite successfully for several years with traffic under 1G, I decided to use it as the edge router of a customer VPN (MPLS).
So basically, the CHR VM is the default route for all sites (20 sites, total traffic under 200Mbps), then it does the NAT to the Internet.

The issue is that all sites (or almost all) suffer from almost total losses of connectivity to Internet from time to time (a matter of minutes).
From a site, a btest (with a MK beyond the CHR VM) will then fail, and suddenly will work again.
A btest from the CHR VM to Internet works perfectly.
I took a lof of traces and found nothing significant.

CHR has 2 cores, 2GB RAM, and is using VirtIO (tried E1000 and VMXNET3: same).
The NAT connection tracking table typically contains around 5000 entries.
The only specific thing with NAT is that I use src-nat to a /32 which is on a bridge interface (kind of like Cisco’s Loopback). The rest of the network learns this /32 through OSPF.

I tried CHR v7 (latest stable) and also v6 (latest stable).

Before looking for issues on Proxmox or in my MPLS backbone, I wonder if anyone has encountered a such issue in the past with CHR.

Thank you

David