Periodically cannot access modem and therefore internet

Hi, I have been having an issue for quite some time where I lose all access to the internet.

Some info on my current setup:

Modem - Virgin Media Business Hitron: 192.168.0.1
Router - RB3011UiAS: 192.168.1.1

I have five static IPs with my ISP but I am only ever using one at a time.

The problem
Over time (this can range from hours, days or weeks) my internet will become so slow that it will finally cut out altogether.
When this happens, when I try to ping my modem via the router all packets are lost.
Often, rebooting the modem will fix this, but not always.
Other times, switching to a different IP from my range will give me internet access again (and this will also allow me to ping my modem via the router again).

It may be that this is an issue with my ISP or modem, though Virgin have been running many diagnosis on my connection and they have also replaced my modem with a new one. Unfortunately, nothing has worked so far.

Would anybody be able to point me in the right direction please? I am more than happy to provide any more information.