I found the problem. I had connected port 1 to the modem which is a 1Gb port on the routerboard. It was auto negotiating to 10Mb though. I tried turning off auto negotiation which didn't work at all, changed over all the cables just in case and tried a different 1Gb port as well but it kept auto negotiating to 10Mb. I changed to a 100Mb port and reconnected and I now get similar speeds to using the ISPs router so its effectively fixed, but I have no idea why auto negotiating is failing. The max speed of the connection is 72Mb so not a problem but I may try it again with a 1Gb switch connected in between to see what happens.
To recap on my setup (as I couldn't find any other instructions on setting up BT infinity with a routerboard
* connect a 100Mb port directly to the supplied modem
* create a pppoe-client using the connected port (e6 in my case), username email@example.com
and blank password. Leave everything else as defaults
* In IP>firewall change any rules referring to e1-*-gateway to refer to the pppoe interface and also change the masquerade rule to refer to the same interface.
It should just work after that.