Help, 2 RB's as access controllers, Radius Server contacted only by one over and over.

I have 2 Routerboards an RB1100AHx2 and a 2011UAS, both are routers to different Internet routes

I have them configured to use Freeradius, server which uses MySQL and Daloradius

I need to have different users go out over different routers, so I set the NAS Identifier as a user check attribute along with user name and password in the Radius server, what should happen is that the users router contacts an access controller either one, which in turn contacts the radius server, if it is rejected because the NAS identifier does not match it should relinquish control to the next access server which should then contact the radius server and will get accepted because the NAS identity matches.

What is happening is that a client router contacts the 2011UAS which contacts the radius server, gets rejected because it is the wrong NAS Identifier, but it keeps trying over and over again, if I disable the radius entry in the RB, then control immediately passes to the RB1100AHx2 and the user get accepted and is the connected to the internet via that router.

Everything is fine for users that are supposed to go out over the 2011UAS, they connect straight away because their NAS identifier matches.

How can I stop the 2011 from continually trying to contact the radius for a specific user that is rejected.