Sometimes strange things remain “sticky” so rebooting is generally advised if something has been changed and it doesn’t work, even if it should.
By rebooting you are effectively resettting anything that is created dynamically (when the router was booted with a different configuration) and that may not be updated by the configuration change.
The issue is that most changes actually work immediately and you never really know which changes actually need a reboot to take effect properly.
I think I will add this piece of info to the GP & CSA thread:
http://forum.mikrotik.com/t/gp-csa-for-mikrotik-devices/182176/1