After further debugging i found the problem. I has nothing to do with
the routing preference.
FYI I have had the following vrrp config on that port which causes that these odd routing behavior is triggered.
LAN port 2000:4711::2/64
VRRP port 2000:4711::1/64
and if the VRRP is master the forwarding does not work.
if I change the prefix of the vrrp address to 128 everything is fine.