Ok, I just needed to set "add-arp for leases" option in dhcp-server.
But I also have couple of static ip addresses on that interface, I added them manually to arp table and they still doesn't work. Now only dynamically added arps work. Why is that ?
My interface is VLAN if that's any difference.