No not at all.
Are you located by the router (aka office) or are you by a switch ?
What I would do is the following
create vlan99 for management with you on it and nobody else.
Ensure vlan99 has access to all other vlans.
Put all smart devices on VLAN99 as their IP address for management purposes.
Done.
Vlan 99 can be use for ports 3,4 on the hapac, keeping 5 for 192.168.2.01 (but dont need to create a subnet just an IP: address. that way you can always log into the router from that IP: by setting the ip ip of your laptop to something like 192.168.2.2 just in case the bridge falls apart at any time.
You can ensure vlan 99 reaches the other house as well, and from there dedicate a switch port to that VLAN and only allow your IP address to access the vlan as per y our input chain rules.