First thing to find out is to see if IoT communications can pass over router. There are two aspects:
- can you manually set IP address of IoT controller (or individual devices, whatever applies to your IoT swarm) in smart phone app. Or does it insist on auto-discovery instead? Autodiscovery generally doesn’t work across router (unless router runs some helper application).
- are IoT devices able to use gateway for communication outside their own subnet? Quite likely they can (at least to update their software and to connect to producer’s cloud), but it’s not certain.
If answer to both questions is a yes, then you could set up 2.4GHz radio as separate subnet (with separate addressing, DHCP server, whatnot) and 5GHz radio member of usual LAN (so essentially adding 5GHz wifi interface as port to LAN bridge). After that it’s up to routing/firewall to control what kind of communication can pass in each of 6 directions (IoT->LAN, IoT->internet, LAN->IoT, LAN->internet, internet->IoT, internet->LAN).
The problem you’re facing is that you have 3 2.4GHz radios and you want to make them member of same subnet. And that’s where VLANs come into the play. Only central router(UTM) will control traffic between the 3 networks (internet, LAN and IoT), so you’ll set up rules centrally.
If you don’t know much about VLANs, then I suggest you to search around internet and learn about concepts of VLAN. After you get a good idea about VLAN, then go through tutorial about VLANs in RouterOS to implement them. Beware that your LAN will almost certainly be inoperable for a while (in part or the whole of it) when you’ll reconfigure devices into VLAN.