I don't think you can use the switch chip to do this because you need access to the AP still, and don't want VLANs.
What do you want clients connected via port 4 to be able to do? Do you want all clients on that port to be treated the same?
If you can live with all port 4 clients being blocked from access to your PC, then the simplest way would seem to be a different subnet for that. No VLAN is required if all clients connecting through on that port are treated the same. You'll have to take that port off the common bridge.
Then, in order for traffic to pass from that subnet to your PC's subnet, the router must route it which it will do by default. We can use the firewall to control that. Create a rule to drop traffic originating from port 4 and destined for the PC's port (or anywhere other than WAN if you only want internet access from port 4). That rule should be placed after the 'allow established/related' rules. You might need another rule allowing connections from port 4 to WAN depending on your firewall arrangement.
So... traffic starting from port 4 cannot get to your PC; the firewall drops it. For connections started from the PC going to port 4, create another firewall rule to permit that (if needed according to your firewall setup). This connection will get noted by connection tracking and so the return traffic will be allowed by the established rule.
That is basically what I do here for an IoT subnetwork. Anything on that network is only allowed to be routed to WAN, but my rule to allow new connections from main LAN to IoT mean I can still make contact to configure those devices.
You'll need to create/change DHCP for that subnet of course.
A new subnet for port 4 would be a good idea. However, I don't want to drop all forwards between port 4 (AP) and bridge (LAN) since devices connected to the switch can sometimes use a server in my bridge (LAN) other than the WAN port for intenet connection. thanks