You can set up an address list in the firewall, then put in a filter rule that will accept connections from those IP addresses, and then after that a rule that will drop everything else. Order of rules in the firewall is extremely important. The good thing about an address list is you can have multiple IPs/Subnets in them without having to make a rule for each of them.
Be sure to have a console cable ready, in case it gets messed up, because with the filter rule it will drop all input without the first rule being there. If you wanted to get more specific you can do that by adding more to the firewall matcher. If these rules don’t work, move them up in the list because something before them is accepting your connections.