Ugh - I'm such an idiot
Of course you can't use layer 7 filtering on encrypted connections - d'oh!
I'll try to come up with a better idea. Ideally you'd whitelist the 3DS servers but as there's no known whitelist it does look like limits are the way to do.
Limit to destination port 443 (as far as I can remember browsers display an error if they load http traffic within an https-served page).
Limit time to 15 minutes as payment processors usually time out a transaction after this.
Limit total bytes transferred because it doesn't really take much more than a few MB of traffic to do 3DS.
I've never used hotspot so not sure what you'd trigger the temporary hole with. One idea that springs to mind (assuming hotspot web pages are HTTP-only) is looking for a 30x HTTP redirect to the payment processor?
With commands it'd be something like: (TOTALLY UNTESTED)
/ip firewall layer7-protocol add name=HTTP-payment-redirect regexp="^HTTP/1.1 30.*Location: https://MY-CHOSEN-PAYMENT-PROCESSOR.COM"
/ip firewall address add list=payment-traffic
/ip firewall mangle add chain=prerouting action=mark-packet protocol=tcp src-port=80 new-packet-mark=possible-HTTP-response comment="reduce matching load for next rule"
/ip firewall mangle add chain=prerouting action=add-src-to-address-list address-list=payment-traffic address-list-timeout=15m layer7-protocol=HTTP-payment-redirect packet-mark=possible-HTTP-response comment="add to temporary payment whitelist for 15m"
/ip firewall filter add chain=forward action=drop src-address-list=payment-traffic connection-bytes=20000000-0 comment="drop payment packets if more than 20MB transferred"
/ip firewall filter add chain=forward action=accept src-address-list=payment-traffic protocol=tcp dst-port=443 comment="only allow outgoing HTTPS if in payment whitelist"
This isn't perfect - for example the connection-bytes parameter probably only limits one TCP connection, not all TCP connections to remote 3DS servers. One connection might be to retrieve the 3DS iframe form, another might be to a different server to grab JQuery, etc.
Doesn't really fix the issue of someone getting redirected to your payment processor and then having 15 minutes of wild HTTPS fun (in 20MB blocks) then repeat.