Hotspot user limit seems to work only for uptime:
[admin@TrueHOTSPOT] /ip hotspot user> print detail
Flags: X - disabled, D - dynamic
0 name="T-50:F5:20:41:47:46" password="" mac-address=50:F5:20:41:47:46
profile=default limit-uptime=30m
limit-bytes-total=10000000 uptime=0s bytes-in=0 bytes-out=0 packets-in=0
packets-out=0

I've tried using bytes-out as well, but user can keep browsing till time expires.
This is a trial user that was deleted/copied to static user with added transfer-limit.

Regards

Trial user is limited by uptime.
Use other login methods to apply session/bytes and other limitations.