I was not able to make this work as well, so I used this variation.
/ ip firewall filter
add chain=forward protocol=tcp dst-port=21 src-address-list=black_list_ftp action=drop \
comment="Drop FTP brute forcers" disabled=no log=yes log-prefix="FW_Block_Bruteforce_FTP"
add chain=forward protocol=tcp dst-port=21 connection-state=new \
src-address-list=ftp_stage2 action=add-src-to-address-list address-list=black_list_ftp address-list-timeout=1d \
comment="" disabled=no
add chain=forward protocol=tcp dst-port=21 connection-state=new \
src-address-list=ftp_stage1 action=add-src-to-address-list address-list=ftp_stage2 address-list-timeout=5m \
comment="" disabled=no
add chain=forward protocol=tcp dst-port=21 connection-state=new \
action=add-src-to-address-list address-list=ftp_stage1 address-list-timeout=5m comment="" \
disabled=no
Here is how its work:
When some tries to FTP to you, the bottom rules hits inn.
It will add the IP to an temporary access-list
ftp_stage1 and it stays there for 5 min.
2 attempt
If the user tries again, second rule from bottom will run, since IP is in access-list
ftp_stage1
This will then add user to access-list
ftp_stage2 and it stays there for 5 min.
3 attempt
If the user tries again, third rule from bottom will run, since IP is in access-list
ftp_stage2
This will then add user to access-list
black_list_ftp and it stays there for 1 day.
4 attempt
Top rule will hit since IP is in access-list
black_list_ftp and user will not be able to log inn.
This is a simple way to do it. It works.
You can add more steps, change timers.