If only you drop packets from that address list in firewall raw section to not be processed further. It wouldn't stop attackers and your WAN link still would be saturated with the stream of these packets.
I can see only single good thing about collecting addresses of remote devices doing funky things: one can block traffic from those remote devices also for otherwise allowed services.
E.g. if one is hosting web server and hence traffic with dst-port=443 is in principle allowed, one can still block access to this service from some particular clients.
But even this kind of defense has limited use ... because attacker might skip those (useless) scanning techniques and go directly for the open service.
Some “watchdog” scheduled script can be made to watch count of such list and remove addresses by oldest sort (shorter exp. timeout) from list when reaches some defined threshold count, but still if port scan is performed from very large address pool in short period list can grow much above threshold in such short period before next scheduled event, not sure how much this case is probable unless router is targeted for such ddos.
I have such rules, usually there are 3-5 addresses added to list per day, mostly because I watch streams over torrent and then my public IP is avaiable to trackers and peers and that is good source for finding active IPs for attackers.