just close the tcp sockets and you wont even notice them trying to connect and failing
do you also log everyone who looks at your house? it's a self inflicted problem
At least in the case of VPS my experience has been 99% failed ssh attempts. I just use nftables to rate limit those to 2 failed attempts per minute. Log size is quite modest and can easily filter out failed attempts when viewing.