Yes, sorry should've said I never doubted you!
My comment more addressed to the clients, who can mitigate the issue with appropriate ssh_config.
More specifically, make sure that you aren't forwarding your ssh keys; otherwise the game can impersonate you on any server the keys work on.
Ah, that's worse than a mere privacy issue! Good point.
I was thinking merely about anonymity.
Both can be largely eliminated by having per host or per organization Identities and using IdentitiesOnly.