The fact that signal knows my phone number doesn't bother me.

The fact that anyone who knows my phone number can know that I use signal does bother me.