Signal is open-source [1]. You can compile the code yourself and review each PR if you're that paranoid.
Looks like the build is even reproducible. That makes me trust Signal even more.
https://github.com/signalapp/Signal-Android/blob/main/reprod...