Flying it the high limits of the atmosphere reduces sonic boom intensity due to both the extremely low density of the air and the distance from the ground. Ultimately, sonic boom is a sign of wasted energy, so if (big if) we can make aircraft that conform to the required aerodynamic ideals for efficient hypersonic flight, they will be relatively quiet.

Just like with regular air travel, though, I would expect a few decades of practical application for specialized service before it was commodified to be really useful for large amounts of passengers.