What about noise? So far all we have gotten is companies trying to build planes that still are very loud and produce audible booms all the way along their travel path. Pitched to the empathetic heros that are billionaires. And to sway public opinion they fly test planes at slightly over mach one in specific weather and altitude setting that avoids having the sonic booms reach the ground. While marketing and selling the idea of not mach 1.1 but mach 2-3 supersonics. Which is AFAIK completely impossible without the boom issue, even if made less bad - but arguably still bad - by fancy wing shapes. So all I see is selfish people trying to build something at expense of everyone else, including the environment.
Flying it the high limits of the atmosphere reduces sonic boom intensity due to both the extremely low density of the air and the distance from the ground. Ultimately, sonic boom is a sign of wasted energy, so if (big if) we can make aircraft that conform to the required aerodynamic ideals for efficient hypersonic flight, they will be relatively quiet.
Just like with regular air travel, though, I would expect a few decades of practical application for specialized service before it was commodified to be really useful for large amounts of passengers.
what about noise is you're supersonic noise and sound is about a mile back
I think the concern is for the frothing masses of earthbound unfortunates, not the passengers hurtling gloriously towards destiny miles above the troublesome surface of the planet.