Gotta be taller than the tallest ladder would be my guess. Tower forts are common. You can find them in places like Ireland also.
in a dense city you also have to take into account the height of the neighboring towers too - you don't want to be showered with arrows and later musket balls from a higher tower near by - thus race to the top :)
yep. From reading Bret Devereaux's Acoup.blog, I've learned that especially with arrows, height is an advantage - if you're firing down on someone from a position above them, you can aim easier, and your arrow is accelerated by gravity. Conversely, if you're firing up at someone from a position below them, gravity is working against you, and it's harder to aim.
This is why they build seige towers back in the day - to give the attackers a position higher than the beseiged city's walls from which to fire down on the defenders.