They have land that can be taken or voted away. I don't think only land owners should be able to vote, but it's worth noting worldwide having significant property is one of the most common ways for immigrants to qualify for a resident visa (other two common ways is job or business investment). Right or not it signifies enough skin in the game to many if not most societies to reflect reciprocated integration the community.

Remember in the early days there was almost no immigration control as well, so finding proxies for skin in the game might have been more challenging than today, when emigrating is almost impossible for the poor so they are stuck with their skin in America whether they like it or not.