Endowments have come from people over the entire history of the institution. The vast majority of the endowers are likely deceased and won't be able to agree to change the terms of their endowment.
And yet some trust, estate, or descendant somewhere ultimately has the authority to change those agreements. These things are not immutable facts of the universe, they can be changed.
As for the minority where that is not the case, it also means nobody will have standing to sue if the school decides to stop letting someone who died 200 years ago decide exactly how Harvard's money will be spent.