I think this is a very cool project, but when I saw this statement, "Running Postgres at scale is hard. Eventually, one primary isn’t enough at which point you need to split it up.", I was reminded about this recent article about OpenAI, now one of the most visited apps/sites in the world, running on a single pg primary:
https://news.ycombinator.com/item?id=44071418
Quite from the article:
> At OpenAI, we utilize an unsharded architecture with one writer and multiple readers, demonstrating that PostgreSQL can scale gracefully under massive read loads.
Of course, if you have a lot of write volume this would be an unsuitable architecture, but just a reminder that pg can scale a lot more than many people think with just a single writer.
You can also "soft"-shard your read replicas. I.e. each replica has still has all the data, but you still direct queries to specific replicas as if they were sharded. We had success with this since each replica can have a high cache hit rate by keeping only the data from its shard in cache. This is a lot simpler than real sharding since all replicas can still answer all queries and resharding is very forgiving.
Definitely. I wouldn't be such a big fan of Postgres if this wasn't the case.
That being said, if sharding was natively supported and worked well, they would of done it ages ago.
All massive Pg deployments today are so because we just don't have another option.
But nobody looks at a 10TB table and doesn't worry about what tomorrow will look like.