Always great to see a successful “Single Rack of Failure” project out in the wild. And I don’t mean this as a negative. By being able to contain and control all the business into a single rack, you can more easily set up redundancy than trying to replicate your AWS environment between availability zones or worst, try recreating it in Azure or Google with all the different services, footguns, and so on. Just get another rack in a different datacenter and you’re set.
Congratulations to your team on keeping costs low and running a successful business out of a single rack.
We are just about finished with a total hoist into a self-hosted k8s cluster. We've gone slow due to my concerns about k8s and the additional complexity, but it's actually been pretty smooth and enjoyable. The end goal is the ability to point our helm charts at any cloud provider and have the entire infra stood up in less than a day. We already do offsite backups to rsync.net. We don't need crazy redundancy and are OK with a certain amount of risk to availability and < 1 day of data loss. Only thing that would take a long time is some GIS search data. We use elasticsearch and h3 hex strings encoded to guarantee prefix searching works for different zoom levels, which is a couple terabytes of derived data that takes a while to compute.
I have been enjoying this on-prem renaissance that we've seen over the last couple of years, makes my stubbornness around self-hosting feel smart in hindsight ;)