There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).
This doesn't make sense to me, and I feel like this is "holding it wrong"
But this is also a caveat with "managing it yourself", you get a lot of people having ideas and shooting themselves in the foot with it
Honest question - why would you want stateful workloads in a separate cluster from stateless? Why not just use a namespace?
because namespaces aren’t a failure boundary.
If your api gets hosed, you can create a new cluster and tear down the old one and call it a day.
With a stateful cluster, you can’t do that. As such you put in a lot more care with eg k8s upgrades, or introducing new controllers or admissions/mutating webhooks.
> It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation
Are these not covered by taint/toleration? I guess maybe isolation depending on what exactly they're demanding but even then I'd think it could work.
Most important reason is the ridiculous etcd limit of 8Gb. That alone is the reason for most k8s cluster splits.
I hate etcd probably more than most, but that 8Gb seems to just be a warning, unless you have information otherwise https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...
I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something
It isn't a mere warning. It is strongly recommended as the upper limit.
https://www.perfectscale.io/blog/etcd-8gb
https://github.com/etcd-io/etcd/issues/9771
And yes, I agree not allowing a pluggable replacement is really stupid.
> https://github.com/etcd-io/etcd/issues/9771
> stale bot marked this as completed (by fucking closing it)
Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly
Yes, a few, maybe even 10, 12, but 47? It's also a prime number, so it's not something like running each thing three times for dev, stage and prod.