It sounds like it's not easy to figure out the permissions, envvars, memory size, etc. of your existing system, and that's why the migration is so difficult? That's not really one of Kubernetes' (many) failings.
Yes, and now we are back at the ancestor comment’s original point: “at the end of the day kubernetes feels like complexity trying to abstract over complexity, and often I find that's less successful that removing complexity in the first place”
Which I understand to mean “some people think using Kubernetes will make managing a system easier, but it often will not do that”
Can you elaborate on other things you think Kubernetes gets wrong? Asking out of curiosity because I haven't delved deep into it.
It's good you asked, but I'm not ready to answer it in a useful way. It depends entirely on your use cases.
Some un-nuanced observations as starting points:
- Helm sucks, but so does Kustomize
- Cluster networking and security is annoying to set up
- Observability is awkward. Some things aren't exposed as cluster metrics/events, so you need to look at, say, service and pod state. It's not easy to see, e.g. how many times your app OOMed in the last hour.
- There's a lot of complexity you can avoid for a while, but eventually some "simple" use case will only be solvable that way, and now you're doing service meshes.
Maybe "wrong" is the wrong word, but there are spots that feel overkill, and spots that feel immature.
> - Helm sucks, but so does Kustomize
Helm != Kubernetes, FWIW
I'd argue that Kustomize is the bee's knees but editor support for it sucks (or, I'd also accept that the docs suck, and/or are missing a bazillion examples so us mere mortals could enlighten ourselves to what all nouns and verbs are supported in the damn thing)
> how many times your app OOMed in the last hour.
heh, I'd love to hear those "shell scripts are all I need" folks chime in on how they'd get metrics for such a thing :-D (or Nomad, for that matter)
That said, one of the other common themes in this discussion is how Kubernetes jams people up because there are a bazillion ways of doing anything, with wildly differing levels of "it just works" versus "someone's promo packet that was abandoned". Monitoring falls squarely in the bazillion-ways category, in that it for sure does not come batteries included but there are a lot of cool toys if one has the cluster headroom to install them
https://github.com/google/cadvisor/blob/v0.51.0/metrics/prom... which is allegedly exposed in kubelet since 2022 https://github.com/kubernetes/kubernetes/pull/108004 (I don't have a cluster in front of me to test it, though)
https://github.com/kubernetes/kube-state-metrics/blob/v2.14.... shows the metric that KSM exposes in kube_pod_container_status_terminated_reason
https://opentelemetry.io/docs/specs/semconv/attributes-regis... shows the OTel version of what I suspect is that same one
And then in the "boil the ocean" version one could egress actual $(kubectl get events -w) payloads if using something where one is not charged by the metric: https://github.com/open-telemetry/opentelemetry-collector-co...