otabdeveloper4 3 days ago

Kubernetes is great example of the "second-system effect".

Kubernetes only works if you have a webapp written in a slow interpreted language. For anything else it is a huge impedance mismatch with what you're actually trying to do.

P.S. In the real world, Kubernetes isn't used to solve technical problems. It's used as a buffer between the dev team and the ops team, who usually have different schedules/budgets, and might even be different corporate entities. I'm sure there might be an easier way to solve that problem without dragging in Google's ridiculous and broken tech stack.

5
mrweasel 3 days ago

> It's used as a buffer between the dev team and the ops team, who usually have different schedules/budgets

That depends on your definition. If the ops team is solely responsibly for running the Kubernetes cluster, then yes. In reality that's rarely how things turns out. Developers want Kubernetes, because.... I don't know. Ops doesn't even want Kubernetes in many cases. Kubernetes is amazing, for those few organisations that really need it.

My rule of thumb is: If your worker nodes aren't entire physical hosts, then you might not need Kubernetes. I've seen some absolutely crazy setups where developers had designed this entire solution around Kubernetes, only to run one or two containers. The reasoning is pretty much always the same, they know absolutely nothing about operations, and fail to understand that load balancers exists outside of Kubernetes, or that their solution could be an nginx configuration, 100 lines of Python and some systemd configuration.

I accept that I lost the fight that Kubernetes is overly complex and a nightmare to debug. In my current position I can even see some advantages to Kubernetes, so I was at least a little of in my criticism. Still I don't think Kubernetes should be your default deployment platform, unless you have very specific needs.

YZF 2 days ago

I think I live in the real world and your statement is not true for any of the projects I've been involved in. Kubernetes is absolutely used to solve real technical problems that would otherwise require a lot of work to solve. I would say as a rule it's not a webapp in a slow interpreted language that's hosted in k8s. It truly is about decoupling from the need to manage machines and operating systems at a lower level and being able to scale seamlessly.

I'm really not following on the impedance mismatch from what you're actually trying to do. Where is that impedance mismatch? Let's take a simple example, Elastic Search and the k8s operator. You can edit a single line in your yaml and grow your cluster. That takes care of resources, storage, network etc. Can you do this manually with Elastic running on bare metal or in containers or in VM? Absolutely, it's a nightmare, non-replicable, process that will take you days. You don't need elastic search, or you never need to scale it, fine. You can run it on a single machine and lose all your data if that machine dies - fine.

tacitusarc 2 days ago

I’m curious if you’ve ever built and maintained a k8s cluster capable of reliably hosting an ES cluster? Because I have, and it was painful enough that we swapped to provisioning real HW with ansible. It is much easier to manage.

I should note, we still manage a K8s cluster, but not for anything using persistent storage.

Uehreka 3 days ago

> In the real world, Kubernetes isn't used to solve technical problems. It's used as a buffer between the dev team and the ops team, who usually have different schedules/budgets, and might even be different corporate entities.

At my company I’m both the dev and the ops team, and I’ve used Kubernetes and found it pleasant and intuitive? I’m able to have confidence that situations that arise in production can be recreated in dev, updates are easy, I can tie services together in a way that makes sense. I arrived at K8s after rolling my own scripts and deployment methods for years and I like its well-considered approach.

So maybe resist passing off your opinions as sweeping generalizations about “the real world”.

bavell 1 day ago

Similar experience here at a (very) small company.

rixed 3 days ago

Contrary to popular belief, k8s is not Google's tech stack.

My understanding is that it was initially sold as Google's tech to benefit from Google's tech reputation (exploiting the confusion caused by the fact that some of the original k8s devs where ex-googlers), and today it's also Google trying to pose as k8s inventor, to benefit from its popularity. Interesting case of host/parasite symbiosis, it seams.

Just my impression though, I can be wrong, please comment if you know more about the history of k8s.

jonasdegendt 3 days ago

Is there anyone that works at Google that can confirm this?

What's left of Borg at Google? Did the company switch to the open source Kubernetes distribution at any point? I'd love to know more about this as well.

> exploiting the confusion caused by the fact that some of the original k8s devs where ex-googlers

What about the fact that many active Kubernetes developers, are also active Googlers?

GauntletWizard 2 days ago

I'm an Ex-Google SRE. Kubernetes is not Borg, will never be Borg, and Borg does not need to borrow from k8s - Most of the "New Features" in K8s were things Google had been doing internally for 5+ years before k8s launched. Many of the current new features being added to k8s are things that Google has already studied and rejected - It breaks my heart to see k8s becoming actively worse on each release.

A ton of the experience of Borg is in k8s. Most of the concepts translate directly. The specifics about how borg works have changed over the years, and will continue to change, but have never really matched K8s - Google is in the business of deploying massive fleets, and k8s has never really supported cluster sizes above a few thousand. Google's service naming and service authentication is fully custom, and k8s is... fine, but makes a lot of concessions to more general ideas. Google was doing containerization before containerization was a thing - See https://lkml.org/lkml/2006/9/14/370 ( https://lwn.net/Articles/199643/ doesn't elide the e-mail address) for the introduction of the term to the kernel.

The point of k8s was to make "The Cloud" an attractive platform to deploy to, instead of EC2. Amazon EC2 had huge mindshare, and Google wanted some of those dollars. Google Cloud sponsored K8s because it was a way to a) Apply Google learnings to the wider developer community and b) Reduce AWS lock-in, by reducing the amount of applications that relied on EC2 APIs specifically - K8s genericized the "launch me a machine" process. The whole goal was making it easier for Google to sell it's cloud services, because the difference in deployment models (Mostly around lifetimes of processes, but also around how applications were coupled to infrastructure) were a huge impedance to migrating to the "cloud". Kubernetes was an attempt to make an attractive target - That would work on AWS, but commoditized it, so that you could easily migrate to, or simply target first, GCP.

rixed 2 days ago

Thank you for the exhaustive depiction of the situation. Also an ex SRE from long ago, although not for borg. One of the learnings I took with me is that there is no technical solution that is good for several orders of magnitude. The tool you need for 10 servers is not the one you need for 1000, etc.

jonasdegendt 1 day ago

Thank you, very insightful!

fragmede 3 days ago

Borg isn't going anywhere, Kubernetes isn't Google-scale

maxdo 3 days ago

kubernetes is an API for your cluster, that is portable between providers, more or less. there are other abstractions, but they are not portable, e.g. fly.io, DO etc. so unless you want a vendor lock-in, you need it. for one of my products, I had to migrate due to business reasons 4 times into different kube flavors, from self-manged ( 2 times ) to GKE and EKS.

otabdeveloper4 3 days ago

> there are other abstractions, but they are not portable

Not true. Unix itself is an API for your cluster too, like the original post implies.

Personally, as a "tech lead" I use NixOS. (Yes, I am that guy.)

The point is, k8s is a shitty API because it's built only for Google's "run a huge webapp built on shitty Python scripts" use case.

Most people don't need this, what they actually want is some way for dev to pass the buck to ops in some way that PM's can track on a Gantt chart.

YZF 2 days ago

I'm not an insider but afaik anything heavy lifting in Google is C++ or Go. There's no way you can use Python for anything heavy at Google scale, it's just too slow and bloated.

Most stuff I've seen run on k8s is not crappy webapp in Python. If anything that is less likely to be hosted in k8s.

I'm not sure why you call k8s api shitty. What is the NixOS API for "deploy an auto-scaling application with load balancing and storage"? Does NixOS manager clusters?

How much experience do you have with k8s?

otabdeveloper4 2 days ago

There is no such thing as "auto-scaling".

You can only "auto-scale" something that is horizontally scalable and trivially depends on on the number of incoming requests. I.e., "a shitty web-app". (A well designed web-app doesn't need to be "auto-scaled" because you can serve the world from three modern servers. StackOverflow only uses nine and has done so for years.)

As an obvious example, no database can be "auto-scaled". Neither can numeric methods.

kfir 2 days ago

If you think StackOverflow is the epitome of scale, then your view of the world is somewhat limited. I worked for a flash sale site in 2008 that had to handle 3 million users, all trying to connect to your site simultaneously to buy a minimal supply of inventory. After 15 minutes of peak scale, traffic will scale back down by 80-90%. I am pretty sure StackOverflow never had to deal with such a problem.