Languages should not have a package management system. They all have a all the world is my language blindspot and fail hard when you have anything else. Sometimes you can build plugins in a different language but they still assume the one true language is all you want.
package management belongs to the os - or at least something else.
don't get me wrong, package management is a real problem and needs to be solved. I'm arguing against a language package manager we need a language agnostic package manager.
I think C++ is a living proof that not having a standard tooling around the language makes the language a complete pain in the ass to use, with any other language that does standard package managing/tooling out of the box, I can just pin the versions, commit a file to the repository, and on any computer that I'm working on I just issue a single command and everything is handled for me; meanwhile one of the C++ projects I've been working on, it turned out that I cannot build it on my server because one of the libraries I'm using only worked with clang17 which my desktop OS provides but the Debian I'm using on my server is shipping with clang16, and the library was not compatible with the earlier version of some C++ implementation, meanwhile Arch on my desktop updated to clang18, which also broke the library in some fashion, so now I'm sitting here with two systems, one where I want to deploy my software, and one where I want to develop the software, both of which are completely defunct and unable to build my project anymore; now I have to figure out how to build the specific version of clang on both systems and ensure I override a bunch of environment variables when configuring the builds on both of these systems, and then do the same on every new computer I'm developing/deploying on - with a proper tool I could just tell the project file to "use this compiler with this version with this standard" and things would just work. Some people will tell you "yeah bro just use docker with this and that and you will have a reproducible build system everywhere", but the thing is - I do not want to learn a completely unrelated tool and spend hours writing some scripts just to be able to continue working on my project when in any other programming language (like Go, Rust, JS), I can just install the runtime, clone the repo, run a command, and everything is handled for me seamlessly like it should be in 2024.
The problem for me is a "political" one, not a matter of convenience: When I choose a linux distro I implicitly trust the distro maintainers to not backdoor the liveCD, so I might as well trust them to maintain packages transparently. If something happens upstream, we expect the distro maintainers to patch out undesirable behavior, integrate changes into the system as a whole or warn us of changes. Most distros are the same in functionality: the choice of a certain distro is mostly a choice of which political institution (such as a business or non-profit) that we trust to maintain the interoperability of the OS.
Languages need to be more agnostic than a package manager requires because I should not have to rope another organization into my trust model.
Cargo already goes too far in encouraging a single repository (crates.io) for everything through its default behavior. Who maintains crates.io? Where is the transparency? This is the most important information the user should know when deciding to use crates.io, which is whether or not they can trust the maintainers not to backdoor code, and it is rarely discussed or even mentioned!
The default cargo crate (template?) encourages people to use permissive licensing for their code. So that is an example where you are already making implicit political decisions on behalf of the ecosystem and developers. That is alarming and should not be for the language maintainers to decide at all.
In C/C++ you have a separation of the standard from the implementation. This is really what makes C/C++ code long-lived, because you do not have to worry about the standard being hijacked by a single group. You have a standard and multiple competing implementations, like the WWW. I cannot encourage the use of Rust while there is only a single widely-accepted implementation.
The problem with that is that no Linux distro maintainer will ever put effort into maintaining every version of every library and compiler perpetually for a specific, seemingly random, programming language (or at least, reasonably, within few major versions including all minor releases in between), but with a tool that versions dependencies and allows for, say, git-based upstream with tag-versioned releases, you can expect to pick any specific version and for things to just work; managing library code for a specific programming language, be it any language, does not seem like the responsibility of an operating system, if anything, the package manager from your OS should be able to just supply the tool to manage the said language (like you currently can with npm, cargo or go); that also does not touch the topic of making things work across different platforms, sure, you maybe found a way to solve this issue in your imaginary Linux distro, how do you solve the problem for a co-developer that uses Windows, or macOS?
Additionally, you do not have to necessarily enforce these things on the language level, the standard and the tooling could live as two independent projects coming from the same entity. You could still use the compiler and the libraries from your OS, and build the code like that, or you could just reach out to an optional standardized tool that serves as a glue for all the external tools in a standardized way.
Yes, there are a lot of valid concerns with this approach as well, but personally for me, as a frustrated C++ developer, who is most likely going to still use the language for a decade to come, I feel like all the other languages I had mentioned in my previous post had addressed what is my biggest point of frustration with C++, so it's definitely an issue that could be solved. Many tried to do it independently, but due to personal differences, no funding, and different ideas of what should be the scope of such tooling, we ended up with a very fragmented ecosystem of tools, none of which have yet to date been able to fully address an issue that other languages solved.
> The default cargo crate (template?) encourages people to use permissive licensing for their code. So that is an example where you are already making implicit political decisions on behalf of the ecosystem and developers. That is alarming and should not be for the language maintainers to decide at all.
You and I must be using two very different versions of Cargo, because on mine the default template doesn't specify a license.
What you are asking for is standard command line flags for the compiler. Which probably cannot happen though it would be nice.
That and a better package manager so your clang wrong version problem cannot have. Which is what I was trying to get at.
I'd recommend using upstream apt llvm repos if you are using Debian or debian-derivatives like Ubuntu, to make sure you have the same compiler everywhere.
> Some people will tell you "yeah bro just use docker with this and that and you will have a reproducible build system everywhere", but the thing is - I do not want to learn a completely unrelated tool and spend hours writing some scripts just to be able to continue working on my project
You're working with some seriously hairy technologies, dealing with very knotty compatibility issues, and you don't want to learn... Docker?
I find this odd because it's relatively simple (certainly much simpler that a lot of what you're currently dealing with), well documented, has a very small, simple syntax and would probably solve your problems with much less effort than setting up a third development machine.
Docker solves the problems in some cases. However it forces you to ignore those knotty compatibility issues which is limiting. (You can't run on *BSD, Mac, windows... if you use docker) As such for many docker is not in the list of acceptable answers - in particular any open source project should consider docker not an option to solve their problems.
My understanding of the post I was replying to was that the compatibility issues were due to different versions of Linux having different clang versions. If I've understood correctly then Docker is highly likely to be a good solution.
> any open source project should consider docker not an option to solve their problem
That's generalising far too much.
Specifications for package interchange are absolutely essential, which is distinct from language endorsed package managers.
Python doesn't have a language package manager, you're free to use pip or poetry or uv or whatever, but it does have PEP 517/518, which allow all Python package managers to interact with a common package ecosystem which encompasses polyglot codebases.
C++ is only starting to address this problem with efforts like CPS. We have a plethora of packaging formats, Debian, pkg-config, conan, CMake configs, but they cannot speak fluently to one another so the package ecosystem is fractured, presenting an immense obstacle to any integration effort.
> Python doesn't have a language package manager, you're free to use pip or poetry or uv or whatever, but it does have PEP 517/518, which allow all Python package managers to interact with a common package ecosystem which encompasses polyglot codebases.
This is a long-standing pain point. LWN has a series of reports covering this, one of which is: https://lwn.net/Articles/920832/
Python polyglot code bases are not a solved problem at all. There have been difficulties installing TensorFlow and PyTorch with poetry for some time, and the installs still regularly break. This is the reason so many people use Conda. In HPC people are increasingly using Spack and EasyBuild to stop you having 10 versions of BLAS installed with all your Python dependencies.
Comparing it to other languages isn’t really fair since they don’t have polyglot code bases in the same way, and where native packages exist in for e.g. Npm, then you run into the same problems anyway.
Disagree completely. OS package managers are one of the biggest sources of problems.
Basically, once you have an OS level package manager, you have issues of versioning and ABI. You have people writing to the lowest common denominator - see for example being limited to the compiler and libraries available on an old Red Hat version. This need to maintain ABI compatibility has been one of the hugest issues with evolving C++.
The OS package manager ends up being a Procrustean bed forcing everything into its mold whether or not it actually fits.
Also, this doesn't even have the issue of multiple operating systems and even distros which have different package managers.
Rust and Go having their own package managers has helped greatly with real world usage and evolution.
This is a weird opinion, but I think that the OS package manager's complexity is largely owing to the unix directory structure which it just dumps all binaries in /bin, all configuration files in /etc, all libraries in /lib. It comes from a time where everything on the OS was developed by the same group of people.
By dumping all the same file types in massive top-level directories, you need a separate program (the package manager) to keep track of which files belong to which packages and dealing with their versions and ABI and stuff. Each package represents code developed by a specific group with a certain model of the system's interoperability.
GoboLinux has an interesting play on the problem by changing the directory structure so that the filesystem does most of the heavy lifting.
Interesting point, and I'm included to agree with your main point. I don't think the OS level is preferable, however:
Point 1: I do not want my program to only run on only one OS, or to require custom code to make it multi-platform.
Point 2: What if there's no OS?
>Point 1: I do not want my program to only run on only one OS, or to require custom code to make it multi-platform.
To run on only one OS at build time? I usually just set up cross-compilers from linux if I am making cross-platform C/C++ code.
>Point 2: What if there's no OS?
You can use a system like bitbake I think.
This sets up an untenable N*M explosion:
Will the GhostBSD maintainers pin the right version of Haskell's aeson package?
Will the Fedora Asahi devs stay on top of the latest Ocaml TLS developments?
Will MS package PureScript's code for DOM manipulation?
I think the term "package management system" is a bit over broad a term to talk about.
If we are talking about global shared dependencies, sure it may belong in the OS.
If we are talking about directly shared code, it may as well belong in the language layer.
If we are talking about combining independent opaque libraries, then it might belong in a different "pseudo os" level like NPM.
> package management belongs to the os
It clearly doesn't except if you're a fan of dll hell and outdated packages.
Window's package management is famously bad. However bugs in their implementation cannot be used to shoot down the concept.
If your solution fails on the large majority of computers, can it really be called a solution? 'All the world is my language' blindspots are nothing compared to 'all the world is GNU/Linux' blindspots.
the solution to DLL hell is to patch the applications to all use the same version of the library.
Oh but of course!
The solution to…a problem created directly by a specific approach is to…do even more work ourselves to try and untangle ourselves? And just cross our fingers and just _hope_ that every app/library is fully amenable to being patched this way?
Alternatively, we could realise that this isn’t really feasible at the scale that the ecosystem operates at now, and that instead of taking an approach that requires us to “do extra work to untangle ourselves” we should try and…not have that problem in the first place.
I don't think it's unreasonable to have a system where every program uses the same version of a library.
>And just cross our fingers and just _hope_ that every app/library is fully amenable to being patched this way?
It requires some foresight in designing the application, and whether or not you even choose to use that application in the first place. We should strive to decrease the complexity of the system as a whole. The fact that packages are using different versions of the same library in the first place is a canary and the system should disincentivize that use case to some extent. Using static libraries or a chroot or a sandbox for everything is sweeping the problems under the carpet.
>taking an approach that requires us to “do extra work to untangle ourselves” we should try and…not have that problem in the first place.
I would prefer a system that allows you to link every application to the same library as a default, but also allows for some per-application override, perhaps by using symlinks. That would cover the majority of use cases. But I do not think that dynamic linking is generally in vain.
In my own projects, I try to rely on static linking as much as possible, so I understand your perspective as a developer. But as a user I do not want programs to have their own dependencies separate from the rest of the system.
> I don't think it's unreasonable to have a system where every program uses the same version of a library.
I really think it is. Even at the scale of a single app it may sometimes make sense to have multiple versions of a same library, if for instance it implements a given algorithm in two different ways and both ways have useful properties
I have seen this (linking with multiple versions of the same library) for maintaining backwards compatibility, for example to support parsing a data file from a previous version, but never for selecting different algorithms.
> I don't think it's unreasonable to have a system where every program uses the same version of a library.
If there were guarantees that every library would always be both forwards and backwards compatible, that would be reasonable. Sadly, that's not the case.
Could a more streamlined “conception” of something like Gentoo fix this?
Applications ship their lock files + version constraints. Gets merged into a user/os level set of packages. You update one package, OS can figure out what it has to rebuild and goes off and does that.
Still shit-out-of-luck for anything proprietary, and it’s still super possible for users to end up looking at compile failures, but technically fits the bill?
> The solution to…a problem created directly by a specific approach is to…do even more work ourselves to try and untangle ourselves?
The solution is to be more professional. DLL hell comes from libraries that break compatibility: serious libraries should not break compatibility, or at least not often. Then when they do and you happen to have the issue, it's totally fair to go patch the library you depend on that depends on the breaking lib. Even in proprietary software.
The modern way is to use ZeroVer [1] and language package managers that pull hundreds of dependencies in the blink of an eye. Then asking that people compile everything themselves or use the one system deemed worthy of support (usually Windows and the very latest Ubuntu). And of course not caring about security one bit.
[1]: https://0ver.org/
This only works in the context of a single distribution. The moment you have two competing distributions, you're going to have to fork and end up with distro specific applications. Package maintainers won't be able to keep up and software becomes outdated.
But like…why?
Let’s say we make a “thing” which contains packages for all participating languages.
98% of the time, aren’t users just going to go “filter down to my language” and just continue what they’re doing, except with a somewhat worse overall experience, depending on whatever the “lowest common denominator” API + semantics we use for this shared package management solution.
Multi-language build systems already exist, which happily serve the needs to those projects which find themselves needing cross-language (+distributed) builds. Could there be some easier versions of these? Sure, but I don’t feel like “throw everyone in the same big box” is the solution here.
> I don’t feel like “throw everyone in the same big box” is the solution
It has to be - while nobody needs more than a subset of that big box, the intersection of what everyone needs turns out to be throw everyone in the same big box. If you have anything less than that one big box you end up many standards and then everyone chooses which standard and in turn something important you need choose the other standard and you can't use it (ie the situation we are in now)
Of course making that "one standard to rule them all" easy enough to use is a hard problem. It may be itself impossible and thus everyone drops back to the current mess.
I don’t entirely buy the argument, but I am intrigued.
> the intersection of what everyone needs turns out to be throw everyone in the same big box
I don’t follow: what’s the intersection of JS/Python/Go/Rust package management? What are they all needing that isn’t “download and store packages”? It can’t be OS level configuration, because that’s going to vary by OS and language.
> package management belongs to the os
Os package managers do a fundamentally different task than dependency management tools used in development.
They ship a bunch of applications and the libraries you need to run the applications.
If you need different version of libfoo than e.g. Firefox does, you're out of luck.
Need to support a customer with an older release which needs a different version of libfoo? Not gonna happen.
Unless you're talking about Nix or Guix, your OS package manager is not a substitute for a dependency management tools.
Fair enough. The world needs a package manager that is language agnostics and provides input to OS package managers as well as build tools (which should also be language agnostic).
Agreed. At least, languages should not require its own package management system to be used. There should be a way to invoke the compiler or interpreter without involving that language's own package management system, so that something else (like Bazel) can build on top. Fortunately, most common languages are all like that. You can invoke rustc without cargo. You can use python without pip. You can use javac without maven.
ideally yes. In practice if they wnat to provide a reasonnably good experience they have to do it
Honestly I don’t know why more languages don’t just adopt e.g. npm, maven or NuGet. They’re largely language independent at the binary level anyway.
npm, maven, and NuGet have caused me far more problems in trying to reproduce builds than the OS package manager ever will.