I had missed the fact that Dagger is apparently trying to... replace? Docker for container definitions. That's a pretty big vision.
I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling.
At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...
In theory you could replace much of Docker's low-level tooling with Dagger. In practice, that's not what we're trying to do. There's a lot of inertia to ripping out existing tools, and even if the replacement is better, it may still not be worth the effort.
What we're focusing on is green field application of container tech. Things that should be containerized, but aren't. For example:
- Cross-platform builds
- Complex integration testing environments
- Data processing pipelines
- AI agent workflows (where you give a LLM the ability to perform tasks in a controlled environment)
In those kinds of workflows, there is no dominant tool to replace - Docker or otherwise. Everyone builds their own unique snowflake of a monolith, by gluing together a dozen tools. Dagger aims to replace that glue with a modular, composable system.
this is helpful. and i appreciate the intellectual/technological humility.
i think there's a decent chance we end up giving Dagger a spin this year.
Does it replace Kabuki in non priviledged CI builds? Can one exchange lower / independent layers like with nix container builds?
> Does it replace Kabuki in non priviledged CI builds?
I have never heard of Kabuki, and couldn't find it in a quick web search. Did you mean Kaniko?
> Can one exchange lower / independent layers like with nix container builds?
Yes.
How does that work? Are you building special adaptors that understand how package manager metadata files work? A deb package's postinstall is basically just a blob of bash running as root with access to the whole filesystem. If I tell dagger to install three packages in three different steps, those are not trivially reorderable operations. Nor are they reproducible at all, without Dagger having additional insight into the state of the local apt cache and what is offered by the remote apt repo at the moment of execution.
Yes, Dagger provides the core system API that allows creating and combining filesystem layers in any order you want. However it doesn't provide compatibility with existing package managers: that's something that you, or someone else in the Dagger community, would implement. You can extend the Dagger API with your own types and functions, then share them as reusable modules.
For example, here's a Dagger module for building Alpine Linux containers in an order-independent way. Under the hood it's derived from the 'apko' build tool created by the folks at Chainguard. https://daggerverse.dev/mod/github.com/dagger/dagger/modules...
And here's a Dagger Shell command for building a container using that module:
github.com/dagger/dagger/modules/alpine | container --packages=git,openssh,curl | terminal
Neat! Okay, yes, I can definitely see the value here, particularly if all or most of an image is otherwise able to be expressed in terms of things that are deterministic, such as language ecosystem package managers with lockfiles.
It seems like you're cutting an interesting track on providing a better way than Dockerfiles with their imperative layers, but pragmatic enough to not insist on taking over the whole world like Bazel or Nix.
The Bash compatible syntax is just one representation of an underlying cross-language API. You can also write code in your preferred language, provided there's an SDK for it. (Currently: Python, TypeScript, Go, PHP, Java, Elixir, Rust, .NET)
But if you're talking specifically about shell or a "native" REPL in a safer language - I also want to scratch that itch. Some day I'll dust off https://github.com/vito/dash :)
They even supported CUE at one point. https://dagger.io/blog/ending-cue-support
I worked with the Dagger team on that. It was super hard to marry two DAG engines that work in opposite directions. The team eventually heard from users they wanted to write in the language of their choice, so there are now a bunch of SDKs.
I think one could wrap the Go SDK with CUE, and have oft considered doing it, but the work has not migrated from the backlog yet. I've been toying with a CUE powered monorepo DX CLI that is more akin to BuildPacks, using Dagger as the containerization layer.
I feel this is much more powershell like than bash. The container thing looks much more than a typed object than a plain character device/pipe. Also what is the bourne again part?
It’s def closer to powershell in all the best ways IMO. But the key point is that the container thing is actually a typed object. So is every other primitive in there including once you make yourself.
okay, I think what I ultimately missed is that this is just a new language for the existing engine. I may not be the world's most careful reader.
> That's a pretty big vision.
Is it? Docker is quite long in the tooth and this point and is a long way from perfect. Docker's design is rife with poor choices and unaddressed flaws. One I've been tracking for years: there is no straightforward way to undo several things that get inherited by derived images, such as VOLUME and EXPOSE, and Docker Inc. doesn't care no matter how long the github threads get.
So I think there is ample room for alternatives.
I see this more as a complement. I wouldn’t stop using Dockerfile for standalone definitions, but for shell+docker+shell sandwiches (the second shell being inside the container), this looks very handy.
What kind of tooling do you have in mind to replace?
Also what would you prefer to see instead of bash compatible?
i want to move away from GitHub Actions and I also want to move away from some bespoke monorepo tooling for something that's a better overall orchestrator of 'things'. Dagger scratches some of those itches, or seems to, so I've had my eye on it for a while. Knowing that they're trying to also help me define and use containerized environments - I sort of assumed I'd have to bring those myself.
I'm using Dagger in a monorepo setup (actually several). For the main one, I've gone the route of wrapping the Dagger Go SDK with a custom Go CLI (plus CUE as another centerpiece). For sure, I've been making the team aware of the challenges and rough edges. (of course there are a lot of ways people organize monorepos, we are polyglot with one go.mod but multiple package.json...)
I'm largely replacing Dockerfiles and Makefiles with a custom CLI that wraps Dagger. Being able to use a proper language, functions, packages, imports is so much better by comparison. (ymmv)
I mean if we're going full pie in the sky, fully declarative containers exist today:
https://github.com/nlewo/nix2container#nix2container
Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.
Dagger is fully declarative. It's just built on a dynamic declarative API, instead of a static declarative DSL.
So, if you took Nix, and replaced the static scheme-like DSL with a proper API, and then built SDKs in 5 languages for that API; and then built a bash-like shell also for easy scripting; then you would start to have something that approximates Dagger.
I stand corrected! I shall investigate more closely with this additional context. Thanks.
Thanks for giving it another chance! We need to get better at explaining all this, it's a lot to unpack, although prior experience with declarative systems like Nix or Bazel does help a lot :)
We have a very active discord, feel free to come by and ask all the tough questions!