The BEAM might be the most under-rated tech in all of open source.
Example -- whatsapp.
Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.
>Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.
I work at an Erlang shop.
For Erlang to be useful you need to have massive scale, millions of DAU. Yes Elixir might run your website serving thousands of DAU but Erlang and the BEAM was not invented for you. Few companies have the scale that makes Erlang a reasonable choice.
More pressing these days I believe is that the Erlang ecosystem is all or nothing, the BEAM is like its own little operating system and has extremely (IMHO) complicated setup and configuration with many moving parts: ERTS, epmd, Rebar,... You don't need or shouldn't use container tech and k8s with Erlang because it's already doing that stuff in its own Erlang way - but it's the Erlang way it's extremely unique and unfamiliar to most.
When you have the right use case for Erlang and you see it in action it is truly like black magic, it's been a highlight of my career to work with it.
When we tried to use Elixir Liveview we found that containerizing it (docker compose) was necessary just because of how demanding the runtime environment is regarding keeping modules up to date and synced, and what that requires of the host operating system.
We'll run prod on one server and dev on 3-4 workstations and nothing will match between any of them without a docker container to give this Elixir app a cleanroom environment to work from.
The project we were trying this on eventually ran out of funding before it got off the ground, and we lost access to our main guy who understood Elixir setup really well, so nowadays the rest of us can't even manage to stand up the service to demo it because of all of the moving parts that get angry about not having their environment set up just right.
I've basically found it the only language more difficult than python to set up an environment for. (Well.. the more I think about it, Gradle and every other mobile development stack I have yet seen is literally Satan's armpit..)
With python (though I rarely code in that either) I can stand up almost anything on almost any machine with miniconda and venv and not have to drag Docker into things.
Node/NPM is a prima donna and a PITA but IME as long as you whack it with a wrench long enough following up on each of its complaints then you'll eventually get things running.
My experience still revolves around PHP or Perl or C on the backend, Raw Javascript or sometimes Elm on the front end, and those all seem to be a lot easier to maintain over a timescale of decades as servers update and browsers gobble up new and different features.
---
What I can say in favor of Elixir Liveview is that we built a smooth as hell SPA using that. It was incredibly user friendly to work with and aesthetically amazing, but the maintenance right down at the foundation was the biggest challenge.
What environment issues were you running into? We use mise/asdf to manage which version of erlang/elixir our project is using, and only needed to containerize our database to ease its maintenance.
For the dev experience, I'd also recommend NextLS/Lexical over Elixir LS until the official one is out. It should compile much faster for nontrivial Elixir applications.
https://elixir-lang.org/blog/2024/08/15/welcome-elixir-langu...
Would something like Nix (optionally, NixOS) help in getting all the parts documented and working together?
Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years. I can't speak for Erlang though.
> Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years.
One of your sibling commenters noted...
> I've basically found it [Elixir] the only language more difficult than python to set up an environment for.
Quite the difference.
I'm not saying either one of you is wrong; I'm sure you both experienced what you say you did, it's just interesting to see the dichotomy, literally (as of this writing) next to each other.
We found Erlang to be the right choice at small scale. Namely, we used it in iot applications, where the self-healing property of proper supervision trees resulted in a very stable implementation. We also saw the same benefits in the cloud part of our application. No complicated k8s setup, just simple supervisor trees.
> Few companies have the scale that makes Erlang a reasonable choice.
This is a reality that I wish more people would embrace, for so many things.
elixir softens a whole ton of the sharp edges around getting started and there are only a handful of "gotchas" coming from other hlls (like all terms being immutable when you pass to a function -- but thats honestly almost a problem with the "mainstream" languages).
So you'd do:
x = doubler(x)
Instead of treating the function like a "subroutine"? Like:
x = 10
doubler(x)
print x
I'm unsure why, but I use the former. Probably due to most sources saying global variables are bad and if I do it the latter way I get errors, whereas with the former I only get errors unrelated to scope.
I'm not a professional or even amatuer programmer, that's why I ask. My only exposure to Erlang unfortunately is from coding contests like advent of code and Codegolf stack; and those are "clever" and my brain doesn't abide clever. The inverse of "why use many words when few words will do"
In my own programs, if I go back and read the code I can tell what I copy-pasted vs what I understood as I wrote it, if that makes sense. Because clever stuff that solves the problem I've been unable to solve in the time I alloted will be in my code. You see anything pretending to be, or being a lambda in my code, I lifted it.
I would like to learn Erlang and there's been a few book recommendations aside from TFA. This comment also serves as my reminder to get some books.
well probably part if the reason for example in python, you cannot use the latter for x an integer but you could if x were a dict or a map!!
that's probably why!!
but don't forget, there are core routines that must be used to set certain data that are of the latter form, so at some level you have no choice.
d = {'foo': 1}
d.pop('foo') # <== mutating function callI believe it's purely marketing
Other languages (Java, C#, Go) are supported by massive corporate backers, who have vested interest in seeing them succeed. Erlang's own corporate owner tried to strangle it in the crib, and even since then they've been standoffishness towards anything apart from technical resources.
We didn't really see much marketing like material arise until things like Elixir came about, and even that is more following the ruby model, which is very dev oriented. And the world to which Elixir emerged in 2014 is very different than the world rails sprung into in 2004.
Devs can usually be convinced about how good BEAMlangs are, but without the glossy, you have a harder time convincing the MBA set
Erlang wasn't open sourced until the late 90s. Erlang was never really marketed. Elixir arose fairly quickly.
Erlang and Elixir are nice because concurrency isn't tacked onto the language like those other languages.
Investment gap is what I'd say too. While Rust, Go, Python, etc... have had massive backers that have managed to invest a ton more into things like static analysis, type checking, and developer ergonomics, the Erlang ecosystem hasn't necessarily had the same love, and instead the major users have typically chosen to pivot, or build something outside of the BEAM.
> Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.
1) Because the amount of "high concurrency" you can handle with a single machine with "standard" languages keeps moving up every year.
2) Because you had a single, nigh undocumented, very impenetrable to port implementation of BEAM for the longest time.
3) Because "Erlang" isn't the magic. OTP is the magic. And that requires that you shift your mindset dramatically. And you can do the OTP abstractions in other languages!
4) I think Scala sucked up a lot of Erlang's oxygen for a long time. Erlang is helping with that.
1. Fair, but at the same time - OTP is the only sane concurrency framework. To the point I still struggle to comprehend why it wasn't copied. 2. Fair. 3. I am not aware of a single language doing OTP abstractions, especially implementing preemptive scheduler for green threads. 4. Possibly, but I haven't seen a single instance of a company choosing between the two.
I think marketing is a way better explanation to be honest. Jose lives in Poland, was very active and a lot of Ruby shops moved to Elixir.
There have been multiple attempts in various languages to implement Erlang-like processes and claims of parity, Erlang processes remain unbeaten at what they do.
OTP is the magic but the ability of the Beam to handle that many processes gracefully is part of the magic
C++ and its libraries can do everything erlang does with concurrency, and faster. erlang remains niche mainly for that reason.
You can build anything with anything. Not always a good idea. There are a reason few web apps are built using C++ directly.
everyone is trying to hire the best, but caters to the lowest common denominator.
"But what if we hire an engineer to stupid to learn a new language in a few weeks???" and on and on and on
Because it's a single stack that leaks too many things in my opinion, it's not just code but it's also handling infra, process restart, cache, queue, db ect ... And it only works with Erlang, so it's an all or nothing setup which imo is very restrictive.
Kubernetes does a lot of things that Beam does but better and completely language neutral. Then you have builtin queues and db in Erlang but they are inferior to industry standard and again only work with Erlang.
Honestly, that mixing of concerns is the biggest benefit, at least for me.
While Erlang clearly lost the popularity war, using it (well, Elixir in my case) feels like going to an alternate timeline where it wasn’t a given that a good application used separately developed tools for frontend, backend, database, cache, service discovery, load balancing, containerization, orchestration, and so on. We really could have been on a platform that incorporated all of those things into a single runtime.
The way I explain Erlang to folks is: “it’s not a competitor to Python running Django in Gunicorn; it’s a competitor for Python+Django+Gunicorn in Docker on Kubernetes on Linux, talking via a service mesh and storing data in disk-persisted Redis.” In many ways, swapping out only the top of that stack with Erlang is going to leave you worse off than you were before.
Sadly, the Erlang renaissance was years late to the boom of containerization+orchestration+distributed systems, so we’ll never know how good (or bad) things could have been. Ah well, c’est la vie.
We swapped our Squarespace website out for a Phoenix application yesterday and it was such a delightful experience.
And now with Gleam moving past version 1.0 last year, the BEAM also boasts a decent amount of "flavors" of functional programming. You can go with Erlang/OTP (the alien tech that has been battle tested for almost 30 years), Elixir for it's great webdev ecosystem, Gleam if you like an ML-like programming language on the BEAM. LFE is also another great option if need a BEAM-native LISP.
How I wish that Gleam had OTP!
Elixir is also getting some tooling love in the form of a new language server and some very much appreciated type checking
Forgive my ignorance, I was looking at this the other day, isn't this otp for gleam?
yes but i believe it is still in the ”beta” stage, as gleam is implementing the OTP by itself and not wrapping the erlang implementation.
you have to import the otp package in gleam to use the famous actor architecture
At the same time, the least secret "secret sauce" out there. BBC moved to Elixir (quite recent news), so I guess traction is there.