my thoughts, exactly, for around 10 years, since my first serious data engineering projects. So many projects I worked on, all suffering this problem: complex individual data pipelines.
Sometimes I wrote small frameworks on my own, sometimes I used 3rd party tools. I also tried to create a universal framework for that, trying to solve exactly what you described. Never finished it, because I am more of the no-finishing guy.
However, at some point I came to the conclusion, that it's simply not possible. which is nonsense and more of an excuse. as you said, it's just data in and data out, processed through some simple rules.
so, I'd know the concept, but there are a million other projects in my head.
I know exactly what you mean—it’s the same cycle I’ve seen in countless projects, with every project spiraling into a brittle pipeline that’s hard to maintain. Despite all the promising ideas, we never seem to nail down a universal solution that’s both simple and robust. - There must be a way, and sooner or later it will be figured out.