ferguess_k 5 days ago

Many companies wrote their in-house framework for ETL. For example I have worked in two companies that both chose to write sort of YAML based ingestion framework so that users only need to write YAML.

I guess you can make the case that a general framework is useful, but TBH I think either it takes too long to include all use cases (and each company probably only needs one), or becomes too complicated to be a framework.

Not to discourage you though, it's fun to build your own project and I'd love you to try it out. Worst case, no one uses it but you bag a ton of experience.

1
codingmoh 4 days ago

I get why so many teams roll their yaml based system, as it feels more straightforward to build something tailored to your needs than trying to shoehorn the generalities in. But that’s exactly the tricky part: the moment you try to accommodate every possible use case, the framework either blows up in scope or ends up too niche to be widely adopted.