gpderetta 2 days ago

From just a short skimming of the docs:

- my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio.

- I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement.

- Async need not force the full pipeline to be async. There should be an option to run async funcitons in background event loops. Especially as you already support threaded executions.

2
pyper-dev 2 days ago

Your third point intrigues me a lot. I imagine for the majority of cases, it's generally more useful to work with async functions in the structure of async syntax, but I suppose it's possible to run async functions in a synchronous pipeline.

Even though there's currently no built-in support for this, a workaround could be to just define synchronous helper functions to handle running your async logic in an event loop.

d0mine 2 days ago

asyncio.TaskGroup from stdlib provides structured concurrency

gpderetta 2 days ago

Since 3.11 it seems; I'm currently stuck on 3.8, but hopefully we should be able to upgrade soon. Thanks.