Skip to content
Advertisement

Tag: parallel-processing

Running airflow tasks/dags in parallel

I’m using airflow to orchestrate some python scripts. I have a “main” dag from which several subdags are run. My main dag is supposed to run according to the following overview: I’ve managed to get to this structure in my main dag by using the following lines: What I want airflow to do is to first run the etl_internal_sub_dag1 then

How do I parallelize a simple Python loop?

This is probably a trivial question, but how do I parallelize the following loop in python? I know how to start single threads in Python but I don’t know how to “collect” the results. Multiple processes would be fine too – whatever is easiest for this case. I’m using currently Linux but the code should run on Windows and Mac

Advertisement