Skip to content

how to limit the number of tasks run simultaneously in a specific dag on airflow?

I have a Dag in air-flow looks like that:

start >> [task1,task2,....task16] >> end

I want to limit the tasks that running simultaneously in this dag to be 4 for example.

I know there is 'max_active_tasks_per_dag' parameters, but it affects on all dags and in my case I want to define it only for my dag. how can I do so? it is even possible?

Answer

If you look at the description of the parameter you will find out that you can configure it per dag:

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-active-tasks-per-dag

max_active_tasks_per_dag New in version 2.2.0.

The maximum number of task instances allowed to run concurrently in each DAG. To calculate the number of tasks that is running concurrently for a DAG, add up the number of running tasks for all DAG runs of the DAG. This is configurable at the DAG level with max_active_tasks, which is defaulted as max_active_tasks_per_dag.