I have a Dag
in air-flow
looks like that:
start >> [task1,task2,....task16] >> end
I want to limit the tasks that running simultaneously in this dag to be 4 for example.
I know there is 'max_active_tasks_per_dag'
parameters, but it affects on all dags and in my case I want to define it only for my dag.
how can I do so? it is even possible?
Advertisement
Answer
If you look at the description of the parameter you will find out that you can configure it per dag:
max_active_tasks_per_dag New in version 2.2.0.
The maximum number of task instances allowed to run concurrently in each DAG. To calculate the number of tasks that is running concurrently for a DAG, add up the number of running tasks for all DAG runs of the DAG. This is configurable at the DAG level with max_active_tasks, which is defaulted as max_active_tasks_per_dag.