Skip to content
Advertisement

How to airflow dag’s get serial_number of scheduled instance(for example 1,2,3,4 for weekly scheduler)

Is there any way to get serial_number of scheduled instance(for example 1,2,3,4 for weekly scheduler) so i can change the start/end dates of my database query

there is something like run_id but i am not sure if it the incremental numeric serial number?

Advertisement

Answer

I didnt find the incremental serial number of the scheduled dag, instead i modified my approach. I got the the scheduled date and than performed database query based on that

To access the scheduled(Running date) by {{ dag_run.execution_date }}

You can then pass it into spark you can do this

['spark-submit','/home/hadoop/test.py',"{{ dag_run.execution_date }}"]
User contributions licensed under: CC BY-SA
8 People found this is helpful
Advertisement