Is there any way to get serial_number of scheduled instance(for example 1,2,3,4 for weekly scheduler) so i can change the start/end dates of my database query
there is something like run_id
but i am not sure if it the incremental numeric serial number?
Advertisement
Answer
I didnt find the incremental serial number of the scheduled dag, instead i modified my approach. I got the the scheduled date and than performed database query based on that
To access the scheduled(Running date) by {{ dag_run.execution_date }}
You can then pass it into spark you can do this
['spark-submit','/home/hadoop/test.py',"{{ dag_run.execution_date }}"]