I have multiple sql files in my sql folder. I am not sure how to execute all the sql files within a DAG?
JavaScript
x
5
1
- dags
2
- sql
3
- dummy1.sql
4
- dummy2.sql
5
For a single file, below code works
JavaScript
1
4
1
sql_insert= PostgresOperator(task_id='sql_insert',
2
postgres_conn_id='postgres_conn',
3
sql='sql/dummy1.sql')
4
Advertisement
Answer
With a list
JavaScript
1
4
1
sql_insert= PostgresOperator(task_id='sql_insert',
2
postgres_conn_id='postgres_conn',
3
sql=['sql/dummy1.sql', 'sql/dummy2.sql'])
4
Or you can make it dynamic
JavaScript
1
5
1
import glob
2
sql_insert= PostgresOperator(task_id='sql_insert',
3
postgres_conn_id='postgres_conn',
4
sql=glob.glob("sql/*.sql")]
5