I am new to Airflow, and I am wondering, how do I load a file from a GCS Bucket to BigQuery? So far, I have managed to do BigQuery to GCS Bucket: Can someone help me to modify my current code, so I can load a file from a GCS Bucket and load it to BigQuery? Answer For your requirement,
Tag: directed-acyclic-graphs
Read run_id in airflow operator for non templated fields
I am trying to read run_id for DAG in SnowflakeOperator to set a session parameter, query_tag. But it seems like the session parameter is not templated. How can I reference run_id and use it as an input here? Answer You need to make the non-templated field templated. Then you can use it as:
Is there a function in snakemake to make the list of output dependent on the arguments passed into the shell command
I have a snakemake rule that calls a python program, the output of the python program is dependent on the arguments passed. I would like to make snakemake aware of the differences between the expected output when a certain parameter is passed (Boolean) or not passed. My current solution is to create a list of output list_phen_gen_output dependent on the
How to handle line breaks in a templated file included in a Jinja2 JSON template
I am writing a Jinja2 template to output a JSON DAG where one value is a string containing a list of commands from an external shell script, (containing templated variables. This works if the commands in the script are semicolon-separated on a single line: But, it fails to render the template when formatted with each command on its own line:
Return minimum “sub-DAG” generated from dictionary
I have an input data and some transformation functions t1, t2, t3, t4, t5, t6. Each of them requires some columns as input and outputs some columns. The DAG associated with these transformations is I get as input the columns I want to generate and I should return the sub-DAG required to obtain it (I am not interested only in
How to run Spark code in Airflow?
Hello people of the Earth! I’m using Airflow to schedule and run Spark tasks. All I found by this time is python DAGs that Airflow can manage. DAG example: The problem is I’m not good in Python code and have some tasks written in Java. My question is how to run Spark Java jar in python DAG? Or maybe there
DAG not visible in Web-UI
I am new to Airflow. I am following a tutorial and written following code. On running the script, it doesn’t show any errors but when I check for dags in Web-UI it doesn’t show under Menu->DAGs But I can see the scheduled job under Menu->Browse->Jobs I also cannot see anything in $AIRFLOW_HOME/dags. Is it supposed to be like this only?