Is there any way to get serial_number of scheduled instance(for example 1,2,3,4 for weekly scheduler) so i can change the start/end dates of my database query there is something like run_id but i am not sure if it the incremental numeric serial number? Answer I didnt find the incremental serial number of the scheduled dag, instead i modified my approach.
Tag: airflow
How to control the parallelism or concurrency of an Airflow installation?
In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler doesn’t appear to be fully loaded. How can I increase the number of DAGs or tasks that can run concurrently? Similarly, if my installation is under high load and I want to limit how quickly my Airflow workers
Running airflow tasks/dags in parallel
I’m using airflow to orchestrate some python scripts. I have a “main” dag from which several subdags are run. My main dag is supposed to run according to the following overview: I’ve managed to get to this structure in my main dag by using the following lines: What I want airflow to do is to first run the etl_internal_sub_dag1 then
How do I check when my next Airflow DAG run has been scheduled for a specific dag?
I have airflow set up and running with some DAGs scheduled for once a day “0 0 * * *”. I want to check when is the next time a specific dag has been scheduled to run, but I can’t see where I can do that within the admin. Answer If you want you use the Airflow’s CLI, there’s next_execution
Add GCP credentials to airflow via command line
Airflow allows us to add connection information via command-line airflow connections. This can help with automated deployment of airflow installations via ansible or other dev-ops tools. It is unclear how connections to google cloud platform (service accounts) can be added to ariflow via command line. Answer Pre airflow 1.9 the following example outlines how to use a DAG to add
Accessing configuration parameters passed to Airflow through CLI
I am trying to pass the following configuration parameters to Airflow CLI while triggering a dag run. Following is the trigger_dag command I am using. My problem is that how can I access the con parameters passed inside an operator in the dag run. Answer This is probably a continuation of the answer provided by devj. At airflow.cfg the following
How to run Spark code in Airflow?
Hello people of the Earth! I’m using Airflow to schedule and run Spark tasks. All I found by this time is python DAGs that Airflow can manage. DAG example: The problem is I’m not good in Python code and have some tasks written in Java. My question is how to run Spark Java jar in python DAG? Or maybe there
DAG not visible in Web-UI
I am new to Airflow. I am following a tutorial and written following code. On running the script, it doesn’t show any errors but when I check for dags in Web-UI it doesn’t show under Menu->DAGs But I can see the scheduled job under Menu->Browse->Jobs I also cannot see anything in $AIRFLOW_HOME/dags. Is it supposed to be like this only?
Unable to start Airflow worker/flower and need clarification on Airflow architecture to confirm that the installation is correct
Running a worker on a different machine results in errors specified below. I have followed the configuration instructions and have sync the dags folder. I would also like to confirm that RabbitMQ and PostgreSQL only needs to be installed on the Airflow core machine and does not need to be installed on the workers (the workers only connect to the