Tag: airflow-scheduler

For a pipeline like below, sensor was set to softfail= True I’m trying to figure out how to only skip certain tasks when the sensor fails. For example only have B and D fail but still execute C and E. Many thanks for your help. Sensor A >> B >> C >> D >> E Answer I think you could

I am new to Airflow, and I am trying to create a Python pipeline scheduling automation process. My project youtubecollection01 utilizes custom created modules, so when I run the DAG it fails with ModuleNotFoundError: No module named ‘Authentication’. This is how my project is structured: This is my dag file: I am importing main function from the __main__.py, however inside

I have implemented email alerts on success and failure using on_success_callback and on_failure_callback. According to Airflow documentation, a context dictionary is passed as a single parameter to this function. How can I pass another parameter to these callback methods? Here is my code I intend to move the callbacks to another package and pass the email address as parameter. Answer