Recently Microsoft has withdrawn the basic authentication service for accessing the Exchange Service. Therefore I am working on switching the authentication from basic to OAuth based. However, during the testing, I come across the below exception. I went through the suggestions mentioned here https://github.c…
Tag: airflow
Update Airflow XCOM value in Database
How can I update any value in the XCOM table with: Is there any option like the following? Answer I solved my own question:
Configure volumes in airflow GKEStartPodOperator operator
I have a google cloud composer environment. In my DAG I want to create a pod in GKE. When I come to deploy a simple app based on a docker container that doesn’t need any volume configuration or secrets, everything works fine, for example: But when I have an application that need to access to my GKE clus…
how to limit the number of tasks run simultaneously in a specific dag on airflow?
I have a Dag in air-flow looks like that: I want to limit the tasks that running simultaneously in this dag to be 4 for example. I know there is ‘max_active_tasks_per_dag’ parameters, but it affects on all dags and in my case I want to define it only for my dag. how can I do so? it is even possibl…
How do I retrieve airflow url link to my latest task log in a DAG?
In individual DAG task, how do I set up the url link with the help from python operator as I am intending to send an url link of the latest log directly to the user whenever errors occur so that they can access to the page directly skipping the step of navigating. Answer You can define a callback function, wh…
How to return False when file exists with FIleSensor
I’m trying to reverse FileSensor(): that is, with It will return True if the file exists. But, how can it return False when the file exists and True when it does not? Answer You could create a PythonSensor that checks for the existence of a file with the logic that you want.
airflow on kubernetes: How to install extra python packages?
i installed airflow using bitnami repo: To install extra python packages i mounted an extra volume I prepared my requirements.txt file and then i created a ConfigMap using kubectl create -n airflow configmap requirements –from-file=requirements.txt after this i upgraded airflow using helm upgrade……
Attempting to delete files in s3 folder but the command is removing the entire directory itself
I have an s3 bucket which has 4 folders now of which is input/. After the my airflow DAG Runs at the end of the py code are few lines which attempt to delete all files in the input/. Now, this sometimes deletes all files and sometimes deletes the directory itself. I am not sure why it is deleting even
Airflow: ‘ERR_CONNECTION_REFUSED’ on port 8080 & 8081
I installed airflow (v 2.2.4) on the ami below: The folder it currently lives in is here (But should I relocate this somewhere else per these docs?) AMI INFO: And the issue I am having is that after I run airflow initdb – which works great, I run airflow webserver and when I try to copy in the http valu…
How to execute multiple sql files in airflow using PostgresOperator?
I have multiple sql files in my sql folder. I am not sure how to execute all the sql files within a DAG? For a single file, below code works Answer With a list Or you can make it dynamic