When trying a new Airflow version, I got this error:
E ValueError: The name 'my_airflow_plugin' is already registered for this blueprint. Use 'name=' to provide a unique name.
With Apache Airflow you can define a plugin using an entry_point.
I managed to track it down to a call to importlib_metadata.distributions()
which returns the same object twice.
Why does it return twice?
Advertisement
Answer
The importlib_metadata.distributions()
call uses your PYTHONPATH
environment variable, accessible via sys.path
in your python project.
When I inspected my sys.path
, it turns out, I had duplicates in there. When removing these duplicates I also fixed the PYTHONPATH issue.
I added the following code that I used to deduplicate it:
import sys from typing import List def deduplicate_python_path() -> None: """ Our python path may contain duplicates that will lead to discovering our adyen plugin multiple times. To avoid that, we deduplicate the python path first while remaining an ordering based on first occurrences. """ new_list: List[str] = [] for item in sys.path: if item not in new_list: new_list.append(item) sys.path = new_list