Flask blueprint already registered due to setuptools returns duplicate distributions

When trying a new Airflow version, I got this error:

E           ValueError: The name 'my_airflow_plugin' is already registered for this blueprint. Use 'name=' to provide a unique name.

With Apache Airflow you can define a plugin using an entry_point. I managed to track it down to a call to importlib_metadata.distributions() which returns the same object twice. Why does it return twice?

Answer

The importlib_metadata.distributions() call uses your PYTHONPATH environment variable, accessible via sys.path in your python project. When I inspected my sys.path, it turns out, I had duplicates in there. When removing these duplicates I also fixed the PYTHONPATH issue.

I added the following code that I used to deduplicate it:

import sys
from typing import List

def deduplicate_python_path() -> None:
    """
    Our python path may contain duplicates that will lead to discovering our adyen plugin multiple times.
    To avoid that, we deduplicate the python path first while remaining an ordering based on first occurrences.
    """
    new_list: List[str] = []
    for item in sys.path:
        if item not in new_list:
            new_list.append(item)
    sys.path = new_list