Poetry doesn`t use system-global Cython for compiling dependencies from source

I have a dependency package hdbscan that is compiled from source and requires a Cython to be present.

Now, the dependencies are managed through Poetry, and it seems that while compiling hdbscan it uses a dedicated virtualenv with no Cython available in it.

  The following additional packages will be installed:
    dh-python libpython3-dev libpython3.7 libpython3.7-dev python3-dev
    python3.7-dev
  Suggested packages:
    cython-doc
  The following NEW packages will be installed:
    cython3 dh-python libpython3-dev libpython3.7 libpython3.7-dev python3-dev
    python3.7-dev
  0 upgraded, 7 newly installed, 0 to remove and 8 not upgraded.
  Need to get 51.9 MB of archives.
  After this operation, 96.1 MB of additional disk space will be used.
  Get:1 http://deb.debian.org/debian buster/main amd64 cython3 amd64 0.29.2-2 [1392 kB]
  Get:2 http://deb.debian.org/debian buster/main amd64 dh-python all 3.20190308 [99.3 kB]
  Get:3 http://deb.debian.org/debian buster/main amd64 libpython3.7 amd64 3.7.3-2+deb10u2 [1498 kB]
  Get:4 http://deb.debian.org/debian buster/main amd64 libpython3.7-dev amd64 3.7.3-2+deb10u2 [48.4 MB]
  Get:5 http://deb.debian.org/debian buster/main amd64 libpython3-dev amd64 3.7.3-1 [20.1 kB]
  Get:6 http://deb.debian.org/debian buster/main amd64 python3.7-dev amd64 3.7.3-2+deb10u2 [510 kB]
  Get:7 http://deb.debian.org/debian buster/main amd64 python3-dev amd64 3.7.3-1 [1264 B]
  debconf: delaying package configuration, since apt-utils is not installed
  Fetched 51.9 MB in 3s (18.8 MB/s)
.......
  Installing dependencies from lock file
  
    PackageInfoError
  
    Unable to determine package info for path: /opt/teamcity-agent/temp/buildTmp/pypoetry-git-hdbscank82kqn8_
  
    Fallback egg_info generation failed.
  
    Command ['/opt/teamcity-agent/temp/buildTmp/tmpov_cp376/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output:
    setup.py:8: UserWarning: No module named 'Cython'
      warnings.warn(e.args[0])
    setup.py:92: UserWarning: Due to incompatibilities with Python 3.7 hdbscan nowrequires Cython to be installed in order to build it
      warnings.warn('Due to incompatibilities with Python 3.7 hdbscan now'
    Traceback (most recent call last):
      File "setup.py", line 94, in <module>
        raise ImportError('Cython not found! Please install cython and try again')
    ImportError: Cython not found! Please install cython and try again
  
    at /opt/venv/lib/python3.7/site-packages/poetry/inspection/info.py:503 in _pep517_metadata
        499│                     venv.run("python", "setup.py", "egg_info")
        500│                     return cls.from_metadata(path)
        501│                 except EnvCommandError as fbe:
        502│                     raise PackageInfoError(
      → 503│                         path, "Fallback egg_info generation failed.", fbe
        504│                     )
        505│                 finally:
        506│                     os.chdir(cwd.as_posix())
        507│
  Process exited with code 1

I tried installing cython3 in apt before poetry install.

I tried pip install cython before poetry install

I read through some SO questions and tried adding

[build-system]
requires = ["poetry>=1.0.9", "Cython"]

to my pyproject.toml file, but it has no effect (maybe it is only important when the very package defined in config is built )

The only thing that works is installing cython and hdbscan in pip inside the same virtualenv or system-global, but that doesn`t sound like reasonable

Answer

It turned out it was the same problem as I described in https://github.com/python-poetry/poetry/issues/3744.

We use a private pypi mirror and without explicitly setting

 pip config set global.index-url ${PYPI_URL}

poetry was unable to install a Cython in a build virualenv even though a system-wide Cython was present

Advertisement

Answer