Skip to content
Advertisement

Importing cython generated *.so-module with another python-version or on another OS

How should a file myModule.cpython-35m-x86_64-linux-gnu.so be imported in python? Is it possible?

I tried the regular way:

import myModule

and the interpreter says:

`ModuleNotFoundError: No module named 'myModule'`

This is a software that I can’t install in the cluster that I am working at so I just extracted the .deb package and it does not have a wheel file or structure to install.

Advertisement

Answer

It is problematic to use a C-extension built for one Python version in another Python version. Normally (at least for Python3) there is a mechanism in place to differentiate C-extensions for different Python versions, so they can co-exist in the same directory.

In your example, the suffix is cpython-35m-x86_64-linux-gnu so this C-extension will be picked up by a CPython3.5 on a x86_64 Linux. If you try to import this extension with another Python-version or on another plattform, the module isn’t visible and ModuleNotFoundError is raised.

It is possible to see, which suffixes are accepted by the current Python version, e.g. via:

>>> import _imp
>>>_imp.extension_suffixes()
['.cpython-36m-x86_64-linux-gnu.so', '.abi3.so', '.so']

A possibility is to use the stable C-API which could be used with multiple Python versions without recompilation. Cython start to support it in version 3.0 (see this PR), see also this SO-post about setuptools and stable C-API.

One might want to be clever and rename the extension to simple .so, so it can be picked up by the Finder – this can/does work for some Python-version combinations on some platforms for some extension – yet this approach cannot be sustained in the long run and is not the right thing to do.

The right thing to do, is to build the C-extension for/with the right Python-version on the right OS/platform or to use the right wheel (or use stable C-API).


In general, a C-extension built for a python-version (let’s say PythonA.B) cannot be used by another Python version (let’s say PythonC.D), because those extensions/modules are linked against a special Python-library and the needed functionality might no longer/not yet be present in the library of another version.

This different to *.py-files and more similar to *.pyc-files which cannot be used with a different version.

While PEP-3147 regulates the suffices of *.pyc-files, PEP-3149 does the same for the C-extensions. PEP-3149 is however not the state-of-the-art, as some of the problems where fixed only in Python3.5, the whole discussion can be found here.

User contributions licensed under: CC BY-SA
1 People found this is helpful
Advertisement