I have some protocol buffer definitions which need to be built to Python source as part of the pip install
process. I’ve subclassed the setuptools.command.install
command in setup.py
but I think it’s trying to run the Makefile after the package is installed so the sources aren’t recognised.
I can’t find information about what happens during a pip installation. Can anyone shed any light?
setup.py:
import subprocess import sys from setuptools import setup from setuptools.command.install import install class Install(install): """Customized setuptools install command - builds protos on install.""" def run(self): protoc_command = ["make", "python"] if subprocess.call(protoc_command) != 0: sys.exit(-1) install.run(self) setup( name='myprotos', version='0.0.1', description='Protocol Buffers.', install_requires=[], cmdclass={ 'install': Install, } )
Output of $ pip install -vvv .
:
Processing /path/to/myprotos Running setup.py (path:/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py) egg_info for package from file:///path/to/myprotos Running command python setup.py egg_info running egg_info creating pip-egg-info/myprotos.egg-info writing pip-egg-info/myprotos.egg-info/PKG-INFO writing top-level names to pip-egg-info/myprotos.egg-info/top_level.txt writing dependency_links to pip-egg-info/myprotos.egg-info/dependency_links.txt writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt' reading manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt' writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt' Source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build has version 0.0.1, which satisfies requirement myprotos==0.0.1 from file:///path/to/myprotos Building wheels for collected packages: myprotos Running setup.py bdist_wheel for myprotos: started Destination directory: /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel- Running command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel- --python-tag cp27 running bdist_wheel running build installing to build/bdist.macosx-10.12-x86_64/wheel running install # THIS IS MY MAKEFILE RUNNING Grabbing github.com/google/protobuf... Building Python protos... # MAKEFILE COMPLETE running install_egg_info running egg_info creating myprotos.egg-info writing myprotos.egg-info/PKG-INFO writing top-level names to myprotos.egg-info/top_level.txt writing dependency_links to myprotos.egg-info/dependency_links.txt writing manifest file 'myprotos.egg-info/SOURCES.txt' reading manifest file 'myprotos.egg-info/SOURCES.txt' writing manifest file 'myprotos.egg-info/SOURCES.txt' Copying myprotos.egg-info to build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1-py2.7.egg-info running install_scripts creating build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1.dist-info/WHEEL Running setup.py bdist_wheel for myprotos: finished with status 'done' Stored in directory: /Users/jds/Library/Caches/pip/wheels/92/0b/37/b5a50146994bc0b6774407139f01d648ba3a9b4853d2719c51 Removing source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build Successfully built myprotos Installing collected packages: myprotos Found existing installation: myprotos 0.0.1 Uninstalling myprotos-0.0.1: Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/DESCRIPTION.rst Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/INSTALLER Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/METADATA Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/RECORD Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/WHEEL Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/metadata.json Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/top_level.txt Successfully uninstalled myprotos-0.0.1 Successfully installed myprotos-0.0.1 Cleaning up...
Should my Makefile be running early in the process to generate the source files? Do the files need to be there before egg_info
runs for example?
If I manually run the Makefile and then install the package then it works.
Update
Here is the structure of my project:
myprotos ├── Makefile ├── README.md ├── document.proto ├── myprotos # Generated by Makefile │ ├── __init__.py # Generated by Makefile │ └── proto_pb2.py # Generated by Makefile └── setup.py
Here is the section of the Makefile which generates the Python source from Potocol Buffer definitions:
python: protoc deps # the protoc and deps command above just downloads # the `protoc` binary to a local bin directory @echo "Building Python protos..." @mkdir -p "${PYTHON_OUT}" @touch "${PYTHON_OUT}"/__init__.py @printf "__all__ = ['proto_pb2']" > "${PYTHON_OUT}"/__init__.py @PATH="${LOCAL_BINARY_PATH}:$$PATH" protoc --proto_path="${BASE}" --proto_path="${GOPATH}/src/github.com/google/protobuf/src" --python_out="${PYTHON_OUT}/" ${PROTOS}
Advertisement
Answer
OK, there are three things you need to change here:
Add
Makefile
anddocument.proto
to a new fileMANIFEST.in
.Makefile document.proto
If you do that, the
.zip
file created bypython setup.py sdist
(which is also uploaded to PyPI) will contain those files.You need to run your
make
command duringpython setup.py build
, not duringinstall
. Since you are generating Python code, you will need to change thebuild_py
command here:import sys import subprocess from setuptools import setup from setuptools.command.build_py import build_py class Build(build_py): """Customized setuptools build command - builds protos on build.""" def run(self): protoc_command = ["make", "python"] if subprocess.call(protoc_command) != 0: sys.exit(-1) super().run() setup( name='buildtest', version='1.0', description='Python Distribution Utilities', packages=['buildtest'], cmdclass={ 'build_py': Build, } )
If your
Makefile
generated machine code, i.e. from C or any other compiled language, you should change thebuild_ext
command:import sys import subprocess from setuptools import setup from setuptools.command.build_ext import build_ext class Build(build_ext): """Customized setuptools build command - builds protos on build.""" def run(self): protoc_command = ["make", "python"] if subprocess.call(protoc_command) != 0: sys.exit(-1) super().run() setup( name='buildtest', version='1.0', description='Python Distribution Utilities', packages=['buildtest'], has_ext_modules=lambda: True, cmdclass={ 'build_ext': Build, } )
Lastly, you need to tell
setuptools
to install the resulting package oninstall
by defining an attributepackages
insetup()
:setup( ... packages=['myprotos'] )
The reason for deciding to run build_py
or build_ext
lies in the situation when the two are run:
build_py
is also run when creating source distributions, which have to be cross-platform. Compiled extensions are usually not cross-platform, so you cannot compile them in this step.build_ext
is only run when you are creating binary distributions, which are platform-specific. It is OK to compile to platform-specific machine code here.