I’m trying to create an AWS Lambda Deployment Package, so I can use it as an AWS Lambda Layer. Specifically, I’m trying to use the ipopt sovler, in conjuction with the pyomo and cython. I can get the pyomo packaged without issue, but when I try to pip install ipopt for the package I get an error:
ERROR: Command errored out with exit status 1: command: 'c:python36python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"'; __file__='"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:UsersusernameAppDataLocalTemppip-record-d1o_oqjyinstall-record.txt' --single-version-externally-managed --compile --install-headers 'c:python36Includeipopt' cwd: C:UsersusernameAppDataLocalTemppip-install-rir_unvzipopt Complete output (20 lines): running install running build running build_py creating build creating buildlib.win-amd64-3.6 creating buildlib.win-amd64-3.6ipopt copying ipoptipopt_wrapper.py -> buildlib.win-amd64-3.6ipopt copying ipoptversion.py -> buildlib.win-amd64-3.6ipopt copying ipopt__init__.py -> buildlib.win-amd64-3.6ipopt running build_ext skipping 'srccyipopt.c' Cython extension (up-to-date) building 'ipopt.cyipopt' extension creating buildtemp.win-amd64-3.6 creating buildtemp.win-amd64-3.6Release creating buildtemp.win-amd64-3.6Releasesrc C:Program Files (x86)Microsoft Visual Studio 14.0VCBINx86_amd64cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iinclude_mt/coin -Ic:python36libsite-packagesnumpycoreinclude -Ic:python36include -Ic:python36include "-IC:Program Files (x86)Microsoft Visual Studio 14.0VCINCLUDE" "-IC:Program Files (x86)Windows Kits10include10.0.10240.0ucrt" "-IC:Program Files (x86)Windows Kits8.1includeshared" "-IC:Program Files (x86)Windows Kits8.1includeum" "-IC:Program Files (x86)Windows Kits8.1includewinrt" /Tcsrccyipopt.c /Fobuildtemp.win-amd64-3.6Releasesrccyipopt.obj cyipopt.c c:python36libsite-packagesnumpycoreincludenumpynpy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION srccyipopt.c(569): fatal error C1083: Cannot open include file: 'IpStdCInterface.h': No such file or directory error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe' failed with exit status 2 ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:python36python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"'; __file__='"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:UsersusernameAppDataLocalTemppip-record-d1o_oqjyinstall-record.txt' --single-version-externally-managed --compile --install-headers 'c:python36Includeipopt' Check the logs for full command output.
I am trying to run the pyomo/ipopt package in an AWS lambda function. This package isn’t in the SDK, so I’ve read that I’ll have to compile the dependencies into a zip, and then upload the zip into the lambda function.
I’ve done some research and tried out the links below, but when I try to pip install ipopt I get error messages – I’m on a Windows box, but I’ve also tried it on the Linux EC2 instance
Appreciate the help!
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/
Unable to install cvxpy into virtualenv for AWS lambda
https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e
https://github.com/coin-or/Ipopt
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
AWS Lambda – unable to import module ‘lambda_function’
Advertisement
Answer
TL;DR, here is the working lambda deployment package that’s compatible with Python 3.6 runtime. [gdrive download link]
It turns out it’s super tricky to pack everything needed into one zip. I spent quite a while (several hours) manually copying library files over. Here are the steps that I took to generate this package.
- Spin up a local lambda Python 3.6 runtime docker container
docker run -it --rm lambci/lambda:build-python3.6 bash
- Pip install all the dependencies into a local directory
mkdir ipopt-lib PYTHONPATH=$PYTHONPATH:~/ipopt-lib pip install Cython numpy -t ipopt-lib/
- Followed the official guide to build and install Ipopt from source code
yum install lapack-devel git clone https://github.com/coin-or/Ipopt.git cd Ipopt IPOPTDIR=`pwd` mkdir $IPOPTDIR/build cd $IPOPTDIR/build $IPOPTDIR/configure make make install
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pip install ipopt -t ipopt-lib/
Manually copying all the library files over
cp -t ~/ipopt-lib/ /usr/local/lib/lib* cp -t ~/ipopt-lib/ /usr/lib64/libblas.so* /usr/lib64/liblapack* cp -t ~/ipopt-lib/ /usr/lib64/libgfortran.so.3* cp -t ~/ipopt-lib/ /usr/lib64/libstdc++.so* cp -t ~/ipopt-lib/ /usr/lib64/libquadmath.so.0*
- Zipping all together
cd ~/ipopt-lib zip -r /root/ipopt-lib.zip *
- Copy to the host machine
docker cp <CONTAINER_ID>:/root/ipopt-lib.zip ~/Downloads/
Here are all the Python packages available in this deployment package (including the default ones installed in the runtime):
$ pip freeze appdirs==1.4.3 arrow==0.15.5 attrs==19.3.0 aws-lambda-builders==0.8.0 aws-sam-cli==0.47.0 aws-sam-translator==1.22.0 awscli==1.18.41 binaryornot==0.4.4 boto3==1.12.41 botocore==1.15.41 certifi==2020.4.5.1 chardet==3.0.4 chevron==0.13.1 click==7.1.1 colorama==0.4.3 cookiecutter==1.6.0 Cython==0.29.17 dateparser==0.7.4 distlib==0.3.0 docker==4.2.0 docutils==0.15.2 filelock==3.0.12 Flask==1.0.4 future==0.18.2 idna==2.8 importlib-metadata==1.6.0 importlib-resources==1.4.0 ipopt==0.1.9 itsdangerous==1.1.0 Jinja2==2.11.2 jinja2-time==0.2.0 jmespath==0.9.5 jsonschema==3.2.0 MarkupSafe==1.1.1 numpy==1.18.3 pipenv==2018.11.26 poyo==0.5.0 pyasn1==0.4.8 pyrsistent==0.16.0 python-dateutil==2.8.0 pytz==2019.3 PyYAML==5.3.1 regex==2020.4.4 requests==2.22.0 rsa==3.4.2 s3transfer==0.3.3 serverlessrepo==0.1.9 six==1.14.0 tomlkit==0.5.8 tzlocal==2.0.0 urllib3==1.25.8 virtualenv==20.0.18 virtualenv-clone==0.5.4 websocket-client==0.57.0 Werkzeug==1.0.1 whichcraft==0.6.1 zipp==3.1.0
Testing on lambda using Python 3.6 runtime
import sys try: sys.path.index('/opt') except ValueError: sys.path.append('/opt') import os os.environ['LD_LIBRARY_PATH'] += os.pathsep + '/opt' import json import numpy import ipopt def lambda_handler(event, context): print(numpy.__version__) print(ipopt.__version__)
Result
Response: null Request ID: "9bd9fd88-d340-4976-a954-8a727e173933" Function Logs: START RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 Version: $LATEST 1.18.3 0.1.9 END RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 REPORT RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 Duration: 0.45 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 74 MB Init Duration: 343.99 ms