Skip to content
Advertisement

Install newer version of sqlite3 on AWS Lambda for use with Python

I have a Python script running in a Docker container on AWS Lambda. I’m using the recommended AWS image (public.ecr.aws/lambda/python:3.9), which comes with SQLite version 3.7.17 (from 2013!). When I test the container locally on my M1 Mac, I see this:

JavaScript

However, I use newer SQLite features, so I need to find a way to use a newer version of the library. The most straightforward solution would be to install a binary package as suggested in this answer. The docs say it should be as simple as installing using pip. Unfortunately, when I attempt to use this approach inside the Docker container, I get this:

JavaScript

And I get the same error when I attempt to install it outside the container using pipenv (which is what I’m actually using for package management):

JavaScript

Am I doing something wrong? And if not, how can I get a recent version of SQLite which Python can use in this container? Do I really need to use a separate build stage in the Dockerfile as suggested here and copy the rpm components into place as laid out here? That feels like a lot of work for something that many people presumably need to do all the time.


Update: I tried the rpm approach inside the container using version 3.26 from EPEL8 (IIUC) and it failed with a bunch of dependency errors like this:

JavaScript

When I try --skip-broken, it just skips installing the 3.26 package altogether.


Update 2: I’ve tried downloading the Python 3.9 wheel from pysqlite3-binary manually. However, it looks like that project only produces wheels for x86_64, not the aarch64 platform which Lambda uses. (This is not correct, see answer.) So presumably that’s why pip is not finding it.

Advertisement

Answer

The problem was that I was running Docker locally to do my testing, on an M1 Mac. Hence the aarch64 architecture. Lambda does allow you to use ARM, but thankfully it still defaults to x86_64. I confirmed that my Lambda function was running x86_64, which is what the binary wheel uses, so that’s good:

enter image description here

So I needed to do three things:

  1. Change my Pipfile to conditionally install the binary package only on x86_64:

    JavaScript
  2. Tweak the sqlite import, as described in the original answer:

    JavaScript
  3. Set my Docker container to launch in x86 emulation mode locally.

    JavaScript

Et, voilà!

JavaScript
User contributions licensed under: CC BY-SA
5 People found this is helpful
Advertisement