Install newer version of sqlite3 on AWS Lambda for use with Python

I have a Python script running in a Docker container on AWS Lambda. I’m using the recommended AWS image (public.ecr.aws/lambda/python:3.9), which comes with SQLite version 3.7.17 (from 2013!). When I test the container locally on my M1 Mac, I see this:

$ docker run --env-file .env --entrypoint bash -ti my-image
bash-4.2# uname -a
Linux e9ed14d35cbe 5.10.104-linuxkit #1 SMP PREEMPT Thu Mar 17 17:05:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
bash-4.2# sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668

However, I use newer SQLite features, so I need to find a way to use a newer version of the library. The most straightforward solution would be to install a binary package as suggested in this answer. The docs say it should be as simple as installing using pip. Unfortunately, when I attempt to use this approach inside the Docker container, I get this:

bash-4.2# pip3 install pysqlite3-binary
ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary

And I get the same error when I attempt to install it outside the container using pipenv (which is what I’m actually using for package management):

🕙 01:08:24 ❯ pipenv install pysqlite3-binary
Installing pysqlite3-binary...
Error:  An error occurred while installing pysqlite3-binary!
Error text:
ERROR: Could not find a version that satisfies the requirement pysqlite3-binary (from versions: none)
ERROR: No matching distribution found for pysqlite3-binary

✘ Installation Failed

Am I doing something wrong? And if not, how can I get a recent version of SQLite which Python can use in this container? Do I really need to use a separate build stage in the Dockerfile as suggested here and copy the rpm components into place as laid out here? That feels like a lot of work for something that many people presumably need to do all the time.

Update: I tried the rpm approach inside the container using version 3.26 from EPEL8 (IIUC) and it failed with a bunch of dependency errors like this:

bash-4.2# curl --output-dir /tmp -sO https://vault.centos.org/centos/8/BaseOS/aarch64/os/Packages/sqlite-3.26.0-15.el8.aarch64.rpm
bash-4.2# yum localinstall /tmp/sqlite-3.26.0-15.el8.aarch64.rpm
Loaded plugins: ovl
Examining /tmp/sqlite-3.26.0-15.el8.aarch64.rpm: sqlite-3.26.0-15.el8.aarch64

# etc.

--> Finished Dependency Resolution
Error: Package: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
           Requires: libc.so.6(GLIBC_2.28)(64bit)

# Plus 6 other package dependency errors

Error: Package: nss-softokn-3.67.0-3.amzn2.0.1.aarch64 (@amzn2-core)
           Requires: libsqlite3.so.0()(64bit)
           Removing: sqlite-3.7.17-8.amzn2.1.1.aarch64 (@amzn2-core)
               libsqlite3.so.0()(64bit)
           Updated By: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
               Not found
           Obsoleted By: sqlite-3.26.0-15.el8.aarch64 (/sqlite-3.26.0-15.el8.aarch64)
               Not found
           Available: sqlite-3.7.17-8.amzn2.0.2.aarch64 (amzn2-core)
               libsqlite3.so.0()(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

When I try --skip-broken, it just skips installing the 3.26 package altogether.

Update 2: I’ve tried downloading the Python 3.9 wheel from pysqlite3-binary manually. However, it looks like that project only produces wheels for x86_64, not the aarch64 platform ~~which Lambda uses.~~ (This is not correct, see answer.) So presumably that’s why pip is not finding it.

Answer

The problem was that I was running Docker locally to do my testing, on an M1 Mac. Hence the aarch64 architecture. Lambda does allow you to use ARM, but thankfully it still defaults to x86_64. I confirmed that my Lambda function was running x86_64, which is what the binary wheel uses, so that’s good:

So I needed to do three things:

Change my Pipfile to conditionally install the binary package only on x86_64:
```
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }
```

Tweak the sqlite import, as described in the original answer:

try:
    import pysqlite3 as sqlite3
except ModuleNotFoundError:
    import sqlite3  # for local testing because pysqlite3-binary couldn't be installed on macos

print(f"{sqlite3.sqlite_version=}")

Set my Docker container to launch in x86 emulation mode locally.

$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build -t my-image .
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker run -ti my-image

Et, voilà!

sqlite3.sqlite_version='3.39.2'

Advertisement

Answer