Problems installing lxml on M1 mac

So, I’m having the classic trouble install lxml.

Initially I was just pip installing, but when I tried to free up memory using Element.clear() I was getting the following error:

Python(58695,0x1001b4580) malloc: *** error for object 0x600000bc3f60: pointer being freed was not allocated

I thought this must be because lxml is using the system’s libxml2 which is probably out of date.

So I used homebrew to install libxml2 and libxlt, and I force linked them both.

I then tried to install using the following command:

❯ STATIC_DEPS=true pip install lxml --no-cache-dir                                                13:01:46
Collecting lxml
  Downloading lxml-4.8.0.tar.gz (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 5.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: lxml
  Building wheel for lxml (setup.py) ... done
  Created wheel for lxml: filename=lxml-4.8.0-cp310-cp310-macosx_12_0_arm64.whl size=1683935 sha256=47912c1ba66d274c3ad7b2a2db00243f96d334a3fd5e439725f5005a7a72a602
  Stored in directory: /private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-ephem-wheel-cache-4_v4ov7s/wheels/e4/52/34/64064e2e2f1ce84d212a6dde6676f3227846210a7996fc2530
Successfully built lxml
Installing collected packages: lxml
Successfully installed lxml-4.8.0

..but then when I tried to import etree I would get this error:

Traceback (most recent call last):
  File "/Users/human/Code/ia_book_images/viewer/book_image_downloader.py", line 4, in <module>
    from lxml import etree as ET
ImportError: dlopen(/Users/human/.virtualenvs/ia_book_images/lib/python3.10/site-packages/lxml/etree.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '___htmlDefaultSAXHandler'

So then I thought let’s make 100% sure that it’s using the right versions of libxml2 using CFLAGS and got the following result:

❯ CFLAGS="-I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include" STATIC_DEPS=true pip install lxml --no-cache-dir
Collecting lxml
  Downloading lxml-4.8.0.tar.gz (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 4.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [199 lines of output]
      Checking for gcc...
      Checking for shared library support...
      Building shared library libz.1.2.12.dylib with gcc.
      Checking for size_t... Yes.
      Checking for off64_t... No.
      Checking for fseeko... Yes.
      Checking for strerror... Yes.
      Checking for unistd.h... Yes.
      Checking for stdarg.h... Yes.
      Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
      Checking for vsnprintf() in stdio.h... Yes.
      Checking for return value of vsnprintf()... Yes.
      Checking for attribute(visibility) support... Yes.
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN -I. -c -o example.o test/example.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o adler32.o adler32.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o crc32.o crc32.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o deflate.o deflate.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o infback.o infback.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o inffast.o inffast.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o inflate.o inflate.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o inftrees.o inftrees.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o trees.o trees.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o zutil.o zutil.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o compress.o compress.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o uncompr.o uncompr.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o gzclose.o gzclose.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o gzlib.o gzlib.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o gzread.o gzread.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN  -c -o gzwrite.o gzwrite.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN -I. -c -o minigzip.o test/minigzip.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/adler32.o adler32.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/crc32.o crc32.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/deflate.o deflate.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/infback.o infback.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/inflate.o inflate.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/inffast.o inffast.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/inftrees.o inftrees.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/trees.o trees.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/zutil.o zutil.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/gzclose.o gzclose.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/uncompr.o uncompr.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/compress.o compress.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/gzlib.o gzlib.c
      libtool -o libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/gzread.o gzread.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN  -DPIC -c -o objs/gzwrite.o gzwrite.c
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN -o example example.o -L. libz.a
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN -o minigzip minigzip.o -L. libz.a
      gcc -dynamiclib -install_name /private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/build/tmp/libxml2/lib/libz.1.dylib -compatibility_version 1 -current_version 1.2.12 -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -fPIC -DHAVE_HIDDEN -o libz.1.2.12.dylib adler32.lo crc32.lo deflate.lo infback.lo inffast.lo inflate.lo inftrees.lo trees.lo zutil.lo compress.lo uncompr.lo gzclose.lo gzlib.lo gzread.lo gzwrite.lo  -lc -arch x86_64
      ld: warning: ignoring file crc32.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file adler32.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file deflate.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file infback.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file inffast.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file inflate.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file inftrees.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file trees.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file compress.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file zutil.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file uncompr.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file gzread.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file gzlib.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file gzclose.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      ld: warning: ignoring file gzwrite.lo, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
      rm -f libz.dylib libz.1.dylib
      ln -s libz.1.2.12.dylib libz.dylib
      ln -s libz.1.2.12.dylib libz.1.dylib
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN -o examplesh example.o -L. libz.1.2.12.dylib
      gcc -I/opt/homebrew/opt/libxslt/include -I/opt/homebrew/opt/libxml2/include -DHAVE_HIDDEN -o minigzipsh minigzip.o -L. libz.1.2.12.dylib
      ld: warning: ignoring file libz.1.2.12.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
      ld: warning: ignoring file libz.1.2.12.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
      Undefined symbols for architecture arm64:
        "_gzclose", referenced from:
            _gz_compress in minigzip.o
            _gz_uncompress in minigzip.o
        "_gzdopen", referenced from:
            _main in minigzip.o
        "_gzerror", referenced from:
            _gz_compress in minigzip.o
            _gz_uncompress in minigzip.o
        "_gzopen", referenced from:
            _file_compress in minigzip.o
            _file_uncompress in minigzip.o
            _main in minigzip.o
        "_gzread", referenced from:
            _gz_uncompress in minigzip.o
        "_gzwrite", referenced from:
            _gz_compress in minigzip.o
      ld: symbol(s) not found for architecture arm64
      Undefined symbols for architecture arm64:
        "_compress", referenced from:
            _test_compress in example.o
           (maybe you meant: _test_compress)
        "_deflate", referenced from:
            _test_deflate in example.o
            _test_large_deflate in example.o
            _test_flush in example.o
            _test_dict_deflate in example.o
           (maybe you meant: _test_large_deflate, _test_deflate , _test_dict_deflate )
        "_deflateEnd", referenced from:
            _test_deflate in example.o
            _test_large_deflate in example.o
            _test_flush in example.o
            _test_dict_deflate in example.o
        "_deflateInit_", referenced from:
            _test_deflate in example.o
            _test_large_deflate in example.o
            _test_flush in example.o
            _test_dict_deflate in example.o
        "_deflateParams", referenced from:
            _test_large_deflate in example.o
        "_deflateSetDictionary", referenced from:
            _test_dict_deflate in example.o
        "_gzclose", referenced from:
            _test_gzio in example.o
        "_gzerror", referenced from:
            _test_gzio in example.o
        "_gzgetc", referenced from:
            _test_gzio in example.o
        "_gzgets", referenced from:
            _test_gzio in example.o
        "_gzopen", referenced from:
            _test_gzio in example.o
        "_gzprintf", referenced from:
            _test_gzio in example.o
        "_gzputc", referenced from:
            _test_gzio in example.o
        "_gzputs", referenced from:
            _test_gzio in example.o
        "_gzread", referenced from:
            _test_gzio in example.o
        "_gzseek", referenced from:
            _test_gzio in example.o
        "_gztell", referenced from:
            _test_gzio in example.o
        "_gzungetc", referenced from:
            _test_gzio in example.o
        "_inflate", referenced from:
            _test_inflate in example.o
            _test_large_inflate in example.o
            _test_sync in example.o
            _test_dict_inflate in example.o
           (maybe you meant: _test_large_inflate, _test_inflate , _test_dict_inflate )
        "_inflateEnd", referenced from:
            _test_inflate in example.o
            _test_large_inflate in example.o
            _test_sync in example.o
            _test_dict_inflate in example.o
        "_inflateInit_", referenced from:
            _test_inflate in example.o
            _test_large_inflate in example.o
            _test_sync in example.o
            _test_dict_inflate in example.o
        "_inflateSetDictionary", referenced from:
            _test_dict_inflate in example.o
        "_inflateSync", referenced from:
            _test_sync in example.o
        "_uncompress", referenced from:
            _test_compress in example.o
        "_zlibCompileFlags", referenced from:
            _main in example.o
        "_zlibVersion", referenced from:
            _main in example.o
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      ld: symbol(s) not found for architecture arm64
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      make: *** [minigzipsh] Error 1
      make: *** Waiting for unfinished jobs....
      make: *** [examplesh] Error 1
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/setup.py", line 270, in <module>
          **setup_extra_options()
        File "/private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/setup.py", line 162, in setup_extra_options
          ext_modules = setupinfo.ext_modules(
        File "/private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/setupinfo.py", line 74, in ext_modules
          XML2_CONFIG, XSLT_CONFIG = build_libxml2xslt(
        File "/private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/buildlibxml.py", line 428, in build_libxml2xslt
          cmmi(zlib_configure_cmd, zlib_dir, multicore, **call_setup)
        File "/private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/buildlibxml.py", line 352, in cmmi
          call_subprocess(
        File "/private/var/folders/g9/lqph46sj36n9kkvjt1pzdxhm0000gn/T/pip-install-kl4hmrrk/lxml_4ecb3c255ad049e39a89a66ee0a50e76/buildlibxml.py", line 335, in call_subprocess
          raise Exception('Command "%s" returned code %s' % (cmd_desc, returncode))
      Exception: Command "make -j6" returned code 2
      Building lxml version 4.8.0.
      Latest version of zlib is 1.2.12
      Downloading zlib into libs/zlib-1.2.12.tar.gz from https://zlib.net/zlib-1.2.12.tar.gz
      Unpacking zlib-1.2.12.tar.gz into build/tmp
      Latest version of libiconv is 1.16
      Downloading libiconv into libs/libiconv-1.16.tar.gz from https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz
      Unpacking libiconv-1.16.tar.gz into build/tmp
      Latest version of libxml2 is 2.9.12
      Downloading libxml2 into libs/libxml2-2.9.12.tar.gz from http://xmlsoft.org/sources/libxml2-2.9.12.tar.gz
      Unpacking libxml2-2.9.12.tar.gz into build/tmp
      Latest version of libxslt is 1.1.34
      Downloading libxslt into libs/libxslt-1.1.34.tar.gz from http://xmlsoft.org/sources/libxslt-1.1.34.tar.gz
      Unpacking libxslt-1.1.34.tar.gz into build/tmp
      Starting build in build/tmp/zlib-1.2.12
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Do I need to do something special to build lxml on an m1 mac?

Answer

It turned out that installing lxml with a simple pip install was working fine.

The reason for my malloc error was the fact that I was trying to clear the element before the end tag had been seen. Turns out this isn’t possible and you need to wait for the end tag even if you already know you aren’t interested in the element.