heroku-python-obscure-libs.md

Using "obscure" libraries with the Heroku Python buildpack

For a project I required dm.xmlsec.binding on Heroku.

To build this package on Ubuntu, libxmlsec1-dev needs to be installed.

Okay, let's add heroku-buildpack-apt and the following Aptfile:

libxmlsec1-dev

That should work, right?

Unfortunately, it doesn't:

remote:            In file included from src/_xmlsec.c:567:0:
remote:            src/cxmlsec.h:1:10: fatal error: xmlsec/crypto.h: No such file or directory
remote:             #include <xmlsec/crypto.h>
remote:                      ^~~~~~~~~~~~~~~~~

What went wrong here?

Searching in the Heroku documentation, I ended up at this article stating Python packages requiring "obscure" C libraries are not supported by the heroku-python buildpack, and recommending to use Docker for deployment instead.

As this would be quite a drastic change to the current deployment process, I was somewhat hesitant and decided to do some more research into why the build is failing, and if we cannot make it work:tm: some other way.

Looking at the error, it seems the header cannot be found.

Doing some debugging with find, I realized the apt buildpack doesn't install the packages to / but to $BUILD_DIR/.apt instead.

The xmlsec1-config tool, that is used by dm.xmlsec.binding to look up the search paths for compiling the C extension, doesn't reflect that though:

$ xmlsec1-config --cflags
... -I/usr/include/xmlsec1 -I/usr/include/libxml2 -DXMLSEC_CRYPTO_OPENSSL=1

The necessary file is not in /usr/include/xmlsec1 but in $BUILD_DIR/.apt/user/include/xmlsec1.

To fix the build, I decided to create a wrapper for the config tool that ensures the correct paths are output:

#!/usr/bin/env bash

set -eo pipefail
xmlsec1-config.orig "$@" | sed -e 's@/usr@'"$BUILD_DIR"'/.apt/usr@g'

To make sure this wrapper is used instead of the original binary, I saved it as bin/xmlsec1-config.wrapper and created a bin/pre_compile script that installs the wrapper before the Python packages are installed:

#!/usr/bin/env bash
#
# This script is executed by the Heroku buildpack for Python at the beginning of the compile step.
set -eo pipefail
echo "-----> Updating xmlsec1-config"
XMLSEC1_CONFIG="$(which xmlsec1-config)"
mv -v "$XMLSEC1_CONFIG" "${XMLSEC1_CONFIG}.orig"
cp -v "bin/xmlsec1-config.wrapper" "$XMLSEC1_CONFIG"
echo "-----> Done"

With this in place, the build succeeds:

remote:        Building wheels for collected packages: dm.xmlsec.binding
remote:          Building wheel for dm.xmlsec.binding (setup.py): started
remote:          Building wheel for dm.xmlsec.binding (setup.py): finished with status 'done'
remote:          Created wheel for dm.xmlsec.binding: filename=dm.xmlsec.binding-1.3.7-cp27-cp27mu-linux_x86_64.whl size=292043 sha256=273e229799431fac69bf8ab93673ed22639cc4f22c9dbd8b3647176fda83834e
remote:          Stored in directory: /tmp/pip-ephem-wheel-cache-47ES_g/wheels/04/8d/fb/457fb8e8c1025aff43cede7f4312189612bd1e3f81354b8f8c
remote:        Successfully built dm.xmlsec.binding

And to my surprise, the produced wheel does work at runtime and I did not run into any more weird dynamic linking issues that I was expecting. 🎉