This post shows how to cross compile Python (using the CPython implementation) for use on an armv7l chip. This can likely be extrapolated to other chip archetectures, but the proper cross-compilation toolchain would need to be substituded. The Python version utilized here is version 3.7.13. The driving force behind this effort was to get python-can functional on an armv7l platform. This compilation process is done on a Ubuntu 20.04.4 LTS operating system. I have utilized various online references and will do my best to provide citation and links.
In my case the device with an armv7l chip contains a root file system and shared objects that are compiled using a version of buildroot. The buildroot menu configuration allows for the selection of Python 3 for installation, but not all of the basic packages are included. For example, importing the python-can package into an environment using Python 3 installed by buildroot resulted in the following error message:
ModuleNotFoundError: No module named 'zlib'
Full Python installations typically ship with zlib. Therefore, this post will go over how to compile and then link some of the core python modules.
The requirements are a functioning Python installation and the cross-compilation toolchain. For the Python 3.7.13 version, I utilize the Anaconda package manager:
$ cd /tmp/
$ wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh
$ bash Anaconda3-2021.11-Linux-x86_64.sh
In my case the shared libraries on the device are compiled by arm-linux-gnueabihf. It is important that the toolchain utilized for building this Python version is what was used to build the shared objects on the device. Here is how to get the toolchain and then extract it onto your system:
$ cd $HOME/Downloads
$ wget https://releases.linaro.org/components/toolchain/binaries/7.5-2019.12/arm-linux-gnueabihf/gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf.tar.xz
$ sudo mkdir /opt/ && tar -xvf gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf.tar.xz -C /opt/
Create a Python 3.7.13 environment with anaconda:
(base) user@ubuntu:~$ conda create -n ENV python=3.7
(base) user@ubuntu:~$ conda activate ENV
(ENV) user@ubuntu:~$
Add the cross-compilation toolchain to path (source):
$ export PATH=/opt/Embedix/tools/arm-linux/bin:$PATH
$ echo $PATH
/opt/gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf/bin:/home/user/anaconda3/envs/ssb/bin:/home/user/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
In this section I'll download all of the tarballs (.tar.gz) files that are going to be requied in this process. Not every project may need as many of the default modules that I am going to build here, and some may need more. The versions here matter.
$ cd $HOME/Downloads
$ wget https://github.com/python/cpython/archive/refs/tags/v3.7.13.tar.gz # cpython version 3.7.13
$ wget https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz # libffi version 3.3
$ wget https://github.com/openssl/openssl/archive/refs/tags/OpenSSL_1_1_1o.tar.gz # openssl version 1.1.1o
$ wget https://sqlite.org/2022/sqlite-autoconf-3380500.tar.gz # sqlite3 version 3.38.0-500
$ wget https://github.com/madler/zlib/archive/refs/tags/v1.2.11.tar.gz # zlib version 1.2.11
Before getting too deep into the weeds it would be good to verify that the toolchain being utilized is functioning properly (source).
$ tar -xvf v3.7.13.tar.gz && cd cpython-3.7.13 # extract and change into cpython directory
$ CHOST=arm-linux-gnueabihf CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ AR=arm-linux-gnueabihf-ar LD=arm-linux-gnueabihf-ld RANLIB=arm-linux-gnueabihf-ranlib ./configure --prefix=$HOME/pythonArm --host=arm-linux-gnueabihf --target=arm --with-zlib --build=x86_64-linux-gnu --disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no --with-ensurepip=install
$ make -j($nproc) # make with all available processors, or use `make` to use one processors
Among many other things, near the end of the output something like the following should be shown:
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2 _curses _curses_panel
_dbm _gdbm _hashlib
_lzma _sqlite3 _ssl
_tkinter readline zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time
Failed to build these modules:
_ctypes _uuid
Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381
It is possible that your Python needs may be such that you don't require any of the optional modules. To finish up the installation process run the following command:
$ make install
The directory $HOME/pythonArm should now exist. The can be zipped into a tarball and secure shell copied to the remote device.
Helpful resources used in this section:
- https://stackoverflow.com/a/14541123/11637415
- https://stackoverflow.com/a/21398721/11637415
- https://stackoverflow.com/a/59420165/11637415
- https://stackoverflow.com/a/68175370/11637415
- https://www.linuxquestions.org/questions/linux-software-2/how-to-use-multiple-paths-in-configure-env-variables-213384/#post1115253
- https://stackoverflow.com/a/15783521/11637415
Create an environment variable named cross.
$ export cross=arm-linux-gnueabihf
libffi
$ tar -xvf libffi-3.3.tar.gz && cd libffi-3.3/
$ CHOST="${cross}" CC="${cross}-gcc" CXX="${cross}-g++" AR="${cross}-ar" LD="${cross}-ld" RANLIB="${cross}-ranlib" ./configure --prefix=$HOME/libffiArm --host="${cross}"
$ make
$ make install
openssl
$ tar -xvf OpenSSL_1_1_1o.tar.gz && cd openssl-OpenSSL_1_1_1o
$ ./Configure linux-generic32 shared -DL_ENDIAN --prefix=$HOME/opensslArm --openssldir=$HOME/opensslArm
$ make CC="${cross}"-gcc RANLIB="${cross}"-ranlib LD="${cross}"-ld MAKEDEPPROG="${cross}"-gcc PROCESSOR=ARM
$ make install
sqlite3
$ tar -xvf sqlite-autoconf-3380500.tar.gz
$ cd sqlite-autoconf-3380500/
$ CHOST="${cross}" CC="${cross}-gcc" CXX="${cross}-g++" AR="${cross}-ar" LD="${cross}-ld" RANLIB="${cross}-ranlib" ./configure --prefix=$HOME/sqliteArm --host="${cross}"
$ make -j 8
$ make install
zlib
$ tar -xvf v1.2.11.tar.gz && cd cd zlib-1.2.11/
$ CHOST="${cross}" CC="${cross}-gcc" CXX="${cross}-g++" AR="${cross}-ar" LD="${cross}-ld" RANLIB="${cross}-ranlib"
$ ./configure --prefix=$HOME/zlibArm
$ make -j 8
$ make install
$ cd cpython-3.7.13/
$ CHOST="${cross}" CC="${cross}"-gcc CXX="${cross}"-g++ AR="${cross}"-ar LD="${cross}"-ld RANLIB="${cross}"-ranlib CFLAGS="-I$HOME/zlibArm/include -I$HOME/sqliteArm/include -I$HOME/libffiArm/include -I$HOME/opensslArm/include" LDFLAGS="-L$HOME/zlibArm/lib -L$HOME/sqliteArm/lib -L$HOME/libffiArm/lib -L$HOME/opensslArm/lib" CPPFLAGS="-I$HOME/zlibArm/include -I$HOME/sqliteArm/include -I$HOME/libffiArm/include -I$HOME/opensslArm/include" ./configure --prefix=$HOME/pythonArm --host="${cross}" --target=arm --build=x86_64-linux-gnu --disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no --with-ensurepip=install --enable-loadable-sqlite-extensions --enable-optimizations
$ make -j 8
Something similar to the following output should appear near the end. Notice that the number of optional modules not found are reduced from when it was done previously prior to linking the build libraries.
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2 _curses _curses_panel
_dbm _gdbm _lzma
_tkinter readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time
Failed to build these modules:
_uuid
Now copy the $HOME/pythonArm folder to the cross device and run the following command:
path/to/python3 -m ensurepip --default-pip