Skip to content

Instantly share code, notes, and snippets.

@the-moog
Last active March 19, 2023 00:15
Show Gist options
  • Save the-moog/f2aa5d6ea5f77c1e07d67bd7b7f7f8e2 to your computer and use it in GitHub Desktop.
Save the-moog/f2aa5d6ea5f77c1e07d67bd7b7f7f8e2 to your computer and use it in GitHub Desktop.
Improved fix for dynamic build and load of shared library support when using pyenv virtualenv (Python)
### pyenv hook script to prepare a build env
#
# 0: Intro
# $MYLOC points to your local sideload root (usually ~/.local)
MYLOC=$HOME/.local
#
export PYTHON_CONFIGURE_OPTS="--with-openssl=$MYLOC --enable-shared --enable-optimizations --with-lto"
export PYTHON_CFLAGS="-march=native -mtune=native"
echo ""
echo "This is $(basename $0) hook 'build.bash':"
echo " Location: $(pyenv root)/pyenv.d/install/build.bash"
echo ""
echo "This add-on performs additional tasks and creates additional"
echo "environment variables to pass to python build process"
echo "when called by the pyenv-build plugin:"
echo ""
echo " PYTHON_CONFIGURE_OPTS='${PYTHON_CONFIGURE_OPTS}'"
echo " CONFIGURE_OPTS='${CONFIGURE_OPTS}'"
echo " PYTHON_CFLAGS='${PYTHON_CFLAGS}'"
echo " CFLAGS='${CFLAGS}'"
echo ""
#
# Use: with_log "string" cmd...
function with_log () {
local TTTT=$(mktemp -d)
local MSG=$1; shift
"$@" >$TTTT/log 2>$TTTT/log
echo "$MSG results in $TTTT"
test -f $TTTT/err && \
echo -e "ERRORS\nError log in $TTTT/err\n$(tail $TTTT/err)\nDONE: There were ERRORS" || \
echo "$MSG DONE: No errors, results in $TTTT/log"
echo ""
return $?
}
# Use: update_pyenv
function update_pyenv () {
PYENV=$(pyenv root)
if [ -d $PYENV/.git ]; then
echo "Updating pyenv"
pushd $PYENV
git show-ref -q --heads mine || git branch mine
git branch -q --set-upstream-to=origin/master master
git checkout -q mine
git merge -q -m "auto merge" origin/master
fi
echo ""
}
#
# 1: Make sure pyenv is up to date
with_log "Update pyenv" update_pyenv
#
# 2: Build openssl and side-load it
OPENSSLVER=1.1.1t
echo "Side loading openssl $OPENSSLVER into $MYLOC"
if [ ! -d $MYLOC/bin/openssl ] || ! ( $MYLOC/bin/openssl version | grep $OPENSSLVER ); then
BUILDDIR=$(mktemp -d)
pushd $BUILDDIR
git clone https://github.com/openssl/openssl
cd openssl
git checkout -b build OpenSSL_$(echo "$OPENSSLVER" | tr '.' '_')
with_log "Config" ./config --prefix=$MYLOC --openssldir=$MYLOC/ssl
echo "Building openssl, please wait...."
with_log "Build" make
echo "Installing openssl, please wait...."
with_log "Install" make install_sw
popd
rm -rf $BUILDDIR
fi
echo ""
# 3: Build python, passing some extra options
echo "Building Python, please wait...."
@the-moog
Copy link
Author

I was thinking of adding detection of an environment variable or config file with the contents like this:
PYENV_BUILD_SHARED_=Y|N
PYENV_BUILD_SHARED_DEFAULT=Y|N
e.g.
.pyenv/pyenv.cfg

# Use regex compatible fragments that express pyenv version or pyenv virtualenv venv string with spaces ' ' replacing dots '\.'
# (That is to save on parsing and make it easier to read without knowing reged too well, as versions never have spaces in anyway)
# The # character and anything after it are ignored until the end of the line
# A special use of # is on the end of a line where #; Any text is printed at the start and end of the process if it is a script or exe
#
# Each line is read in sequence, with the Y or N being the result of the last regex match
# If Y|y|T|t|1 then the --enable-shared would be included
# If N|n|F|f|0 then the --enable-shared would be excluded
# If there is no match then the system default is used
# So if the the last line is .* then that will override the default
#
[--enabled-shared]
.*        = N        #; Shared libraries are not being built
3 [5-7] = Y
3 8 16= N        # No for 3.8.16
3 9     = Y        # Yes for 3.9.*
2        = N       # No for 2.*.*
myvenv = Y
testv.*  = Y
.*          = N

The same mechanism could be extended to pass any arbitraty configuration to a given pyenv-build build or pyenv virtualenv, through the use of a simple set of additional syntax elements in addition to Y/N, E.g.

  • enabling optional modules,
  • building extra tools INCLUDE
  • setting environment variables SET<x=y>
  • run pre or post scripts

e.g.
[--set-env]
3.9.1 = DEBUG=2
3.8 = LD_LIBRARY_PATH=$PYENV/versions/lib/$PY_VERSION

[--enable]
3.8 = zlib

[--sideload]
3.9.5 = git clone https://github.com/mozillazg/pypy #; "Installing PyPy"

[--post-build]
2.7 = scripts/deprecation_warning severity=10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment