Skip to content

Instantly share code, notes, and snippets.

@smoser smoser/README.md
Last active Feb 26, 2016

Embed
What would you like to do?
py2or3: attempt at an executable allowing programs to use python2 or python3 if they'd work in both.

The Problem

During transition to python3, much code is now written that works on python2 or python3. Python 2 is becoming less common, and has even been removed from default installs of Ubuntu Server in 15.10 and going forward.

A summary of Ubuntu releases and python inclusion in the default install of Ubuntu Server:

  • 12.04 precise: python 2 only (2.7.3)
  • 14.04 trusty: python 2 only (2.7.5) and python3 (3.4.0)
  • 14.10 utopic: python 2 (2.7.8) and python3 (3.4.2)
  • 15.04 vivid: python 2 (2.7.9) and python3 (3.4.3)
  • 15.10 wily: python 2 (2.7.9) and python3 (3.4.3)
  • 16.04 xenial: python 3 only (3.5.1)
  • going forward: python 3 only

Note also:

  • both python 2 and python 3 are installable via 'apt-get install' on any of the above.
  • the modules other than standard library modules installed by default in each of the above releases differs greatly.

As a result, it is less than trivial to simply run python code that you've taken care to port to both python 2 and python 3. This is because you have to know what platform you're expecting to run it on to provide the proper '#!'.

This is even more complicated if you have some modules that you need.

The solution: py2or3

The solution provided here is py2or3. This posix shell script is intended to live in /usr/bin/py2or3. And can be utilized from the '#!' of your python executables.

It can:

  • find a python with suitable dependencies
  • install those dependencies if requested and the dependencies are not available.

For example:

  • #!/usr/bin/py2or3 -m yaml

    Will use python2 or python3 that is able to 'import yaml'

  • #!/usr/bin/py2or3 -m novaclient -m glanceclient -2 -I

    need python-novaclient and python-glanceclient. this will attempt import novaclient and import glanceclient. -2 indicates that python2 should be tried first.

    If neither python 2 python 3 is available then install the standard named packages 'python-novaclient' and 'python-glanceclient' (or python3- versions).

TODO

A few things that could be improved on.

  • The selection of the default python especially for install. It would make sense for 14.04 to default to installing python2 dependencies and other releases to install python 3 dependencies. It might make sense to consult a /etc/py2or3.conf file to determine which.
  • -I is less than ideal currently for many reasons.
    • attempted installation as non-root is surprising.
    • stderr and stdout (and stdin) are unexpectedly affected by apt
    • currently do not run 'apt-get update', which is actually a requirement, or otherwise must assert up to date package database for this to work.
#!/bin/sh
#
# https://gist.github.com/smoser/8904199bb8f00a90dd04
#
_me=$_
Usage() {
cat <<EOF
Usage:
[Import Options]
-m2 IMPORT[,pkg] python2 needs python IMPORT
-m3 IMPORT[,pkg] python3 has a dependency on IMPORT
-m IMPORT[,pkg] both python2 and python3 have dependency on IMPORT
IMPORT is available in the debian package
python-IMPORT or python3-IMPORT
The string 'IMPORT' can be a simple top level python packagename
or it can be a sub module. Examples:
yaml attempt 'import yaml'
apt.package attempt from apt import package
the optional pkg name declares how to install the package.
If using '-m2' or '-m3' then the complete package name must be
provided. If using '-m', then the prefix 'python-' or 'python3-'
will be added.
-m2 yaml,python-yaml add package depends on python-yaml in py2
-m yaml add package depends on python{,3}-yaml
-m3 apt.package python3 package depend on python3-apt
-m3 apt.package,mypkg python3 package depepnd on 'mypkg'
-m mysrc, import mysrc, no package install can help
[Package Options]
-P2 PKG debian package PKG is needed for running in python2
-P3 PKG debian package PKG is needed for running in python3
-P PKG both python2 and python3 need package PKG
The string PKG is a debian package to be installed. Examples:
-P2 util-linux python2 needs util-linux
-P util-linux both python2 and python3 need util-linux
[Install Flags]
-I2 if no suitable python is found, install python2 deps
-I3 if no suitable python is found, install python3 deps
-I pick a python if no suitable python is found.
[Preference]
-2 attempt python 2 check first
-3 attempt python 3 check first
Examples:
* use python with yaml installed, install if necessary
#!/usr/bin/py2or3 -m yaml -I
* need python-novaclient and python-glanceclient -I
#!/usr/bin/py2or3 -m novaclient -m glanceclient -2
this will attempt 'import novaclient' and 'import glanceclient'
try python2 first.
If neither python2 or python3 is available then install
the standard named packages 'python-novaclient' and 'python-glanceclient'
(or python3-<library> versions).
* need python-apt if not available, install python-apt
and python-software-properties
#!
EOF
}
fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
error() { echo "$@" 1>&2; }
debug() {
[ "$1" -le "${_PY2OR3_DEBUG}" ] || return 0
shift
error "$@"
}
pycheck() {
local python="$1"
command -v "$python" >/dev/null 2>&1 || return 1
shift
[ $# -eq 0 ] && return 0
debug 2 "checking $python for $*"
$python -c '
import sys, importlib
failed=0
for i in sys.argv[1:]:
(name, dot, pkg) = i.rpartition(".")
try:
package = pkg
if pkg and not name:
package = None
importlib.import_module(name=name or pkg, package=package)
except Exception as e:
failed += 1
sys.exit(failed)
' "$@"
}
install() {
local sudo=""
[ "$(id -u)" = "0" ] || sudo="sudo"
error "sudo apt-get -qy install" "$@"
$sudo apt-get -qy install "$@"
}
tok_import() {
local d="," im="" pk=""
im=${1%$d*}
pk=${1#*$d}
if [ "$im" = "$1" ]; then
# no comma, package from import
import="$1"
pkg=${import%%.*}
elif [ -z "$pk" ]; then
# <import>, (no package)
import="$im"
pkg="$pk"
else
import=$im
pkg=$pk
fi
return
}
handleargs() {
# _n stores the number of arguments consumed
# so that the caller can shift off of their "$@"
_n=0
local c="" m2="" m3="" p2="" p3="" pref="" ipref="" avail="" pyver=""
local install=false checkonly=false flag=false myargs=""
local import pkg
[ "$1" = "--check-only" ] && checkonly=true && shift
[ "$1" = "-h" -o "$1" = "--help" ] && { Usage; exit 0; }
while [ $# -gt 0 ]; do
flag=false
case "$1" in
-[23]|-I[23]|-I) flag=true;;
esac
case "$1" in
-m2) tok_import "$2"; m2="$m2 $import"; p2="$p2 $pkg";;
-m2) tok_import "$2"; m3="$m3 $import"; p3="$p3 $pkg";;
-m) tok_import "$2"
m2="$m2 $import"; m3="$m3 $import"
p2="$p2 python-$pkg"; p3="$p3 python3-$pkg";;
-P2) p2="$p2 $2";;
-P3) p3="$p3 $2";;
-P) p2="$p2 $2"; p3="$p3 $2";;
-2|-3) pref="$pref ${1#-}";;
-I2|-I3) ipref="$ipref ${1#-I}"; install=true;;
-I) install=true;;
--) _n=$(($_n+1)); break; shift;;
*) break;;
esac
if $flag; then
myargs="$myargs $1"
_n=$(($_n+1))
shift 1
else
myargs="$myargs $1 $2"
_n=$(($_n+2))
shift 2
fi
done
myargs=${myargs# }
m3=${m3# }
m2=${m2# }
ipref=${ipref# }
pref=${pref# }
p2=${p2# }
p3=${p3# }
for c in 3 2; do
if command -v "python$c" >/dev/null 2>&1; then
[ -z "$_defpy" ] && _defpy="python$c"
avail="$avail $c";
fi
done
avail=${avail# }
[ -z "$pref" ] && pref="$avail"
local imports=""
debug 2 "pref: $pref avail=$avail"
debug 2 "p2=$p2 , p3=$p3 , m2=$m2 , m3=$m3"
for pyver in $pref; do
case "$pyver" in
2) imports="$m2";;
3) imports="$m3";;
esac
[ "${avail#*$pyver}" = "${avail}" ] && continue
pycheck "python$pyver" $imports && _python="python$pyver" && return 0
done
$checkonly && return 1
if $install; then
if [ -z "$ipref" ]; then
if [ -n "$pref" ]; then
ipref=${pref%% *}
elif [ -n "$avail" ]; then
ipref=${avail%%* }
else
ipref=3
fi
fi
local pkgs=""
pyver=${ipref# }
pyver=${pyver%% *}
case "$pyver" in
2) pkgs="python $p2";;
3) pkgs="python3 $p3";;
esac
install $pkgs ||
{ error "WARN: install of $pkgs failed."; return 1; }
handleargs --check-only $myargs && return
error "WARN: installation of $pkgs did not fix deps"
return 3
fi
error "no suitable python found."
return 1
}
# These are the only non-local variables used. setting a shell variable
# that was present in the environment overrides that variable even
# if it were not exported here. So _ prefix to avoid pollution.
_PY2OR3_DEBUG=${PY2OR3_DEBUG:-0}
_defpy=""
_python=""
if [ "${_me}" = "$0" ] ||
[ "${_me##*/}" = "sh" -o "${_me##*/}" = "bash" ]; then
# invoked as py2or3 or as 'sh ./py2or3'
# _me : path to py2or3 or /bin/sh or /bin/bash
# $0 : same as _me
# $1..$N : options maybe for us, end at first unknown arg or --
# 0=./py2or3 me=./py2or3 args=./my.py --help
debug 2 "py2or3 executed me 0=$0 me=$_me 2=$2 args=$*"
handleargs "$@" || exit
debug 2 "shifting $_n from $*"
shift $_n
elif [ "$2" = "$_me" ]; then
# invoked via shebang.
# _me : the python file that had the shebang
# $0 : path to py2or3
# $1 : all options on the shebang line as one argument
# $2 : same as _me
debug 2 "shebang executed me 0=$0 me=$_me 2=$2 args=$1"
handleargs $1 || exit
[ $# -eq 0 ] || shift
else
fail "do not know how i was invoked: 0=$0 me=$_me 2=$2 \$\#=$# args=$*"
fi
if [ -z "$_python" ]; then
if [ -n "$_defpy" ]; then
_python="$_defpy"
else
fail "no available python"
fi
fi
debug 1 "$_python $*"
exec "$_python" "$@"
#!/usr/bin/python3
import sys
import os
import argparse
import importlib
def tokenize_args(args):
# return a list of args to py2or3, and args to its program
if len(args) == 1:
return [], []
elif len(args) > 1:
return args[1].split(), args[2:]
def which(program):
# Return path of program for execution if found in path
def is_exe(fpath):
return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
_fpath, _ = os.path.split(program)
if _fpath:
if is_exe(program):
return program
else:
for path in os.environ.get("PATH", "").split(os.pathsep):
path = path.strip('"')
exe_file = os.path.join(path, program)
if is_exe(exe_file):
return exe_file
return None
def find_available():
ret = {}
ret[sys.version_info.major] = sys.executable
for version, name in ((2, 'python'), (3, 'python3')):
if version in ret:
continue
exe = which(name)
if exe:
ret[version] = exe
return ret
class ImportRef(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
cur = getattr(namespace, self.dest, None)
if cur is None:
cur = {2: [], 3: []}
if option_string == "-m2":
cur[2].append(values)
elif option_string == "-m3":
cur[3].append(values)
else:
cur[2].append(values)
cur[3].append(values)
setattr(namespace, self.dest, cur)
def check_imports(imports):
missing = []
for i in imports:
try:
importlib.import_module(i)
except Exception:
missing.append(i)
return missing
def main():
py_exe = sys.executable
py_major = sys.version_info.major
py_minor = sys.version_info.minor
myargs, args = tokenize_args(sys.argv)
mypath = __file__
parser = argparse.ArgumentParser()
parser.add_argument('-2', action='store_const', dest='pref', const=2,
default=None, help="prefer python2")
parser.add_argument('-3', action='store_const', dest='pref', const=3,
default=None, help="prefer python3")
parser.add_argument('-t', action='append', dest='tried', type=int,
default=[])
parser.add_argument('-m2', action=ImportRef, dest='imports',
help='python2 needs specified import',
metavar='IMPORT[,pkg]')
parser.add_argument('-m3', action=ImportRef, dest='imports', default=[],
help='python3 needs specified import',
metavar='IMPORT[,pkg]')
parser.add_argument('-m', action=ImportRef, dest='imports', default=[],
help='python2 and python3 need specified import',
metavar='IMPORT[,pkg]')
print("py_major: %s" % py_major)
print("myargs: %s" % myargs)
print("args: %s" % args)
parsed = parser.parse_args(myargs)
print(parsed)
available = find_available()
if (parsed.pref != py_major and parsed.pref in available and
parsed.pref not in parsed.tried):
os.execv(available[parsed.pref],
[available[parsed.pref], __file__] + [' '.join(myargs)] + args)
if parsed.imports is None:
parsed.imports = {'2': [], 3: []}
missing = check_imports(parsed.imports[py_major])
if not missing:
# we could try to avoid exec here and just tell python to execute
os.execv(available[py_major], [available[py_major]] + args)
tried = parsed.tried + [py_major]
remaining = [f for f in available.keys() if f not in tried]
if not remaining:
sys.stderr.write("Sorry, couldn't get imports. missing: %s\n"
% missing)
sys.exit(1)
pyexe = available[remaining[0]]
flatargs = '-t %s %s' % (py_major, ' '.join(myargs))
os.execv(pyexe, [pyexe, __file__, flatargs] + args)
if __name__ == '__main__':
main()
# vi: ts=4 expandtab syntax=python
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.