py2or3: attempt at an executable allowing programs to use python2 or python3 if they'd work in both.

README.md

The Problem

During transition to python3, much code is now written that works on python2 or python3. Python 2 is becoming less common, and has even been removed from default installs of Ubuntu Server in 15.10 and going forward.

A summary of Ubuntu releases and python inclusion in the default install of Ubuntu Server:

• 12.04 precise: python 2 only (2.7.3)
• 14.04 trusty: python 2 only (2.7.5) and python3 (3.4.0)
• 14.10 utopic: python 2 (2.7.8) and python3 (3.4.2)
• 15.04 vivid: python 2 (2.7.9) and python3 (3.4.3)
• 15.10 wily: python 2 (2.7.9) and python3 (3.4.3)
• 16.04 xenial: python 3 only (3.5.1)
• going forward: python 3 only

Note also:

• both python 2 and python 3 are installable via 'apt-get install' on any of the above.
• the modules other than standard library modules installed by default in each of the above releases differs greatly.

As a result, it is less than trivial to simply run python code that you've taken care to port to both python 2 and python 3. This is because you have to know what platform you're expecting to run it on to provide the proper '#!'.

This is even more complicated if you have some modules that you need.

The solution: py2or3

The solution provided here is py2or3. This posix shell script is intended to live in /usr/bin/py2or3. And can be utilized from the '#!' of your python executables.

It can:

• find a python with suitable dependencies
• install those dependencies if requested and the dependencies are not available.

For example:

• #!/usr/bin/py2or3 -m yaml

  Will use python2 or python3 that is able to 'import yaml'

• #!/usr/bin/py2or3 -m novaclient -m glanceclient -2 -I

  need python-novaclient and python-glanceclient. this will attempt import novaclient and import glanceclient. -2 indicates that python2 should be tried first.

  If neither python 2 python 3 is available then install the standard named packages 'python-novaclient' and 'python-glanceclient' (or python3- versions).

TODO

A few things that could be improved on.

• The selection of the default python especially for install. It would make sense for 14.04 to default to installing python2 dependencies and other releases to install python 3 dependencies. It might make sense to consult a /etc/py2or3.conf file to determine which.
• -I is less than ideal currently for many reasons.
  • attempted installation as non-root is surprising.
  • stderr and stdout (and stdin) are unexpectedly affected by apt
  • currently do not run 'apt-get update', which is actually a requirement, or otherwise must assert up to date package database for this to work.

py2or3

#!/bin/sh
#
# https://gist.github.com/smoser/8904199bb8f00a90dd04
#
_me=$_
Usage() {
   cat <<EOF
Usage: 

  [Import Options]
  -m2 IMPORT[,pkg]   python2 needs python IMPORT
  -m3 IMPORT[,pkg]   python3 has a dependency on IMPORT
  -m  IMPORT[,pkg]   both python2 and python3 have dependency on IMPORT
                     IMPORT is available in the debian package
                     python-IMPORT or python3-IMPORT

     The string 'IMPORT' can be a simple top level python packagename
     or it can be a sub module.  Examples:
         yaml               attempt 'import yaml'
         apt.package        attempt from apt import package

     the optional pkg name declares how to install the package.
     If using '-m2' or '-m3' then the complete package name must be
     provided.  If using '-m', then the prefix 'python-' or 'python3-'
     will be added.

         -m2 yaml,python-yaml   add package depends on python-yaml in py2
         -m yaml                add package depends on python{,3}-yaml
         -m3 apt.package        python3 package depend on python3-apt
         -m3 apt.package,mypkg  python3 package depepnd on 'mypkg'
         -m  mysrc,             import mysrc, no package install can help


  [Package Options]
  -P2 PKG            debian package PKG is needed for running in python2
  -P3 PKG            debian package PKG is needed for running in python3
  -P  PKG            both python2 and python3 need package PKG

      The string PKG is a debian package to be installed.  Examples:
         -P2 util-linux    python2 needs util-linux
         -P  util-linux    both python2 and python3 need util-linux

  [Install Flags]
  -I2                if no suitable python is found, install python2 deps
  -I3                if no suitable python is found, install python3 deps
  -I                 pick a python if no suitable python is found.

  [Preference]
  -2                 attempt python 2 check first
  -3                 attempt python 3 check first

Examples:
  * use python with yaml installed, install if necessary
      #!/usr/bin/py2or3 -m yaml -I

  * need python-novaclient and python-glanceclient -I
      #!/usr/bin/py2or3 -m novaclient -m glanceclient -2

    this will attempt  'import novaclient' and 'import glanceclient'
    try python2 first.
    If neither python2 or python3 is available then install
    the standard named packages 'python-novaclient' and 'python-glanceclient'
    (or python3-<library> versions).

  * need python-apt if not available, install python-apt
    and python-software-properties

    #!

EOF
}

fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
error() { echo "$@" 1>&2; }
debug() {
   [ "$1" -le "${_PY2OR3_DEBUG}" ] || return 0
   shift
   error "$@"
}

pycheck() {
   local python="$1"
   command -v "$python" >/dev/null 2>&1 || return 1
   shift
   [ $# -eq 0 ] && return 0
   debug 2 "checking $python for $*"
   $python -c '
import sys, importlib
failed=0
for i in sys.argv[1:]:
    (name, dot, pkg) = i.rpartition(".")
    try:
        package = pkg
        if pkg and not name:
            package = None
        importlib.import_module(name=name or pkg, package=package)
    except Exception as e:
        failed += 1
sys.exit(failed)
' "$@"
}

install() {
   local sudo=""
   [ "$(id -u)" = "0" ] || sudo="sudo"
   error "sudo apt-get -qy install" "$@"
   $sudo apt-get -qy install "$@"
}

tok_import() {
   local d="," im="" pk=""
   im=${1%$d*}
   pk=${1#*$d}
   if [ "$im" = "$1" ]; then
      # no comma, package from import
      import="$1"
      pkg=${import%%.*}
   elif [ -z "$pk" ]; then
      # <import>, (no package)
      import="$im"
      pkg="$pk"
   else
      import=$im
      pkg=$pk
   fi
   return
}

handleargs() {
   # _n stores the number of arguments consumed
   # so that the caller can shift off of their "$@"
   _n=0
   local c="" m2="" m3="" p2="" p3="" pref="" ipref="" avail="" pyver=""
   local install=false checkonly=false flag=false myargs=""
   local import pkg
   [ "$1" = "--check-only" ] && checkonly=true && shift
   [ "$1" = "-h" -o "$1" = "--help" ] && { Usage; exit 0; }
   while [ $# -gt 0 ]; do
      flag=false
      case "$1" in
         -[23]|-I[23]|-I) flag=true;;
      esac
      case "$1" in
         -m2) tok_import "$2"; m2="$m2 $import"; p2="$p2 $pkg";;
         -m2) tok_import "$2"; m3="$m3 $import"; p3="$p3 $pkg";;
         -m)  tok_import "$2"
              m2="$m2 $import"; m3="$m3 $import"
              p2="$p2 python-$pkg"; p3="$p3 python3-$pkg";;
         -P2) p2="$p2 $2";;
         -P3) p3="$p3 $2";;
         -P)  p2="$p2 $2"; p3="$p3 $2";;
         -2|-3) pref="$pref ${1#-}";;
         -I2|-I3) ipref="$ipref ${1#-I}"; install=true;;
         -I) install=true;;
         --) _n=$(($_n+1)); break; shift;;
         *) break;;
      esac
      if $flag; then
         myargs="$myargs $1"
         _n=$(($_n+1))
         shift 1
      else
         myargs="$myargs $1 $2"
         _n=$(($_n+2))
         shift 2
      fi
   done

   myargs=${myargs# }
   m3=${m3# }
   m2=${m2# }
   ipref=${ipref# }
   pref=${pref# }
   p2=${p2# }
   p3=${p3# }

   for c in 3 2; do
      if command -v "python$c" >/dev/null 2>&1; then
         [ -z "$_defpy" ] && _defpy="python$c"
         avail="$avail $c";
      fi
   done
   avail=${avail# }
   [ -z "$pref" ] && pref="$avail"

   local imports=""
   debug 2 "pref: $pref avail=$avail"
   debug 2 "p2=$p2 , p3=$p3 , m2=$m2 , m3=$m3"
   for pyver in $pref; do
      case "$pyver" in
         2) imports="$m2";;
         3) imports="$m3";;
      esac
      [ "${avail#*$pyver}" = "${avail}" ] && continue
      pycheck "python$pyver" $imports && _python="python$pyver" && return 0
   done

   $checkonly && return 1

   if $install; then
      if [ -z "$ipref" ]; then
         if [ -n "$pref" ]; then
            ipref=${pref%% *}
         elif [ -n "$avail" ]; then
            ipref=${avail%%* }
         else
            ipref=3
         fi
      fi
      local pkgs=""
      pyver=${ipref# }
      pyver=${pyver%% *}
      case "$pyver" in
         2) pkgs="python $p2";;
         3) pkgs="python3 $p3";;
      esac
      install $pkgs ||
         { error "WARN: install of $pkgs failed."; return 1; }
      handleargs --check-only $myargs && return
      error "WARN: installation of $pkgs did not fix deps"
      return 3
   fi

   error "no suitable python found."
   return 1
}

# These are the only non-local variables used.  setting a shell variable
# that was present in the environment overrides that variable even
# if it were not exported here. So _ prefix to avoid pollution.
_PY2OR3_DEBUG=${PY2OR3_DEBUG:-0}
_defpy=""
_python=""

if [ "${_me}" = "$0" ] ||
   [ "${_me##*/}" = "sh" -o "${_me##*/}" = "bash" ]; then
   # invoked as py2or3 or as 'sh ./py2or3'
   #   _me : path to py2or3 or /bin/sh or /bin/bash
   #   $0  : same as _me
   #   $1..$N : options maybe for us, end at first unknown arg or --
   # 0=./py2or3 me=./py2or3 args=./my.py --help
   debug 2 "py2or3 executed me 0=$0 me=$_me 2=$2 args=$*"
   handleargs "$@" || exit
   debug 2 "shifting $_n from $*"
   shift $_n
elif [ "$2" = "$_me" ]; then
   # invoked via shebang.
   #   _me : the python file that had the shebang
   #   $0  : path to py2or3
   #   $1  : all options on the shebang line as one argument
   #   $2  : same as _me
   debug 2 "shebang executed me 0=$0 me=$_me 2=$2 args=$1"
   handleargs $1 || exit
   [ $# -eq 0 ] || shift
else
   fail "do not know how i was invoked: 0=$0 me=$_me 2=$2 \$\#=$# args=$*"
fi
if [ -z "$_python" ]; then
   if [ -n "$_defpy" ]; then
      _python="$_defpy"
   else
      fail "no available python"
   fi
fi

debug 1 "$_python $*"
exec "$_python" "$@"

py2or3.py

#!/usr/bin/python3
import sys
import os
import argparse
import importlib


def tokenize_args(args):
    # return a list of args to py2or3, and args to its program
    if len(args) == 1:
        return [], []
    elif len(args) > 1:
        return args[1].split(), args[2:]

def which(program):
    # Return path of program for execution if found in path
    def is_exe(fpath):
        return os.path.isfile(fpath) and os.access(fpath, os.X_OK)

    _fpath, _ = os.path.split(program)
    if _fpath:
        if is_exe(program):
            return program
    else:
        for path in os.environ.get("PATH", "").split(os.pathsep):
            path = path.strip('"')
            exe_file = os.path.join(path, program)
            if is_exe(exe_file):
                return exe_file

    return None

def find_available():
    ret = {}
    ret[sys.version_info.major] = sys.executable
    for version, name in ((2, 'python'), (3, 'python3')):
        if version in ret:
            continue
        exe = which(name)
        if exe:
            ret[version] = exe

    return ret


class ImportRef(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        cur = getattr(namespace, self.dest, None)
        if cur is None:
            cur = {2: [], 3: []}

        if option_string == "-m2":
            cur[2].append(values)
        elif option_string == "-m3":
            cur[3].append(values)
        else:
            cur[2].append(values)
            cur[3].append(values)

        setattr(namespace, self.dest, cur)


def check_imports(imports):
    missing = []
    for i in imports:
        try:
            importlib.import_module(i)
        except Exception:
            missing.append(i)
    return missing


def main():
    py_exe = sys.executable
    py_major = sys.version_info.major
    py_minor = sys.version_info.minor
    myargs, args = tokenize_args(sys.argv)
    mypath = __file__

    parser = argparse.ArgumentParser()
    parser.add_argument('-2', action='store_const', dest='pref', const=2,
                        default=None, help="prefer python2")
    parser.add_argument('-3', action='store_const', dest='pref', const=3,
                        default=None, help="prefer python3")
    parser.add_argument('-t', action='append', dest='tried', type=int,
                        default=[])
    parser.add_argument('-m2', action=ImportRef, dest='imports',
                        help='python2 needs specified import',
                        metavar='IMPORT[,pkg]')
    parser.add_argument('-m3', action=ImportRef, dest='imports', default=[],
                        help='python3 needs specified import',
                        metavar='IMPORT[,pkg]')
    parser.add_argument('-m', action=ImportRef, dest='imports', default=[],
                        help='python2 and python3 need specified import',
                        metavar='IMPORT[,pkg]')

    print("py_major: %s" % py_major)
    print("myargs: %s" % myargs)
    print("args: %s" % args)
    parsed = parser.parse_args(myargs)
    print(parsed)
    available = find_available()

    if (parsed.pref != py_major and parsed.pref in available and
            parsed.pref not in parsed.tried):
        os.execv(available[parsed.pref],
                 [available[parsed.pref], __file__] + [' '.join(myargs)] + args)

    if parsed.imports is None:
        parsed.imports = {'2': [], 3: []}

    missing = check_imports(parsed.imports[py_major])
    if not missing:
        # we could try to avoid exec here and just tell python to execute
        os.execv(available[py_major], [available[py_major]] + args)

    tried = parsed.tried + [py_major]
    remaining = [f for f in available.keys() if f not in tried]

    if not remaining:
        sys.stderr.write("Sorry, couldn't get imports. missing: %s\n"
                         % missing)
        sys.exit(1)

    pyexe = available[remaining[0]]
    flatargs = '-t %s %s' % (py_major, ' '.join(myargs))
    os.execv(pyexe, [pyexe, __file__, flatargs] + args)


if __name__ == '__main__':
    main()

# vi: ts=4 expandtab syntax=python