tox-venv and venv: easily activate tox created environments or create virtualenv envs.

README.md

tox and virtual-env tools

Some tools are provided here for interacting with virtual-env and tox.

venv

venv is probably a recreation of other tools. It overlaps greatly with virtualenvwrapper (workon).

The big difference is mostly a preference. I prefer running subshells to modifying the current shell environment.

Create a new virtual env

$ venv create restview

Pip install something into it

$ venv restview pip install restview

Activate that virtual env

$ venv exec restview
(venv:restview) ~/$

Run a command in it:

$ venv exec restview your-command
or, 'e' is an alias for exec.
$ venv e restview your-command your-arg1 your-arg1

Destroy it:

$ venv delete --force restview

~/bin wrappers

If you use a program from venv and do not like then length of:

venv exec venv-name command

Then you can write a wrapper with:

$ venv wrapper softlayer slcli
#!/bin/sh
exec venv exec softlayer slcli "$@"

That writes the wrapper to stdout by default. Give it -o/--output to write to a file:

$ venv wrapper --output ~/bin/slcli softlayer slcli

Or you could alias that also.

tox-venv

tox-venv: run a command inside a tox environment.

If you use tox, then you likely have .tox/<venv> directories that you can enter with tox command. However, sometimes you want to poke around or run a command inside there without modifying your tox.ini file.

Assuming 'tox.ini' in the local directory, it will determine where your tox environments are correctly.

tox-venv is copied into cloud-init's source tree at tools/tox-venv. The two versions have diverged slightly.

tox-venv could/should probably be done in python possibly importing tox. This version is much faster than python -c "import tox" (or even python -c pass). The faster path is used when providing a command (tox-venv py3 /bin/true).

Example:

Run python in pep8 tox environment. This will use tox to create the enviroment if it does not exist.

$ tox-venv pep8 python

Run pep8 on the file my.py:

$ tox-venv pep8 pep8 my.py

Run the default commands for the py3 environment passing it the provided posargs.

$ time ./tools/tox-venv py3 - tests/unittests/test_log.py

Just enter the virtualenv that provided in your 'doc' tox environment.

$ tox-venv doc /bin/bash
(tox:docs) $

List what tox environments there are. Those with a * are existing environments.

$ tox-venv --list
  py26
  docs*
  py27*

By default, tox-venv will run the 'commands' that it reads for the environment provided.

tox-venv

#!/bin/sh
# https://gist.github.com/smoser/2d4100a6a5d230ca937f

CR='
'
error() { echo "$@" 1>&2; }
fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
get_env_dirs() {
    # read 'tox --showconfig'. return list of
    #  envname:dir
    local key="" equal="" val="" curenv="" out=""
    while read key equal val; do
        case "$key" in
           "[testenv:"*)
                curenv=${key#*:};
                curenv=${curenv%%"]"*};
                continue;;
        esac
        if [ "${key#*=}" != "$key" ]; then
            # older tox shows key=value or key=   value
            # newer tox shows: key =    value
            key=${key%%=*}
            val=${equal}
        fi
        [ "$key" = "envdir" ] || continue
        out="${out:+${out}${CR}}${curenv}:$val"
    done
    echo "$out"
}

load_config() {
    local tox_ini="$1" out="" envs=""
    if [ "$tox_ini" = "${CACHED_ENVS_INI}" ]; then
        _RET="$CACHED_ENVS"
        return
    fi
    out=$(tox -c "$tox_ini" --showconfig) || return 1
    envs=$(echo "$out" | get_env_dirs) || return 1
    CACHED_ENVS="$envs"
    CACHED_ENVS_INI="$tox_ini"
    _RET="$envs"
}

list_environments() {
    local tox_ini="$1" prefix="  " out="" envs="" oifs="$IFS"
    load_config "$tox_ini" || return 1
    envs="${_RET}"
    IFS="$CR"
    for d in ${envs}; do
        env=${d%%:*}
        dir=${d#*:}
        [ -f "$dir/bin/activate" ] && s="*" || s=""
        echo "${prefix}$env$s";
    done
    IFS="$oifs"
}

get_command() {
    local tox_ini="$1" env="$2" out=""
    shift 2
    out=$(
        sed -e ':x; /\\$/ { N; s/\\\n[ ]*//; tx };' "${tox_ini}" |
        awk '
        $1 ~ /^\[testenv.*\]/ {
            name=$1;
            sub("\\[", "", name); sub(".*:", "", name);
            sub("].*", "", name);
            curenv=name; };
        $1 == "basepython" && (name == "testenv" || name == n) { python=$3 }
        $1 == "commands" && (name == "testenv" || name == n) {
            sub("commands = ", ""); cmd = $0; };
        END {
            sub("{envpython}", python, cmd);
            sub("{toxinidir}", toxinidir, cmd);
            if (inargs == "") replacement = "\\1"
            else replacement = inargs
            cmd = gensub(/{posargs:?([^}]*)}/, replacement, "global", cmd)
            print(cmd);
            }' n="$env" toxinidir="$(dirname $tox_ini)" inargs="$*" \
                python=python)
    if [ -z "$out" ]; then
        error "Failed to find command for $env in $tox_ini"
        return 1
    fi
    echo "$out"
}

get_env_dir() {
    local tox_ini="$1" env="$2" oifs="$IFS" t="" d="" envs=""
    if [ "${TOX_VENV_SHORTCUT:-1}" != "0" ]; then
        local stox_d="${tox_ini%/*}/.tox/${env}"
        if [ -e "${stox_d}/bin/activate" ]; then
            _RET="${stox_d}"
            return
        fi
    fi
    load_config "$tox_ini" && envs="$_RET" || return 1
    IFS="$CR"
    for t in $envs; do
        [ "$env" = "${t%%:*}" ] && d="${t#*:}" && break
    done
    IFS=${oifs}
    [ -n "$d" ] || return 1
    _RET="$d"
}

Usage() {
    local tox_ini="$1"
    cat <<EOF
Usage: ${0##*/} [--no-create] tox-environment [command [args]]
   run command with provided arguments in the provided tox environment
   command defaults to 'cmd' (see below).

   run with '--list' to show available environments

   if 'command' above is literal 'cmd' or '-', then the 'command' will
   be read from tox.ini.  This allows you to do:
      tox-venv py27 - tests/some/sub/dir
   and have the 'command' read correctly and have that execute:
      python -m nose tests/some/sub/dir
EOF

    if [ -f "$tox_ini" ]; then
        local oini=${tox_ini}
        [ "${tox_ini}" -ef "$PWD/tox.ini" ] && oini="./tox.ini"
        echo
        echo "environments in $oini"
        list_environments "$tox_ini"
    fi
}

if [ -f tox.ini ]; then
    tox_ini="$PWD/tox.ini"
else
    tox_ini="${0%/*}/../tox.ini"
fi

[ $# -eq 0 ] && { Usage "$tox_ini" 1>&2; exit 1; }
[ "$1" = "-h" -o "$1" = "--help" ] && { Usage "$tox_ini"; exit 0; }

[ -f "$tox_ini" ] || fail "$tox_ini: did not find tox.ini"

if [ "$1" = "-l" -o "$1" = "--list" ]; then
    list_environments "$tox_ini"
    exit
fi

nocreate="false"
if [ "$1" = "--no-create" ]; then
    nocreate="true"
    shift
fi

env="$1"
shift
[ "$1" = "--" ] && shift
get_env_dir "$tox_ini" "$env" && activate="$_RET/bin/activate" || activate=""

if [ -z "$activate" -o ! -f "$activate" ]; then
    if $nocreate; then
        fail "tox env '$env' did not exist, and no-create specified"
    elif [ -n "$activate" ]; then
        error "attempting to create $env:"
        error "    tox -c $tox_ini --recreate --notest -e $env"
        tox -c "$tox_ini" --recreate --notest -e "$env" ||
            fail "failed creation of env $env"
    else
        error "$env: not a valid tox environment?"
        error "found tox_ini=$tox_ini"
        error "try one of:"
        list_environments "$tox_ini" 1>&2
        fail
    fi
fi
. "$activate"

[ $# -eq 0 ] && set -- cmd
if [ "$1" = "cmd" -o "$1" = "-" ]; then
   shift
   out=$(get_command "$tox_ini" "$env" "$@") || exit
   eval set -- "$out"
fi
echo "inside tox:$env running: $*" 1>&2
debian_chroot="tox:$env" exec "$@"

# vi: ts=4 expandtab

venv

#!/bin/sh
# https://gist.github.com/smoser/2d4100a6a5d230ca937f

# this is where your virtual envs live, it is searched if the
# virtual-env argument is not a path
VENV_BASE_D="$HOME/pub/venvs"

list_environments() {
   local venv_base_d="$1" prefix="  "
   echo "existing environments in $venv_base_d"
   ( cd "$venv_base_d" &&
     for d in *; do [ -f "$d/bin/activate" ] && echo "${prefix}$d"; done )
}

error() { echo "$@" 1>&2; }
fail() { [ $# -eq 0 ] || error "$@"; exit 1; }

Usage() {
   local venv_base_d="$1"
   cat <<EOF
Usage: ${0##*/} [mode] your-venv [command [args]]
   run command with provided arguments in 'your-venv'
   command defaults to \${SHELL:-/bin/sh}.

   if the first argument is not a valid mode, then 'activate'
   is assumed.

   modes are:
     activate: enter the named environment and execute a command
         --change-dir    change the working directory to the virtual env dir

     create:   create a new virtual env
         --py2 | --py3   create with python2 or python3
         --activate      enter after creating
                         this is the default if you give additional args

     destroy:  delete the named environment
         --force         do not prompt before deleting

     list:     list enviroments in ${VENV_BASE_D}

     wrapper:  write a wrapper script for executing command.

   Example:
     # create it and run 'pip install restview' inside
     venv create restview pip install restview

     # now run restview inside
     venv restview restview

     # now I'm all done
     venv destroy restview
EOF
   if [ -d "$venv_base_d" ]; then
      echo
      list_environments "$venv_base_d"
   fi
}

list() {
   list_environments "$VENV_BASE_D"
   return
}

find_env() {
   local error=false
   [ "$1" = "--error" ] && { error=true; shift; }

   local env="$1" activate="" env_d="" cand=""
   if [ -f "$env" ] && [ "${env##*/}" = "activate" ]; then
      activate="$env"
      env_d="${env%/*/*}"
   elif [ -f "$env/bin/activate" ]; then
      activate="$env/bin/activate"
      env_d="$env"
   elif [ -f "$VENV_BASE_D/$env/bin/activate" ]; then
      activate="$VENV_BASE_D/$env/bin/activate"
      env_d="$VENV_BASE_D/$env"
   else
      if $error; then
         error "$env: not a valid env?"
         error "try one of:"
         list_environments "$VENV_BASE_D" 1>&2
      fi
      return 1
   fi
   _R_activate="$activate"
   _R_env_d="$env_d"
   return 0
}

activate() {
   local change_d=false
   [ "$1" = "--change-dir" -o "$1" = "-C" ] && shift && change_d=true
   local activate="" env="$1" env_d=""
   shift

   find_env --error "$env" &&
      activate="$_R_activate" && env_d="$_R_env_d" || return 1
   . "$activate"

   ! $change_d || cd "$env_d"
   [ "$#" -gt 0 ] || set -- ${SHELL:-/bin/bash}
   debian_chroot="venv:$env" VENV_D="$env_d" exec "$@"
}

create() {
   local python="" pyexe="" activate=false virtualenv="" out=""
   if [ "$1" = "--py3" -o "$1" = "--py2" ]; then
      python="python${1#--py}"
      pyexe=$(which "$python") || { error "no '$python' in path"; return 1; }
      shift
   else
      pyexe=$(which python3) || { error "no python3, try --py2"; return 1; }
      python=python3
   fi
   # allow passthrough options like --system-site-packages
   # venv create --py3 --system-site-packages foo
   local pt=""
   while [ "${1#--}" != "$1" ]; do
       pt="${pt:+$pt }$1"
       shift
   done

   if command -v virtualenv >/dev/null; then
      virtualenv=virtualenv
   elif out=$($python -c 'import virtualenv'); then
      virtualenv="$python -m virtualenv"
   else
      cat 1>&2 <<EOF
No 'virtualenv' command and '$python -c "import virtualenv"' failed.  Maybe:
    apt-get install virtualenv
or
    apt-get install $python-virtualenv
EOF
      return 1
   fi
   if [ "$1" = "--activate" ]; then
      shift
      activate=true
   fi

   local env="$1" venv_base_d="${VENV_BASE_D}" env_d=""
   shift
   case "$env" in
      */*|/*) venv_base_d=$(dirname "$env");;
      *) venv_base_d="$VENV_BASE_D";;
   esac
   env_d="${venv_base_d}/${env##*/}"
   [ -d "$venv_base_d" ] || mkdir -p "$venv_base_d" ||
      { error "failed mkdir $venv_base_d"; return 1; }
   $virtualenv ${pyexe:+--python=$pyexe} $pt "$env_d"

   [ $# -eq 0 ] || activate=true
   if $activate; then
       activate "$env_d" "$@"
   fi
}

destroy() {
   local force=false
   [ "$1" = "--force" ] && force=true
   local env="$1" env_d=""
   find_env --error "$env" && env_d="$_R_env_d" || return 1
   [ -d "$env_d" ] || { error "$env_d: not a dir?"; return 1; }
   if ! $force; then
      local resp=""
      error "remove '$env_d' [y/n]?"
      read resp || { error "failed read response"; return 1; }
      [ "$resp" = "y" -o "$resp" = "Y" ] ||
         return 0
   fi
   rm -Rf "$env_d"
}

wrapper_Usage() {
   cat <<EOF
Usage: $0 wrapper venv [command]

    write a wrapper to execute command in venv.
    If command is not provided, then only the command
    will be the same as: 'venv exec <env>'

    options:
       -o | --output  FILE    write to FILE (default=-)
EOF
}

wrapper() {
    local short_opts="ho:"
    local long_opts="help,output:"
    local getopt_out=""
    getopt_out=$(getopt --name "${0##*/}" \
        --options "${short_opts}" --long "${long_opts}" -- "$@") &&
        eval set -- "${getopt_out}" ||
        { wrapper_Usage 1>&2; return 1; }

    local cur="" next="" output="-"
    while [ $# -ne 0 ]; do
        cur="$1"; next="$2";
        case "$cur" in
            -h|--help) Usage ; exit 0;;
            -o|--output) output=$next; shift;;
            --) shift; break;;
        esac
        shift;
    done

    [ $# -ge 1 ] || {
      wrapper_Usage; echo "Got $# args, expect 2 or more."; return 1; } 1>&2

    local venv="$1" cmd=""
    shift
    cmd="$*"

    [ "$output" = "-" ] || exec > "$output" ||
        { error "failed to write to '$output'"; return 1; }
    printf "%s\n%s %s\n" "#!/bin/sh" \
      "exec venv exec $venv${cmd:+ $cmd}" '"$@"'

    if [ "$output" != "-" ]; then
       chmod +x "$output" ||
          { error "failed to make '$output' executable."; return 1; }
       error "wrote to $output"
    fi
    return 0
}


[ $# -eq 0 ] && { Usage "$VENV_BASE_D" 1>&2; exit 1; }
[ "$1" = "-h" -o "$1" = "--help" ] && { Usage "$VENV_BASE_D"; exit 0; }

case "$1" in
   create|list|destroy|activate|wrapper) mode="$1"; shift;;
   exec|e) mode="activate"; shift;;
   *) Usage 1>&2; error "unknown mode '$1'"; return;;
esac

"$mode" "$@"