Virtualenv's `bin/activate` is Doing It Wrong

gistfile1.md

This article is now published on my website: Prefer Subshells for Context.

I can only get this to work with bash not zsh

After an ubuntu upgrade, my virtualenvs broke, I got tired of that setup, and I saw how this could be useful for making a virtualenvwrapper that doesn't even have to touch the shell from which it is invoked...

So, I made a rewrite of virtualenvwrapper in pure python... and I unimaginatively called it "invewrapper" :)

https://github.com/berdario/invewrapper

It works on bash, zsh, fish... and it mostly works on windows too! (there's a bug right now that affects sitepackages_dir and related commands)

Please let me have some feedback :)

A year after reading this, I'm still chewing on it. I think I completely agree, and I don't know why it hasn't gained wider adoption.

@datagrok, do you attend PyCon? If so, you should really do at least a lightning talk on this. It's subtly and elegantly revolutionary IMO.

@lyndsysimon have you tried invewrapper? it could be a way for it to gain wider adoption :D

I've done a bit more work on this--you can see it in my feature/subshell-activate branch of virtualenv. It's not completely working but I think I have a good proof-of-concept. I've implemented a means to do the "activation" in a subshell (the way I like it) without breaking workflow at all for users who prefer to source bin/activate or one of its flavors. (I reimplement the latter in terms of the former.) I'll be pushing more commits whenever I have time.

Also, I just submitted a PyCon 2014 talk proposal based on this work.

Thanks all for your feedback, recommendations, alternative proposals, and encouragement.

@datagrok - I hope to catch your talk; PyCon's time and place is actually convenient for once. :)

@Apreche take a look here:

http://rosettacode.org/wiki/Multiline_shebang

I start a lot of my scripts this way....

#!/bin/sh
if "true" : '''\'
then

# set up your execution environment
source /some/stuff

exec python "$0" "$@"

exit 127
fi
'''

print "now we're in Python"

@tsal ah, my proposal was declined. I'll try to boil it down into a lightning talk and submit that when I'm there though. Oh, and will try to finish up my proposed changes to virtualenv, that too. But @berdario's invewrapper looks pretty awesome, he might have gone and eliminated the need for any of my changes! We'll see :)

@datagrok wrote:

Unfortunately, I don't know if this "eval the output of a command" technique works for all possible shells.

Yes it can work. But with many quirks depending on each shell: for example, you must avoid any use of \n and consecutive spaces if you want the code to work even if the user forgot to use quotes around your command (eval "$(xxx)" vs eval $(xxx)).
I'm using this technique for the launch of my prompt engine named angel-PS1 which is portable (on various shells: bash, zsh, dash, ksh, fish, tcsh).

Any thoughts on why this doesn't work with zsh, or what the fix will be to make it more portable?

Mark: Virtualenv the right way.

@Apreche it looks like pew in ve_name your_command might be the answer you're looking for (confirmed for shell scripts, not sure about Python scripts or modules just yet) - see https://stackoverflow.com/questions/22018185/how-can-i-use-pew-in-a-bash-python-fabric-sh-script

Very important note: the provided inve will fail if you do not have the python-dev package installed, as there will not be include/python*/Python.h

@datagrok You should make a note of this or modify the script to function without python-dev.

I've added a pull request in pypa/virtualenv#581 to introduce login/logout shell logic. It is a first step that could be expanded later if merged.

There's a problem using this with oh-my-zsh. The $PATH variable gets overwritten by the stock ~/.zshrc when you launch the new shell, appending something like this to your ~/.zshrc will fix the problem.

if [ -z "$VIRTUAL_ENV" ]; then
else
    export PATH="$VIRTUAL_ENV/bin:$PATH"
fi

Thanks for the link!

I guess by now I have written every major variation on activating virtualenvs, at least in prototype, and it's clear that different interfaces have different merits. I think it's completely valid, depending on what you want, to use raw virtualenv and direct paths to pythons or bin/ scripts for speed especially in non-interactive cases, simple shell aliases like ave for maximum interactive speed in a favorite shell, invewrapper for maximal replacement of most virtualenvwrapper features or vex for a more minimalist interface that functions like sudo. And several of these offer some completion definitions and such.

What these options don't have is a sensible default that everyone uses; the venerable virtualenvwrapper is as close as we get. But if you maybe want to teach newbies to use virtualenv, making them edit bashrc is painful, so we get virtualenv-burrito to hack your bashrc for you. And all this resting on a super ornate pile of shell functions that are not very hard to disrupt accidentally. Still, if you want 'standard,' it is still the clear choice for that purpose.

Just a question. Where does PS1_CONTEXT come from? Is it supposed to be created in .bashrc?

@orodbhen, PS1_CONTEXT is just an example I made up of "various ways to stuff things into your prompt without clobbering $PS1." You could have other scripts that assign the current git branch to that variable whenever you're in a git repo, for example. If it's unset, it doesn't do anything.

It's a good observation that adding the venv setting to the current shell is a bit problematic. Unfortunately, I think you throw the baby out with the bathwater and create a couple of new problems in the process.

Don't do away with activate, keep it and use it instead. Then there's no need to duplicate its logic in inve, just use this:

#/bin/bash
"`dirname $0`/activate"
exec "${@:-$SHELL}"

That way you don't break everyone who edited activate to add custom logic (like jdawsongit). That also allows you to get rid of the eval hack as well - just keep using activate for these occasions. No need to bend over backwards for something that activate gives you for free.

Finally, the system-level inve is broken. It seems to assume it's executed with a $PWD somewhere under the venv directory. From my understanding, it's not common to actually have a shell open under the venv path, since you typically have your project source code next to the venv, not inside it (so you can nuke and recreate your venv, if need be). The system-level inve also changes the current directory, which is a horrible idea. That breaks many use-cases, for example

inve py.test ./tests    # Yeah, '.' is not what you thought it was :-(

So there's a good idea here, but I think the solution creates more problems than it solves. Especially the system-level inve can't be used as-is. I'm not sure whether it's salvageable, you'd need a way to discover venvs in parent directories, which seems tricky and error-prone. At the very least, use pushd/popd to avoid changing the current directory for the childs you execute!

I'm relatively new to Python though I've been around long enough to appreciate Unix-friendly approaches. My first tries at getting activate to work failed, so I turned to The Internet for instructions. Your article came up, and pointed me at vex. Vex worked great, and it feels much more in line with the tools I've come to expect from the CLI. This will probably go double when I start mucking about with FreeBSD, I know BASH is not the standard shell over there.

Thanks for your article and the included recommendations.

I've been using conda for a while and the same problem was bothering me.
Conda has an virtualenv's style activate and it was very annoying to use the envs on a non-console application (IDE).

So I created exec-wrappers to be able to create wrappers ready to use with minimum overhead.
It also happens to create wrappers for virtualenv. So if someone is interested, please take a look.

All I can say is +1

great article. It is exactly what people need when python is integrated to other technologies (like java). python code probably is called with not terminal in the middle. This article solved my problem.

forget pip - pip3
use conda https://conda.io/docs/_downloads/conda-cheatsheet.pdf

This is great. I would just note that there is no need to hard code the path of the virtualenv in shell and batch scripts. It can and should be relative.

Please also look at vrun [1] which is super lightweight and simply activates a virtualenv for the duration of a command. You run something like env/bin/vrun <cmd> and it is executed with all of the virtualenv scripts etc on the path. This is similar to vex but with fewer assumptions and should be mentioned in the same conversation.

[1] https://pypi.org/project/vrun/

Helpful! Thank you!!

404 on the link 👎

still 404

@nmaupu , @benedikt-bartscher: here is a copy of the page on the Wayback Machine from archive.org.