Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Virtualenv's `bin/activate` is Doing It Wrong
@metamatt

This comment has been minimized.

Copy link

metamatt commented Apr 11, 2012

Yes. Thank you.

@datagrok

This comment has been minimized.

Copy link
Owner Author

datagrok commented Apr 11, 2012

I've taken this rant to the virtualenv maintainers and now I'm working on a patch that might eventually get accepted upstream! :-D Just need a bit more time to improve it.

pypa/virtualenv#247
https://groups.google.com/d/topic/python-virtualenv/XzI8GStvKDw/discussion

@mk270

This comment has been minimized.

Copy link

mk270 commented Feb 27, 2013

I think virtualenv is goodish, but you're write that it's not Unix enough.

A particular gripe is that . bin/activate seems to break when called from a shell script running with

set -eu

I'd like to be able to use -eu as it makes bash scripting a lot less error-prone, but virtualenv's active doesn't want to know

@NewbiZ

This comment has been minimized.

Copy link

NewbiZ commented Feb 27, 2013

I don't really feel like activate is a real problem. Most of the time, you just end up spawning a bash before entering a virtualenv, and that's it. If you're lazy:
alias inve='bash --init-file '

Btw, you often end up automating the use of virtualenvs (e.g. using the excellent fabric library) and dealing with a single source is way easier than using a subshell and pipe once-more-escaped commands to it.

@Apreche

This comment has been minimized.

Copy link

Apreche commented Feb 27, 2013

I agree. But the real problem I have with virtualenv is how it works on a non-interactive shell. What if I write a Python program using a virtualenv, and then I want to execute it from init.d, supervisor, upstart, cron, etc.? Those can't activate.

The trick I have been using is to hardcode the full path to the virtualenv python binary. For example in a cron I will

*/5 * * * * /home/username/.virtualenv/project/bin/python /home/username/src/project/program.py

In a perfect world it would look something more like this

*/5 * * * * virtualenv_activate projectname -e /home/username/src/project/program.py

@jdawsongit

This comment has been minimized.

Copy link

jdawsongit commented Feb 27, 2013

activate also modifies $LD_LIBRARY_PATH. inve needs to set that too.

EDIT:

Standard activate does not modify $LD_LIBRARY_PATH. I just remembered that's something I patched in. Where I work, we use the virtual Python environment to be the general container for all the stuff our application needs. We include gevent in our virtual environment, so we also put its dependencies (like libevent, needed by our version of gevent) in the virtual environment under the $VIRTUAL_ENV/lib. It's worked out well for us. It lets us deploy new virtual environment snapshots with libraries upgraded as needed. The virtual environment is the atomic unit of deployment, complete with all libraries and their dependencies. I wonder why this isn't common practice.

@tclancy

This comment has been minimized.

Copy link

tclancy commented Feb 27, 2013

Not to be a pain, but to speak up for the great unwashed, please don't forget about Windows in the drive to be more Unix-y. It's possible some of the problems come from Windows compatibility as activate (well, activate.bat) is an executable on Windows.

@bbarrows

This comment has been minimized.

Copy link

bbarrows commented Feb 27, 2013

This is pretty nice, I always put my virtual env in _venv so I wrote an alias that would do something like:
alias venv='source _venv/bin/activate'
but I like this much better.

One thing I do think would be nice is modifying the shell prompt. For example, how virtual env prepends to the users prompt after activating. Its nice having that visual reminder so that you know which shells are actively using which virtual environment..

@vishnubob

This comment has been minimized.

Copy link

vishnubob commented Feb 27, 2013

ssh-agent(1) doesn't start a sub-shell. It generates environment variables that you source into your current environment. dircolors(1) does the same thing. I would argue that the pattern of modifying your environment in place is very consistent with how UNIX programmers have solved this problem. Sub-shells are simply an alternative.

You make a lot of great points about the simplicity of solving this problem with sub-shells, and I completely agree that it should be made available as an option -- but nothing about modifying your environment in place is "abhorrently wrong and un-unix-y".

@selenamarie

This comment has been minimized.

Copy link

selenamarie commented Feb 27, 2013

An incremental name suggestion for 'inve' - 'intove' is a little more evocative.

Thanks for a great perspective, code and attention to UX issues that benefit everyone when fixed. :)

@e000

This comment has been minimized.

Copy link

e000 commented Feb 28, 2013

Thanks! I've been super annoyed having to switch from fishfish to bash to use virtualenv. :3

@datagrok

This comment has been minimized.

Copy link
Owner Author

datagrok commented Feb 28, 2013

Wow, almost a year after I wrote this thing it has been receiving a lot of attention on Hacker News (link) and Reddit (link). I'm very grateful for all the feedback and intend to incorporate as much as I can.

@NewbiZ

This comment has been minimized.

Copy link

NewbiZ commented Feb 28, 2013

The trick I have been using is to hardcode the full path to the virtualenv python binary. For example in a cron I will

*/5 * * * * /home/username/.virtualenv/project/bin/python /home/username/src/project/program.py

You are not forced to do this.
If your program was properly installed inside the virtualenv (as it should be), then its shebang should have been replaced by the full path of the virtualenv's python interpreter. This was made to answer this problem.

Standard activate does not modify $LD_LIBRARY_PATH. I just remembered that's something I patched in. Where I work, we use the virtual Python environment to be the general container for all the stuff our application needs. We include gevent in our virtual environment, so we also put its dependencies (like libevent, needed by our version of gevent) in the virtual environment under the $VIRTUAL_ENV/lib.

Yeah, we do the exact same thing.

I wonder why this isn't common practice.

Because this isn't really working all that well:

  • (if you use pip to provision your venvs:)pip only handle runtime dependencies. It only ensure that after install, every dependency is present, so the installation order of you dependencies is not guaranteed to match your dependency graph => you can not rely on pip to build source packages. You are limited to distribute binaries (and since pip was made to handle source packages, it does not provide anything to support multiple platforms for prebuilt binaries, thus forcing you to handle this at the bootstrap layer (a.k.a. multiple repositories for different platforms, and distributing a pip.conf) or the package maintainer level (the package contains all platforms, and setup.py picks at installation time which one to install)
  • Relying on LD_LIBRARY_PATH is kind of hackish since it directly hooks up library resolution in ld.so, thus impacting your dynamic loader (program execution) AND ld (the linker). This means that any attempt to build something in your virtualenv could not be trusted.
@honza

This comment has been minimized.

Copy link

honza commented Feb 28, 2013

Doesn't seem to work for me on the fish shell. Creates a subshell just fine but doesn't actually "activate" the virtualenv.

@esoergel

This comment has been minimized.

Copy link

esoergel commented Feb 28, 2013

I absolutely agree with this idea, it makes a lot more sense conceptually. If you're developing for a server, you should basically be creating a local version of that server, and a local version of ssh is more analogous. I use virtualenvwrapper, which simpler to use, but it just abstracts away that complication rather than fixing it.

@nvie

This comment has been minimized.

Copy link

nvie commented Feb 28, 2013

@honza You should check that, because it does work on fish for me without further hacks.

@caruccio

This comment has been minimized.

Copy link

caruccio commented Mar 1, 2013

Do you known Kenneth Reitz's autoenv?
One simply "cd " and BANG!, virtualenv activated.
Then, "cd .." and BANG!, virtualenv deactivated.

The picture says for itself... Please give it a try:

https://github.com/kennethreitz/autoenv

@dustinlacewell

This comment has been minimized.

Copy link

dustinlacewell commented Mar 2, 2013

To be fair, I only give auto-activating a virtualenv as an example of using capn which is a generic directory-based hooking mechanism. It doesn't really "use" the anti-pattern described here. @caruccio, that might interest you based on your comment, though autoenv looks nice.

That said, I agree with what has been said here. I wonder, like @tclancy how much of this has to do with Windows.

@j0hnsmith

This comment has been minimized.

Copy link

j0hnsmith commented Mar 3, 2013

@e000 in fish (or fishfish) you can do . virtualenv/bin/activate.fish

@mangecoeur

This comment has been minimized.

Copy link

mangecoeur commented Mar 14, 2013

This is a very elegant approach. I used it to create a simple activate/workon script for environments created with the conda tool, which is part of the Anaconda scientific python distribution. check it out https://gist.github.com/mangecoeur/5161488

@mondaugen

This comment has been minimized.

Copy link

mondaugen commented May 15, 2013

I can only get this to work with bash not zsh

@berdario

This comment has been minimized.

Copy link

berdario commented May 26, 2013

After an ubuntu upgrade, my virtualenvs broke, I got tired of that setup, and I saw how this could be useful for making a virtualenvwrapper that doesn't even have to touch the shell from which it is invoked...

So, I made a rewrite of virtualenvwrapper in pure python... and I unimaginatively called it "invewrapper" :)

https://github.com/berdario/invewrapper

It works on bash, zsh, fish... and it mostly works on windows too! (there's a bug right now that affects sitepackages_dir and related commands)

Please let me have some feedback :)

@lyndsysimon

This comment has been minimized.

Copy link

lyndsysimon commented Jun 25, 2013

A year after reading this, I'm still chewing on it. I think I completely agree, and I don't know why it hasn't gained wider adoption.

@datagrok, do you attend PyCon? If so, you should really do at least a lightning talk on this. It's subtly and elegantly revolutionary IMO.

@berdario

This comment has been minimized.

Copy link

berdario commented Jun 29, 2013

@lyndsysimon have you tried invewrapper? it could be a way for it to gain wider adoption :D

@datagrok

This comment has been minimized.

Copy link
Owner Author

datagrok commented Sep 17, 2013

I've done a bit more work on this--you can see it in my feature/subshell-activate branch of virtualenv. It's not completely working but I think I have a good proof-of-concept. I've implemented a means to do the "activation" in a subshell (the way I like it) without breaking workflow at all for users who prefer to source bin/activate or one of its flavors. (I reimplement the latter in terms of the former.) I'll be pushing more commits whenever I have time.

Also, I just submitted a PyCon 2014 talk proposal based on this work.

Thanks all for your feedback, recommendations, alternative proposals, and encouragement.

@tsal

This comment has been minimized.

Copy link

tsal commented Sep 30, 2013

@datagrok - I hope to catch your talk; PyCon's time and place is actually convenient for once. :)

@ericfrederich

This comment has been minimized.

Copy link

ericfrederich commented Nov 27, 2013

@Apreche take a look here:

http://rosettacode.org/wiki/Multiline_shebang

I start a lot of my scripts this way....

#!/bin/sh
if "true" : '''\'
then

# set up your execution environment
source /some/stuff

exec python "$0" "$@"

exit 127
fi
'''

print "now we're in Python"
@datagrok

This comment has been minimized.

Copy link
Owner Author

datagrok commented Jan 3, 2014

@tsal ah, my proposal was declined. I'll try to boil it down into a lightning talk and submit that when I'm there though. Oh, and will try to finish up my proposed changes to virtualenv, that too. But @berdario's invewrapper looks pretty awesome, he might have gone and eliminated the need for any of my changes! We'll see :)

@dolmen

This comment has been minimized.

Copy link

dolmen commented Jan 13, 2014

@datagrok wrote:

Unfortunately, I don't know if this "eval the output of a command" technique works for all possible shells.

Yes it can work. But with many quirks depending on each shell: for example, you must avoid any use of \n and consecutive spaces if you want the code to work even if the user forgot to use quotes around your command (eval "$(xxx)" vs eval $(xxx)).
I'm using this technique for the launch of my prompt engine named angel-PS1 which is portable (on various shells: bash, zsh, dash, ksh, fish, tcsh).

@cmeiklejohn

This comment has been minimized.

Copy link

cmeiklejohn commented Feb 19, 2014

Any thoughts on why this doesn't work with zsh, or what the fix will be to make it more portable?

@manageyp

This comment has been minimized.

Copy link

manageyp commented Feb 25, 2014

Mark: Virtualenv the right way.

@kwill

This comment has been minimized.

Copy link

kwill commented Feb 25, 2014

@Apreche it looks like pew in ve_name your_command might be the answer you're looking for (confirmed for shell scripts, not sure about Python scripts or modules just yet) - see https://stackoverflow.com/questions/22018185/how-can-i-use-pew-in-a-bash-python-fabric-sh-script

@Dakta

This comment has been minimized.

Copy link

Dakta commented Mar 6, 2014

Very important note: the provided inve will fail if you do not have the python-dev package installed, as there will not be include/python*/Python.h

@datagrok You should make a note of this or modify the script to function without python-dev.

@techtonik

This comment has been minimized.

Copy link

techtonik commented Mar 19, 2014

I've added a pull request in pypa/virtualenv#581 to introduce login/logout shell logic. It is a first step that could be expanded later if merged.

@joedevivo

This comment has been minimized.

Copy link

joedevivo commented Apr 7, 2014

There's a problem using this with oh-my-zsh. The $PATH variable gets overwritten by the stock ~/.zshrc when you launch the new shell, appending something like this to your ~/.zshrc will fix the problem.

if [ -z "$VIRTUAL_ENV" ]; then
else
    export PATH="$VIRTUAL_ENV/bin:$PATH"
fi
@sashahart

This comment has been minimized.

Copy link

sashahart commented Jul 4, 2014

Thanks for the link!

I guess by now I have written every major variation on activating virtualenvs, at least in prototype, and it's clear that different interfaces have different merits. I think it's completely valid, depending on what you want, to use raw virtualenv and direct paths to pythons or bin/ scripts for speed especially in non-interactive cases, simple shell aliases like ave for maximum interactive speed in a favorite shell, invewrapper for maximal replacement of most virtualenvwrapper features or vex for a more minimalist interface that functions like sudo. And several of these offer some completion definitions and such.

What these options don't have is a sensible default that everyone uses; the venerable virtualenvwrapper is as close as we get. But if you maybe want to teach newbies to use virtualenv, making them edit bashrc is painful, so we get virtualenv-burrito to hack your bashrc for you. And all this resting on a super ornate pile of shell functions that are not very hard to disrupt accidentally. Still, if you want 'standard,' it is still the clear choice for that purpose.

@orodbhen

This comment has been minimized.

Copy link

orodbhen commented Nov 19, 2014

Just a question. Where does PS1_CONTEXT come from? Is it supposed to be created in .bashrc?

@datagrok

This comment has been minimized.

Copy link
Owner Author

datagrok commented Jun 14, 2015

@orodbhen, PS1_CONTEXT is just an example I made up of "various ways to stuff things into your prompt without clobbering $PS1." You could have other scripts that assign the current git branch to that variable whenever you're in a git repo, for example. If it's unset, it doesn't do anything.

@Grk0

This comment has been minimized.

Copy link

Grk0 commented Jun 16, 2015

It's a good observation that adding the venv setting to the current shell is a bit problematic. Unfortunately, I think you throw the baby out with the bathwater and create a couple of new problems in the process.

Don't do away with activate, keep it and use it instead. Then there's no need to duplicate its logic in inve, just use this:

#/bin/bash
"`dirname $0`/activate"
exec "${@:-$SHELL}"

That way you don't break everyone who edited activate to add custom logic (like jdawsongit). That also allows you to get rid of the eval hack as well - just keep using activate for these occasions. No need to bend over backwards for something that activate gives you for free.

Finally, the system-level inve is broken. It seems to assume it's executed with a $PWD somewhere under the venv directory. From my understanding, it's not common to actually have a shell open under the venv path, since you typically have your project source code next to the venv, not inside it (so you can nuke and recreate your venv, if need be). The system-level inve also changes the current directory, which is a horrible idea. That breaks many use-cases, for example

inve py.test ./tests    # Yeah, '.' is not what you thought it was :-(

So there's a good idea here, but I think the solution creates more problems than it solves. Especially the system-level inve can't be used as-is. I'm not sure whether it's salvageable, you'd need a way to discover venvs in parent directories, which seems tricky and error-prone. At the very least, use pushd/popd to avoid changing the current directory for the childs you execute!

@ptvirgo

This comment has been minimized.

Copy link

ptvirgo commented Apr 5, 2016

I'm relatively new to Python though I've been around long enough to appreciate Unix-friendly approaches. My first tries at getting activate to work failed, so I turned to The Internet for instructions. Your article came up, and pointed me at vex. Vex worked great, and it feels much more in line with the tools I've come to expect from the CLI. This will probably go double when I start mucking about with FreeBSD, I know BASH is not the standard shell over there.

Thanks for your article and the included recommendations.

@gqmelo

This comment has been minimized.

Copy link

gqmelo commented May 20, 2016

I've been using conda for a while and the same problem was bothering me.
Conda has an virtualenv's style activate and it was very annoying to use the envs on a non-console application (IDE).

So I created exec-wrappers to be able to create wrappers ready to use with minimum overhead.
It also happens to create wrappers for virtualenv. So if someone is interested, please take a look.

@jerzygangi

This comment has been minimized.

Copy link

jerzygangi commented Jan 7, 2017

All I can say is +1

@pabloa

This comment has been minimized.

Copy link

pabloa commented Feb 3, 2017

great article. It is exactly what people need when python is integrated to other technologies (like java). python code probably is called with not terminal in the middle. This article solved my problem.

@johndpope

This comment has been minimized.

Copy link

johndpope commented Jun 18, 2017

@Acrisel

This comment has been minimized.

Copy link

Acrisel commented Jul 14, 2017

This is great. I would just note that there is no need to hard code the path of the virtualenv in shell and batch scripts. It can and should be relative.

@mmerickel

This comment has been minimized.

Copy link

mmerickel commented Sep 8, 2017

Please also look at vrun [1] which is super lightweight and simply activates a virtualenv for the duration of a command. You run something like env/bin/vrun <cmd> and it is executed with all of the virtualenv scripts etc on the path. This is similar to vex but with fewer assumptions and should be mentioned in the same conversation.

[1] https://pypi.org/project/vrun/

@apoliakov

This comment has been minimized.

Copy link

apoliakov commented Nov 3, 2017

Helpful! Thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.