Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

An example use-case

I’m a noob about python dependency management—here’s what’s confusing me. Using virtualenvwrapper, with no existing venvs, let’s say I create a new directory and save a copy of the system packages I have installed with pip:

coby at vulcan in ~/tmp
➜ mkdir python-test && cd python-test
 
coby at vulcan in ~/tmp/python-test
➜ pip freeze > system-requirements.txt

coby at vulcan in ~/tmp/python-test
➜ wc system-requirements.txt
      54      54     971 system-requirements.txt

Next, I create a new virtual environment, and explicitly say I don’t want to bring over the system packages I have:

coby at vulcan in ~/tmp/python-test
➜ mkvirtualenv --no-site-packages myenv
New python executable in myenv/bin/python2.7
Also creating executable in myenv/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
(myenv)

Cool, looks like that worked, now let’s save out the virtualenv’s installed packages and compare them to the system ones:

coby at vulcan in ~/tmp/python-test
(myenv) ➜ pip freeze > myenv-requirements.txt
(myenv)
 
coby at vulcan in ~/tmp/python-test
(myenv) ➜ diff system-requirements.txt myenv-requirements.txt
50d49
< vboxapi==1.0
(myenv)
 
coby at vulcan in ~/tmp/python-test
(myenv) ➜ wc myenv-requirements.txt
      53      53     958 myenv-requirements.txt
(myenv)

So only one of the packages wasn’t copied over? Shouldn’t the diff be much bigger than that?


My desired use-case

The reason I’m wanting to understand this better is that I want to upgrade fastmonkeys/sentry-on-heroku from Sentry v6.3.2 to v6.3.3—what should be a simple thing to upgrade.

What I need to be able to do is create a virtualenv containing just the packages specified in that project’s requirements.txt so I can do a pip install --upgrade sentry or similar. However as the terminal output above shows, it looks like a large number of my system pip packages get ported over to any virtualenv I create, even though I specify --no-site-packages, so doing pip upgrades and a subsequent pip freeze > requirements.txt will polute the diff for the requirements.txt of that project with all kinds of packages from my system, and I’ve no way of telling which changes were legit upgrades as dependencies of the new version of sentry, or which were just packages that my system happened to have newer version of.

Here’s what I’m trying to do:

coby at vulcan in ~/tmp/python-test
➜ cd ~/code/sentry-on-heroku

coby at vulcan in ~/code/sentry-on-heroku ⌥ master •
± git pull origin master
From https://github.com/fastmonkeys/sentry-on-heroku
 * branch            master     -> FETCH_HEAD
Already up-to-date.

coby at vulcan in ~/code/sentry-on-heroku ⌥ master •
± gco -b sentry-6.3.3
Switched to a new branch 'sentry-6.3.3'

coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
± mkvirtualenv --no-site-packages sentry-heroku
New python executable in sentry-heroku/bin/python2.7
Also creating executable in sentry-heroku/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
(sentry-heroku)

coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
(sentry-heroku) ± pip install -r requirements.txt

[lots of output]

(sentry-heroku)

coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
(sentry-heroku) ± pip freeze > requirements.txt
(sentry-heroku)

Now, when I do a diff of the requirements.txt file, here’s what I see:

coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
(sentry-heroku) ± gd
diff --git a/requirements.txt b/requirements.txt
index 849355d..ab56ba8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,5 @@
 BeautifulSoup==3.2.1
-Django==1.5.4
+Django==1.5.5
 Pygments==1.6
 South==0.8.2
 amqp==1.0.13
@@ -7,39 +7,47 @@ anyjson==0.3.3
 billiard==2.7.3.34
 celery==3.0.24
 cssutils==0.9.10
+distribute==0.6.34
 dj-database-url==0.2.2
 django-bcrypt==0.9.2
 django-celery==3.0.23
 django-crispy-forms==1.2.8
 django-paging==0.2.5
-django-picklefield==0.3.0
+django-picklefield==0.3.1
 django-secure==1.0
 django-social-auth==0.7.28
 django-static-compiler==0.3.3
 django-templatetag-sugar==0.1
-gevent==0.13.8
+gevent==1.0
 greenlet==0.4.1
 gunicorn==0.17.4
+gyp==0.1
 httpagentparser==1.2.2
 httplib2==0.8
 kombu==2.5.16
+libxml2-python==2.9.1
 logan==0.5.9.1
+mercurial==2.7.2
 mock==1.0.1
 nydus==0.10.6
 oauth2==1.5.211
 psycopg2==2.5.1
-py-bcrypt==0.3
+py-bcrypt==0.4
 pynliner==0.5.0
 python-dateutil==1.5
 python-memcached==1.53
 python-openid==2.2.5
-pytz==2013.7
-raven==3.5.0
+pytz==2013.8
+raven==3.5.2
 redis==2.8.0
-sentry==6.3.2
+sentry==6.3.3
 sentry-github==0.1.2
-setproctitle==1.1.7
+setproctitle==1.1.8
 simplejson==3.3.1
 six==1.4.1
-sourcemap==0.1.7
+sourcemap==0.1.8
+stevedore==0.13
+virtualenv==1.10.1
+virtualenv-clone==0.2.4
+virtualenvwrapper==4.1.1
 wsgiref==0.1.2
(sentry-heroku)

I’m just guessing wildly here, but I doubt distribute, gyp, libxml2-python, mercurial, stevedore, virtualenv, virtualenv-clone, and virtualenvwrapper are all completely new additions to the dependencies of the project. I think they’re getting in there from my local system pip packages.

What am I missing? What am I doing wrong?

@paddycarey

This comment has been minimized.

Copy link

commented Nov 29, 2013

distribute'll get a different version because it's installed when you create the venv before you install the packages, that's normal

the others do look like they're coming from somewhere else though, had a quick gawk at sentry's commit history and see nothing jumping out that'd explain new deps

can you try using regular virtualenv instead of virtualenvwrapper? see if that helps at all

virtualenv --no-site-packages ~/myvdir
source ~/myvdir/bin/activate
pip install -r requirements.txt

@paddycarey

This comment has been minimized.

Copy link

commented Nov 29, 2013

btw, when I follow the exact steps as in your desired use case (after a pip install --upgrade sentry) above my diff looks like

diff --git a/requirements.txt b/requirements.txt
index 849355d..546d383 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,18 +1,20 @@
BeautifulSoup==3.2.1
-Django==1.5.4
+Django==1.5.5
Pygments==1.6
South==0.8.2
amqp==1.0.13
anyjson==0.3.3
+argparse==1.2.1
billiard==2.7.3.34
celery==3.0.24
cssutils==0.9.10
+distribute==0.6.34
dj-database-url==0.2.2
django-bcrypt==0.9.2
django-celery==3.0.23
django-crispy-forms==1.2.8
django-paging==0.2.5
-django-picklefield==0.3.0
+django-picklefield==0.3.1
django-secure==1.0
django-social-auth==0.7.28
django-static-compiler==0.3.3
@@ -33,12 +35,12 @@ pynliner==0.5.0
python-dateutil==1.5
python-memcached==1.53
python-openid==2.2.5
-pytz==2013.7
-raven==3.5.0
+pytz==2013.8
+raven==3.5.2
redis==2.8.0
-sentry==6.3.2
+sentry==6.3.3
sentry-github==0.1.2
-setproctitle==1.1.7
+setproctitle==1.1.8
simplejson==3.3.1
six==1.4.1
sourcemap==0.1.7

which is as expected

@cobyism

This comment has been minimized.

Copy link
Owner Author

commented Nov 29, 2013

Thanks for the replies. I’ll have a crack with just regular virtualenv and see what happens 😃

@cobyism

This comment has been minimized.

Copy link
Owner Author

commented Nov 29, 2013

Nope, using regular virtualenv instead of virtualenvwrapper produces the exact same output for me. Looking at the diff between versions 6.3.2 and 6.3.3 of Sentry, it looks like literally the only thing that changed was South being explicitly set to 0.8.2, so I’m really sceptical about basically all of the other the diff entries in both of our respective outputs 😕

@paddycarey

This comment has been minimized.

Copy link

commented Nov 29, 2013

fwiw i'm pretty confident in my diff above, the package upgrades can be explained by following some of the dependency chains, it's only the extra packages in your diff that i'm unsure about

i can only assume there's something screwy going on with your python/virtualenv install in general :(

@cobyism

This comment has been minimized.

Copy link
Owner Author

commented Nov 29, 2013

Right, I see. Yeah it could very well be something odd going on with my setup. I don’t work with Python much, so I wouldn’t notice if something was out of whack. I guess I just want to be able to explicitly know that things I’m doing pertain only to the environment of a given project, and are completely independent of my system packages like I can in Ruby with Bundler and bundle exec. I’m sure there’s a way to do that here, I just don’t have the Python experience to know what to look for.

Thanks for helping troubleshoot—I really appreciate it! 😃⚡️👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.