I’m a noob about python dependency management—here’s what’s confusing me. Using virtualenvwrapper, with no existing venvs, let’s say I create a new directory and save a copy of the system packages I have installed with pip:
coby at vulcan in ~/tmp
➜ mkdir python-test && cd python-test
coby at vulcan in ~/tmp/python-test
➜ pip freeze > system-requirements.txt
coby at vulcan in ~/tmp/python-test
➜ wc system-requirements.txt
54 54 971 system-requirements.txt
Next, I create a new virtual environment, and explicitly say I don’t want to bring over the system packages I have:
coby at vulcan in ~/tmp/python-test
➜ mkvirtualenv --no-site-packages myenv
New python executable in myenv/bin/python2.7
Also creating executable in myenv/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
(myenv)
Cool, looks like that worked, now let’s save out the virtualenv’s installed packages and compare them to the system ones:
coby at vulcan in ~/tmp/python-test
(myenv) ➜ pip freeze > myenv-requirements.txt
(myenv)
coby at vulcan in ~/tmp/python-test
(myenv) ➜ diff system-requirements.txt myenv-requirements.txt
50d49
< vboxapi==1.0
(myenv)
coby at vulcan in ~/tmp/python-test
(myenv) ➜ wc myenv-requirements.txt
53 53 958 myenv-requirements.txt
(myenv)
So only one of the packages wasn’t copied over? Shouldn’t the diff be much bigger than that?
The reason I’m wanting to understand this better is that I want to upgrade fastmonkeys/sentry-on-heroku from Sentry v6.3.2 to v6.3.3—what should be a simple thing to upgrade.
What I need to be able to do is create a virtualenv containing just the packages specified in that project’s requirements.txt so I can do a pip install --upgrade sentry or similar. However as the terminal output above shows, it looks like a large number of my system pip packages get ported over to any virtualenv I create, even though I specify --no-site-packages, so doing pip upgrades and a subsequent pip freeze > requirements.txt will polute the diff for the requirements.txt of that project with all kinds of packages from my system, and I’ve no way of telling which changes were legit upgrades as dependencies of the new version of sentry, or which were just packages that my system happened to have newer version of.
Here’s what I’m trying to do:
coby at vulcan in ~/tmp/python-test
➜ cd ~/code/sentry-on-heroku
coby at vulcan in ~/code/sentry-on-heroku ⌥ master •
± git pull origin master
From https://github.com/fastmonkeys/sentry-on-heroku
* branch master -> FETCH_HEAD
Already up-to-date.
coby at vulcan in ~/code/sentry-on-heroku ⌥ master •
± gco -b sentry-6.3.3
Switched to a new branch 'sentry-6.3.3'
coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
± mkvirtualenv --no-site-packages sentry-heroku
New python executable in sentry-heroku/bin/python2.7
Also creating executable in sentry-heroku/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
(sentry-heroku)
coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
(sentry-heroku) ± pip install -r requirements.txt
[lots of output]
(sentry-heroku)
coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
(sentry-heroku) ± pip freeze > requirements.txt
(sentry-heroku)
Now, when I do a diff of the requirements.txt file, here’s what I see:
coby at vulcan in ~/code/sentry-on-heroku ⌥ sentry-6.3.3 •
(sentry-heroku) ± gd
diff --git a/requirements.txt b/requirements.txt
index 849355d..ab56ba8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,5 @@
BeautifulSoup==3.2.1
-Django==1.5.4
+Django==1.5.5
Pygments==1.6
South==0.8.2
amqp==1.0.13
@@ -7,39 +7,47 @@ anyjson==0.3.3
billiard==2.7.3.34
celery==3.0.24
cssutils==0.9.10
+distribute==0.6.34
dj-database-url==0.2.2
django-bcrypt==0.9.2
django-celery==3.0.23
django-crispy-forms==1.2.8
django-paging==0.2.5
-django-picklefield==0.3.0
+django-picklefield==0.3.1
django-secure==1.0
django-social-auth==0.7.28
django-static-compiler==0.3.3
django-templatetag-sugar==0.1
-gevent==0.13.8
+gevent==1.0
greenlet==0.4.1
gunicorn==0.17.4
+gyp==0.1
httpagentparser==1.2.2
httplib2==0.8
kombu==2.5.16
+libxml2-python==2.9.1
logan==0.5.9.1
+mercurial==2.7.2
mock==1.0.1
nydus==0.10.6
oauth2==1.5.211
psycopg2==2.5.1
-py-bcrypt==0.3
+py-bcrypt==0.4
pynliner==0.5.0
python-dateutil==1.5
python-memcached==1.53
python-openid==2.2.5
-pytz==2013.7
-raven==3.5.0
+pytz==2013.8
+raven==3.5.2
redis==2.8.0
-sentry==6.3.2
+sentry==6.3.3
sentry-github==0.1.2
-setproctitle==1.1.7
+setproctitle==1.1.8
simplejson==3.3.1
six==1.4.1
-sourcemap==0.1.7
+sourcemap==0.1.8
+stevedore==0.13
+virtualenv==1.10.1
+virtualenv-clone==0.2.4
+virtualenvwrapper==4.1.1
wsgiref==0.1.2
(sentry-heroku)I’m just guessing wildly here, but I doubt distribute, gyp, libxml2-python, mercurial, stevedore, virtualenv, virtualenv-clone, and virtualenvwrapper are all completely new additions to the dependencies of the project. I think they’re getting in there from my local system pip packages.
What am I missing? What am I doing wrong?
distribute'll get a different version because it's installed when you create the venv before you install the packages, that's normal
the others do look like they're coming from somewhere else though, had a quick gawk at sentry's commit history and see nothing jumping out that'd explain new deps
can you try using regular virtualenv instead of virtualenvwrapper? see if that helps at all
virtualenv --no-site-packages ~/myvdir
source ~/myvdir/bin/activate
pip install -r requirements.txt