jvns/pip-install-user.md Secret

## pip-install-user.md

      
    Raw
  

              pip-install-user.md
            
          
    some quick python packaging explorations: (please do not post this to Hacker News or whatever, I wrote this pretty quickly, I'm 100% sure it has factual errors, and I have no interest in starting yet another repetitive flamewar about Python packaging)
the question: how can pip install --user break system packages?

In Homebrew, pip install --user requests prints out this scary error message. But
why? How can installing user packages break my system programs? That seems
surprising.
why don't I just use virtualenv?

A lot of people asked this so in case anyone cares: I basically only use like 10 Python packages (click, requests, pyyaml, lxml, numpy, pandas, matplotlib, jupyter, maybe sqlite-utils) which I don't care about the version of at all
so for me it's a waste of time to carefully create virtualenvs for my random throwaway scripts that use them, it's much simpler to just install them all globally and then my scripts can just use whatever version I ended up with. It's worked well for me in the past without any problems.
(I do actually use virtualenv sometimes if I care about pinning specific versions of packages but that doesn't happen often these days)
Also I'm a pretty stubborn person with no fear of breaking things so my reaction to messages like "you should not do this, it might break your Python install" is "well I don't understand exactly HOW it will break my Python install, I'll just ignore that".
So let's see how.
Homebrew: does installing user packages override system programs?

When I install httpiewith Homebrew, it
uses a virtualenv.  (I assume it's the same for other Python programs, though that may not be true)
$ ls /opt/homebrew//Cellar/httpie/3.2.4/libexec/lib/python3.13/site-packages/
...
charset_normalizer                  httpie-3.2.4.dist-info          multidict                  requests-2.32.3.dist-info          socks.py
charset_normalizer-3.4.0.dist-info  idna                            multidict-6.1.0.dist-info  requests_toolbelt                  sockshandler.py
... rest of deps ...

Let's check if installing a user package changes httpie's dependencies:
$ pip3 install --user --break-system-packages requests==2.32.2
$ /opt/homebrew/Cellar/httpie/3.2.4/libexec/bin/python3
>>> import requests
>>> print(requests.__version__)
2.32.3

Seems like the answer is no: the python3 from httpie is still using 2.32.3.
The version from its virtualenv takes precedence.
So it looks like the answer is no, installing user packages doesn't override system packages.
what about Ubuntu Linux?

Next I tried the same thing on Ubuntu.
$ sudo apt install python3-pip httpie

I edited /usr/bin/http to include this at the beginning:
import requests
print(requests.__version__)

Now let's check if installing a user package changes httpie's dependencies:
$ http
2.31.0 <- the original version is 2.32.1
$ pip install --user --break-system-packages requests==2.32.3 
$ http
2.32.3

It does change! So it looks like the answer on Ubuntu is yes, installing user packages does override system packages.
To understand why this is we just have to look at sys.path:
$ python3
>>> import sys
>>> for p in sys.path:
        print(p)
   
/usr/lib/python312.zip
/usr/lib/python3.12
/usr/lib/python3.12/lib-dynload
/home/bork/.local/lib/python3.12/site-packages
/usr/local/lib/python3.12/dist-packages
/usr/lib/python3/dist-packages
/usr/lib/python3.12/dist-packages

On Ubuntu, my local PATH (/home/bork/.local/lib/) comes before the system packages, so they'll override the system packages.
Homebrew and Ubuntu don't behave the same way at all

It seems like using --break-system-packages is MUCH safer in Homebrew than it is on Linux. On Ubuntu it really seems like it would very easily break system packages because programs you install system-wide will use your user-installed versions instead.
Also on Mac OS the core OS is not managed by Homebrew which makes it safer too.
why not "just" ignore user packages

I also wondered -- why doesn't Ubuntu's httpie ignore user packages (maybe with python -S or something), or prioritize the distribution versions above the user versions?
PEP 668 explains why it's important to include user packages sometimes:

In some cases, however, it’s useful and intentional to install a Python package from outside of the distro that influences the behavior of distro-shipped commands. This is common in the case of software like Sphinx or Ansible which have a mechanism for writing Python-language extensions.

but also according to @ancoghlan on Mastodon, Fedora does now ignore user packages when running system packages.
So this issue of user packages being included when running system packages might mostly be a Debian-specific issue at this point?
my conclusions

For Homebrew's python, it feels safe enough to me to set up my pip.conf like this.
$ cat ~/.pip/pip.conf
[global]
break-system-packages = true
user = true

I think the biggest issue with using Homebrew in this way is that Homebrew might upgrade Pytohn and delete the old one, breaking old virtualenvs, but I don't feel so bothered by that.
I think I probably wouldn't do the same thing on Debian/Ubuntu, though I'm not sure. I'm a pretty chaotic person so I might still do it but I'd be more cautious. I'd probably use more virtualenvs or start using uv.