Skip to content

Instantly share code, notes, and snippets.

@pradyunsg

pradyunsg/pip_upgrade.rst

Last active August 6, 2019 18:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pradyunsg/a5abeac4af90fbdc54bb266c32c0d2d8 to your computer and use it in GitHub Desktop.
Save pradyunsg/a5abeac4af90fbdc54bb266c32c0d2d8 to your computer and use it in GitHub Desktop.
Download ZIP
Pip Upgrade Writeup
Raw
pip_upgrade.rst

This is done now. We went with changing the default, using --upgrade-strategy as a transitory mechanism for users.

There's still a few missing references here.

Abstract

This document discusses the issues with pip's upgrade behaviour and proposes a resolution for the various issues.

Motivation

There has been a lot of discussion and half-done work related to modifying pip to add an upgrade command to pip. Some of these discussions are spread across various Github Issues and Pull Request and even buried in Mailing List Archives. This document is the result of an attempt to assimilate all the ideas, proposals and problems discussed in all these places into one place.

Definitions

It will be helpful to establish a minimum vocabulary to simplify the reading and discussion of this document. This section defines certain terms for the scope of this document.

Upgrade Strategy

Logic used to determine which packages should be upgraded.

There are a few types of upgrade strategies that are repeatedly mentioned/talked about in this Post, it makes sense to explicitly define them as well.

Recursive

Recursively apply the upgrade strategy to the dependencies of the new parent package. A recursive upgrade must also be one of Eager or Non-Eager.

Eager Recursive

Upgrade dependencies regardless of whether they currently satisfy the new parent version-specification.

Non-Eager Recursive

Upgrade dependencies if and only if they do not satisfy the new parent version-specification.

Non-Recursive

Ignore dependencies and do not upgrade them.

The phrase "<upgrade strategy type> upgrade" is equivalent to the phrase "upgrade using <upgrade strategy type> upgrade strategy".

Do note that the parent package is always upgraded. This is because that is the intent of running the upgrade.

Current State of Affairs

pip currently has only one way to upgrade a package, install --upgrade. This command-option pair follows an eager recursive upgrade strategy1.

pip handles multiple version-specifiers for a single package in a "first found wins" manner, the first version-specifier is acknowledged and latter version-specifiers are ignored.2

How it is flawed

The eager recursive strategy followed by install --upgrade also uninstalls a package which currently matches the requirement of the new parent package, to install the latest version that matches the version-specifications. While it is useful on some occasions but this can cause unexpected behaviour on many occasions3.

Eager updating may result in unnecessary upgrading of packages45. This updating may even be undesirable in some cases (eg. some packages may need extra configuration to work).

This issue is so problematic that various prominent scientific packages lie and don't list their dependencies if they are already installed, to (understandably) avoid a reinstall of such packages (scipy setup.py, statsmodels setup.py).

pip ignoring version-specifiers along with defaulting to eager recursive updating can easily result in breaking the user's environment. It is possible that installing/upgrading a package can result in an environment in which a package does not have a correct version of a dependency6. Breaking the environment is bad enough. Doing so silently is even worse.

Even if eager recursive upgrades may be useful in some cases, they can cause surprising behaviour in other cases, which is undesirable. Thus, it does not serve as a good default upgrade strategy.

Fixing the behaviour

Non-eager recursive upgrades are the desired behaviour in most cases. There is consensus that it should be the default upgrade strategy78. It has been expressed that providing an opt-in to eager recursive upgrade is also a required.9. The packages that lie about their dependencies have to be given an incentive to stop lying.

While the requirements have been decided, the changes to be made to user-side interface are still undecided. There are multiple options about what the exact interface should be for exposing pip's new upgrade strategy. These are listed below.

Fix the default

There is a counter proposal to the upgrade command, to change the behaviour of the "--upgrade" switch of pip to be non-eager recursive. 1 new option or 2 new flags would be added to pip, for choosing the upgrade strategy - eager or non-eager.

Pros

  • Causes minimal changes to the interface, keep the same UX.
  • It "fixes" the broken default.

Cons

  • Backwards Incompatible
  • Involves a warning period informing users of the change.

Undecided

  • Exact message warning
  • Exact option spelling
  • How would the warning message be suppressed?

This proposal seems to have been "vetoed" by a pip developer, in favour of backwards compatibility10.

Add a "pip upgrade" command

A proposal adds an "upgrade" command, that moves the upgrade-a-package UX to pip upgrade and defaults to non-eager recursive upgrade. It will have a switch for using the (current) eager recursive upgrade strategy. The current install --upgrade switch would be deprecated.

Pros

  • Backwards Compatible
  • A command for explicitly upgrading packages
  • Agreed to by pip developers

Cons

  • "upgrade" and "install" perform very similar tasks.
  • Exact semantics (and naming); may get complicated/confusing

Undecided

  • Exact option spelling
  • should "upgrade" install missing packages?
  • should "install" upgrade installed packages?

Make "pip install" upgrade by default

Another proposal is to move the upgrade-a-package UX to pip install to upgrade an already installed package, using the non-eager recursive upgrade strategy. This would make the --upgrade option redundant and thus would be removed. A switch would be added to enable non-eager recursive upgrade.

Pros

  • A short, quick-to-type UX
  • Consistent with OS-level package managers

Cons

  • Backwards Incompatible
  • It changes how one should think about pip's upgrading behaviour
  • No distinction between out-of-date and not-installed packages
  • No way to say "install this package unless already installed"

Undecided

  • Exact option spelling

Proposal

The author shall propose an interface after some discussion about this issue with the pip developers.

NOTE

  1. Normalize Pros-Cons to be more comprehensive. Help would be appreciated.
  2. Add more proposals if any shows up. Or if I missed any.

  1. https://pip.pypa.io/en/latest/user_guide/#only-if-needed-recursive-upgrade↩

  2. pypa/pip#174↩

  3. pypa/pip#59 (comment)↩

  4. pypa/pip#3216↩

  5. pypa/pip#3619↩

  6. pypa/pip#1337↩

  7. https://groups.google.com/forum/#!searchin/pypa-dev/pip$20upgrade/pypa-dev/vVLmo1PevTg/oBkHCPBLb9YJ↩

  8. pypa/pip#3194 (comment)↩

  9. pypa/pip#3194 (comment)↩

  10. NOTE: I couldn't locate any text/document/comment where they did so. I do see that @rgommers said so in a Github comment.↩

@pradyunsg
Copy link
Author

Please be kind. This is my first non-zero visibility contribution. 😇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment