Skip to content

Instantly share code, notes, and snippets.

@tiran
Last active Sep 13, 2021
Embed
What would you like to do?
Negative Python user experience on Debian/Ubuntu

Negative Python user experience on Debian/Ubuntu

The user experience of Python on a minimal Debian or Ubuntu installation is bad. Core features like virtual environments, pip bootstrapping, and the ssl module are either missing or do not work like designed and documented. Some Python core developers including me are worried and consider Debian/Ubuntu's packaging harmful for Python's reputation and branding. Users don't get what they expect.

Reproducer

The problems can be easily reproduced with official Debian and Ubuntu containers in Docker or Podman. Debian Stable (Debian 10 Buster) comes with Python 3.7.3. Ubuntu Focal (20.04 LTS) has Python 3.8.5.

Run Debian container

$ docker run -ti debian:stable

Run Ubuntu container

$ docker run -ti ubuntu:focal

Install Python3

# apt update
# apt install python3

venv is broken

venv is another Python standard library module. It provides support for creating lightweight "virtual environments". The venv module is available but dysfunctional. It cannot create virtual environments out of the box.

# python3 -m venv /tmp/venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/tmp/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']

Update Julien Palard wrote that one of his students ran into another issue with venv. Debian's venv can give an invalid advise when a user has multiple Python versions installed.

ensurepip is missing

The ensurepip package is part of Python's standard library and provides support for bootstrapping the pip installer into an existing Python installation or virtual environment. The ensurepip package is missing on Debian/Ubuntu.

# python3 -m ensurepip
/usr/bin/python3: No module named ensurepip
# pip
bash: pip: command not found

After installation of python3-venv, the ensurepip package is failing with a different error message:

# python3 -m ensurepip
ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.
# echo $?
1

distutils is stripped down and missing most code

The distutils package is mostly missing. Only the package root and distutils.version is available. The remaining code has been moved to python3-distutils by Debian/Ubuntu packagers. The python3-distutils is not installed with python3 and must be installed separately.

# python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)  
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> from distutils import sysconfig
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: cannot import name 'sysconfig' from 'distutils' (/usr/lib/python3.7/distutils/__init__.py)

ssl module cannot verify connections

A minimal installation has no CA certificates because neither the python3 package nor OpenSSL libraries depend on ca-certificates.

>>> import urllib.request
>>> urllib.request.urlopen("https://pypi.org/")
Traceback (most recent call last):
 ...
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
   return opener.open(url, data, timeout)
 File "/usr/lib/python3.7/urllib/request.py", line 525, in open
   response = self._open(req, data)
 File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
   '_open', req)
 File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
   result = func(*args)
 File "/usr/lib/python3.7/urllib/request.py", line 1367, in https_open
   context=self._context, check_hostname=self._check_hostname)
 File "/usr/lib/python3.7/urllib/request.py", line 1326, in do_open
   raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

Incompatible OpenSSL downstream patch

Debian/Ubuntu have applied downstream patches to OpenSSL. The patches have caused breakage of user applications or Python's CI tests. Examples for issues and workarounds:

lib2to3 is missing

The lib2to3 package is moved to python3-lib2to3 package, which is not installed by default.

tkinter is in an extra package (ok)

The tkinter package is not part of the default distribution. For once this is a good decision. tkinter depends on libtk and whole lot of X11 libraries. Graphical user interface libraries should not be installed by default on headless servers and containers. I just find it confusing that the tkinter package is provided by a python3-tk package and not by python3-tkinter.

Python 3.9 is missing dependency on tzdata

Paul Ganssle added a zoneinfo implementation with timezons to Python 3.9, see PEP 615. The feature requires tzdata database. As of 2020-11-13 Debian and Ubuntu's python3.9 package are missing a dependency on the tzdata package. The zoneinfo module does not work without tzdata:

>>> import zoneinfo
>>> zoneinfo.available_timezones()
set()
>>> zoneinfo.ZoneInfo("CET")
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/zoneinfo/_common.py", line 24, in load_tzdata
    raise ZoneInfoNotFoundError(f"No time zone found with key {key}")
zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key CET'

NOTE The issue has been fixed by Anthony Sottile in Deadsnakes PPA, see comment.

UPDATE My launchpad bug 1904271 was closed as Invalid. Matthias wrote that tzdata is a required package and pointed to Debian policy. However the package is not installed by default in the official Debian and Ubuntu container images.

New virtualenvs contain unwanted libraries

Virtualenvs contain de-vendored dependencies of pip and setuptools, https://bugs.launchpad.net/ubuntu/+source/python-virtualenv/+bug/1904945

Expectations and Proposal

Minimalization of Python installation is a legitimate effort. However a minimal installation of Python with core features missing should not be called a Python installation. Users should expect that package-manager install python3 gets them a working Python interpreter with majority of stdlib packages (with exception to tkinter GUI and test package).

I propose

  1. Debian's current minimized Python package python3 should rather be called python3-minimal or something similar. This package would still users to get a stripped down interpreter if they explicitly ask for it.
  2. apt install python3 should provide a Python installation with working venv, ensurepip (*), distutils, and ssl modules.

(*) I define working ensurepip as python3 -m ensurepip does not fail and python3 -m pip works afterwards. It does not imply that stdlib's pip bundle must be shipped with Python distribution package. Debian could also provide an API compatible ensurepip facade and make python3 package depend on python3-pip.

@JulienPalard

This comment has been minimized.

Copy link

@JulienPalard JulienPalard commented Nov 3, 2020

Long time Debian user here. I concur with Christian here: I'd love to see Debian packaging venv and ensurepip back with python3.

I remember searching why it's like it, and I remember I stumbled upon https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816740 stating that venv and ensurepip are dev tools that should not be packaged in the main package.

But I do not agree, some people are using venv in production, and some people are using Debian / Ubuntu to develop, both need the "standard" standard library, I don't know if it's the only argument, but I'd like to see it reevaluated, or get more feedback from Debian packagers on it.

I'm unsure about tkinter, I understand it's splitted to avoid a huge dependency tree (python3-tk would add some fonts, libtk, libtcl, libx11, ...), but if we go for the separation between python3 and python3-minimal, I bet a user would expect a full Python in the python3 package, with tkinter, but I'm no packaging expert and I don't have a clear opinion on tkinter.

@webknjaz

This comment has been minimized.

Copy link

@webknjaz webknjaz commented Nov 4, 2020

Another problem that's been preventing us from adding focal to CI: ansible/ansible#69203.

@sfermigier

This comment has been minimized.

Copy link

@sfermigier sfermigier commented Nov 4, 2020

Long time Ubuntu and Python user, and I'm still bitten by the issues raised by @tiran each time I have to install a new server.

Of course pip and venv are production tools, at least in my company.

@whitequark

This comment has been minimized.

Copy link

@whitequark whitequark commented Nov 5, 2020

Not entirely sure if it's Debian-specific or not (maybe pypa/pip#7953 is related?) but if I create a pyproject.toml file in my setup.py based project, I can no longer use pip install -e . or pip install --user -e . with this error:

Installing collected packages: nmigen
  Running setup.py develop for nmigen
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/whitequark/Projects/nmigen/setup.py'"'"'; __file__='"'"'/home/whitequark/Projects/nmigen/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
         cwd: /home/whitequark/Projects/nmigen/
    Complete output (28 lines):
    running develop
    WARNING: The user site-packages directory is disabled.
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied: '/usr/local/lib/python3.7/dist-packages/test-easy-install-81031.write-test'
@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Nov 5, 2020

Debian patches Python to add additional installation layout schemata. The default site-packages directory is /usr/lib/python3.7/site-packages. Notice that your error message refers to a dist-packages directory in /usr/local: /usr/local/lib/python3.7/dist-packages

distutils-install-layout.diff and sysconfig-debian-schemes.diff might interfere with develop mode and break it.

@whitequark

This comment has been minimized.

Copy link

@whitequark whitequark commented Nov 5, 2020

This leaves me with a choice of:

  • not using editable installs (at least on Debian),
  • not using pyproject.toml,
  • stuffing import site; site.ENABLE_USER_SITE = True into my setup.py.
    Right?
@JulienPalard

This comment has been minimized.

Copy link

@JulienPalard JulienPalard commented Nov 6, 2020

@whitequark:

This leaves me with a choice of:

* not using editable installs (at least on Debian),

* not using `pyproject.toml`,

* stuffing `import site; site.ENABLE_USER_SITE = True` into my `setup.py`.
  Right?

I use pyproject.toml and editable installs on Debian daily, trick is to not use the Debian packaged Python to develop or create venvs. I think that when I develop a module I need to test it with different Python versions, so I compile some of them instead of using the single one provided by Debian.

@whitequark

This comment has been minimized.

Copy link

@whitequark whitequark commented Nov 6, 2020

trick is to not use the Debian packaged Python to develop or create venvs

True, this is a fourth choice, effectively amounting to not using Debian (at least in the context of Python). Unfortunately it's not really something I can ask downstream users of my libraries do.

@webknjaz

This comment has been minimized.

Copy link

@webknjaz webknjaz commented Nov 6, 2020

@whitequark there's pyenv that is useful for having many Pythons in your userspace for development (that are obviously isolated from the system one). And also, there's deadsnakes that are packaged by @asottile and usually provide a better UX. OTOH I understand why the downstream is important but why would they use editable installs in the first place? Normal packaging flow doesn't need editables, they are for development...

@whitequark

This comment has been minimized.

Copy link

@whitequark whitequark commented Nov 6, 2020

OTOH I understand why the downstream is important but why would they use editable installs in the first place? Normal packaging flow doesn't need editables, they are for development...

For several reasons:

  • The project I maintain integrates with a large amount of complex third-party software and hardware. Anyone who is using it for serious work is inevitably going to need to improve these integrations (which are kept in-tree partly because they are tightly coupled to a core component of the project, and partly to build a centralized repository everyone benefits from), so quite a few users are involved in development.
  • It is also a fairly young project, so it is common for people to develop against a regularly updated git checkout rather than latest release. (Also, everyone who had to improve the integrations I mentioned above is stuck with a git version until the next release.) While this is not necessarily a good idea in general, we spend significant effort to make it tractable in practice--e.g. through time-based deprecation cycles for unreleased but nevertheless used features that turned out to have a major design flaw.
@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Nov 13, 2020

I have updated the text with tzdata issue.

@JulienPalard

This comment has been minimized.

Copy link

@JulienPalard JulienPalard commented Nov 20, 2020

In this serie, today I had a student (remotely so I gathered just what's needed, I do not have everything) that got bitten by it, tl;dr:

  • python3 -m venv .venv → Give the message telling to apt install python3-venv
  • sudo pip install python3-venv → already installed # student lost

(https://wyz.fr/1A-ON)

So I bet the student has its Ubuntu in a strange setup : it looks like it have python3 pointing to a Python 3.8 installed from the the repositories (thus printing "apt install python3-venv"), but python3-venv installed things for Python 3.6, so the 3.8 don't see them and whine again about missing python3-venv. I sadly have no more info about why and how the machine is in that state, we won't be able to debug it, I temporarily "fixed" the student issue by making him use Python 3.6 for the day, so we can continue without lossing too much time on it.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Nov 20, 2020

Thanks @JulienPalard! I updated the text with your issue and with recent news about tzdata issue. My Launchpad bug https://bugs.launchpad.net/bugs/1904271 was closed as invalid.

@nitzmahone

This comment has been minimized.

Copy link

@nitzmahone nitzmahone commented Dec 3, 2020

@tiran thanks for putting this list together! The Ansible team is definitely weary of figuring/documenting/working around "what novel way is Python busted this time?" after every new Debian/Ubuntu release- both for ourselves and our users, so we're glad it's not just us. We've also struggled trying to get traction on bugs (and even backports of fixes), though I think they've accepted at least one of our proposed backports recently in 20.04. Let me know if there's any effort where we can lend our voices or hands.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Dec 10, 2020

@hynek ran into an issue with latest virtualenv on Bionic:

Ubuntu Bionic + latest virtualenv + deadsnakes’s Python 3.9, NEEDS python3.9-distutils, otherwise you’ll get a misleading RuntimeError: failed to find interpreter for Builtin discover of python_spec=‘3.9’

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Dec 27, 2020

A user on the Python mailing list encountered the apt-get install python3-venv issue today, https://mail.python.org/pipermail/python-list/2020-December/900072.html

@flowerbug

This comment has been minimized.

Copy link

@flowerbug flowerbug commented Dec 28, 2020

A user on the Python mailing list encountered the apt-get install python3-venv issue today, https://mail.python.org/pipermail/python-list/2020-December/900072.html

Please note that this was me, it was on Debian testing and an issue with the packages not being in sync yet and brought in from unstable. I do not know why and I've not filed a bug with Debian as it might still be a part of the transition to python3.9. Thanks! :)

When running testing/unstable it is acceptable to have issues at times, so I'm ok with that, but I also like to provide a heads-up for those using them so they know what is going on. That is precisely why I run testing and unstable so I can catch things before they go further.

@pradyunsg

This comment has been minimized.

Copy link

@pradyunsg pradyunsg commented Dec 30, 2020

This came up again for me today -- what's our next steps here? Who/where at Debian do we reach out to for having this discussion move forward in some way?

@brettcannon

This comment has been minimized.

Copy link

@brettcannon brettcannon commented Jan 4, 2021

My plan, if I ever document this sort of thing, is to point at deadsnakes for Debian/Ubuntu and explicitly advise against installing the system package (having lived through this with @amcinnes87; it was not entirely smooth and included a lot of questions of "why?!?").

@hynek

This comment has been minimized.

Copy link

@hynek hynek commented Jan 5, 2021

I don't think deadsnakes solves any of those problems and there's a bit of extra problems. For instance there's not deadsnakes for Python 3.8 on focal due to the complexities involved.

What you'd need is someone pyenving and packaging the Pythons to /opt/python/3.9 etc. deadsnakes will always reflect debian's policy for better or for worse.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Jan 5, 2021

I don't think deadsnakes solves any of those problems and there's a bit of extra problems. For instance there's not deadsnakes for Python 3.8 on focal due to the complexities involved.

What Hynek said! Deadsnakes provides more Python versions additional to system Python. From the PPA:

Note: Python2.7 (all), Python 3.5 (xenial), Python 3.6 (bionic), Python 3.8 (focal) are not provided by deadsnakes as upstream ubuntu provides those packages.

The deadsnakes packages are basically forward and backward ports of Debian packages. The packages are split up the same way and have almost the same downstream patches. There are very few minor deviations, e.g. tzdata fix for 3.9.

@brettcannon

This comment has been minimized.

Copy link

@brettcannon brettcannon commented Jan 5, 2021

What we really want, I think, is relocatable manylinux builds of Python that can be downloaded and unpacked from a zip file or something.

@pradyunsg

This comment has been minimized.

Copy link

@pradyunsg pradyunsg commented Jan 5, 2021

IMO, that'll solve a different problem with some overlap with this one.

What I really want is that Debian stops meddling with the Python interpreter + pip, to the point of making every wider discussion about how to get set up / Python packaging aimed at beginners, have a giant asterisk "because Debian does X differently". It'll also move us closer to having a consistent story across the various Linux distributions', which is a good thing.

I'm happy to work with them, but it can't be that they're not accomodating at all in their approach and only expect upstream to make the changes they want. We do constantly deal with users having a (IMO) degraded experience due to their choices - this has support, technical design and perception costs for us.

To be clear, I also come into this as someone who agrees with the opinion that "if Debian doesn't want to improve this situation, they should stop calling what they ship Python". I also do respect that there's a lot of pieces at play in a distro like Debian, but I don't particularly like that the most popular Linux distro has such a horrible Python story.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 18, 2021

Debian Python person here. Someone alerted me to this thread. I am not the cpython package maintainer, but I know the package well and maintain the PyPy packages.

I think the best way to improve the situation is to discuss why the changes were made, and try to find solutions that work for all parties. Some of these are choices made with different constraints, some are gnarly issues with no simple solution. Fundamentally, I think the divide is between a "system python" providing a python runtime for python applications packaged in Debian, and providing a good python developer experience. A few stdlib modules were broken out into separate packages to avoid bringing in their dependencies unless needed (e.g. Tcl, Tk, CA certificates, pip wheels). And a few things are de-vendored to avoid duplicating code in other Debian packages.

venv is broken

This is because the venv module has dependencies on pip wheels. It should just work if the user follows the instructions and installs python3-venv. If it doesn't, that's a bug. The issue @flowerbug linked seems to be something about creating venvs from a venv, but I can't reproduce that here.

ensurepip is missing

pip is packaged as python3-pip. We wouldn't want the ensurepip module to do anything other than install that package, or it would be installing files that would not be package managed. We could make ensurepip run apt install python3-pip, but instead we print a message explaining to the user how to do that.

distutils is stripped down and missing most code

I don't know for certain, but I'm assuming this is to reduce the size of the stdlib and reduce duplication. Distutils isn't really needed in the python runtime, only when building packages.

ssl module cannot verify connections

Packages are expected to depend on ca-certificates, if they require them.

Incompatible OpenSSL downstream patch

This seems unrelated to this discussion. Just a gripe at Debian.

lib2to3 is missing

Presume to reduce the size of the stdlib. Only needed by developers porting Python code.

tkinter is in an extra package (ok)

Reduce the size of the stdlib. Most python users will never need the TK module and its dependencies, so it's a separate package to avoid pulling them in.

Python 3.9 is missing dependency on tzdata

I disagree with @doko42 there, the relevant policy §3.5 is about not depending on Essential: yes packages, not Priority: required packages.

The docker image should probably include tzdata, though. That sounds like a bug.

Debian's current minimized Python package python3 should rather be called python3-minimal
apt install python3 should provide a Python installation with working venv, ensurepip (*), distutils, and ssl modules

Debian already has a python3-minimal package. This was historically used in Ubuntu live CDs, to reduce the footprint of the interpreter and continue to fit on the CD. Ubuntu still makes an effort to keep the install DVD size down, slowly increasing the space budget every release.

So, either this would need to be increased to cover everything in the current python3 stdlib packages, at the cost of install image size, or we'd need a python3-system package. And every python library / app package would have to depend on that instead of python3.

It makes sense to me that the python3 package should provide the python that Python-developers expect. Making that change would impact every python-related package in the archive, though.

A lighter weight option would be to add a python3-full package that includes the entire stdlib. But then users need to be told to use that...

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Jan 19, 2021

Thanks for your feedback! I'll reply to your other answers in the next couple of days.

Incompatible OpenSSL downstream patch

This seems unrelated to this discussion. Just a gripe at Debian.

It's really not a gripe on Debian. On several occasions Debian's or Ubuntu's downstream patches of OpenSSL have caused breakage of the ssl module or Python's test suite. Some examples for issues and workarounds are:

ssl module cannot verify connections
Packages are expected to depend on ca-certificates, if they require them.

Yes, I agree. Python's ssl module requires ca-certificates. Debian's Python package is missing a dependency on ca-certificates package. I have opened Debian bug in May last year. The issues was cloned to Ubuntu.

For what it's worth ca-certificates should probably be an essential package or hard dependency of libssl.

@pganssle

This comment has been minimized.

Copy link

@pganssle pganssle commented Jan 19, 2021

NOTE: The following is my personal opinion, I am speaking as an individual and not on behalf of an organization like PSF, PyPA or Python core developers.

I think the best way to improve the situation is to discuss why the changes were made, and try to find solutions that work for all parties. Some of these are choices made with different constraints, some are gnarly issues with no simple solution. Fundamentally, I think the divide is between a "system python" providing a python runtime for python applications packaged in Debian, and providing a good python developer experience. A few stdlib modules were broken out into separate packages to avoid bringing in their dependencies unless needed (e.g. Tcl, Tk, CA certificates, pip wheels). And a few things are de-vendored to avoid duplicating code in other Debian packages.

I disagree with many of Debian's policy rationales, but at the end of the day, I don't think it's really the job of core development (either CPython or PyPA) to make decisions about Debian's packaging policies, which is why I'm not sure how productive it would be for us to rehash the rationales for why these changes were made. Please don't take it as dismissive to say that none of the rationales you've provided here are new to me, and I suspect they are not new to many of us on the CPython / PyPA side who have been unhappy with the status quo for some time now.

In my opinion, I think it is best for Python core to make clear what expectations we have for users — the ones we support and document — and leave it to Debian to decide how to satisfy those expectations. At the end of the day, Debian is providing a product called "Python" that end users associate with the PSF, the PyPA and Python core development. End users do not realize what parts of their experience are provided by Debian, which means that Debian's modifications reflect on Python, not on Debian. At the end of the day, I think that if Debian is going to continue to advertise that it provides Python, it should provide as close to an unmodified experience as possible. Where deviations are absolutely necessary because of peculiarities in Debian's system, they should be very clearly labeled as such. And to be clear, I would not consider "it makes the executable smaller" to be something that is an absolute necessity. If the executable is bloated, that will properly reflect on Python.

I don't think that this should be hugely controversial. The user experience on Fedora and Arch Linux at least are much better, and both distributions have many similar policy considerations and preferences.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

On several occasions Debian's or Ubuntu's downstream patches of OpenSSL have caused breakage of the ssl module or Python's test suite

To some degree that's to be expected, though. Distros differ and are going to do things that are going to break upstream test suites. Handling these quirks in tests seems about the best you can do. Yes, the SSL >= 1.2 policy is rather heavy handed, but OpenSSL doesn't provide a whole lot of options for lighter-weight distro policies.

For what it's worth ca-certificates should probably be an essential package or hard dependency of libssl.

Yeah, that'd be reasonable. In the modern world, it's pretty essential. Getting packages added to Essential is hard, this will probably require some project-level discussion.

Generally Debian tries to remove things from essential rather than add to them. And historically Debian has tried to avoid admitting subservince to the CA cartel. So, there are political minefields there.

A developer-focussed python package that depends on ca-certificates is by far the easier option.

Please don't take it as dismissive to say that none of the rationales you've provided here are new to me, and I suspect they are not new to many of us on the CPython / PyPA side who have been unhappy with the status quo for some time now.

The Good to hear. The document reads as if the author has no insight into any of the reasons behind these changes.

As somebody who maintains some PyPA packages in Debian, but doesn't follow upstream development much, the unhappyness is new to me. I thought we had a fairly good working relationship. I know there's a movement towards having a "system python" that's different to the development Python, but I don't tend to think about that a whole lot...

In my opinion, I think it is best for Python core to make clear what expectations we have for users — the ones we support and document — and leave it to Debian to decide how to satisfy those expectations.

Not particularly convinced about that, it sounds like declaring an ultimatum and stirring a fight rather than having a productive discussion. Debian isn't new to upstreams flexing their muscles like this, Firefox for example. I don't think the outcome of that was best for the upstream brand or Debian's brand. Python is so core to the system that removal is not an option. Compromises have to be found.

The only way things work out is if the two sides understand each other's needs and agree to work towards the common aim. That change is going to require work.

At the end of the day, Debian is providing a product called "Python" that end users associate with the PSF, the PyPA and Python core development. End users do not realize what parts of their experience are provided by Debian, which means that Debian's modifications reflect on Python, not on Debian. At the end of the day, I think that if Debian is going to continue to advertise that it provides Python, it should provide as close to an unmodified experience as possible.

There is often a tension in Debian, between packager and upstream. Distros often need to change the behaviour of packages. Usually in very minimal ways, and usually this can be forwarded back upstream. But not always. At least, the upstream should understand what's being changed, and why.

IMHO the upstream should be happy with the way their work is being packaged. They may have to do some work to accommodate distros, of course.

Where deviations are absolutely necessary because of peculiarities in Debian's system, they should be very clearly labeled as such.

I'm afraid that's not always practical. There usually isn't anywhere visible to put these labels. They end up in descriptions of patches, comments in source code, and README files buried in /usr/share/doc...

And to be clear, I would not consider "it makes the executable smaller" to be something that is an absolute necessity. If the executable is bloated, that will properly reflect on Python.

I'm not speaking about the executeable size, but rather the number and size of packages getting pulled in when someone install something. Installing a little utility that's written in python doesn't need to bring in Tk. There are some bits of the stdlib that can be split out, and save end-users a lot of downloads. In that scenario, it reflects poorly on the utility and on Debian, not Python. Python is an implementation-detail. Batteries-included has downsides...

I don't think that this should be hugely controversial. The user experience on Fedora and Arch Linux at least are much better, and both distributions have many similar policy considerations and preferences.

This is one of those times I wish there was more cross-distro collaboration. I know very little about how they handle python...

@pganssle

This comment has been minimized.

Copy link

@pganssle pganssle commented Jan 19, 2021

Not particularly convinced about that, it sounds like declaring an ultimatum and stirring a fight rather than having a productive discussion.

I am not saying that Python core cannot be helpful, and I think in the long run the right way to do things is for Debian to make the case upstream for their more radical changes. Frankly, you will get very little disagreement from at least me and presumably from Christian about at least some of the pruning of the standard library. We likely all have the same goals in mind and we are no strangers to the tension between distributions and upstream.

That said, I am not sure that Python core has the right combination of time, resources and expertise to take on the task of fixing Debian specifically. My point was that I think it is more feasible for us to explain the damage that Debian is doing to Python's reputation with its policies and ask you, the experts, to please find a solution, and to make the case for the features you would like supported upstream.

The fact that your response indicates that you don't really follow what goes on upstream, that you were unaware of these problems that we've been complaining about for years, and that you know very little about how other distros handle Python makes me think that I might be accurate in saying that Debian maintainers could probably do more to explore their part of the solution space. I do not want to put you on blast for admitting ignorance — I think it is a very admirable trait and one I try to live up to — I just want to point out that possibly with a more open and active communication strategy some of these problems may have been avoided.

Finally, I should clarify that yes, I think that this is the first step down the road to PSF issuing an ultimatum with legal force saying, "Fix the problems with your distro or rename it." (Reminder: These are my personal opinions, I have no authority to speak on behalf of the PSF), but really I want to open your eyes to the fact that this is something you should take seriously. We've been raising these concerns for years to no avail. Obviously we would prefer a situation where Debian has a great Python story. There are many people motivated to make changes to CPython to make that easier for Debian. But at the end of the day Debian has to recognize that this is a problem and be interested and willing to fix it.

I am sorry if any of this comes off as unnecessarily hostile. You sound like you are just wading into this, and you may not realize that there is a long and antagonistic history between some Debian maintainers and a large number of upstream maintainers. Unfortunately, our cynicism is borne of experience .

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

Yeah, I've about reached the limit of how useful I can be in this thread. I'm not the CPython maintainer in Debian, and Debian has strong maintainer-ownership policies. All I can do is provide backstory and try to facilitate discussion. I can bring this thread to the attention of the debian-python mailing list.

The fact that your response indicates that you don't really follow what goes on upstream, that you were unaware of these problems that we've been complaining about for years,

I've definitely heard complaints about how Debian patches Python over the years, but not these specific issues. Mostly dist-packages, etc. I think we tend to become rather immune to those complaints, because most of the time the complainer doesn't understand the context for the changes.

and that you know very little about how other distros handle Python makes me think that I might be accurate in saying that Debian maintainers could probably do more to explore their part of the solution space

I think most distro-people are deeply knowledgeable about a specific distro (and possibly derivatives of it). I can't think of anyone seriously involved in more than one family of distros. I'm sure there are many issues out there that can be attributed to this. :(

You're a huge project being represented by a single maintainer. (Who has this as this his day job, but also has significant other responsibilities too.) I think the biggest issue here is going to be motivation for the work required. I hope that legal force isn't the best answer there :/

@ehashman

This comment has been minimized.

Copy link

@ehashman ehashman commented Jan 19, 2021

Speaking on behalf of myself and NOT on behalf of the Debian Technical Committee in this post.

I'm a longtime system Python on Debian user and I haven't run into essentially any of the problems above, but I am a relatively sophisticated user who knows the internals of Python and Debian packaging and is aware of the core split of Python. It's not reasonable to assume that every Python user on Debian would be as knowledgeable as me. I have also built CPython both from upstream source and the Debian packages.

@stefanor wrote:

venv is broken

It should just work if the user follows the instructions and installs python3-venv.

ensurepip is missing

pip is packaged as python3-pip.

distutils is stripped down and missing most code

I don't know for certain, but I'm assuming this is to reduce the size of the stdlib and reduce duplication. Distutils isn't really needed in the python runtime, only when building packages.

Is distutils now replacing setuptools? I think it's a really bad idea to bundle non-core distribution functionality into the core language. There should be a way to separate this out.

ssl module cannot verify connections

Packages are expected to depend on ca-certificates, if they require them.

lib2to3 is missing

Presume to reduce the size of the stdlib. Only needed by developers porting Python code.

Given that Python 2.x has been deprecated for over a year now, this seems like a non-core concern, but I think this is also split out into a separate package.

tkinter is in an extra package (ok)

Reduce the size of the stdlib. Most python users will never need the TK module and its dependencies, so it's a separate package to avoid pulling them in.

Python 3.9 is missing dependency on tzdata

The docker image should probably include tzdata, though. That sounds like a bug.

The docker images for Ubuntu especially have been pretty buggy in my experience, especially around tzdata issues. I don't know who maintains them but this is managed by a separate group.

Debian's current minimized Python package python3 should rather be called python3-minimal
apt install python3 should provide a Python installation with working venv, ensurepip (*), distutils, and ssl modules

Debian already has a python3-minimal package. This was historically used in Ubuntu live CDs, to reduce the footprint of the interpreter and continue to fit on the CD. Ubuntu still makes an effort to keep the install DVD size down, slowly increasing the space budget every release.
...
A lighter weight option would be to add a python3-full package that includes the entire stdlib. But then users need to be told to use that...

This is what I'd suggest (and initially suggested on IRC). It'd solve most of the above problems, with the exception of the openssl concerns and the issue limited to docker. For the ensurepip issue, based on the comments above there are also probably some genuine bugs in the patch there. We should fix those, but they'd likely be avoided in most cases by a metapackage. I am a little worried about the distutils issue but for the rest it seems like an easy win.

Adding a python3-full metapackage is precedented (e.g. latex-live-full) and could happen before bullseye freeze? We have another month of NEW packages, I think (until Feb. 16) and there's no real copyright review required. Then it would just be a matter of educating beginner users to use this. I don't know if it makes sense for this to be the default Python installed, because plenty of Debian users don't use Python at all.


@pganssle wrote:

I think that this is the first step down the road to PSF issuing an ultimatum with legal force saying, "Fix the problems with your distro or rename it." (Reminder: These are my personal opinions, I have no authority to speak on behalf of the PSF) ... I am sorry if any of this comes off as unnecessarily hostile.

It does, and it is unnecessary. In all the time I've been contributing to the Debian project and Python no one from Python has attempted to work with Debian collaboratively on this. All I've seen are issues raised piecemeal. Core Python needs to make an effort to understand the problems the distro needs to solve, which are not the same problems that CPython or even PyPA need to solve, and Debian needs to work in the best interest of our users, who are not being served by the status quo.

Jumping to legal threats (when you don't even have the authority to be making them) is heavy-handed and should be your last resort. As a PSF donor, I think that engaging in legal action against other FOSS projects when it won't even solve the technical issues at hand is a massive waste of the organization's funds. How do you collaborate with someone when on your first interaction you're threatened with legal action?

It's particularly concerning when CPython doesn't offer prebuilt Python binaries for Linux and is already offloading all that effort to distros. It's one thing to say "don't use system Python, use upstream"; it's another entirely to say "you have to build it yourself, but not like that, and we're not willing to work with you: it must be 100% on our terms or we'll apply legal force." None of this serves end users, and I encourage you to remove this comment as it's needlessly inflammatory.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Jan 19, 2021

To some degree that's to be expected, though. Distros differ and are going to do things that are going to break upstream test suites. Handling these quirks in tests seems about the best you can do. Yes, the SSL >= 1.2 policy is rather heavy handed, but OpenSSL doesn't provide a whole lot of options for lighter-weight distro policies.

OpenSSL does provide other means for crypto policies that do not cause issue. Debian's dwonstream patch can cause internal error in OpenSSL. Python users have been complaining about the breakage. Fedora-based distributions have TLS 1.0 and 1.1 disabled in a different way that a) allows Python to detect the policy, and b) does not break user software.

A developer-focussed python package that depends on ca-certificates is by far the easier option.

This package should be called python / python3. For comparison curl, ruby, and php all pull in ca-certificates package.

stdlib packages like lib2to3, distutils, and venv are not optional component that are primarily used by developers. For example lib2to3 contains a parser that is used by 3rd parties. The black Python code reformatter is based on lib2to3. These days it ships a modified copy. 3rd party packages use helpers from distutils to parse version numbers, execute helper processes, and more. venv is a core piece of Python. Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.

I would be willing to accept and take a python3-full metapackage as an intermediate workaround. It's certainly better than doing nothing. It's a compromise and concession, because next Debian stable is going into freeze.

By the way I have only reported one of the issue to Debian, because I have not received any feedback on the first bug. I also find Debian's email-based bug reporting process tedious and user-unfriendly. And it's not like I'm new to Linux. I'm using Linux for almost 25 years and had used Debian-based distros on my primary desktop and production servers for about 15 years.

@pradyunsg

This comment has been minimized.

Copy link

@pradyunsg pradyunsg commented Jan 19, 2021

First off, thank you to @stefanor and @ehashman! Your inputs and perspectives are genuinely welcome and appreciated. ^.^


I'm gonna clarify some terms, just to make sure we've all got the same vocabulary here - it'll help to have a common baseline. I'm sure that many of us here are familiar with them, but these names are very similar/loaded and mean very different things sooooo it can't hurt to be redundant. 🤷🏻‍♂️

  • CPython: the reference implementation of Python, usually this is what someone is referring when they say Python
  • PyPy (pie-pie): an alternate implementation of Python
  • PyPI (pie-pee-aye): Python Package Index, i.e. https://pypi.org
  • PyPA: Python Packaging Authority, a bunch of volunteers who banded together to maintain tooling for distribution of software written in Python. pip and setuptools are a part of this.

I guess we're all mentioning this so I'll note that these are my personal opinions.


Is distutils now replacing setuptools?

No. It's the other way round. :)

I think it's a really bad idea to bundle non-core distribution functionality into the core language. There should be a way to separate this out.

I think many of us here agree (and there's a PEP for removing distutils from the standard library as well, where we have involved @doko42 in the discussions). But, as it stands, distutils is a part of the Python standard library and splitting it out in the manner that Debian has, does result in a degraded experience for Python-on-Debian users.

IMHO the upstream should be happy with the way their work is being packaged. They may have to do some work to accommodate distros, of course.

Well, if the general tone of this discussion is anything to go by -- upstream is not happy.

I think a large part of the frustration here is that many Python-on-Debian bug reports/suggestions for improvements have been seemingly stonewalled, with pointers to Debian policy without sufficient explanation. That leaves us with a strong sense of "this won't be fixed unless someone is forced to make changes", rather than a sense that downstream wants to collaborate with us to make changes and improvements.

If I were to attempt to summarise the situation in my own words: Debian is making changes that are incompatible with how upstream Python/PyPA tooling work, and Python-on-Debian users are left with a sour taste for "Python" / "pip" / "setuptools" (not Debian/Ubuntu) when those tools fail to work on Debian.

I am very much not trying to point fingers, but I've now seen multiple individuals involved in some capacity with Debian [2] express a sense that the current situation with Python is "okay" or even "good" somehow. If that's really the general "feel" of the situation for the folks involved with Debian, I find it very concerning: I've not met/read/heard-of any Python developer who found the Python-on-Debian experience to be a generally positive one, unless they were completely circumventing Debian's packaging.


I guess this is a good place to note: I think any legal action by anyone should only be a last resort. That said, I also don't find it surprising that someone else would feel differently -- I think many upstream maintainers feel that we'd need to "force" Debian to make changes to how it's packaging/handling Python for any improvements to happen in the Python-on-Debian story.


I can probably elaborate as an upstream maintainer of pip -- I'm genuinely unhappy at the situation because the package is being modified in ways that causes a degraded user experience, additional support overhead for me and my fellow maintainers and contributes significantly to a negative perception of the package.

As I'm sure many of us here are aware - when users have a degraded experience, they don't go to their Linux distributions' issue tracker but rather go to the upstream issue tracker (this is not just because it's significantly easier to do so, but that also doesn't help). And often, it doesn't really matter if it was because of certain decisions made downstream by their distro -- they'll be left with a sour taste for the upstream package/tool.

As an easy example, pypa/pip#5599 surfaced a lot of such users. We haven't named anyone in that issue, but IIRC most of the complaints about it were on Debian/Ubuntu. [1] Further, in that issue, our users have also called out various areas where we'd like to work with Debian maintainers to make improvements, but (AFAIK) we haven't been able to make any headway.


All that said, I don't think this is a situation where we want to be pointing fingers or whatever. I really do want us to have a better story for Python-on-Debian -- these are users of both Debian and Python, and both ecosystems would benefit from any effort to improve the overall "story" here.

To be a bit "sound bite"-y, I really want that the most popular Linux distro and the most popular programming language today work together well and have a good user experience.

[1] this might be a sampling bias (more users on those vs other distros), but the overall point still stands.
[2] bah, I don't like this phrasing because it's too "us vs them", but I can't come up with anything nicer.

@pganssle

This comment has been minimized.

Copy link

@pganssle pganssle commented Jan 19, 2021

I do not think that my continued participation in this discussion would be useful, so I'm going to bow out at this point, but I'll be happy to participate again when we get past the hostility and into the productive phase where we start determining what it means to be a Python distribution.

I would like to address one point, however, because I feel my position has been mischaracterized:

Jumping to legal threats (when you don't even have the authority to be making them) is heavy-handed and should be your last resort. As a PSF donor, I think that engaging in legal action against other FOSS projects when it won't even solve the technical issues at hand is a massive waste of the organization's funds. How do you collaborate with someone when on your first interaction you're threatened with legal action?

I tried to be very clear that I am not making any legal threats — in fact the section you quoted includes a parenthetical aside in the middle of the sentence to remind the reader that I have no authority to make such threats specifically to make it abundantly clear that no threats are being issued. The very paragraph you quoted is actually agreeing with you that legal threats are the last resort, just with a different emphasis. My point was that this is not our first interaction with Debian, and we are not the first maintainers to get burned out tilting at this particular windmill. It seems that Stefano even agrees with me that the biggest problem here is motivation:

You're a huge project being represented by a single maintainer. (Who has this as this his day job, but also has significant other responsibilities too.) I think the biggest issue here is going to be motivation for the work required. I hope that legal force isn't the best answer there :/

Right now, Debian has no motivation to fix this, because the brunt of the work and reputational damage is being borne by upstream maintainers. This is exactly what trademark law exists for — upstream has no authority in the Debian package and doesn't want that responsibility anyway, but upstream is directly affected by Debian's decisions. Contrary to your implication, this is not our first interaction with Debian, and we have not failed to understand Debian's position here.

If I may use an analogy to explain the position I'm taking here: imagine that Python and Debian are neighbors, and Python is complaining that Debian's trash can doesn't close all the way, so sometimes raccoons get into it, or the wind blows it open, so trash gets all over Python's lawn. Python has asked Debian to get a different trash can lid or put some bungee cords on it, and Debian has said, "Well, the garbage cans with strong lids take up a lot of space in my garage, and the bungee cords are hard for me to work with, and I haven't been terribly bothered by the garbage when it gets all over the place myself..." In this analogy, I would say that my position is like that of a third neighbor who points out that the city code does say that you must ensure that your trash doesn't get all over someone else's lawn. I am not trying to call the city on Debian for their trash-habits, I'm just saying that the city already made rules about this for a reason.

Yes, I think it would be a huge pain in everyone's asses and a waste of everyone's time to pursue any sort of legal action over this. No, I do not think we're anywhere near that point. But I do think it's important for Debian to recognize that they are doing real harm to Python's reputation and polluting our support channels. Even if we never involve lawyers and we all agree that we don't want it to get that far, it's not a bad idea to think of this in terms of trademarks, because that's a body of law that exists to mitigate this particular form of externality — both for producers and consumers. If we reach an impasse, it at least opens the possibility for Debian to do things like rename their fork of Python, or work to make users fully aware that their modifications are Debian-specific, and that users should not raise issues about it on upstream projects.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

OpenSSL does provide other means for crypto policies that do not cause issue

Speaking as somebody not intimately familiar with Fedora or OpenSSL, from what I can see, Fedora built these mechanisms in their patches. I asked the OpenSSL maintainer, he didn't recall the details of they're doing, offhand, either.

If there was a better mechanism Debian could be using, and arguments for that, it would make sense to describe that (in some technical detail) in a bug against the openssl package.

By the way I have only reported one of the issue to Debian, because I have not received any feedback on the first bug. I also find Debian's email-based bug reporting process tedious and user-unfriendly. And it's not like I'm new to Linux. I'm using Linux for almost 25 years and had used Debian-based distros on my primary desktop and production servers for about 15 years.

Yeah, you're not alone there. It's not user-friendly. We know about this and constantly debate about replacements. But nobody has stepped up to do it. And whoever does will have a lot of work ahead of them (technically, and politically). Volunteer projects aren't all rainbows and unicorns.

Yes, many bugs go unanswered for a long time. I'm responsible for ignoring bugs too. There are a lot of bugs, and we're all spread quite thin.

If you're not getting a response from serious bugs, I suggest escalating to private email with the package maintainer. Or finding them on IRC. Writing a manifesto of complaints is just likely to poison the pot more.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Jan 19, 2021

Speaking as somebody not intimately familiar with Fedora or OpenSSL, from what I can see, Fedora built these mechanisms in their patches. I asked the OpenSSL maintainer, he didn't recall the details of they're doing, offhand, either.

Fedora platforms use openssl.cnf to configure crypto policies like mininum protocol version. There is minor patch to an include an external policy file in openssl.cnf.

# /etc/pki/tls/openssl.cnf

openssl_conf = default_modules

[ default_modules ]
ssl_conf = ssl_module

[ ssl_module ]
system_default = crypto_policy

[ crypto_policy ]
.include = /etc/crypto-policies/back-ends/opensslcnf.config
# /etc/crypto-policies/back-ends/opensslcnf.config
...
MinProtocol = TLSv1.2
MaxProtocol = TLSv1.3
...

Yeah, you're not alone there. It's not user-friendly. We know about this and constantly debate about replacements. But nobody has stepped up to do it. And whoever does will have a lot of work ahead of them (technically, and politically). Volunteer projects aren't all rainbows and unicorns.

I know, I'm an unpaid volunteer for CPython core and Python's security team. In fact all core developers and PyPA developers are unpaid volunteers (with the exception of very few people who can spent a couple of hours work time on OSS).

Yes, many bugs go unanswered for a long time. I'm responsible for ignoring bugs too. There are a lot of bugs, and we're all spread quite thin.

I'm going to tell you one weird trick how you can reduce your workload: collaborate and work with us instead against us, follow our advise and try to get as many of Debian's downstream patches into upstream as possible. This will reduce both your and our workload and make our mutual users happy.

If you're not getting a response from serious bugs, I suggest escalating to private email with the package maintainer. Or finding them on IRC. Writing a manifesto of complaints is just likely to poison the pot more.

Your recommendation doesn't scale. CPython core lacks resources to chase down and poke distro maintainers. Anyhow I tried that approach for my ssl / ca-certificate bug already. I left follow-up pings on the Launchpad bug, talked with doko on IRC (last time in November) as well as the maintainer of Deadsnakes packages, and even twitted at people. It was even picked up by the Bulletproof TLS Newsletter. Today the bug is still unprocessed. The fix is a trivial one-liner like in Debian's Ruby, PHP, and curl packages.

@brettcannon

This comment has been minimized.

Copy link

@brettcannon brettcannon commented Jan 19, 2021

Apologies for pulling a management-like move, but if someone from the Debian side can specifically tell me who the Python steering council should reach out to in order to see if there's a way to resolve this that would be much appreciated.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

Fedora platforms use openssl.cnf to configure crypto policies like mininum protocol version

As does Debian.

As I said before, I suggest discussing this in an alternative thread, it's not related to your desire for a better python developer experience.

I'm going to tell you one weird trick how you can reduce your workload: collaborate and work with us instead against us

Ah, you've come around to my PoV :)

Your recommendation doesn't scale. CPython core lacks resources to chase down and poke distro maintainers.

And Debian's Python people don't have the resources to follow everything going on in CPython. Somewhere in the middle lies the answer.

Thanks for filing bugs. That's good. But please don't stop if they aren't all being dealt with. Bugs often go unanswered, in all projects. IIRC some of Debian's cpython patches have been forwarded to the python bugtracker decades ago, with little traction. It usually takes motivation and effort to push change through.

Looking back at the list of complaints in this gist:

  1. venv is broken: I don't really see the issue there. The error message clearly explains how to get a functional venv. Would be improved by a python3-full package.
  2. ensurepip is missing: I don't think we can ever have an an ensurepip that does more than noop.
  3. distutils is stripped. Would be improved by a python3-full package.
  4. ssl can't verify. Would be improved by a python3-full package.
  5. Incompatible SSL patches. Can be resolved by workarounds in python test suite.
  6. lib2to3 is missing. Would be improved by a python3-full package.
  7. tkinter is an extra package. Would be improved by a python3-full package.
  8. Python 3.9 is missing adependency on tzdata. Was resolved in 3.9.1~rc1-1.

1, 3, 4, 6, 7 are all the same issue in that they are the result of breaking up the stdlib, and would all be improved by a python3-full package. I think we can get that to happen

I still don't see a systemic issue here. I see a handful of bugs, and some difficulty in getting them resolved. This is not insurmountable.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

Apologies for pulling a management-like move, but if someone from the Debian side can specifically tell me who the Python steering council should reach out to in order to see if there's a way to resolve this that would be much appreciated.

Your first call is the maintainers of the packages in question. @doko42 for python, @kroeckx for OpenSSL.

There are thousands of python-related packages in Debian, so there are teams of people who have a stake in this stuff. So, if you hit a brick wall with the maintainers, I suggest a mail to the python team mailing list: https://lists.debian.org/debian-python/ or the IRC channel #debian-python on OFTC.

The Technical Committee can overrule maintainers, as a last resort. But they're pragmatists and, in a volunteer project, limited in what they can ask of project members. So, without volunteers within the project to drive your actions and/or take over maintenance of packages in question, it's hard to imagine you finding much success through this avenue.

Third, there's the project leader, leader@debian.org (currently, @highvoltage). He'd handle any legal action. But more practically, he's the point of call for any incoming issues that you don't know where to direct.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

@pradyunsg:

I am very much not trying to point fingers, but I've now seen multiple individuals involved in some capacity with Debian [2] express a sense that the current situation with Python is "okay" or even "good" somehow. If that's really the general "feel" of the situation for the folks involved with Debian, I find it very concerning: I've not met/read/heard-of any Python developer who found the Python-on-Debian experience to be a generally positive one, unless they were completely circumventing Debian's packaging.

As a primarily-Python developer, who uses Debian, I find the situation on Debian excellent, and always have. But then I'm deep in the weeds on this and don't get to experience the issues that newcomers have, very often. My experience with onboarding new Software Developers, at a company using a lot of Python has been that on Debian/Ubuntu they have minimal issues, and on macOS, there's a non-ending stream of problems that we had to help them with. So, shrug :/

I totally hear your concerns in this thread, though. And agree many of them. Splitting up the stdlib confuses people. Having ca-certicates be an optional leaf package isn't great. Missing tzdata is obviously a problem.

But I think we're making some progress:

People seem interested in increasing the priority of ca-certificates. Not to essential, but it really should be default on installs. Docker may be another issue, though. The images aren't produced by Debian, and I'm sure they try hard to reduce the image size.

python3.9 now Recommends: ca-certifactes, which means it will be installed if you install python3 unless you've gone out of your way to disable installing recommends.

Doko seems willing to do a python3-full package. This isn't exactly what you want. This feels like something that would benefit from a public discussion on the debian-python list...

I can probably elaborate as an upstream maintainer of pip -- I'm genuinely unhappy at the situation because the package is being modified in ways that causes a degraded user experience, additional support overhead for me and my fellow maintainers and contributes significantly to a negative perception of the package.

I haven't touched the pip package recently, but I have been involved in maintaining it. I try to look after the virtualenv stack and pip is part of it.

I've not been aware of your unhappiness with pip in Debian.

I routinely see complaints about pip being ancient, but we're a distro, old packages are kind of what we do.

I'd be happy to have a separate conversation with you about what issues you see, and how we can improve the situation.

@highvoltage

This comment has been minimized.

Copy link

@highvoltage highvoltage commented Jan 19, 2021

Hi, thanks for the poke, @stefanor. It sounds like most of these issues can be resolved (with some already being solved or in the process).

I don't use GitHub actively so if you'd like to reach out to the Debian Project Leader for this or future issues then you are more than welcome to e-mail leader@debian.org at any time.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Jan 19, 2021

Stefano, thanks for your proposal. A python3-full package is not an acceptable long-term solution. In my opinion your proposal is a barely acceptable compromise for the upcoming Debian stable release. A long-term solution is:

  1. python3 package must provide a fully working interpreter with all stdlib packages with an exception for test directory, documentation, header files, and tkinter/idle. If Debian needs a minimal Python installation, then the package should be called something like python3-minimal or system-python.
  2. The python or python3 command in default PATH must be a fully working interpreter with all stdlib with same exception as above. If Debian requires a minimal system Python interpreter, then the executable name should reflect this, e.g. /usr/bin/system-python or /usr/lib/system-python

tl;dr: if you call it Python, then it has to be Python. If Debian wants a minimal system Python, then please don't call it python. For example RHEL 8 has an internal, hidden platform-python and an end-user python3 command.

Fedora platforms use openssl.cnf to configure crypto policies like mininum protocol version

As does Debian.

Correction, it's only Ubuntu. Debian does not have Ubuntu's tls1.2-min-seclevel2.patch patch that causes the issue.

Debian has another issue. The libssl1.1 package does not contain openssl.cnf. The file is in the openssl package and not installed by default. That means a minimal Debian installation of the debian Docker container image does not set minimum protocol version. The behavior of libssl changes after I manually install the openssl package:

$ podman run -ti debian:latest
# apt update
# apt install -y python
# stat /etc/ssl/openssl.cnf
stat: cannot stat '/etc/ssl/openssl.cnf': No such file or directory
# python3
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version, ctx.maximum_version
(<TLSVersion.MINIMUM_SUPPORTED: -2>, <TLSVersion.MAXIMUM_SUPPORTED: -1>)
>>> exit()
# apt install -y openssl
# python3
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version, ctx.maximum_version
(<TLSVersion.TLSv1_2: 771>, <TLSVersion.MAXIMUM_SUPPORTED: -1>)
# Fedora 33
>>> ctx.minimum_version, ctx.maximum_version
(<TLSVersion.TLSv1_2: 771>, <TLSVersion.TLSv1_3: 772>)

As I said before, I suggest discussing this in an alternative thread, it's not related to your desire for a better python developer experience.

Incompatible SSL patches. Can be resolved by workarounds in python test suite.

I do not agree with you. As the primary maintainer and main developer of Python's ssl module, I can ensure you, that these issues matter and directly affect our users negatively. Once my PEP 644 gets accepted I will also remove most to all workaround from our tests.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Jan 19, 2021

People seem interested in increasing the priority of ca-certificates. Not to essential, but it really should be default on installs. (Docker may be another issue, though. The images aren't produced by Debian, and I'm sure they try hard to reduce the image size)

The Debian Docker container image https://hub.docker.com/_/debian is called debian, uses Debian's logo and documented as Maintained by: Debian Developers tianon and paultag. The Ubuntu container image is documented as Maintained by: Canonical and Tianon (Debian Developer). For outsiders like me, these facts make the image as official and produced by Debian / Canonical as it can get.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

Your long-term requirements sound like a good idea to encode in an informational PEP. I'd suggest getting people from the distros involved in that. You are asking for invasive changes, so be prepared for this to take a few years to happen.

python3 package must provide a fully working interpreter with all stdlib packages with an exception for test directory, documentation, header files, and tkinter/idle. If Debian needs a minimal Python installation, then the package should be called something like python3-minimal or system-python.

This is largely how things were until the creation of python3-stdlib-extensions. I don't the motivation there.

The python or python3 command in default PATH must be a fully working interpreter with all stdlib with same exception as above. If Debian requires a minimal system Python interpreter, then the executable name should reflect this, e.g. /usr/bin/system-python or /usr/lib/system-python

This is tricky, as it up-ends everything.

These are corner cases, of course. The vast majority of users are probably well served by things the way they are. But clearly the support volume for the remainder is enough to be an issue.

Speaking personally, I do see the argument for the change. This sounds like something to discuss at the beginning of the next cycle. I don't know if the project will be able to find the motivation to make this level of change. I can't personally make any promises to drive this.
And I'm sure that the change itself will cause problems for users, that we'll all need to deal with.

That means a minimal Debian installation of the debian Docker container image does not set minimum protocol version.

Is there a practical detrimental effect of this? All of the values listed in the paste seem reasonable.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

Context: As I understand it, the docker images are Docker's official Debian images, not Debian's official Docker images. Yeah, not confusing at all :(
Bugs against those images are definitely bugs, but may be an issue with the image generation rather than Debian. I wouldn't trust "installed by default in the docker image" as a way to determine if a package is installed by default in a Debian installation.

I'm part of the Ubuntu release team, and I don't know anything about the Ubuntu Docker images. Shrug.

@kroeckx

This comment has been minimized.

Copy link

@kroeckx kroeckx commented Jan 19, 2021

@njsmith

This comment has been minimized.

Copy link

@njsmith njsmith commented Jan 19, 2021

@brettcannon

Apologies for pulling a management-like move, but if someone from the Debian side can specifically tell me who the Python steering council should reach out to in order to see if there's a way to resolve this that would be much appreciated.

My understanding is:

  • @highvoltage who posted above is the current Debian Project Leader.
  • @ehashman who posted above is a member of the Debian Technical Committee, and also a former auditwheel maintainer, PSF Fellow, frequent PyCon attendee, and one of @tiran's coworkers.

So you have two good choices! Personally I think my first stop would be Elana, since she has a lot of context already. Also, a confusing thing about Debian is that the Tech Committee is actually more powerful than the Project Leader, I think? She's like the Debian equivalent of a Supreme Court Justice, so if she wants to help sort this out then she's in a very good position to do that :-)

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 19, 2021

Except that the technical committee is the court of last resort. If you haven't done everything you can to resolve an issue before bringing it to them, that's what they'll ask you to do.

@njsmith

This comment has been minimized.

Copy link

@njsmith njsmith commented Jan 19, 2021

@stefanor Same with the Supreme Court, actually :-). I wasn't saying that anyone should formally bring this to debian-ctte, just that someone who's on the committee will have a lot of relevant connections and understanding of how to work more effectively with Debian, even when participating in her personal capacity.

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Jan 20, 2021

Debian Docker image issue for including ca-certificates: debuerreotype/docker-debian-artifacts#15
Debian Discussion on installing ca-certificates by default: https://lists.debian.org/debian-devel/2021/01/msg00305.html

@Yhg1s

This comment has been minimized.

Copy link

@Yhg1s Yhg1s commented Feb 9, 2021

Hi, I’m Thomas Wouters, and I’m one of the people on the Python Steering Council. (For the record, I’m also on the PSF Board of Directors, but I’m responding here solely with my Steering Council hat on. We did not discuss this response with the PSF Board, although they are aware of this discussion.)

We’ve been discussing this within the Steering Council for a while, and I think at this point our position is that this is a real, disruptive, common problem, but that it’s a Debian problem, not so much a Python one -- and that we would like the Debian project to acknowledge this and to continue to try and improve the situation.

I’m going to focus on the Python/virtualenv/pip situation here, since I can’t say much about the OpenSSL issues. I want to stress the fact that this is a real, wide-spread problem with a lot of user impact. It affects both new Python programmers and experienced ones. I see this in the #python IRC channel on Freenode on a daily basis, and often after people have tried to work around the problems in ways that mess up their system. While the correct way to deal with this may seem obvious to the Debian packagers, it’s decidedly not clear to end users, and they often end up following outdated or incorrect advice that leaves their Debian installations in an undesirable state. This then leads to seemingly innocuous changes breaking Python or something written in Python. Other steering council members have had similar experiences where people are confused as to why instructions on how to get started with Python do not work on Debian-based distributions.

I think we all understand that packaging something like Python is difficult, especially since it comes with a third-party package management tool, pip. Nevertheless users have a real need for pip, and often without understanding the difference between system installations, virtualenv, and --user installs. In the Python ecosystem it’s simply too common to install PyPI packages, or install directly from GitHub repos, or install different versions in different places. The Debian package management system can’t satisfy those needs (and I don’t believe it was ever intended to).

And I think it’s important to realise here that we’re not just talking about users who choose to use Debian. Debian is the basis for many other distros, for Docker images and other containers, for Raspberry Pi OS, etc, etc. This impacts many different types of users.

At the same time, I’m very hesitant to dictate solutions here -- even if it’s in the form of an informational PEP that duplicates what’s already in the Makefile -- because the Python core developers, as a group, are not the right people to do this. I say this for four reasons, really:

  • The core developers are not representative of the end user. We don’t really know how and why they use Python; whether they’re writing code in Python or just running third-party code; how the code is packaged up, how the Debian system they’re using is maintained, etc. We simply don’t know what Debian users care about in terms of their Python installation. We just see a lot of complaints that originate from the same root causes.

  • The core developers are by and large not Debian packagers, and don’t share their considerations. There can certainly be good reasons for Debian to split packages the way it does, or to not install pip. Mixing package managers that aren’t aware of each other is one of the problems people run into with Debian right now, and we don’t want to exacerbate that problem by dictating ‘python -m pip’ should always work.

  • The core developers aren’t the driving force behind the tools that are central to this problem -- virtualenv, pip and setuptools -- nor the best practices surrounding them. Virtualenv was a third-party development that was replicated in the standard library; pip, while bundled with Python, remains a separate project; setuptools is a fork of distutils that tried out yet another package format that’s since been deprecated (eggs). Best practices and extended tooling around this all (tox, nox, pipenv, poetry, etc) are all community efforts. An informational PEP dictating what should work might solve the current problem, but it’d always be lagging behind actual community practices.

  • Finally, and this may sound a little harsh, fundamentally the problem here is one between Debian and its users. It’s a problem that’s been dragging along for many years now, going at least as far back as when distutils was split off from the regular Python install. I don’t know if the answer is to change the default installation or to educate users better. The choice to apply the Debian packaging policy differently to provide a better user experience, or to make it more obvious what end users should be doing, or whatever other option there is to mitigate the end users’ problems, is one Debian packagers and policy setters need to make, not something the core developers should be dictating.

The main question now is whether the Debian project is aware of the problems, and whether there’s agreement on the seriousness of the issues. If the case is really that the best solution for end users requires Python, as a project, to dictate that solution -- well, so be it, but I would hope Debian already wants to do what’s best for users. Even though we don’t think we’re the best people to solve these problems, we are happy to provide guidance if desired.

@flowerbug

This comment has been minimized.

Copy link

@flowerbug flowerbug commented Feb 9, 2021

https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e#gistcomment-3598489

@stefanor thank you for writing that, you can ask me about what i was doing and i'm willing to help out on any issue i report as best i can. i did admit i was using Debian testing and perhaps it was a transition or update in progress issues. it did get resolved somehow, but i'm not sure how. :)

i'm actually probably a pretty good newbie user because while i do have computer experience i do not have python experience, so when i do hit something and report it it is likely something a novice user might be confused by. my e-mail address is available in the mailing list for debian-users, and that is a good place to respond as it does reach others besides me who might have a use for that information.

i'm also willing to help out any other way i can, but i'm overwhelmed at times like anyone else. the best i can do and have been doing is using Debian testing and reporting things as i come across them, that is my primary way of helping. if i'm submitting a bug or issue i'm always willing to dig into something, please don't be afraid to ask.

my primary problem in understanding Python and how it is set up in any system is figuring out how to get back to a guaranteed default state where i know that all Debian and the base Python packages are installed, verifieably so, and all caches are cleared out so what i am seeing is not the result of some cached code that is being used instead of what i think is being used.

the virtual environment is the primary way i hope to accomplish this. :)

i have both a stable and testing partition available and i can also set up unstable or experimental if needed, so breakages are often something i can work around for the short term. if you want me to test or try something out or have a question ask away.

as for Debian packaging of Python, i don't understand the issues or conflicts with upstream, i've read this whole conversation and it looks like there's a lot of history. knowing certain DDs (debian developers) and their communcations styles for many years (from reading bug reports and various mailing lists i don't interact with them often so they wouldn't know me personally) it is an issue that is personality and sometimes conflict driven and one that causes friction even in the best of times let alone when there is pressure and a release coming up.

@tiran

This comment has been minimized.

Copy link
Owner Author

@tiran tiran commented Feb 9, 2021

Y'all,

could you please move the discussion to a more appropriate place? I propose https://discuss.python.org/ .

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Feb 10, 2021

@flowerbug: Ah, I misunderstood your issue earlier. I have no idea what I was reading, maybe I confused you with somebody else. The danger of trying to respond to too many things in one thread..

Yeah that looks like a broken state in the 3.8->3.9 transition, caused by a distutils mismatch to the stdlib.

That's reproduce-able with 3.9 and 3.10 co-installed. Let's see if we can improve it...

Debian Bug: https://bugs.debian.org/979819

@ehashman

This comment has been minimized.

Copy link

@ehashman ehashman commented Feb 10, 2021

I am going to follow up with a post to the Debian Python mailing list and will not continue monitoring this gist.

@ssbarnea

This comment has been minimized.

Copy link

@ssbarnea ssbarnea commented Mar 22, 2021

Apparently ansible package install by ubuntu/(debian?) is also broken, it cannot be called when a virtualenv is activated. The rpm version on fedora is not affected by the same bug and if someone installs it using pip, it would work.

Because Github has the "inspiration" of preinstalling ansible (from apt) on their GHA images, they also provide a partially broken set of ansible binaries, ones that work only on some cases. I am going to raise an issue with them, asking them to remove ansible from the preinstalled list, or replace it with a pip/pipx installed version, preferably at user level instead of root.

actions/virtual-environments#3001 -- request to remove broken ansible from GHA

@webknjaz

This comment has been minimized.

Copy link

@webknjaz webknjaz commented Mar 22, 2021

Apparently ansible package install by ubuntu/(debian?) is also broken, it cannot be called when a virtualenv is activated. The rpm version on fedora is not affected by the same bug and if someone installs it using pip, it would work.

AFAIK Fedora uses full interpreter paths in their installed scripts so they aren't affected by the env vars...

@stefanor

This comment has been minimized.

Copy link

@stefanor stefanor commented Mar 27, 2021

Debian strongly encourages full interpreter paths for that reason, too.

@sfermigier

This comment has been minimized.

Copy link

@sfermigier sfermigier commented Mar 30, 2021

FYI:

root@puer:~# lsb_release -d
Description:	Ubuntu 20.04.2 LTS
root@puer:~# python -m ensurepip
ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

# But...
root@puer:~# apt-get install python-pip
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package python-pip

# 💡
root@puer:~# apt-get install python3-pip
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-pip is already the newest version (20.0.2-5ubuntu1.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

# 😖
root@puer:~# pip

Command 'pip' not found, but there are 18 similar ones.

I have 25 years of experience with Python. I wonder how a beginner would feel in the same situation.

(BTW: the solution was to use pip3 instead of pip).

@barneygale

This comment has been minimized.

Copy link

@barneygale barneygale commented Apr 4, 2021

I'm a casual Python and Linux Mint user. Naive question: could pip be made to generate a .deb, and then hand off to apt/dpkg for the final installation? I think that would prevent apt and pip stomping on eachother, and so might ameliorate some of the Debian concerns for including ensurepip. Virtualenv has been split up into plugins for different tasks. Could the same happen to pip, with a DebPackager plugin interfacing with apt/dpkg?

@webknjaz

This comment has been minimized.

Copy link

@webknjaz webknjaz commented Aug 26, 2021

could pip be made to generate a .deb, and then hand off to apt/dpkg for the final installation?

Pip is not the right tool for this. It should be a PEP 517 capable build front-end that works within the Debian packaging ecosystem. And even so, it has nothing to do with Debian's wish to delete/relocate parts of packages (OS-level package managers already use proper artifacts with all the files to build stuff).

@webknjaz

This comment has been minimized.

Copy link

@webknjaz webknjaz commented Aug 26, 2021

I think that would prevent apt and pip stomping on eachother

There's an INSTALLER file in the installed dist metadata. It contains string pip if it was installed by pip, other package managers are supposed to put their names there and only modify a package if they installed it, based on that file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment