Negative Python user experience on Debian/Ubuntu

python-on-debian.md

Negative Python user experience on Debian/Ubuntu

The user experience of Python on a minimal Debian or Ubuntu installation is bad. Core features like virtual environments, pip bootstrapping, and the ssl module are either missing or do not work like designed and documented. Some Python core developers including me are worried and consider Debian/Ubuntu's packaging harmful for Python's reputation and branding. Users don't get what they expect.

Reproducer

The problems can be easily reproduced with official Debian and Ubuntu containers in Docker or Podman. Debian Stable (Debian 10 Buster) comes with Python 3.7.3. Ubuntu Focal (20.04 LTS) has Python 3.8.5.

Run Debian container

$ docker run -ti debian:stable

Run Ubuntu container

$ docker run -ti ubuntu:focal

Install Python3

# apt update
# apt install python3

venv is broken

venv is another Python standard library module. It provides support for creating lightweight "virtual environments". The venv module is available but dysfunctional. It cannot create virtual environments out of the box.

# python3 -m venv /tmp/venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/tmp/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']

Update Julien Palard wrote that one of his students ran into another issue with venv. Debian's venv can give an invalid advise when a user has multiple Python versions installed.

ensurepip is missing

The ensurepip package is part of Python's standard library and provides support for bootstrapping the pip installer into an existing Python installation or virtual environment. The ensurepip package is missing on Debian/Ubuntu.

# python3 -m ensurepip
/usr/bin/python3: No module named ensurepip
# pip
bash: pip: command not found

After installation of python3-venv, the ensurepip package is failing with a different error message:

# python3 -m ensurepip
ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.
# echo $?
1

distutils is stripped down and missing most code

The distutils package is mostly missing. Only the package root and distutils.version is available. The remaining code has been moved to python3-distutils by Debian/Ubuntu packagers. The python3-distutils is not installed with python3 and must be installed separately.

# python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)  
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> from distutils import sysconfig
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: cannot import name 'sysconfig' from 'distutils' (/usr/lib/python3.7/distutils/__init__.py)

ssl module cannot verify connections

A minimal installation has no CA certificates because neither the python3 package nor OpenSSL libraries depend on ca-certificates.

• Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960869
• Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1879310

>>> import urllib.request
>>> urllib.request.urlopen("https://pypi.org/")
Traceback (most recent call last):
 ...
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
   return opener.open(url, data, timeout)
 File "/usr/lib/python3.7/urllib/request.py", line 525, in open
   response = self._open(req, data)
 File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
   '_open', req)
 File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
   result = func(*args)
 File "/usr/lib/python3.7/urllib/request.py", line 1367, in https_open
   context=self._context, check_hostname=self._check_hostname)
 File "/usr/lib/python3.7/urllib/request.py", line 1326, in do_open
   raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

Incompatible OpenSSL downstream patch

Debian/Ubuntu have applied downstream patches to OpenSSL. The patches have caused breakage of user applications or Python's CI tests. Examples for issues and workarounds:

• https://bugs.python.org/issue41561
• https://bugs.python.org/issue31453
• https://bugs.python.org/issue26867
• https://github.com/python/cpython/blob/v3.6.12/Lib/test/test_ssl.py#L133-L147
• https://github.com/python/cpython/blob/v3.6.12/Lib/test/test_ssl.py#L149-L178
• https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878
• https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1917625

lib2to3 is missing

The lib2to3 package is moved to python3-lib2to3 package, which is not installed by default.

tkinter is in an extra package (ok)

The tkinter package is not part of the default distribution. For once this is a good decision. tkinter depends on libtk and whole lot of X11 libraries. Graphical user interface libraries should not be installed by default on headless servers and containers. I just find it confusing that the tkinter package is provided by a python3-tk package and not by python3-tkinter.

Python 3.9 is missing dependency on tzdata

Paul Ganssle added a zoneinfo implementation with timezons to Python 3.9, see PEP 615. The feature requires tzdata database. As of 2020-11-13 Debian and Ubuntu's python3.9 package are missing a dependency on the tzdata package. The zoneinfo module does not work without tzdata:

>>> import zoneinfo
>>> zoneinfo.available_timezones()
set()
>>> zoneinfo.ZoneInfo("CET")
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/zoneinfo/_common.py", line 24, in load_tzdata
    raise ZoneInfoNotFoundError(f"No time zone found with key {key}")
zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key CET'

NOTE The issue has been fixed by Anthony Sottile in Deadsnakes PPA, see comment.

UPDATE My launchpad bug 1904271 was closed as Invalid. Matthias wrote that tzdata is a required package and pointed to Debian policy. However the package is not installed by default in the official Debian and Ubuntu container images.

New virtualenvs contain unwanted libraries

Virtualenvs contain de-vendored dependencies of pip and setuptools, https://bugs.launchpad.net/ubuntu/+source/python-virtualenv/+bug/1904945

Expectations and Proposal

Minimalization of Python installation is a legitimate effort. However a minimal installation of Python with core features missing should not be called a Python installation. Users should expect that package-manager install python3 gets them a working Python interpreter with majority of stdlib packages (with exception to tkinter GUI and test package).

I propose

1. Debian's current minimized Python package python3 should rather be called python3-minimal or something similar. This package would still users to get a stripped down interpreter if they explicitly ask for it.
2. apt install python3 should provide a Python installation with working venv, ensurepip (*), distutils, and ssl modules.

(*) I define working ensurepip as python3 -m ensurepip does not fail and python3 -m pip works afterwards. It does not imply that stdlib's pip bundle must be shipped with Python distribution package. Debian could also provide an API compatible ensurepip facade and make python3 package depend on python3-pip.

Fedora platforms use openssl.cnf to configure crypto policies like mininum protocol version

As does Debian.

As I said before, I suggest discussing this in an alternative thread, it's not related to your desire for a better python developer experience.

I'm going to tell you one weird trick how you can reduce your workload: collaborate and work with us instead against us

Ah, you've come around to my PoV :)

Your recommendation doesn't scale. CPython core lacks resources to chase down and poke distro maintainers.

And Debian's Python people don't have the resources to follow everything going on in CPython. Somewhere in the middle lies the answer.

Thanks for filing bugs. That's good. But please don't stop if they aren't all being dealt with. Bugs often go unanswered, in all projects. IIRC some of Debian's cpython patches have been forwarded to the python bugtracker decades ago, with little traction. It usually takes motivation and effort to push change through.

Looking back at the list of complaints in this gist:

1. venv is broken: I don't really see the issue there. The error message clearly explains how to get a functional venv. Would be improved by a python3-full package.
2. ensurepip is missing: I don't think we can ever have an an ensurepip that does more than noop.
3. distutils is stripped. Would be improved by a python3-full package.
4. ssl can't verify. Would be improved by a python3-full package.
5. Incompatible SSL patches. Can be resolved by workarounds in python test suite.
6. lib2to3 is missing. Would be improved by a python3-full package.
7. tkinter is an extra package. Would be improved by a python3-full package.
8. Python 3.9 is missing adependency on tzdata. Was resolved in 3.9.1~rc1-1.

1, 3, 4, 6, 7 are all the same issue in that they are the result of breaking up the stdlib, and would all be improved by a python3-full package. I think we can get that to happen

I still don't see a systemic issue here. I see a handful of bugs, and some difficulty in getting them resolved. This is not insurmountable.

Apologies for pulling a management-like move, but if someone from the Debian side can specifically tell me who the Python steering council should reach out to in order to see if there's a way to resolve this that would be much appreciated.

Your first call is the maintainers of the packages in question. @doko42 for python, @kroeckx for OpenSSL.

There are thousands of python-related packages in Debian, so there are teams of people who have a stake in this stuff. So, if you hit a brick wall with the maintainers, I suggest a mail to the python team mailing list: https://lists.debian.org/debian-python/ or the IRC channel #debian-python on OFTC.

The Technical Committee can overrule maintainers, as a last resort. But they're pragmatists and, in a volunteer project, limited in what they can ask of project members. So, without volunteers within the project to drive your actions and/or take over maintenance of packages in question, it's hard to imagine you finding much success through this avenue.

Third, there's the project leader, leader@debian.org (currently, @highvoltage). He'd handle any legal action. But more practically, he's the point of call for any incoming issues that you don't know where to direct.

@pradyunsg:

I am very much not trying to point fingers, but I've now seen multiple individuals involved in some capacity with Debian [2] express a sense that the current situation with Python is "okay" or even "good" somehow. If that's really the general "feel" of the situation for the folks involved with Debian, I find it very concerning: I've not met/read/heard-of any Python developer who found the Python-on-Debian experience to be a generally positive one, unless they were completely circumventing Debian's packaging.

As a primarily-Python developer, who uses Debian, I find the situation on Debian excellent, and always have. But then I'm deep in the weeds on this and don't get to experience the issues that newcomers have, very often. My experience with onboarding new Software Developers, at a company using a lot of Python has been that on Debian/Ubuntu they have minimal issues, and on macOS, there's a non-ending stream of problems that we had to help them with. So, shrug :/

I totally hear your concerns in this thread, though. And agree many of them. Splitting up the stdlib confuses people. Having ca-certicates be an optional leaf package isn't great. Missing tzdata is obviously a problem.

But I think we're making some progress:

People seem interested in increasing the priority of ca-certificates. Not to essential, but it really should be default on installs. Docker may be another issue, though. The images aren't produced by Debian, and I'm sure they try hard to reduce the image size.

python3.9 now Recommends: ca-certifactes, which means it will be installed if you install python3 unless you've gone out of your way to disable installing recommends.

Doko seems willing to do a python3-full package. This isn't exactly what you want. This feels like something that would benefit from a public discussion on the debian-python list...

I can probably elaborate as an upstream maintainer of pip -- I'm genuinely unhappy at the situation because the package is being modified in ways that causes a degraded user experience, additional support overhead for me and my fellow maintainers and contributes significantly to a negative perception of the package.

I haven't touched the pip package recently, but I have been involved in maintaining it. I try to look after the virtualenv stack and pip is part of it.

I've not been aware of your unhappiness with pip in Debian.

I routinely see complaints about pip being ancient, but we're a distro, old packages are kind of what we do.

I'd be happy to have a separate conversation with you about what issues you see, and how we can improve the situation.

Hi, thanks for the poke, @stefanor. It sounds like most of these issues can be resolved (with some already being solved or in the process).

I don't use GitHub actively so if you'd like to reach out to the Debian Project Leader for this or future issues then you are more than welcome to e-mail leader@debian.org at any time.

Stefano, thanks for your proposal. A python3-full package is not an acceptable long-term solution. In my opinion your proposal is a barely acceptable compromise for the upcoming Debian stable release. A long-term solution is:

1. python3 package must provide a fully working interpreter with all stdlib packages with an exception for test directory, documentation, header files, and tkinter/idle. If Debian needs a minimal Python installation, then the package should be called something like python3-minimal or system-python.
2. The python or python3 command in default PATH must be a fully working interpreter with all stdlib with same exception as above. If Debian requires a minimal system Python interpreter, then the executable name should reflect this, e.g. /usr/bin/system-python or /usr/lib/system-python

tl;dr: if you call it Python, then it has to be Python. If Debian wants a minimal system Python, then please don't call it python. For example RHEL 8 has an internal, hidden platform-python and an end-user python3 command.

Fedora platforms use openssl.cnf to configure crypto policies like mininum protocol version

As does Debian.

Correction, it's only Ubuntu. Debian does not have Ubuntu's tls1.2-min-seclevel2.patch patch that causes the issue.

Debian has another issue. The libssl1.1 package does not contain openssl.cnf. The file is in the openssl package and not installed by default. That means a minimal Debian installation of the debian Docker container image does not set minimum protocol version. The behavior of libssl changes after I manually install the openssl package:

$ podman run -ti debian:latest
# apt update
# apt install -y python
# stat /etc/ssl/openssl.cnf
stat: cannot stat '/etc/ssl/openssl.cnf': No such file or directory
# python3
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version, ctx.maximum_version
(<TLSVersion.MINIMUM_SUPPORTED: -2>, <TLSVersion.MAXIMUM_SUPPORTED: -1>)
>>> exit()
# apt install -y openssl
# python3
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version, ctx.maximum_version
(<TLSVersion.TLSv1_2: 771>, <TLSVersion.MAXIMUM_SUPPORTED: -1>)

# Fedora 33
>>> ctx.minimum_version, ctx.maximum_version
(<TLSVersion.TLSv1_2: 771>, <TLSVersion.TLSv1_3: 772>)

As I said before, I suggest discussing this in an alternative thread, it's not related to your desire for a better python developer experience.

Incompatible SSL patches. Can be resolved by workarounds in python test suite.

I do not agree with you. As the primary maintainer and main developer of Python's ssl module, I can ensure you, that these issues matter and directly affect our users negatively. Once my PEP 644 gets accepted I will also remove most to all workaround from our tests.

People seem interested in increasing the priority of ca-certificates. Not to essential, but it really should be default on installs. (Docker may be another issue, though. The images aren't produced by Debian, and I'm sure they try hard to reduce the image size)

The Debian Docker container image https://hub.docker.com/_/debian is called debian, uses Debian's logo and documented as Maintained by: Debian Developers tianon and paultag. The Ubuntu container image is documented as Maintained by: Canonical and Tianon (Debian Developer). For outsiders like me, these facts make the image as official and produced by Debian / Canonical as it can get.

Your long-term requirements sound like a good idea to encode in an informational PEP. I'd suggest getting people from the distros involved in that. You are asking for invasive changes, so be prepared for this to take a few years to happen.

python3 package must provide a fully working interpreter with all stdlib packages with an exception for test directory, documentation, header files, and tkinter/idle. If Debian needs a minimal Python installation, then the package should be called something like python3-minimal or system-python.

This is largely how things were until the creation of python3-stdlib-extensions. I don't the motivation there.

The python or python3 command in default PATH must be a fully working interpreter with all stdlib with same exception as above. If Debian requires a minimal system Python interpreter, then the executable name should reflect this, e.g. /usr/bin/system-python or /usr/lib/system-python

This is tricky, as it up-ends everything.

These are corner cases, of course. The vast majority of users are probably well served by things the way they are. But clearly the support volume for the remainder is enough to be an issue.

Speaking personally, I do see the argument for the change. This sounds like something to discuss at the beginning of the next cycle. I don't know if the project will be able to find the motivation to make this level of change. I can't personally make any promises to drive this.
And I'm sure that the change itself will cause problems for users, that we'll all need to deal with.

That means a minimal Debian installation of the debian Docker container image does not set minimum protocol version.

Is there a practical detrimental effect of this? All of the values listed in the paste seem reasonable.

Context: As I understand it, the docker images are Docker's official Debian images, not Debian's official Docker images. Yeah, not confusing at all :(
Bugs against those images are definitely bugs, but may be an issue with the image generation rather than Debian. I wouldn't trust "installed by default in the docker image" as a way to determine if a package is installed by default in a Debian installation.

I'm part of the Ubuntu release team, and I don't know anything about the Ubuntu Docker images. Shrug.

A default install of Debian installs Recommends, and should install ca-certificates and openssl. But we've made it possible to reduce the minimal amount of packages by not installing Recommends. What I hear is that the minimal install is too minimal for some use cases, and that the Docker image is probably using such a minimal install. My understanding is that Ubuntu sets the default at compile time, but you can override them in the config file or at runtime, and so it does not depend on the presence of the openssl binary package to set the defaults. I could do the same in Debian. But you seem to indicate that this causes problems? Other options are: - Having libssl1.1 Depend on openssl, creating a circular dependency - Moving the config file to the library package and moving from 1 package to an other on each soname change. - Creating an openssl-common package to ship the config file, and have libssl1.1 Depend on that.

@brettcannon

Apologies for pulling a management-like move, but if someone from the Debian side can specifically tell me who the Python steering council should reach out to in order to see if there's a way to resolve this that would be much appreciated.

My understanding is:

• @highvoltage who posted above is the current Debian Project Leader.
• @ehashman who posted above is a member of the Debian Technical Committee, and also a former auditwheel maintainer, PSF Fellow, frequent PyCon attendee, and one of @tiran's coworkers.

So you have two good choices! Personally I think my first stop would be Elana, since she has a lot of context already. Also, a confusing thing about Debian is that the Tech Committee is actually more powerful than the Project Leader, I think? She's like the Debian equivalent of a Supreme Court Justice, so if she wants to help sort this out then she's in a very good position to do that :-)

Except that the technical committee is the court of last resort. If you haven't done everything you can to resolve an issue before bringing it to them, that's what they'll ask you to do.

@stefanor Same with the Supreme Court, actually :-). I wasn't saying that anyone should formally bring this to debian-ctte, just that someone who's on the committee will have a lot of relevant connections and understanding of how to work more effectively with Debian, even when participating in her personal capacity.

Debian Docker image issue for including ca-certificates: debuerreotype/docker-debian-artifacts#15
Debian Discussion on installing ca-certificates by default: https://lists.debian.org/debian-devel/2021/01/msg00305.html

Hi, I’m Thomas Wouters, and I’m one of the people on the Python Steering Council. (For the record, I’m also on the PSF Board of Directors, but I’m responding here solely with my Steering Council hat on. We did not discuss this response with the PSF Board, although they are aware of this discussion.)

We’ve been discussing this within the Steering Council for a while, and I think at this point our position is that this is a real, disruptive, common problem, but that it’s a Debian problem, not so much a Python one -- and that we would like the Debian project to acknowledge this and to continue to try and improve the situation.

I’m going to focus on the Python/virtualenv/pip situation here, since I can’t say much about the OpenSSL issues. I want to stress the fact that this is a real, wide-spread problem with a lot of user impact. It affects both new Python programmers and experienced ones. I see this in the #python IRC channel on Freenode on a daily basis, and often after people have tried to work around the problems in ways that mess up their system. While the correct way to deal with this may seem obvious to the Debian packagers, it’s decidedly not clear to end users, and they often end up following outdated or incorrect advice that leaves their Debian installations in an undesirable state. This then leads to seemingly innocuous changes breaking Python or something written in Python. Other steering council members have had similar experiences where people are confused as to why instructions on how to get started with Python do not work on Debian-based distributions.

I think we all understand that packaging something like Python is difficult, especially since it comes with a third-party package management tool, pip. Nevertheless users have a real need for pip, and often without understanding the difference between system installations, virtualenv, and --user installs. In the Python ecosystem it’s simply too common to install PyPI packages, or install directly from GitHub repos, or install different versions in different places. The Debian package management system can’t satisfy those needs (and I don’t believe it was ever intended to).

And I think it’s important to realise here that we’re not just talking about users who choose to use Debian. Debian is the basis for many other distros, for Docker images and other containers, for Raspberry Pi OS, etc, etc. This impacts many different types of users.

At the same time, I’m very hesitant to dictate solutions here -- even if it’s in the form of an informational PEP that duplicates what’s already in the Makefile -- because the Python core developers, as a group, are not the right people to do this. I say this for four reasons, really:

• The core developers are not representative of the end user. We don’t really know how and why they use Python; whether they’re writing code in Python or just running third-party code; how the code is packaged up, how the Debian system they’re using is maintained, etc. We simply don’t know what Debian users care about in terms of their Python installation. We just see a lot of complaints that originate from the same root causes.

• The core developers are by and large not Debian packagers, and don’t share their considerations. There can certainly be good reasons for Debian to split packages the way it does, or to not install pip. Mixing package managers that aren’t aware of each other is one of the problems people run into with Debian right now, and we don’t want to exacerbate that problem by dictating ‘python -m pip’ should always work.

• The core developers aren’t the driving force behind the tools that are central to this problem -- virtualenv, pip and setuptools -- nor the best practices surrounding them. Virtualenv was a third-party development that was replicated in the standard library; pip, while bundled with Python, remains a separate project; setuptools is a fork of distutils that tried out yet another package format that’s since been deprecated (eggs). Best practices and extended tooling around this all (tox, nox, pipenv, poetry, etc) are all community efforts. An informational PEP dictating what should work might solve the current problem, but it’d always be lagging behind actual community practices.

• Finally, and this may sound a little harsh, fundamentally the problem here is one between Debian and its users. It’s a problem that’s been dragging along for many years now, going at least as far back as when distutils was split off from the regular Python install. I don’t know if the answer is to change the default installation or to educate users better. The choice to apply the Debian packaging policy differently to provide a better user experience, or to make it more obvious what end users should be doing, or whatever other option there is to mitigate the end users’ problems, is one Debian packagers and policy setters need to make, not something the core developers should be dictating.

The main question now is whether the Debian project is aware of the problems, and whether there’s agreement on the seriousness of the issues. If the case is really that the best solution for end users requires Python, as a project, to dictate that solution -- well, so be it, but I would hope Debian already wants to do what’s best for users. Even though we don’t think we’re the best people to solve these problems, we are happy to provide guidance if desired.

https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e#gistcomment-3598489

@stefanor thank you for writing that, you can ask me about what i was doing and i'm willing to help out on any issue i report as best i can. i did admit i was using Debian testing and perhaps it was a transition or update in progress issues. it did get resolved somehow, but i'm not sure how. :)

i'm actually probably a pretty good newbie user because while i do have computer experience i do not have python experience, so when i do hit something and report it it is likely something a novice user might be confused by. my e-mail address is available in the mailing list for debian-users, and that is a good place to respond as it does reach others besides me who might have a use for that information.

i'm also willing to help out any other way i can, but i'm overwhelmed at times like anyone else. the best i can do and have been doing is using Debian testing and reporting things as i come across them, that is my primary way of helping. if i'm submitting a bug or issue i'm always willing to dig into something, please don't be afraid to ask.

my primary problem in understanding Python and how it is set up in any system is figuring out how to get back to a guaranteed default state where i know that all Debian and the base Python packages are installed, verifieably so, and all caches are cleared out so what i am seeing is not the result of some cached code that is being used instead of what i think is being used.

the virtual environment is the primary way i hope to accomplish this. :)

i have both a stable and testing partition available and i can also set up unstable or experimental if needed, so breakages are often something i can work around for the short term. if you want me to test or try something out or have a question ask away.

as for Debian packaging of Python, i don't understand the issues or conflicts with upstream, i've read this whole conversation and it looks like there's a lot of history. knowing certain DDs (debian developers) and their communcations styles for many years (from reading bug reports and various mailing lists i don't interact with them often so they wouldn't know me personally) it is an issue that is personality and sometimes conflict driven and one that causes friction even in the best of times let alone when there is pressure and a release coming up.

Y'all,

could you please move the discussion to a more appropriate place? I propose https://discuss.python.org/ .

@flowerbug: Ah, I misunderstood your issue earlier. I have no idea what I was reading, maybe I confused you with somebody else. The danger of trying to respond to too many things in one thread..

Yeah that looks like a broken state in the 3.8->3.9 transition, caused by a distutils mismatch to the stdlib.

That's reproduce-able with 3.9 and 3.10 co-installed. Let's see if we can improve it...

Debian Bug: https://bugs.debian.org/979819

I am going to follow up with a post to the Debian Python mailing list and will not continue monitoring this gist.

Apparently ansible package install by ubuntu/(debian?) is also broken, it cannot be called when a virtualenv is activated. The rpm version on fedora is not affected by the same bug and if someone installs it using pip, it would work.

Because Github has the "inspiration" of preinstalling ansible (from apt) on their GHA images, they also provide a partially broken set of ansible binaries, ones that work only on some cases. I am going to raise an issue with them, asking them to remove ansible from the preinstalled list, or replace it with a pip/pipx installed version, preferably at user level instead of root.

actions/runner-images#3001 -- request to remove broken ansible from GHA

Apparently ansible package install by ubuntu/(debian?) is also broken, it cannot be called when a virtualenv is activated. The rpm version on fedora is not affected by the same bug and if someone installs it using pip, it would work.

AFAIK Fedora uses full interpreter paths in their installed scripts so they aren't affected by the env vars...

Debian strongly encourages full interpreter paths for that reason, too.

FYI:

root@puer:~# lsb_release -d
Description:	Ubuntu 20.04.2 LTS
root@puer:~# python -m ensurepip
ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt-get install python-<module name>

Install the python-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

# But...
root@puer:~# apt-get install python-pip
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package python-pip

# 💡
root@puer:~# apt-get install python3-pip
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-pip is already the newest version (20.0.2-5ubuntu1.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

# 😖
root@puer:~# pip

Command 'pip' not found, but there are 18 similar ones.

I have 25 years of experience with Python. I wonder how a beginner would feel in the same situation.

(BTW: the solution was to use pip3 instead of pip).

I'm a casual Python and Linux Mint user. Naive question: could pip be made to generate a .deb, and then hand off to apt/dpkg for the final installation? I think that would prevent apt and pip stomping on eachother, and so might ameliorate some of the Debian concerns for including ensurepip. Virtualenv has been split up into plugins for different tasks. Could the same happen to pip, with a DebPackager plugin interfacing with apt/dpkg?

could pip be made to generate a .deb, and then hand off to apt/dpkg for the final installation?

Pip is not the right tool for this. It should be a PEP 517 capable build front-end that works within the Debian packaging ecosystem. And even so, it has nothing to do with Debian's wish to delete/relocate parts of packages (OS-level package managers already use proper artifacts with all the files to build stuff).

I think that would prevent apt and pip stomping on eachother

There's an INSTALLER file in the installed dist metadata. It contains string pip if it was installed by pip, other package managers are supposed to put their names there and only modify a package if they installed it, based on that file.

@tiran If there's been some improvements on this front or concensus on actionable items here, it might make sense to update this gist to note those at the end.

PS: Happy new year folks! Hope you're doing well, to the extent that you can in the current state of things. :)

@tiran If there's been some improvements on this front or concensus on actionable items here, it might make sense to update this gist to note those at the end.

There have been some improvements in latest Debian and Ubuntu. However bug https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1879310 is not still fixed for Ubuntu 20.04 LTS and Debian 10. Python's ecosystem cannot rely on presence of CA certificates for another 3 to 8 years.