boppreh/PyPI rant.md Secret

## PyPI rant.md

      
    Raw
  

              PyPI rant.md
            
          
    new PyPI: a rant from a package maintainer

(Package names and versions have been changed to protect the innocent. This is an edited version of real events, reproduced to the best of my memory. And it goes without mention that I was constantly googling and trying different things.)
(This rant was created for the distutils-sig mailing list. Click here to read the responses from the community and PyPI developeres.)
TL;DR:

Everything is "legacy".
Uploads fail for unfathomable reasons.
Metadata fails to update and manual changes are not possible.
Documentation is all over the place and outdated.
TestPyPI is extremely confusing/confused.
What should have taken 30 seconds spiraled into consuming hours of my life and most of my sanity.

Task: upload a new version of a package.
Tools at my disposal:

Account and package registered.
New version ready to go in my machine.
Script with steps required, boiling down to:

Bump version at setup.py.
python setup.py sdist --format=zip bdist --format=zip bdist_wheel --universal bdist_wininst upload
Or python setup.py register to just update the description.


Pre-created configuration file ~/.pypirc.

[distutils]
index-servers=
    pypi

[pypi]
username = BoppreH
password = hunter2


Battle tested scripts that had performed releases like this dozens of times.
The fact that the package is a universal pure Python package with zero dependencies (it's one of the selling points).
Updated pip, updated setuptools, Python 3.6 64 bits, Windows 10 64 bits, sane environment.


Act I: "Just take the damn files"

Everything is going well, until python setup.py [...] upload chokes during upload of one of the files. The transfer hangs, a timeout occurs. Oh dear. No problem, I'll just python setup.py upload...
error: No dist file created in earlier command

Short memory much? So I run the full command again...
Uploading distributions to https://upload.pypi.org/legacy/
Uploading package-1.2.3.zip
HTTPError: 400 Client Error: File already exists. for url: https://upload.pypi.org/legacy/

It's the exact same file. You could add a check for it and ignore, but no, it had to be an error. Then I have to play the game of "let's figure out which part of the command created which of 6 files and upload one by one". This was a pain, but manageable.
Checking the release I see the description is just the name of the package. It seems that long_description in setup.py was missed entirely. I check my tools, and they seem to be sending the correct data. Regardless, I'll just edit the description manually in https://pypi.python.org/... but then my browser greets me with: ...

Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about
migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html)

Act II: "Please just tell me which one you want me to use"

It seems that changing the description is not possible via the website. No problem, I'll just run python setup.py register, which should update the documentation. By the way, why was this feature included in the register command? I feel weird using it. Or should I say "used to feel weird", because...
$ python setup.py register
Registering package to https://upload.pypi.org/legacy/
Server response (410): Project pre-registration is no longer required or supported, so continue directly to uploading files.

Hmm. Maybe the answer to my conundrum is in this "new PyPI.org" mentioned before. I visit the page and I'm greeted with a red banner with a warning sign: "This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).". I login in this new website, visit my package page and... there's no way to edit the description there either.
At this point I have to stop and look at the bigger picture of this migration:

My setup.py has been communicating with https://upload.pypi.org/legacy/ by default. Note the legacy part.
The pypi.python.org website has pages that were "deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI". But the "new APIs" seem to be missing at last one important feature, and I must have missed the deprecation period that preceded the complete removal of the old APIs. A friendly email would have been nice if you expect me to migrate to a new solution.
Looking around, it seems that I should be using a new tool called twine to upload my files after setup.py has created them. This should fix the short memory problem and ensure TLS.
The kicker? Twine also uploads to https://upload.pypi.org/legacy/ by default. Editing my .pypirc to force the URL to be https://upload.pypi.org/ gives me 405 Client Error: Method Not Allowed.

The new tool seems to be using legacy APIs, the old website is responding with "Gone" to reasonable requests, the new website is "pre-production" and missing the feature that was removed from the old one, and I see no way to use non-legacy APIs. I feel like someone is telling me to fuck-off.
After all this I still didn't solve my problem, and have to bump my software version to update the description in PyPI. So I bump my version, publish a new release that just says "Fixed README, sorry", confuse my users, and...
$ twine upload dist/*
[...]
Uploading package-1.2.4.win-amd64.exe
HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/

What? But I just uploaded a wizard a moment ago! Why is this error happening now? Why are there exactly 0 results on Google for this error? It seems that my users will have to live without an installation wizard and a weird release.
Act III: Test wasteland

I notice I am confused. I'm clearly under some profound misunderstandings here. I don't want to piss my users off any more, so I look for a way to test all this in a safe environment. There is TestPyPI, built exactly for this situation: https://wiki.python.org/moin/TestPyPI. Finally something makes sense.
Haha, no.
Step 3 in that tutorial is to register your package:
$ python setup.py register -r https://testpypi.python.org/pypi
running register
running egg_info
writing package.egg-info\PKG-INFO
writing dependency_links to package.egg-info\dependency_links.txt
writing top-level names to package.egg-info\top_level.txt
reading manifest file 'package.egg-info\SOURCES.txt'
writing manifest file 'package.egg-info\SOURCES.txt'
running check
Registering package to https://testpypi.python.org/pypi
Server response (410): Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html)

To make matters worse, the tutorial also says: "Note that the test server uses an older implementation of the upload interface and hence does not support auto-registration the way the live service does when using twine's default configuration."
I stumble about in the dark for a while, then find this notice in the website for TestPyPI:

Package Authors
Submit packages with "python setup.py upload". The index hosts package docs. You may also use the web form. You must register. Testing? Use testpypi.

I'm not sure why TestPyPI is telling me to check out TestPyPI, so let's ignore that. See that "web form" link? Here's the entire contents of that "form":

To submit information to this index, you have two options:

Use twine!
Use the setup.py "register" command.


This must be a prank. The tutorial just told me that TestPyPI does not support "auto-registration [...] using twine's default configuration", so if you are telling me to use Twine, you better give more details than an exclamation mark. And "python setup.py register" has been killed, we've been through this before.
Maybe the tutorial is outdated, and TestPyPI supports auto-registration now? Sure, let's go with that.
$ python setup.py sdist
[...]
$ twine upload dist\package-1.3.4.tar.gz -r testpypi
Uploading distributions to https://testpypi.python.org/pypi
Uploading package-1.3.4.tar.gz
HTTPError: 410 Client Error: Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html) for url: https://testpypi.python.org/pypi

God dammnit, so I cannot even upload files to TestPyPI? Let's check out that mailing list announcement at the end of the error message. Wait, this looks interesting:

[...]
For those of you who are using TestPyPI, that will also be affected, and the
required URL for the new upload endpoint for TestPyPI is
https://test.pypi.org/legacy/.
[...]
For TestPyPI the change to disable uploads to legacy will be made in the next
couple of days, likely this weekend.

So the tutorial, the only page that seems to have any details on how to use TestPyPI, is telling users to send data to the wrong URL. Also, what do they mean by "disable uploads to legacy"? Is it https://testpypi.python.org/pypi, the URL from the tutorial, which is "legacy" in the sense of being the old one, or https://test.pypi.org/legacy/, which is explicitly legacy in the name? Oh well, the tutorial one is not working, so let's try the latter. I edit my .pypirc, rerun twine and...
$ twine upload dist\package-1.3.4.tar.gz -r testpypi
Uploading distributions to https://test.pypi.org/legacy/
Uploading package-1.3.4.tar.gz
$

Finally, something worked. I go visit my newly created package at https://test.pypi.org/project/package/ and... the description is still missing. I download my own .tar.gz from there, check PKG-INFO, and the description is there. python setup.py -rms to check my setup and no problems are reported. The new website doesn't show me the raw package metadata, so I go over to the legacy website at https://testpypi.python.org/, more specifically https://testpypi.python.org/pypi?name=package&version=1.3.4&:action=display_pkginfo (not a fan o the URL, by the way).
Metadata-Version: 1.1
Name: package
Version: 1.3.4
Author: BoppreH
Author-email: boppreh at gmail com
Home-page: https://github.com/boppreh/package
Summary: My short summary
License: MIT
Description: package

Some data is correct, but the description that I uploaded has been replaced with just the package name, and also the keywords and trove classifiers are missing entirely. I can't python setup.py register to upload metadata, and I can't python setup.py sdist upload because I already uploaded a source distribution, so I try python setup.py bdist upload:
# python setup.py bdist upload -r testpypi
[...]
Submitting C:\Users\BoppreH\Desktop\source\package\dist\package-1.3.4.win-amd64.zip to https://upload.pypi.org/legacy/
Upload failed (400): Unknown type of file.
error: Upload failed (400): Unknown type of file.

You gotta be kidding me. It shouldn't make a difference, but let's try with twine:
$ twine upload dist\package-1.3.4.win-amd64.zip
Uploading distributions to https://upload.pypi.org/legacy/
Uploading package-1.3.4.win-amd64.zip
HTTPError: 400 Client Error: Only one sdist may be uploaded per release. for url: https://test.pypi.org/legacy/

But... but... It's not an sdist, it's a bdist! Maybe one implies the other? Let's see what the docs say:
$ python setup.py --help | grep bdist
$ 

Nothing. The website https://docs.python.org/3/distutils/builtdist.html also doesn't mention why twine would confuse the two.

Act IV: I give up.

I can't even specify the description of a package, not even during registration or upload. Either this whole PyPI thing is an absolute clusterfuck, or I'm under some severe misunderstandings that no documentation, error message or mailing list announcement was able to fix.
I cannot even report the issues. First, because I'm so burned that I don't care to create individual reports for all the issues. Second, because there's no clear location to report them. For example, if my package description is missing, do I report to:

setuptools because it might have created a file with wrong syntax?
twine because it might have uploaded incorrect metadata?
warehouse because it's their API that I'm hitting?
pypi-legacy because the API is emulating it, or maybe it's just a display issue on the website?
or to the generic packaging-problems repository, which almost looks like the correct place, until you realize that no page has linked to it, it's empty except for a bunch of random issues and a wiki with a roadmap for PyPA developers last updated 5 months ago?

I am trying to donate my precious free time to the Python community by offering free software, receiving absolutely no compensation whatsoever, and every step of the experience has been shitty. Python packages have always been full of quirks, but this time it seems like all the quirks broke at once, and the result is incomprehensible.
I know other people are using these tools, so it's not like everything is broken for everyone. But my bad experiences happened, and they must have an explanation. What do I expect from this rant? Either someone to tell me "yes, PyPI made mistakes, we're going to fix it", or "hey, it's your fault because you did X wrong".
I don't want to throw my work away just because dealing with PyPI is excruciatingly painful.