Demonstration of pypi_simple and parsing wheel (PEP427) filenames

README.md

Demonstrates how to use pypi_simple to parse and manipulate PyPI wheel package metadata.

It takes quite some effort to figure out if a wheel is compatible with your system (platform_tag) and your Python version (python_tag), not to mention other features such as abi-tag, yanked status etc.. All in all, this points at a need for a high-level interface for package resolving with pip. (e.g. pip-tools, poetry, pip/wheel itself)

Stdout as of 2020-02-04:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/matplotlib/ HTTP/1.1" 200 58154
Choose 3.1.3: matplotlib-3.1.3-cp38-cp38-manylinux1_x86_64.whl
DEBUG:__main__:Found 101 versions for "matplotlib": 
  1.5.2
  1.5.2
  1.5.3
  1.5.3
  1.5.3
  1.5.3
  2.0.0b1
  2.0.0b1
  2.0.0b1
  2.0.0b1
  2.0.0b2
  2.0.0b2
  2.0.0b3
  2.0.0b3
  2.0.0b4
  2.0.0b4
  2.0.0rc1
  2.0.0rc1
  2.0.0rc2
  2.0.0rc2
  2.0.0
  2.0.0
  2.0.0
  2.0.0
  2.0.0
  2.0.0
  2.0.1
  2.0.1
  2.0.1
  2.0.2
  2.0.2
  2.0.2
  2.1.0rc1
  2.1.0rc1
  2.1.0rc1
  2.1.0
  2.1.0
  2.1.0
  2.1.1
  2.1.1
  2.1.1
  2.1.2
  2.1.2
  2.1.2
  2.2.0rc1
  2.2.0rc1
  2.2.0rc1
  2.2.0
  2.2.0
  2.2.0
  2.2.2
  2.2.2
  2.2.2
  2.2.2
  2.2.3
  2.2.3
  2.2.3
  2.2.3
  2.2.4
  2.2.4
  2.2.4
  2.2.4
  2.2.5
  2.2.5
  2.2.5
  2.2.5
  3.0.0rc2
  3.0.0rc2
  3.0.0rc2
  3.0.0
  3.0.0
  3.0.0
  3.0.1
  3.0.1
  3.0.1
  3.0.2
  3.0.2
  3.0.2
  3.0.3
  3.0.3
  3.0.3
  3.1.0rc1
  3.1.0rc1
  3.1.0rc2
  3.1.0rc2
  3.1.0
  3.1.0
  3.1.1
  3.1.1
  3.1.2
  3.1.2
  3.1.2
  3.1.3
  3.1.3
  3.1.3
  3.2.0rc1
  3.2.0rc1
  3.2.0rc1
  3.2.0rc3
  3.2.0rc3
  3.2.0rc3

Process finished with exit code 0

example.py

"""
Uses https://github.com/jwodder/pypi-simple to filter a package and give
back the greatest version that fits our requirements.

"""

from dataclasses import dataclass
import logging
import typing

from packaging.version import Version, parse
from pypi_simple import PyPISimple, DistributionPackage


log = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG)


@dataclass
class WheelMeta:
    #: Distribution aka. package name
    distribution: str
    version: str
    build_tag: str
    python_tag: str
    abi_tag: str
    platform_tag: str


def parse_wheel_filename_format(filename):
    """
    Return the (pkg_name, version, build_tag or 0, python tag, abi tag, platform_tag) tuple for the given wheel filename

    PEP427 states that a wheel's filename convention is:

    {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

    Ref: https://www.python.org/dev/peps/pep-0427/#file-name-convention

    """
    if filename.endswith('.whl'):
        filename = filename[:-4]
    parts = filename.split('-')
    names = ['distribution', 'version', 'build_tag', 'python_tag', 'abi_tag', 'platform_tag']
    kwargs = {}
    if len(parts) == len(names)-1:
        names.remove('build_tag')
        kwargs['build_tag'] = ''
    if len(parts) != len(names):
        raise ValueError(f'Unexpected number of parts to the wheel filename: {parts} {names}')
    kwargs.update(zip(names, parts))
    return WheelMeta(**kwargs)




def pypi_simple_example(pkg_name: str) -> DistributionPackage:
    client = PyPISimple()
    # client = PyPISimple(endpoint='http://example.com/custom-endpoint/simple')

    # Get *all* of the packages on the repository. There isn't really a
    # downside to not filtering at this point, as the simple endpoint doesn't
    # support filtering!
    packages: typing.List[DistributionPackage] = client.get_project_files(f'{pkg_name}')
    if not packages:
        raise RuntimeError('No packages found')

    # Quickly filter out the packages that have been removed (yanked), and
    # those which aren't wheels.
    packages = [pkg for pkg in packages
                if pkg.package_type == 'wheel' and pkg.yanked is None]

    # Quickly filter out only those packages that are suitable for manylinux and python3.
    packages = [pkg for pkg in packages
                if 'manylinux' in parse_wheel_filename_format(pkg.filename).platform_tag
                and 'cp3' in parse_wheel_filename_format(pkg.filename).python_tag
                ]

    def greatest_release(pkg: DistributionPackage):
        """
        Return a sort key that gives us the "greatest release", where
        pre-releases are sorted first/lowest.

        """
        vn = parse(pkg.version)
        is_pre = bool(vn.is_prerelease or vn.is_devrelease or vn.is_postrelease or False)
        return -int(is_pre), vn.release

    # packages include pre/dev/post releases, so sort them out.
    pkgs = sorted(packages, key=greatest_release)
    latest: DistributionPackage = pkgs[-1]

    versions = [parse(pkg.version) for pkg in packages]
    _debug_versions = ''.join([f'\n  {version}' for version in versions])
    log.debug(f'Found {len(packages)} versions for "{pkg_name}": {_debug_versions}')

    return latest


if __name__ == '__main__':
    latest = pypi_simple_example('matplotlib')

    print(f'Choose {latest.version}: {latest.filename}')