Skip to content

Instantly share code, notes, and snippets.

@FFY00

FFY00/version-matching-research.md

Created February 24, 2023 22:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save FFY00/dd56cde19e68a9138219ecf44bcb879d to your computer and use it in GitHub Desktop.
Save FFY00/dd56cde19e68a9138219ecf44bcb879d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
version-matching-research.md

Version matching

General approach for this kind of standards:

  • Optimize for the most use-cases
  • Provide a escape hatch for use-cases our general approach doesn't work

Background

Version

Python versions are defined by PEP 440, but here's a quick summary.

({epoch}!){release}({pre})({post})({dev})(+{local})
Note: {} denotes a placeholder, () denotes optional parts, and [] means "one of"
  • Epoch (epoch)
    • Way to override the release section, enabling forced downgrades (eg. 1!1.2.3 is greater than 2.0.0)
    • Format
      • integer
  • Release (release)
    • The main element of the release
    • Format
      • .-separated list of integers (eg. 1.2.3.4.5)
        • Must contain at least one element, and there's not limit for the number elements
  • Pre-release modifier (pre)
    • Makes the version a pre-release, meaning it comes before the normal version with the same release value (eg. 1.2.3.a0 is less than 1.2.3)
    • Format: ([-_.]){pre_letter}([-_.]){pre_number} (eg. a0, -a0, _a0, .a0, a.0, a-0, -a-0, etc.)
      • Letter (pre_letter)
        • One of the following:
          • alpha, a
          • beta, b,
          • preview, pre
          • c
          • rc
      • Number (pre_number)
        • integer
  • Post-release modifier
    • Makes the version a post-release, meaning it comes after the normal version with the same release value (eg. 1.2.3.post0 is less than 1.2.3)
    • Format (one of)
      • -{post_number} (eg. -1)
      • ([-_.]){post_letter}([-_.]){post_number} (eg. r0, -r0, _r0, .r0, r.0, -0, -r-0, etc.)
        • Letter (pre_letter)
          • One of the following: post, rev, r
        • Number (pre_number)
          • integer
  • Dev-release modifier
    • Makes the version a development release, meaning it comes before the normal version with the same release value (eg. 1.2.3.dev0 is less than 1.2.3)
    • Format: ([-_.])dev([-_.]){dev} (eg. dev0, -dev0, _dev0, .dev0, dev.0, dev-0, -dev-0)
      • Number (dev)
        • integer
  • Local identifier (local)
    • Purely informational identifier, which can be use for eg. to differentiate artifacts with otherwise the same name
Note: integers, in this context, are whole numbers, and can be either single or multi-digit (eg. 1, 123, 989898989)

Requirement strings

Python requirement strings are defined by PEP 508, but here's a quick summary.

{operator}{version}(;{env_marker})
Note: {} denotes a placeholder, and () denotes optional parts
  • Operator (operator)
    • Comparison operation to perform
    • Defined by PEP 440
    • Format (one of)
      • ==, =! (matching/exclusion operators)
        • Perform an equals / not equals operation
        • When using this operators, one of the release element might be a *, denoting a wildcard (eg. == 1.2.*, != 1.2.*)
      • <, >, <=, >=
        • Perform a greater/less than/or equal operation
      • ===
        • Performs an arbitrary equality operation (only matches against the exact same version)
      • ~=
        • Performs a "compatible" release operation
          • Equivalent to a greater than operation on the specified version, and a matching operation on the specified version with a wildcard on the last element (eg. ~= X.Y would be equivalent to >= X.Y, == X.* , ~= 1.2.3.4.5 would be equivalent to >= 1.2.3.4.5, == 1.2.3.4.*)
        • Cannot be used on versions single element releases (eg. ~= 1 would not be legal )
        • Cannot be used on versions with local identifiers (eg. ~= 1.2.3+something would not be legal)
  • Version (version)
    • Format
      • A PEP 440 version (described above)
  • Environment marker (env_marker)

Syntax

A: @-matching

A1: w/ custom pre/post/dev release logic

The release section of the version is matched using @ as a placeholder for release parts (eg. @.@.@).

All PEP 440 operators but != can be used (!= never makes sense when matching without arithmetic expressions).

If a dev or pre version is specified, the original version will be pinned, even if a wildcard is used (such cases will be logged with a info level).

Examples

  • 1.2.3 on == @.@.@ > == 1.2.3

  • 1.2.3.4.5 on == @.@.@ > == 1.2.3

  • 1.2.3.post0 on >= @.@.@ > >= 1.2.3

  • 1.2.3.dev0 on == @.@.@ > == 1.2.3.dev0

  • 1.2.3.dev0 on == @.@.* > == 1.2.3.dev0

  • 1.2.3.dev0 on >= @.@.@ > == 1.2.3.dev0

  • 1.2.3.dev0 on >= @ > == 1.2.3.dev0

    (errors)

  • 1.2 on == @.@.@

    • == @.@.@? can be an option to work around this, making the 3rd part optional, but I don't think there's enough need

B: Custom strategy with @-matching

TODO: Define syntax

Brainstorming

  • Matching

    • @ matching

      • On the release section (eg. @.@.@ on 1.2.3.4.5 > 1.2.3)
        • This makes dealing with the variable number of elements very easy
      • On the pre/post/dev sections (eg. @.@.@.dev@ on 1.2.3.dev@ > 1.2.3.dev0)
        • Semantics are very tricky, needs more info (XXX)
          • What about @.@.@.dev@ on 1.2.3.4.5.dev0?
            • Should we allow this? (first instinct: no)

    • Optional matching

      • Allow matching only if a section is present
        • Eg. @.@.@.dev@?
          • 1.2.3 > 1.2.3
          • 1.2.3.dev0 > 1.2.3.dev0
          • 1.2.3.4.5 > 1.2.3
          • 1.2.3.4.5.dev0 > 1.2.3.dev0
            • Should we allow this? (first instinct: no)
        • How do we handle the letter section in pre-releases (eg. 1.2.3.alpha0 and 1.2.3.beta0)

    • Conditional matching

      • Add a conditional to enable a certain constrain, similar to environment markers
        • Examples:
          • @.@.@ ; is_dev
          • @.@.@ ; >= 1.2.3
            • Good for version changes

    • Backup matching

      [tool.mesonpy.binary-runtime-constrains]
packageA = 'match(@.@.@)'
packageB = ['match(@.@.@)', 'match(@.@.@.dev0)']
packageC = ['match(@.@.@)', 'exact']
      • Strategy
        • If the value is a string, match it
        • If the value is a list, match the first element, if the resulting requirement string is valid for the matched version, use it, otherwise go to the next element

    • Custom strategy matching

      • We define certain matching strategies, which the users can use. Strategies can take arguments.
        • exact
        • compatible(...)
          • XXX: How to specify the version?
            • compatible(3) (1.2.3.4.5 > ~= 1.2.3)
            • compatible(@.@.@) (1.2.3.4.5 > ~= 1.2.3)

    • Just do it in code 😅

      • If none of these approaches are deemed good enough, or if matching is too complex for the other approaches, we can just add the runtime dependency constrain as a code option

        • This does not get rid of any of the issues, it just throws them to the user
          • Some of them may be more equipped to deal with these issues, but some probably aren't
          • If the logic is tricky (eg. what to match when a dev release is used), users will almost definitely mess it up
            • If we can avoid making users having to deal with this, we should probably do it
              • Optimizing for the most common use-cases might not be viable though

      • Users create a meson-python-config.py (or a better name) and do the matching themselves

        from collections.abc import Iterator

from mesonpy.types import Version

def runtime_constrains(name: str, version: Version) -> Iterator[str]:
    if name == 'packageA':
        yield f'~= {version.release[:3]}'
    elif name == 'packageB':
        if version.dev:
        	yield f'== {version}'
        else:
            yield f'~= {version.release[:3]}'
    elif name == 'packageC':
        if version >= Version('1.2.3.4'):
            yield f'~= {version.release[:4]}'
        else:
            yield f'~= {version.release[:3]}'

  • Matching issues

    • Failed matching check
      • A matching pattern can succeed, but it might fail to match against the version it was matched against
        • This only happens when we allow patching to dev/pre/post releases

  • Format

    • Full strings

      [tool.mesonpy]
binary-runtime-constrains = [
  'packageA = match(@.@.@)',
  'packageB = exact',
  'packageC = custom:identifier(args)',
]
      • Allows multiple constrains for the same package

    • String-mapping

      [tool.mesonpy.binary-runtime-constrains]
packageA = 'match(@.@.@)'
packageB = 'exact'
packageC = 'custom:identifier(args)'

    • Dict-mapping w/ inferred string default

      [tool.mesonpy.binary-runtime-constrains]
packageA = '@.@.@'
packageB = {strategy='exact'}
packageC = {strategy='custom:identifier' args=...}
      • TOML doesn't support multi-line {}-style mappings, so this would be a but unergonomic on more complex cases

Take-aways

  • @-matching is great for matching the release section
  • We probably want different matching logic for pre/dev versions
    • The most sane approach would probably be to forcibly pin those versions
  • The post part of versions is basically an annoying "minor" release part

Practical Examples

This details some of the examples we want to support.

Cases

Matching

Template: runtime version -> matched version

(generic) 1.2.3.4.5 -> ~= 1.2.3

This is the most common use-case, matching parts of the

  • A: ~= @.@.@

Numpy

(normal) 1.27.0 > ~= 1.27.0
  • A: ~= @.@.@
(dev release) 1.28.0.dev0 > ?

In these cases, we should probably pin the dev release, == 1.28.0.dev0.

  • A1: Custom mapper, the recommended way with @-matching would be ~= @.@.@, which translates into == 1.27.0.dev0
(pre-release) 1.27.0.alpha0 > ?

In these cases, we'd probably want to match to >= 1.27.0.alpha0, < 1.27.0

  • A1: Custom mapper, the recommended way with @-matching would be ~= @.@.@, which translates into == 1.27.0.alpha0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment