pkg_resources alternatives

pkg_resources_alternatives.md

PKG-Resource Entry Points: Problems/Alternatives

Introduction

This gist summarizes some issues with using the python pkg_resources module, in particular its entry_points functionality for python plugin support, specifically in the context of how the OpenTimelineIO project uses it.

Background

The setuptools module is the older system for packaging and installing python code. The pkg_resource module from setuptools can be used to look up information about packages that have been installed.

Unfortunately, in certain situations, using pkg_resources is slow. For examples of this:

pypa/setuptools#510 catkin/catkin_tools#297

The issue appears to be that it validates that packages are installed on import/use.

For OpenTimelineIO there are two main ways that pkg_resouces is used:

• console shims: when you run python setup.py install, the console script shims that are generated by the installed include calls to pkg_resources that will be slower than they ought to be. For OTIO, this generates otiocat, otioconvert, etc.
• entry_points plugins: OTIO's python plugin system supports the entry_points mechanism from pkg_resources for discovering installed plugins without having to configure the OTIO_PLUGIN_MANIFEST_PATH.

Alternatives for generating console shim scripts

The development team has been using (and recommending) that developers use python setup.py install while iterating locally on projects. pip install . (without any options) is considerably slower and doesn't appear to re-use the build state, meaning it is a full clean and rebuild of the C++ library. pip install -e . doesn't appear to work, likely because of the interaction with the cmake system.

pip install rather than setup.py install

Using pip install will generate a shim script that does not use pkg_resources. For example, this is otiocat from python setup.py install:

# EASY-INSTALL-ENTRY-SCRIPT: 'OpenTimelineIO==0.14.0.dev1','console_scripts','otiocat'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'OpenTimelineIO==0.14.0.dev1'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point


def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()


globals().setdefault('load_entry_point', importlib_load_entry_point)


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('OpenTimelineIO==0.14.0.dev1', 'console_scripts', 'otiocat')())

This is the one generated by pip install:

import re
import sys
from opentimelineio.console.otiocat import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Note We have discussed breaking apart the subprojects more. If we did this, it might address the issues with pip install and even enable pip install . -e to work.

use "fast_entry_points" module

Given that python setup.py install is considerably faster than pip install for our current configuration, it would be nice to continue using it while also getting the benefits of faster shim scripts. This fast_entry_points module could allow us to continue using it:

https://github.com/ninjaaron/fast-entry_points

Alternatives for iter_entry_points

Using pkgutils instead

This page has two alternative strategies to use instead of entry_points: https://packaging.python.org/guides/creating-and-discovering-plugins/#using-package-metadata

Neither of these alternatives is a 1:1 replacement for the way that entry points works.

• would require changing the plugin discovery system in OTIO
• would require changing any plugins that use the existing mechanism

Using the reentry module

the "reentry" module claims: "A plugin manager based on setuptools entry points with 10x the speed".

https://pypi.org/project/reentry/1.2.0a2/

• would only need to be added to the OTIO side (plugins wouldn't need to change)
• looks like minor changes to the core

Off switch for pkg_resource feature

Another option is to provide a switch to disable pkg_resource features (and importing) in python_plugin.py. This could be done with an environment variable.

This PR does that: AcademySoftwareFoundation/OpenTimelineIO#928

Recommendations/Conclusions

• Add an environment variable to disable pkg_resources for users who are dealing with immediate performance issues (DONE: AcademySoftwareFoundation/OpenTimelineIO#928)
• Re-architect the plugin system to use the newer pkgutil libraries to locate plugins automatically
• Breaking the repository up might address the issues with pip install ., but will be a lot of work.