[Apache Beam] Python pkg_resources incompatibility

README.md

Apache Beam's Python SDK does not play nicely with the pkg_resources.require API. More generally, it fails due to badly defined requirement versions between grpcio and protobuf.

This makes using apache-beam in any python project which is consumed as a library itself incapable of using the SDK.

To see the failure, use the code in this gist and run:

virtualenv apache-beam-requirements && source apache-beam-requirements/bin/activate
pip install -e .
python runner.py

A stack trace should be produced, similar to the following:

Traceback (most recent call last):
  File "runner.py", line 9, in <module>
    main()
  File "runner.py", line 5, in main
    pkg_resources.require('library')
  File "<venv>/lib/python2.7/site-packages/pkg_resources/__init__.py", line 984, in require
    needed = self.resolve(parse_requirements(requirements))
  File "<venv>/lib/python2.7/site-packages/pkg_resources/__init__.py", line 875, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (protobuf 3.3.0 (<venv>/lib/python2.7/site-packages), Requirement.parse('protobuf>=3.5.0.post1'), set(['grpcio']))

For another, likely more applicable use case, try creating a pyramid web application and depending on apache-beam. The PasteDeploy module archicture makes explicit use of pkg_resources.require during application load time, which will cause the given pyramid app to bail before starting.

library.py

print("Package require suceeded.")

runner.py

import pkg_resources


def main():
    pkg_resources.require('library')


if __name__ == '__main__':
    main()

setup.py

from setuptools import setup
from setuptools import find_packages

REQUIREMENTS = ['apache-beam[gcp]==2.2.0']

setup(
    name='library',
    version='0.0',
    author='',
    author_email='',
    packages=find_packages(),
    install_requires=REQUIREMENTS
)