Skip to content

Instantly share code, notes, and snippets.

@JeanChristopheMorinPerso

JeanChristopheMorinPerso/Notes on multi-platform wheels in OTIO.md

Last active December 20, 2021 21:18
Show Gist options
  • Save JeanChristopheMorinPerso/f5cd307243a8164b1da9ebbf63152c93 to your computer and use it in GitHub Desktop.
Save JeanChristopheMorinPerso/f5cd307243a8164b1da9ebbf63152c93 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Notes on multi-platform wheels in OTIO.md

Platforms:

  • Linux
  • Windows
  • macOS

Python versions: 2.7, 3.7, 3.8 and 3.9

OTIO 0.14.0 has 22 wheels produced and uploaded to PyPI (see https://pypi.org/project/OpenTimelineIO/#files).

cibuildwheel

Wheels are built in GitHub Actions using cibuildwheel. To build for Python 2.7, you will need cibuildwheel 1.12.0 (the last version that supports Python 2).

  • Linux:
  • Windows:
    • Build using Windows Server 2019 with Visual Studio 2019. Even our Python 2.7 wheels are using Visual Studio 2019.
    • The build happens on the build VM with no containerization.
  • macOS:
    • Build with macOS 10.15.
    • The build happens on the build VM with no containerization.
    • After each build it also runs delocate. This is automatically done by cibuildwheel.

auditwheel and delocate are tools mainly used to "repair" a built wheel by copying external library dependencies into the wheel tree and relinking them. There is also delvewheel on Windows, but cibuildwheel doesn't use it yet, stating the project is too young for now. So the goal is to make sure wheels are portable.

A note on Windows and Python 2.7

Building Python 2.7 wheels for Linux and macOS is relatively easy. Doing so on Windows is much more harder.

Why is it harder? Because it is expected that Python 2.7 is compiled with Visual C++ 9.0 (so Visual Studio 2008) (yes you read that right). Visual C++ 9.0 was released in 2008 and this is now impossible to find on the internet. Microsoft pulled the last plug on it in June 2021 (see this link which links to Microsoft Visual C++ Compiler for Python 2.7 which is now a dead link). Note that VCForPython27 can technically be downloaded from the Internet Archive.

So, the only solution left to us is to use what's available in GitHub Actions, which is Visual Studio 2019. Running a wheel built with VS 2019 inside a Python 2.7 interpreter build with VS 2008 works, with one caveat. The VCRUNTIME140.dll and VCRUNTIME140_1.dll needs to be available on the machine.

There is 4 options available:

  1. Document and ask our users to manually these DLLs.
  2. Automatically download and install these DLLs when installing OTIO.
  3. While creating the wheel, copy the DLLs inside the wheel.
  4. Statically link to the libraries (using the /MT flag).

Option 1 is really not user friendly and was discarded quite quickly. Option 2 is a little bit more "user" friendly, but who would like install a python package and get new vcruntimes installed at the same time? Not me, so the option was also discarded. Option 3 looks interesting and we considered it. But Windows being Windows, and DLL Hell being a thing, we instead opted the option 4, static linking. There is also the fact that if you go with option 3, you now have to find where these DLLs are to copy them in the wheel. This is simpler said than done. I mean, it can done done, but will it be robust and work on different systems?

Some history of all this can be found in AcademySoftwareFoundation/OpenTimelineIO#1015.

You can see here waht needs to be done in GitHub Actions for Python 2.7 on Windows: https://github.com/PixarAnimationStudios/OpenTimelineIO/blob/1325927157564989952edf7c5f7c317fb90e1288/.github/workflows/python-package.yml#L97-L111.

Build dependencies

Most of our build dependencies are specified in the pyproject.toml file. This include cmake, setuptools, etc. This allows us to force a specific version of CMake if we want to (which won't affet the system in any way).

VFX Platform compatibility

See AcademySoftwareFoundation/OpenTimelineIO#1094.

Future work

  • Run tests against produced wheels and sdist (AcademySoftwareFoundation/OpenTimelineIO#1027).
    • Confirm that they work as expected and that they contain all the necessary files.
    • The sdist is important as installers fallback to them when no wheel matches the environment. sdist for 0.14.0 was released in a broken state and testing it before the release would have catched the issue.
  • Wheels for macOS M1 (No GitHub issue yet).
  • Python 3.10
  • Automate more of the release process.
    • Some parts of the release process are now automated (wheel creation and upload to PyPI).
    • But more parts could be automated: Version up, release notes generation, tagging, etc.

Notes

  • If you use CMake
    • use CMake 3.18+.
    • Use the new FindPython stuff.
    • Try to not use Pybind11's PYBIND11_FINDPYTHON. Which basically means use what they call FindPython mode.
    • If building inside manylinux docker images, you will need to use the Development.Module component of FindPython (something like find_package(Python REQUIRED COMPONENTS Interpreter Development.Module))
  • Keep your setup.py simple.
  • Statically link your dependencies/libraries. It's much easier to distribute on multiple platforms support multiple python versions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment