Skip to content

Instantly share code, notes, and snippets.

@FFY00
Last active May 9, 2023 16:52
Show Gist options
  • Select an option

  • Save FFY00/a77e7235716089bcb31c2a8e54d838ba to your computer and use it in GitHub Desktop.

Select an option

Save FFY00/a77e7235716089bcb31c2a8e54d838ba to your computer and use it in GitHub Desktop.

Feature or enhancement

sysconfig should export all the information required to build extensions.

Pitch

The new API should try to expose the information in a compiler-agnostic way.

We should probably still keep sysconfig.get_config_vars(), as an escape hatch, but what should we do to get the users to move to the new API? Perhaps just put a warning on the docs, mentioning the data is essentially unstable?

Information to expose

  • Extension suffixes
    • definition: See PEP 3149
      • consideration: PEP 3149 only defines the names for POSIX
      • design: The API should identify the different extensions types (interpreter, stable abi, none)
    • currently: importlib.machinery.EXTENSION_SUFFIXES, sysconfig.get_config_var('EXT_SUFFIX'), sysconfig.get_config_var('SHLIB_SUFFIX')
  • Is a shared libpython available? What's its name? Location?
    • definition: Yes if built with --enable-shared
    • currently:
      • (available?) bool(sysconfig.get_config_var('LDLIBRARY'))
      • (name?) sysconfig.get_config_var('LDLIBRARY')
      • (location?) sysconfig.get_config_var('LIBDIR')
  • Is a static libpython available? What's its name? Location?
    • definition: Yes unless built with --without-static-libpython
    • currently:
      • (available?) bool(sysconfig.get_config_var('LIBRARY'))
      • (name?) sysconfig.get_config_var('LIBRARY')
      • (location?) sysconfig.get_config_var('LIBDIR')
  • Should extensions link against libpython?
    • definition: Yes on Android and Cygwin, no everywhere else (see GH-65735)
    • currently: bool(sysconfig.get_config_var('LIBPYTHON'))
  • Is it a debug build?
    • definition: Yes if built with --with-pydebug
    • currently: bool(sysconfig.get_config_var('Py_DEBUG'))
  • C flags needed when using the C API
    • definition:
      • -fwrapv (until GH-96821)
        • Currently, we have unsafe code, and for a lack of a better option we assume it made into C macros, so extensions that use the C API should use -fwrapv.
  • what else? please comment...
"definition" sections list the answer, if a question, or the value(s) of the item. "currently" sections list the current way of fetching the information.

This is list is a work in progress. Currently, we need to gather all the information that is required to build extensions. If you see anything that's missing, please comment!

Unsure

  • Vendor CFLAGS and LDFLAGS
    • I know how much sense this makes, but on some systems, extensions might need certain CFLAGS/LDFLAGS

API Design

TBD

Vendor configuration

In certain scenarios, it might make sense to allow venders

Unsure

  • Allow vendors to override "Should extensions link against libpython?"
    • Is it wort

Previous discussion

https://discuss.python.org/t/building-extensions-modules-in-a-post-distutils-world/23938

Feature or enhancement

Provide new syconfig API that is aligned to how things have evolved.

Fixes GH-99942

Pitch

Over the years, several aspects of Python installations have evolved and the current sysconfig has trouble keeping up with that (eg. https://discuss.python.org/t/pep-582-python-local-packages-directory/963/387?u=ffy00). The goal is

Sub-issues

A lot of this may require extensive discussions with different parties, so I am splitting it into separate issues for the different parts of the API.

  • Install locations (GH-XXX)
  • Build information
    • Compilation (GH-XXX)
    • Compatibility (GH-XXX)
  • Miscellaneous (GH-XXX)

Previous discussion

This has been mentioned

https://discuss.python.org/t/building-extensions-modules-in-a-post-distutils-world/23938 https://discuss.python.org/t/sysconfig-should-provide-more-information-to-help-packaging/22950 https://discuss.python.org/t/pep-582-python-local-packages-directory/963/391?u=ffy00

@encukou
Copy link
Copy Markdown

encukou commented Apr 5, 2023

Would you consider updating sysconfig's CLI to be in scope?

Currently we ship a separate executable called python-config, which is a pain to keep in sync with python (when you have python3-config, python3.11-config etc.). Could python -m sysconfig accept all the options from python-config, becoming a drop-in replacement?

$ python-config --help
Usage: /usr/bin/python3.10-x86_64-config --prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir|--embed

@mattip
Copy link
Copy Markdown

mattip commented Apr 5, 2023

+1 for this idea

@FFY00
Copy link
Copy Markdown
Author

FFY00 commented Apr 5, 2023

Yes, and I am aware of the CPython issue about this. I am just focusing on the API design right now, as that might change several things.

@mattip
Copy link
Copy Markdown

mattip commented May 9, 2023

Is this moving to a more public forum?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment