Building and linking C extensions in a post-distutils world

[eli-schwartz]

With the removal of distutils from 3.12 alphas, I've recently taken a new look at the use of distutils within https://mesonbuild.com in the hopes that we were finally unblocked and could migrate to sysconfig.

• install schemes turn out to finally be viable since 3.10.3, when the deb_system scheme patch on Debian operating systems for distutils was patched into sysconfig. This hard blocker is gone, thank G-d.
• linking to libpython itself may or may not be an issue, originally I thought it was, but now I think it may not be.

How can a C-extension supporting PEP 517 build backend know the best way to compile and link?

python <3.12

Meson has some code: https://github.com/mesonbuild/meson/blob/master/mesonbuild/modules/python.py#L338-L407

Here's the interesting bit:

def links_against_libpython():
    from distutils.core import Distribution, Extension
    cmd = Distribution().get_command_obj('build_ext')
    cmd.ensure_finalized()
    return bool(cmd.get_libraries(Extension('dummy', [])))

python >=3.8

In bpo-36721, bpo-21536 etc the above code changes from returning True to False, on many Unix platforms.

New additional methods suitable for getting C extension arguments are available. Well, mostly suitable.

• sysconfig.get_config_var('LIBPYTHON')

this is configured into:

• pkg-config --cflags --libs python3
• python-config --embed

There's a couple problems with this:

• on Cygwin and Android, LIBPYTHON and thus pkg-config is hardcoded to link to libpython, but distutils only does so when Py_ENABLE_SHARED
• none of it works on Windows, which since it is built with PCBuild, doesn't have Makefile config vars (and also / consequently? distributes neither of the latter two)
• pkg-config may not always be installed, so we want a fallback for that
• python-config has tons of CPython build-time junk so we cannot use it

...

It feels uncomfortably like there's still way too much undocumented magic here.

@vstinner, I assume the Cygwin/Android inconsistency is probably a configure.ac bug that should behave the same way distutils does/did? I can provide a patch to fix it but would like confirmation of which side to fix.

@FFY00, I think in the long term, sysconfig should expose a function that returns cflags / libs required to build an extension (and works on Windows without _generate_posix_vars, and doesn't include lots of -O3 -pipe -fstack-protector-strong -fdiagnostics-color=always and more -- i.e. works like pkg-config, not like python-config). Even without the question of "whether to link to libpython", there's a 140-line class for figuring out what the name of the library is, which directory to find it in, and what the include directory is too. Of course, this is specific to the case where pkg-config is not available, and most of it is for the Windows case.

Linked PRs

• gh-99942: python.pc on android/cygwin should link to libpython per configure.ac #100356
• gh-99942: correct the pkg-config/python-config flags for cygwin/android #100967

[zooba]

The Windows case really isn't complicated because the linking is all set up to isolate the two sides of the ABI. The only thing that's going to bleed through is C runtime state when you mix a debug build with a release build (which use two different CRT instances). Virtually nobody actually has a debug build of CPython, and those who do will know because the filenames are different, so there's nothing really to pick up from the interpreter.

Now, the complicated bit is finding the compiler, in effect, this bit:

cpython/Lib/distutils/_msvccompiler.py

Lines 66 to 80 in 7571764

	root = os.environ.get("ProgramFiles(x86)") or os.environ.get("ProgramFiles")
	if not root:
	return None, None
	try:
	path = subprocess.check_output([
	os.path.join(root, "Microsoft Visual Studio", "Installer", "vswhere.exe"),
	"-latest",
	"-prerelease",
	"-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
	"-property", "installationPath",
	"-products", "*",
	], encoding="mbcs", errors="strict").strip()
	except (subprocess.CalledProcessError, OSError, UnicodeDecodeError):
	return None, None

This is assuming you want to compile with MSVC. If you want to compile an extension with a different compiler (many of which will work, since the ABI is well defined), you'll need different code. Again, this is best left to the tool figuring out how to run the build - one of the main reasons for dropping distutils is so that compiler options don't get trapped by the CPython release cycle.

Otherwise, the location of Python's header files (typically {prefix}\Include) and import libraries (typically {prefix}\libs) are all you need. They should both be in sysconfig somewhere already, but since you can't use that when cross-compiling, we either need a new data file containing the paths (which I'm in favour of, but it wasn't popular last time we discussed it) or to just stick with the assumptions tools use today.

[eli-schwartz]

They should both be in sysconfig somewhere already, but since you can't use that when cross-compiling, we either need a new data file containing the paths (which I'm in favour of, but it wasn't popular last time we discussed it) or to just stick with the assumptions tools use today.

There's also the undocumented $_PYTHON_SYSCONFIGDATA_NAME environment variable, which is unfortunately only available on posix. But POSIX systems do often make the assumption that they can convince e.g. distutils to cross compile by setting that environment variable before running any build tool (pip, python -m build, python setup.py ...).