gh-92584: Remove the distutils package #99061

vstinner · 2022-11-03T14:33:13Z

Remove the distutils package. It was deprecated in Python 3.10 by PEP 632 "Deprecate distutils module". For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils.

Remove Lib/distutils/ directory
Remove Doc/installing/ directory
Remove test_distutils
Remove references to distutils
Skip test_check_c_globals and test_peg_generator since they use distutils

Issue: PEP 632: Remove the distutils package (Python 3.12) #92584

vstinner · 2022-11-03T14:39:23Z

There are still many references to distutils. I prefer to remove/update them in separated PRs to keep this PR as small as possible.

Doc/distributing/index.rst:7::Email: distutils-sig@python.org
Doc/distributing/index.rst:44:* ``distutils`` is the original build and distribution system first added
Doc/distributing/index.rst:45:  to the Python standard library in 1998. While direct use of ``distutils``
Doc/distributing/index.rst:51:* `setuptools`_ is a (largely) drop-in replacement for ``distutils`` first
Doc/distributing/index.rst:53:  ``distutils`` tools was the ability to declare dependencies on other
Doc/distributing/index.rst:55:  alternative to ``distutils`` that offers consistent support for more
Doc/distributing/index.rst:58:  command to ``distutils``/`setuptools`_. This produces a cross platform

Doc/extending/building.rst:53:Python 3.12 and newer no longer come with distutils. Please refer to the
Doc/extending/newtypes_tutorial.rst:203:   from distutils.core import setup, Extension
Doc/extending/newtypes_tutorial.rst:223:   While this documentation showcases the standard :mod:`distutils` module
Doc/extending/newtypes_tutorial.rst:530:   from distutils.core import setup, Extension
Doc/extending/windows.rst:17:Module authors are encouraged to use the distutils approach for building

Doc/includes/setup.py:1:from distutils.core import setup, Extension
Doc/includes/test.py:192:from distutils.util import get_platform

Doc/install/index.rst:15:   The entire ``distutils`` package has been deprecated and will be
Doc/install/index.rst:18:   :ref:`What's New <distutils-deprecated>` entry for more information.
Doc/install/index.rst:47:In Python 2.0, the ``distutils`` API was first added to the standard library.
Doc/install/index.rst:56:using ``distutils`` directly.
Doc/install/index.rst:634:   that change the installation path will be ignored from all distutils configuration
Doc/install/index.rst:748:| system       | :file:`{prefix}/lib/python{ver}/distutils/distutils.cfg` | \(1)  |
Doc/install/index.rst:750:| personal     | :file:`$HOME/.pydistutils.cfg`                           | \(2)  |
Doc/install/index.rst:760:| system       | :file:`{prefix}\\Lib\\distutils\\distutils.cfg` | \(4)  |
Doc/install/index.rst:762:| personal     | :file:`%HOME%\\pydistutils.cfg`                 | \(5)  |
Doc/install/index.rst:776:   :file:`{prefix}/lib/python1.5/site-packages/distutils`, so the system
Doc/install/index.rst:791:   :file:`C:\\Python\\Lib\\distutils\\distutils.cfg`. Under Python 1.5.2, the
Doc/install/index.rst:794:   :file:`C:\\Program Files\\Python\\distutils\\distutils.cfg` in a standard Python

Doc/library/importlib.metadata.rst:90:``console_scripts``, ``distutils.commands`` and others.  Each group contains a
Doc/library/importlib.metadata.rst:130:    ['console_scripts', 'distutils.commands', 'distutils.setup_keywords', 'egg_info.writers', 'setuptools.installation']

Doc/library/sysconfig.rst:72:Every new component that is installed using :mod:`distutils` or a

Doc/library/zipapp.rst:351:command line tools, or you can take advantage of the fact that distutils
Doc/library/zipapp.rst:354:   >>> from distutils.ccompiler import new_compiler
Doc/library/zipapp.rst:355:   >>> import distutils.sysconfig
Doc/library/zipapp.rst:364:   >>>     cc.add_include_dir(distutils.sysconfig.get_python_inc())

Doc/using/configure.rst:681:the :mod:`distutils` module.
Doc/using/configure.rst:754:   distutils :envvar:`CFLAGS` once Python is installed (:issue:`21121`).
Doc/using/configure.rst:877:   the distutils :envvar:`LDFLAGS` once Python is installed (:issue:`35257`).

Doc/using/mac.rst:130:* Packages can be installed via the standard Python distutils mode (``python
Doc/using/windows.rst:1208:`cx_Freeze <https://cx-freeze.readthedocs.io/en/latest/>`_ is a ``distutils``

Lib/_aix_support.py:9:    # _aix_support is used in distutils by setup.py to build C extensions,

Lib/_osx_support.py:91:    # the build of Python itself (distutils is used to build standard library
Lib/_osx_support.py:396:        # see also distutils.sysconfig
Lib/_osx_support.py:452:    distutils.sysconfig.get_config_vars() is first
Lib/_osx_support.py:458:    requested (in distutils.sysconfig.customize_compiler).
Lib/_osx_support.py:460:    Currently called from distutils.sysconfig
Lib/_osx_support.py:484:    in distutils.sysconfig.customize_compiler.
Lib/_osx_support.py:501:    # called from get_platform() in sysconfig and distutils.util
Lib/_osx_support.py:504:    # distutils' perspective is what MACOSX_DEPLOYMENT_TARGET is set
Lib/_osx_support.py:519:        # Otherwise, distutils may consider this a cross-compiling

Lib/test/test__osx_support.py:316:        # indirectly by test_sysconfig and test_distutils

Lib/site.py:84:# for distutils.commands.install

Lib/sysconfig.py:431:                # This is disabled when called from distutils.sysconfig

Lib/test/support/__init__.py:1853:    # TODO (PEP 632): alternate check without using distutils
Lib/test/support/__init__.py:1854:    from distutils import ccompiler, sysconfig, spawn, errors

Lib/test/test_check_c_globals.py:6:# TODO: gh-92584: c-analyzer uses distutils which was removed in Python 3.12
Lib/test/test_check_c_globals.py:7:raise unittest.SkipTest("distutils has been removed in Python 3.12")

Lib/test/test_importlib/test_windows.py:21:    # Port of distutils.util.get_platform().

Lib/test/test_peg_generator/__init__.py:6:# TODO: gh-92584: peg_generator uses distutils which was removed in Python 3.12
Lib/test/test_peg_generator/__init__.py:7:raise unittest.SkipTest("distutils has been removed in Python 3.12")

Mac/BuildScript/build-installer.py:1332:    # distutils.tests.test_sysconfig.SysconfigTestCase tests that
Mac/BuildScript/build-installer.py:1334:    # distutils.sysconfig is exactly the same for both CFLAGS and

Mac/PythonLauncher/Makefile.in:18:# by distutils

Makefile.pre.in:86:# Use it when a compiler flag should _not_ be part of the distutils CFLAGS
Makefile.pre.in:90:# Use it when a linker flag should _not_ be part of the distutils LDFLAGS
Makefile.pre.in:187:# by distutils. The export statement is needed to ensure that the

PC/pyconfig.h:276:                        generally taken care of by distutils.) */

Tools/c-analyzer/c_parser/preprocessor/__init__.py:228:    # matching distutils.ccompiler.compiler_class:
Tools/c-analyzer/c_parser/preprocessor/common.py:2:import distutils.ccompiler
Tools/c-analyzer/c_parser/preprocessor/common.py:80:    compiler = distutils.ccompiler.new_compiler(

Tools/peg_generator/pegen/build.py:46:    of distutils (this is useful in case you want to use a temporary directory).
Tools/peg_generator/pegen/build.py:52:    import distutils.log
Tools/peg_generator/pegen/build.py:53:    from distutils.core import Distribution, Extension
Tools/peg_generator/pegen/build.py:54:    from distutils.tests.support import fixup_build_ext  # type: ignore
Tools/peg_generator/pegen/build.py:56:    from distutils.ccompiler import new_compiler
Tools/peg_generator/pegen/build.py:57:    from distutils.dep_util import newer_group
Tools/peg_generator/pegen/build.py:58:    from distutils.sysconfig import customize_compiler
Tools/peg_generator/pegen/build.py:61:        distutils.log.set_threshold(distutils.log.DEBUG)

configure.ac:727:# it may influence the way we can build extensions, so distutils
configure.ac:905:  By default, distutils will build C++ extension modules with "$CXX".
configure.ac:1375:# distutils.unixccompiler to know if it should add --enable-new-dtags
configure.ac:1732:  # compile working code using it and both test_distutils and test_gdb are
configure.ac:2450:        # component of the string returned by distutils.get_platform().

vstinner · 2022-11-03T14:47:47Z

The doc build fails when I just removed Doc/installing/index.rst. I reverted this change.

Warning: /home/runner/work/cpython/cpython/Doc/distributing/index.rst:19: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/install/index.rst:23: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/install/index.rst:58: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/library/ensurepip.rst:34: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/tutorial/venv.rst:105: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/tutorial/venv.rst:207: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/tutorial/whatnow.rst:24: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/whatsnew/2.7.rst:2690: WARNING: undefined label: installing-index
Warning: /home/runner/work/cpython/cpython/Doc/whatsnew/3.4.rst:240: WARNING: undefined label: installing-index

It should be removed in a separated PR.

vstinner · 2022-11-03T16:20:10Z

Optional Windows (x86) CI job failed, I re-run the job ("2 re-run tests: test_asyncio test_threading"). It's just a random error, unrelated to this PR.

vstinner · 2022-11-03T16:27:37Z

My colleague @hroncok is already preparing Python packages in Fedora for this removal: Help needed triaging build failures without distutils. 250 packages are impacted: a build dependency to setuptools should be added to get distutils.

bedevere-bot · 2022-11-03T16:33:58Z

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 5334545 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

hroncok · 2022-11-03T16:48:12Z

My colleague @hroncok is already preparing Python packages in Fedora for this removal: Help needed triaging build failures without distutils. 250 packages are impacted: a build dependency to setuptools should be added to get distutils.

A handful of packages also falls to build with setutpools, because the distutils module from setuptools is not backward compatible with distutils.

zware

LGTM (though there's two more lines that can be removed in PC/layout/support).

PC/layout/support/python.props

vstinner · 2022-11-03T16:50:21Z

A handful of packages also falls to build with setutpools, because the distutils module from setuptools is not backward compatible with distutils.

That's... very scary :-(

CAM-Gerlach · 2022-11-03T17:41:37Z

The naming is confusing, but Doc/installing, which is stated to be removed in the OP (but later the docs changes were decided for a separate PR) is not the distutils docs, its the modern(ish), thin-shim guide on installing things with Python, and only mentions distutils one place, in the context of describing it as a deprecated legacy module. So that one sentence should be updated, but the rest is unrelated to distutils and its removal, if desired, is seems out of scope here.

Meanwhile, Doc/install is the actual distutils documentation, which should presumably be removed (it can always be accessed in older Python versions) but is not listed above or removed by this PR.

@vstinner Would you like me to help take care of the docs changes/removals in a followup PR, since I'm fairly familiar with them? They're also somewhat covered by a separate issue, #94249

Remove the distutils package. It was deprecated in Python 3.10 by PEP 632 "Deprecate distutils module". For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils. * Remove Lib/distutils/ directory * Remove test_distutils * Remove references to distutils * Skip test_check_c_globals and test_peg_generator since they use distutils

vstinner · 2022-11-03T18:04:26Z

The test suite passed on most buildbots: great! I will not wait for the few remaining buildbots (like Refleaks buildbots, removing pure Python code is unlikely to introduce such leak).

I addressed @zware's review in a second commit. I will merge my PR as soon as the CI pass on it.

vstinner · 2022-11-03T18:05:53Z

@vstinner Would you like me to help take care of the docs changes/removals in a followup PR

Yes! Help on updating the doc is welcomed. There is a lot of documentation mentioning distutils, and deciding if the doc must be removed or not is not easy.

vstinner · 2022-11-03T18:28:37Z

Ok, I removed the distutils packages. Now test_check_c_globals and test_peg_generator should be updated to no longer use distutils. And remaining references to distutils #99061 (comment) should be removed. @CAM-Gerlach is volunteer to handle the Doc/ part.

encukou · 2022-11-07T09:21:53Z

Lib/test/libregrtest/utils.py

-    # Clear assorted module caches.
-    # Don't worry about resetting the cache if the module is not loaded


This comment isn't specific to distutils. But it's probably not worth putting it back – it's not that hard to see what the code below does.

vstinner requested review from a team, pablogsal and lysnikolaou as code owners November 3, 2022 14:33

bedevere-bot mentioned this pull request Nov 3, 2022

PEP 632: Remove the distutils package (Python 3.12) #92584

Closed

bedevere-bot added the awaiting core review label Nov 3, 2022

vstinner added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 3, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 3, 2022

zware approved these changes Nov 3, 2022

View reviewed changes

PC/layout/support/python.props Outdated Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting core review labels Nov 3, 2022

vstinner added 2 commits November 3, 2022 19:03

Update PC/layout/

eca9891

vstinner merged commit 0faa0ba into python:main Nov 3, 2022

bedevere-bot removed the awaiting merge label Nov 3, 2022

vstinner deleted the remove_distutils branch November 3, 2022 18:27

CAM-Gerlach mentioned this pull request Nov 3, 2022

[Docs] remove distutils legacy ref when setuptools docs are complete #94249

Open

9 tasks

rgommers mentioned this pull request Nov 6, 2022

PEP 632, deprecation of distutils and numpy.distutils numpy/numpy#18588

Closed

encukou reviewed Nov 7, 2022

View reviewed changes

hugovk mentioned this pull request Nov 29, 2022

[Distutils] strtobool #99867

Closed

astrojuanlu mentioned this pull request Jan 31, 2023

Updated distutils.Version to packaging.Version plotly/plotly.py#3897

Merged

5 tasks

fperrin mentioned this pull request Jul 24, 2023

Fix binary type representation robshakir/pyangbind#297

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-92584: Remove the distutils package #99061

gh-92584: Remove the distutils package #99061

vstinner commented Nov 3, 2022 •

edited by bedevere-bot

Loading

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

bedevere-bot commented Nov 3, 2022

hroncok commented Nov 3, 2022

zware left a comment

vstinner commented Nov 3, 2022

CAM-Gerlach commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

encukou Nov 7, 2022 •

edited

Loading

		# Clear assorted module caches.
		# Don't worry about resetting the cache if the module is not loaded

gh-92584: Remove the distutils package #99061

gh-92584: Remove the distutils package #99061

Conversation

vstinner commented Nov 3, 2022 • edited by bedevere-bot Loading

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

bedevere-bot commented Nov 3, 2022

hroncok commented Nov 3, 2022

zware left a comment

Choose a reason for hiding this comment

vstinner commented Nov 3, 2022

CAM-Gerlach commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

encukou Nov 7, 2022 • edited Loading

Choose a reason for hiding this comment

vstinner commented Nov 3, 2022 •

edited by bedevere-bot

Loading

encukou Nov 7, 2022 •

edited

Loading