test_capi leaks references

[Eclips4]

Bug report

Bug description:

./python -m test -R 3:3 test_capi
Using random seed: 2023738098
0:00:00 load avg: 1.04 Run 1 test sequentially
0:00:00 load avg: 1.04 [1/1] test_capi
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XXX XXX
test_capi leaked [160, 160, 160] references, sum=480
test_capi leaked [112, 112, 112] memory blocks, sum=336
test_capi failed (reference leak) in 36.5 sec

== Tests result: FAILURE ==

1 test failed:
    test_capi

Total duration: 36.5 sec
Total tests: run=840 skipped=4
Total test files: run=1/1 failed=1
Result: FAILURE

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-116915: Make _thread._ThreadHandle support GC #116934

[Eclips4]

Bisected to 33da0e8
cc @mpage @colesbury

[mpage]

Ugh, sorry about that. It looks like these are the failing tests:

test.test_capi.test_misc.SubinterpreterTest.test_module_state_shared_in_global
test.test_capi.test_misc.SubinterpreterTest.test_overridden_setting_extensions_subinterp_check
test.test_capi.test_misc.TestPendingCalls.test_isolated_subinterpreter

If I exclude them the refleaks test pass:

> cat ~/local/scratch/exclude.txt
test.test_capi.test_misc.SubinterpreterTest.test_module_state_shared_in_global
test.test_capi.test_misc.SubinterpreterTest.test_overridden_setting_extensions_subinterp_check
test.test_capi.test_misc.TestPendingCalls.test_isolated_subinterpreter
> 
> ./python -m test -R 3:3 test_capi --ignorefile ~/local/scratch/exclude.txt
Using random seed: 583416961
0:00:00 load avg: 4.81 Run 1 test sequentially
0:00:00 load avg: 4.81 [1/1] test_capi
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XX. ...
test_capi passed in 4 min 48 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 4 min 48 sec
Total tests: run=837 (filtered) skipped=4
Total test files: run=1/1 (filtered)
Result: SUCCESS
>

There appears to be a weird interaction with subinterpreters, thread handles that are created with _thread._make_thread_handle, and calling os.register_at_fork(). I've reduced the repro down to the following:

import textwrap
import unittest

from test.support import run_in_subinterp


class ReproLeakTest(unittest.TestCase):
    def test_repro_leak(self):
        run_in_subinterp(textwrap.dedent("""
        import os
        import _thread

        _main_thread = _thread._make_thread_handle(100)

        def after_fork():
            pass

        os.register_at_fork(after_in_child=after_fork)
        """))

which fails with:

> ./python -m test -R 3:3 test_repro_leak
Using random seed: 2054577432
0:00:00 load avg: 2.33 Run 1 test sequentially
0:00:00 load avg: 2.33 [1/1] test_repro_leak
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XXX XXX
test_repro_leak leaked [20, 20, 20] references, sum=60
test_repro_leak leaked [14, 14, 14] memory blocks, sum=42
test_repro_leak failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test_repro_leak

Total duration: 334 ms
Total tests: run=1
Total test files: run=1/1 failed=1
Result: FAILURE
>

I'll dig a bit more later.

[vstinner]

See also issue gh-116946: Heap types implemented in C must have Py_TPFLAGS_HAVE_GC and implement tp_traverse.

[vstinner]

I confirm that ./python -m test -R 3:3 test_capi leaks and that merged PR gh-116934 fix the leak. I close the issue.

Follow-up: see issue gh-116946 "Heap types implemented in C must have Py_TPFLAGS_HAVE_GC and implement tp_traverse".