Python/compile.c:7482: int compute_code_flags(struct compiler *): Assertion `IS_TOP_LEVEL_AWAIT(c) || _PyST_IsFunctionLike(ste)' failed

[alex]

Crash report

What happened?

~/p/cpython ❯❯❯ ./python.exe -c 'compile("assert await u", "", "exec", optimize=2)'
Assertion failed: (IS_TOP_LEVEL_AWAIT(c) || _PyST_IsFunctionLike(ste)), function compute_code_flags, file compile.c, line 7482.
fish: Job 1, './python.exe -c 'compile("asser…' terminated by signal SIGABRT (Abort)

Full stacktrace:

fuzz_pycompile: Python/compile.c:7482: int compute_code_flags(struct compiler *): Assertion `IS_TOP_LEVEL_AWAIT(c) \|\| _PyST_IsFunctionLike(ste)' failed.
--
  | ==24717== ERROR: libFuzzer: deadly signal
  | #0 0x59033612cf71 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
  | #1 0x59033602fd98 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
  | #2 0x590336012ac3 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:231:3
  | #3 0x79353882e41f in libpthread.so.0
  | #4 0x79353862800a in __libc_signal_restore_set /build/glibc-SzIz7B/glibc-2.31/sysdeps/unix/sysv/linux/internal-signals.h:86:3
  | #5 0x79353862800a in raise /build/glibc-SzIz7B/glibc-2.31/sysdeps/unix/sysv/linux/raise.c:48:3
  | #6 0x793538607858 in abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79:7
  | #7 0x793538607728 in __assert_fail_base /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92:3
  | #8 0x793538618fd5 in __assert_fail /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101:3
  | #9 0x5903364b52da in compute_code_flags cpython3/Python/compile.c:7482:9
  | #10 0x5903364b52da in optimize_and_assemble cpython3/Python/compile.c:7578:22
  | #11 0x5903364af709 in compiler_mod cpython3/Python/compile.c:1597:10
  | #12 0x5903364af709 in _PyAST_Compile cpython3/Python/compile.c:422:24
  | #13 0x59033660c8a0 in Py_CompileStringObject cpython3/Python/pythonrun.c:1450:10
  | #14 0x59033660c994 in Py_CompileStringExFlags cpython3/Python/pythonrun.c:1463:10
  | #15 0x590336161d2e in fuzz_pycompile cpython3/Modules/_xxtestfuzz/fuzzer.c:551:24
  | #16 0x590336161d2e in _run_fuzz cpython3/Modules/_xxtestfuzz/fuzzer.c:570:14
  | #17 0x590336161d2e in LLVMFuzzerTestOneInput cpython3/Modules/_xxtestfuzz/fuzzer.c:711:11
  | #18 0x590336013fe0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
  | #19 0x590335ffe774 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6
  | #20 0x59033600420a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9
  | #21 0x590336030602 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
  | #22 0x793538609082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
  | #23 0x590335ff524d in _start
  |  

This regressed somewhere in f621618...690b935

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-121637: Syntax error for optimized-away incorrect await #121656

[alex]

cc: @iritkatriel -- it looks like you've got a few commits in this file in that range, though I don't see one that's obviously relevant.

[JelleZijlstra]

Feels related to #121297.

[sobolevn]

I will add both asyncio and interpreter-core, because both are related.

[JelleZijlstra]

Irit is on vacation, I'll look into this over the next few days.

[JelleZijlstra]

The problem is arguably this:

% python -c 'assert await 42'   
  File "<string>", line 1
SyntaxError: 'await' outside function
% python -O -c 'assert await 42'
%

It used to be that we didn't raise a SyntaxError if there is an await but it is optimized away, and the code would work fine.

Now, however, the symtable sets ste_coroutine when it sees the await, but it still doesn't raise an error about the await, leading to the crash reported here.

I can see two solutions:

• Make the code setting ste_coroutine aware of -O removing asserts, either by reverting gh-121272: compiler no longer modifies the symtable, ste_coroutine always set during symtable construction #121297 or by adding an awareness of asserts into symtable.c. The second suboption would be quite complex, and the first reintroduces the layering violation that gh-121272: compiler no longer modifies the symtable, ste_coroutine always set during symtable construction #121297 tried to fix.
• Make it so it's still a SyntaxError to await outside of a coroutine if the await is optimized away.

The latter seems like the more intuitive behavior for users.

It's also consistent with other cases where optimized-away code leads to a compile error:

% python -O
Python 3.12.1 (main, Apr 22 2024, 00:22:15) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
...     assert (x := 3)
...     global x
... 
  File "<stdin>", line 3
SyntaxError: name 'x' is assigned to before global declaration

So I'll send a PR to make python -O -c 'assert await 1' a SyntaxError.

[alex]

Always doing a syntax error, even if the assert is optimized out, feels like the right behavior to me.

[JelleZijlstra]

I realized there's a few more similar cases (essentially, every place we call compiler_error in compile.c while inside an expression):

% python -c 'assert (yield)'
  File "<string>", line 1
SyntaxError: 'yield' outside function
% python -O -c 'assert (yield)'
% python -O -c 'assert (__debug__ := True)'
% python -c 'assert (__debug__ := True)' 
  File "<string>", line 1
SyntaxError: cannot assign to __debug__
% python -O -c 'assert (__debug__ := True)'
% python -c 'assert [x async for x in ()]' 
  File "<string>", line 1
SyntaxError: asynchronous comprehension outside of an asynchronous function
% python -O -c 'assert [x async for x in ()]'

The latter also crashes on current main:

% ./python.exe -O -c 'assert [x async for x in ()]'
Assertion failed: (IS_TOP_LEVEL_AWAIT(c) || _PyST_IsFunctionLike(ste)), function compute_code_flags, file compile.c, line 7482.
zsh: abort      ./python.exe -O -c 'assert [x async for x in ()]'

[JelleZijlstra]

The linked PR makes symtable.c detect incorrect await and async comprehensions, fixing this issue.

I didn't bother detecting other compiler errors (e.g., invalid yields) because they don't cause similar crashes. I wouldn't be surprised though if there's a crash lurking somewhere with e.g. assert (yield) being optimized away, so maybe if you point your fuzzer in that direction it will find something :)

[gvanrossum]

Always doing a syntax error, even if the assert is optimized out, feels like the right behavior to me.

Of course! It must have been an oversight that we didn't do this from the start.

I'm removing the topic-asyncio label -- just because it mentions await doesn't make it an asyncio issue.

[JelleZijlstra]

Thanks for the report! This will now be a SyntaxError.