Implement and document PEP 669.

[markshannon]

Now that PEP 669 is accepted, it needs to be implemented and documented.

• Implement the PEP
• Decide on and implement policy for monitors seeing the code in other monitors.
• Document the PEP
• Use "macro" instructions to reduce code duplication in bytecodes.c

Linked PRs

• GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython #103083
• GH-103082: Code cleanup from review. #103474
• GH-103082: Turn on branch events for FOR_ITER instructions. #103507
• gh-103082: Fix shifted field initialization in instrumentation.c #103561
• GH-103082: Filter LINE events in VM, to simplify tool implementation. #104387
• GH-103082: Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS #107069
• [3.12] GH-103082: Rename PY_MONITORING_EVENTS to _PY_MONITORING_EVENTS (#107069) #107075
• GH-103082: Document PEP-669: Low Impact Monitoring for CPython #107772
• gh-103082: use IS_VALID_OPCODE instead of _PyOpcode_OpName to check if an opcode is defined #107882
• gh-103082: remove assumption that INSTRUMENTED_LINE is the last instrumented opcode #107978
• [3.12] GH-103082: Document PEP-669: Low Impact Monitoring for CPython (GH-107772) #108909
• [3.12] GH-103082: Clean up the sys.monitoring docs #111048

[markshannon]

Also needs to be added to the "What's new" and the PEP needs to be marked as accepted/final.

[markshannon]

Specific things to document (that aren't clear from the PEP) (from comments by @gvanrossum):

event_set is a "set" constructed by OR-ing various events together (a common C idiom of course but not apparent until you see their integer values).

From GVR: "The naming of the CALL and RETURN/RAISE events is a bit confusing -- we have CALL (universal), C_RETURN/PY_RETURN (per language), but then C_RAISE and RAISE -- why isn't the latter PY_RAISE, just for consistency?"

There are good reasons for this design, so they should be documented.

[pablogsal]

Seems that this PR (#103083) has broken the tracerefs buildbot:

https://buildbot.python.org/all/#/builders/484/builds/3091

[pablogsal]

See also #103083 (comment)

[jpe]

Is there any way to get the active exception from a PY_UNWIND event callback? I was hoping the exception would be passed to the callback but it isn't. This was an issue with the old tracer function debugger / profiler support so it's not really a regression, but I was hoping to avoid the work arounds that we used with it.

Note the the signature for several of the events aren't specified -- including PY_UNWIND and EXCEPTION_HANDLED. They're easily enough determined (though my guess was wrong for PY_UNWIND), but it would be better if they were documented.

[jpe]

It looks like returning DISABLE from a handler for an event >= PY_MONITORING_INSTRUMENTED_EVENTS triggers an assert in a debug build. It would probably be better to just ignore a DISABLE return value since anything else can be returned (I think) or mandate that None be returned. If the current behavior is retained, it should be documented.

Please let me know if you want me to open separate bugs for this and for other sys.monitoring issues. Otherwise, I'lll add to this bug.

[jpe]

An exception raised in a EXCEPTION_HANDLED callback triggers an assert in a debug build of 3.12b4. I was using the following incorrect signature for a callback method and registering it as a bound method --

def _py_handled_event(self, instruction_offset: int, exception: BaseException):

This was my mistake (it was a simple oversight while editing) but it the calling code should handle the exception without crashing.

[jpe]

It looks like no RAISE event is emitted for a bare raise in an except: clause or at the end of a finally: clause that implicitly reraises an exception. I don't know whether this a problem with the implementation or it should simply be documented that this is to be expected. My use case is to track what exception is active so it's available to an unwind handler.

[carljm]

@jpe I think opening a new issue for each separable problem will make these easier to track and triage, and less likely they'll fall through the cracks.

[AA-Turner]

Is this issue now complete given #107772?