Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-114314: ctypes: remove stgdict and switch to heap types #116458

Merged
merged 60 commits into from
Mar 20, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
6155211
Move paramfunc into a new struct, PyStgInfo
encukou Feb 16, 2024
0171c30
Move the StgInfo out of StgDictObject and into the classes themselves
encukou Feb 16, 2024
acd91e9
Move StgInfo init/retrieval right after the corresponding StgDict one
encukou Feb 19, 2024
55af504
Move `size` from StgDict to StgInfo
encukou Feb 19, 2024
27e5c50
Move `align` from StgDict to StgInfo
encukou Feb 19, 2024
01bc637
Move `length` from StgDict to StgInfo, except for sizeof calculation
encukou Feb 19, 2024
c8017e9
Move part of the sizeof calculation
encukou Feb 19, 2024
40b012f
Move `ffi_type_pointer` from StgDict to StgInfo
encukou Feb 19, 2024
537defb
Move `proto` from StgDict to StgInfo. Doesn't work just yet.
encukou Feb 20, 2024
ace6623
Hack: Reserve space for StgInfo in static type objects
encukou Feb 20, 2024
6f775b9
If StgInfo is not initialized, act as if it's not there
encukou Feb 20, 2024
4aaa06a
Move `getfunc` & `setfunc`
encukou Feb 20, 2024
6808524
Move `argtypes`
encukou Feb 20, 2024
f031c00
Move `converters`
encukou Feb 20, 2024
535176c
Move `restype`
encukou Feb 20, 2024
5260003
Move `checker`
encukou Feb 20, 2024
0d3017e
Move `flags`
encukou Feb 20, 2024
269b825
Move `format`
encukou Feb 20, 2024
946f474
Move `ndim`
encukou Feb 20, 2024
607ed23
Move `shape`
encukou Feb 21, 2024
cb126b7
Replace more uses of StgDict
encukou Feb 21, 2024
fd04c91
Don't pass dicts to PyCStgDict_clone
encukou Feb 21, 2024
8dce549
Don't set StgDict as type dict; use name `attrdict` for the type __di…
encukou Feb 21, 2024
6705e0e
Remove more uses of StgDict
encukou Feb 21, 2024
8f99377
Remove StgDict
encukou Feb 21, 2024
daea3e5
Remove StgDict from comments, names, and platform-specific code
encukou Feb 21, 2024
f67211f
Switch PyCSimpleType_new to _init
encukou Feb 23, 2024
02aa9e1
Switch _SimpleCData to a heap type
encukou Feb 23, 2024
13feac7
Convert Struct and Union type _new to _init
encukou Feb 23, 2024
a68417b
Switch PyCPointerType_new to _init
encukou Feb 23, 2024
ad5d7b4
Switch PyCFuncPtrType_new to _init
encukou Feb 23, 2024
8d5e230
Switch CFuncPtr to heap type
encukou Feb 23, 2024
cdfae13
Switch Array to heap type
encukou Feb 23, 2024
2d80960
Switch _Pointer to a heap type
encukou Feb 23, 2024
3337ba4
Switch _ctypes.Union to heap type
encukou Feb 23, 2024
596cb89
Switch Structure to heap type
encukou Feb 23, 2024
91feb64
Make the type-adding macros more regular
encukou Feb 23, 2024
77c64fe
Switch _CData to a heap type
encukou Feb 23, 2024
0be46a1
Visit/decref type in PyCData visit/dealloc
encukou Feb 23, 2024
9be3520
Remove _HackyHeapType
encukou Feb 23, 2024
e523c2f
Decref attrdicts
encukou Mar 1, 2024
fb70e43
Delegate CType_Type deallocation to PyType_Type.tp_dealloc
encukou Mar 5, 2024
cc59dea
Fix Windows-specific crashes
neonene Mar 5, 2024
9bd9eb1
Update comments
encukou Mar 6, 2024
946a0b1
Add tests for metaclass details
encukou Mar 6, 2024
970b41d
Fix getting the module state
encukou Mar 7, 2024
2a450ad
Add a blurb, even though this *should* not affect usage
encukou Mar 7, 2024
9878809
Apply suggestions from code review
encukou Mar 11, 2024
b2ba89a
Inline PyStgInfo_From* & PyStgInfo_Init for performance
encukou Mar 11, 2024
605c30c
Make CType_Type's clear & dealloc safer
encukou Mar 11, 2024
41dea1b
Remove self decrefs in error cases in init functions
encukou Mar 11, 2024
be0888f
Add `static` to PyType_Spec
encukou Mar 11, 2024
ac6eb38
Remove redundant comment
encukou Mar 11, 2024
33be37f
Apply suggestions from code review
encukou Mar 11, 2024
0b9d0a7
Fix comment
encukou Mar 12, 2024
f02bad6
PyErr_WriteUnraisable clears the exception
encukou Mar 12, 2024
16faac7
Call tp_init for argument validation
encukou Mar 20, 2024
2461e50
Add test for swapped type creation
neonene Mar 20, 2024
ab90768
Merge branch 'main' into ctypes-no-stgdict
encukou Mar 20, 2024
de054af
Fix test_swapped_type_creation
encukou Mar 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move the StgInfo out of StgDictObject and into the classes themselves
  • Loading branch information
encukou committed Mar 7, 2024
commit 0171c30c03e62bc6aba78cda81ad787c6fefb652
59 changes: 53 additions & 6 deletions Modules/_ctypes/_ctypes.c
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,48 @@ static PyType_Spec structparam_spec = {
.slots = structparam_slots,
};

/*
CType_Type - a base metaclass. Its instances (classes) have a StgInfo.
*/

static int
CType_Type_traverse(PyObject *self, visitproc visit, void *arg)
{
Py_VISIT(Py_TYPE(self));
return 0;
}

static int
CType_Type_clear(PyObject *self)
{
return 0;
}

static void
CType_Type_dealloc(PyObject *self)
{
PyTypeObject *tp = Py_TYPE(self);
PyObject_GC_UnTrack(self);
(void)CType_Type_clear(self);
erlend-aasland marked this conversation as resolved.
Show resolved Hide resolved
tp->tp_free(self);
Py_DECREF(tp);
}

static PyType_Slot ctype_type_slots[] = {
{Py_tp_traverse, CType_Type_traverse},
{Py_tp_clear, CType_Type_clear},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{Py_tp_clear, CType_Type_clear},

Previously, PyCStgDict_clear() was not a tp_clear function. CType_Type_clear() also seems to work just for CType_Type_dealloc() (mixable).

0, /* tp_traverse */
0, /* tp_clear */

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, it would be less confusing to integrate CDataType_*() into CType_Type_*(), keeping the {Py_tp_clear, CType_Type_clear} slot for subtypes to inherit, roughly like the following:

 static int
 CType_Type_traverse(PyObject *self, visitproc visit, void *arg)
 {
+    ctypes_state *st = GLOBAL_STATE();
+    StgInfo *info;
+    if (PyStgInfo_FromType(st, self, &info) < 0) {
+        return -1;
+    }
+    if (info) {
+        Py_VISIT(info->proto);    // GC-related
+        Py_VISIT(info->argtypes);
+        Py_VISIT(info->converters);
+        Py_VISIT(info->restype);
+        Py_VISIT(info->checker);
+    }
     Py_VISIT(Py_TYPE(self));
-    return 0;
+    return PyType_Type.tp_traverse(self, visit, arg);
 }

 static int
 CType_Type_clear(PyObject *self)
 {
     ctypes_state *st = GLOBAL_STATE();
     StgInfo *info;
     if (PyStgInfo_FromType(st, self, &info) < 0) {
         return -1;
     }
     if (info) {
         Py_CLEAR(info->proto);    // GC-related
         Py_CLEAR(info->argtypes);
         Py_CLEAR(info->converters);
         Py_CLEAR(info->restype);
         Py_CLEAR(info->checker);
     }
-    return 0;
+    return PyType_Type.tp_clear(self);
 }

 static PyType_Slot foo_type_slots[] = {
     ...
-    {Py_tp_traverse, CDataType_traverse},
-    {Py_tp_clear, CDataType_clear},
     ...
 };

 PyType_Spec foo_type_spec = {
     ...
-    .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC |
+    .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
     ...
 };

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would indeed be clearer! I'd prefer leaving it to a follow-up PR though: it'll be easier to debug when we get to isolating the module state.

{Py_tp_dealloc, CType_Type_dealloc},
{0, NULL},
};

static PyType_Spec pyctype_type_spec = {
.name = "_ctypes.CType_Type",
.basicsize = -(Py_ssize_t)sizeof(StgInfo),
.flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_IMMUTABLETYPE |
Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_DISALLOW_INSTANTIATION |
Py_TPFLAGS_BASETYPE ),
.slots = ctype_type_slots,
};

/*
PyCStructType_Type - a meta type/class. Creating a new class using this one as
Expand Down Expand Up @@ -5616,21 +5658,26 @@ _ctypes_add_types(PyObject *mod)
/* StgDict is derived from PyDict_Type */
TYPE_READY_BASE(st->PyCStgDict_Type, &PyDict_Type);

// Common Metaclass
CREATE_TYPE(mod, st->PyCType_Type, &pyctype_type_spec,
&PyType_Type);

/*************************************************
*
* Metaclasses
*/
CREATE_TYPE(mod, st->PyCStructType_Type, &pycstruct_type_spec,
&PyType_Type);
CREATE_TYPE(mod, st->UnionType_Type, &union_type_spec, &PyType_Type);
st->PyCType_Type);
CREATE_TYPE(mod, st->UnionType_Type, &union_type_spec,
st->PyCType_Type);
CREATE_TYPE(mod, st->PyCPointerType_Type, &pycpointer_type_spec,
&PyType_Type);
st->PyCType_Type);
CREATE_TYPE(mod, st->PyCArrayType_Type, &pycarray_type_spec,
&PyType_Type);
st->PyCType_Type);
CREATE_TYPE(mod, st->PyCSimpleType_Type, &pycsimple_type_spec,
&PyType_Type);
st->PyCType_Type);
CREATE_TYPE(mod, st->PyCFuncPtrType_Type, &pycfuncptr_type_spec,
&PyType_Type);
st->PyCType_Type);

/*************************************************
*
Expand Down
2 changes: 1 addition & 1 deletion Modules/_ctypes/ctypes.h
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ typedef struct {
#ifdef MS_WIN32
PyTypeObject *PyComError_Type;
#endif
PyTypeObject *PyCType_Type;
} ctypes_state;

extern ctypes_state global_state;
Expand Down Expand Up @@ -282,7 +283,6 @@ typedef struct {
/* Py_ssize_t *strides; */ /* unused in ctypes */
/* Py_ssize_t *suboffsets; */ /* unused in ctypes */

StgInfo stginfo;
} StgDictObject;

/****************************************************************
Expand Down
19 changes: 10 additions & 9 deletions Modules/_ctypes/stgdict.c
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,14 @@
int PyStgInfo_FromObject(ctypes_state *state, PyObject *obj, StgInfo **result)
{
*result = NULL;
StgDictObject *dict = PyObject_stgdict(obj);
if (!dict) {
PyTypeObject *type = Py_TYPE(obj);
if (!PyObject_IsInstance((PyObject *)type, (PyObject *)state->PyCType_Type)) {
// not a ctypes class.
return 0;
}
StgInfo *info = &dict->stginfo;
StgInfo *info = PyObject_GetTypeData((PyObject *)type, state->PyCType_Type);
encukou marked this conversation as resolved.
Show resolved Hide resolved
if (!info->initialized) {
PyErr_Format(PyExc_SystemError, "%R StgInfo is not initialized.", obj);
PyErr_Format(PyExc_SystemError, "%R StgInfo is not initialized.", type);
return -1;
}
*result = info;
Expand All @@ -43,21 +44,21 @@ int PyStgInfo_FromObject(ctypes_state *state, PyObject *obj, StgInfo **result)
// Initialize StgInfo on a newly created
StgInfo *PyStgInfo_Init(ctypes_state *state, PyTypeObject *type)
{
StgDictObject *dict = PyType_stgdict((PyObject *)type);
if (!dict) {
if (!PyObject_IsInstance((PyObject *)type, (PyObject *)state->PyCType_Type)) {
PyErr_Format(PyExc_SystemError,
"'%s' is not a ctypes class.",
type->tp_name);
return NULL;
}
if (dict->stginfo.initialized) {
StgInfo *info = PyObject_GetTypeData((PyObject *)type, state->PyCType_Type);
if (info->initialized) {
PyErr_Format(PyExc_SystemError,
"StgInfo of '%s' is already initialized.",
type->tp_name);
return NULL;
}
dict->stginfo.initialized = 1;
return &dict->stginfo;
info->initialized = 1;
return info;
}


Expand Down