-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
Improve Py_mod_multiple_interpreters and Py_mod_gil Usability #132861
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I'd love to combine this with a proposal in https://discuss.python.org/t/stable-abi-limited-api-for-free-threaded-builds/86458:
Can it wait until 3.15, or are we in a rush? |
By my count, 614 of the top 15,000 PyPI packages (representing 99%+ of PyPI downloads) need to migrate to multi-phase init. This is then 447 of the top 10k, 268 of the top 5k, 72 of the top 1,000, and 11 of the top 100. We've seen pushback from maintainers on migrating to multi-phase with the current state of affairs (partly due to documentation/marketing, which we're actively improving). My slight concern with changing all of the slots now is that maintainers will question the state of flux and somewhat reasonably ask to delay any change until things have settled down. As a sketch, the below might be needed for a module bridging Python 3.10--3.15: static PyModuleDef_Slot spam_slots[] = {
{Py_mod_exec, spam_exec},
#if PY_VERSION_HEX >= 0x030F00F0 // Python 3.15+
{Py_mod_isolation, Py_MOD_ISOLATED},
{Py_mod_threadsafe, Py_MOD_THREADSAFE},
{Py_mod_external_libs, Py_MOD_EXTERNAL_NOT_ISOLATED},
#else
# if PY_VERSION_HEX >= 0x030C00F0 // Python 3.12+
{Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
# endif
# if PY_VERSION_HEX >= 0x030D00F0 // Python 3.13+
{Py_mod_gil, Py_MOD_GIL_NOT_USED},
# endif
#endif
{0, NULL}
}; The above looks on the surface far more complex and harder to maintain than a 'simple' call to
I don't think the names are a problem, if anything longer and more explicit is useful, and these generally only appear once & at the end of extension modules. A |
IMO, the current slots focus on what CPython does, the proposed ones focus on describing the module. It would help to document how CPython would react to these options (and their combinations), both currently and when future plans are in place. Going into details:
I'd love to try going the other way and reducing the number of slots, even if the result is a bit less granular. |
Uh oh!
There was an error while loading. Please reload this page.
Feature or enhancement
Proposal:
Here are some deficiencies:
Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED
andPy_MOD_PER_INTERPRETER_GIL_SUPPORTED
is unclearPy_MOD_PER_INTERPRETER_GIL_SUPPORTED
applies to the vast majority of extensions and should be the defaultPy_mod_gil
doesn't have an option that explicitly covers the case where an external dependency is not thread-safePy_MOD_MULTIPLE_INTERPRETERS_SUPPORTED
implies an external dependency is not thread-safe and/or isolated to each interpreter; it should be the other way around (and be a synonym forPy_MOD_PER_INTERPRETER_GIL_SUPPORTED
), which would introduce a need for aPy_MOD_PER_INTERPRETER_GIL_NOT_SUPPORTED
We could take a simple approach, working with the existing slots:
Py_MOD_PER_INTERPRETER_GIL_NOT_SUPPORTED
Py_MOD_GIL_USED_FOR_EXTERNAL
I think we could do better, by replacing the existing slots with new ones that focus explicitly on what we care about:
Py_mod_isolation
Py_MOD_NOT_ISOLATED
- the module has process-global state and/or objects (e.g. static types)Py_MOD_ISOLATED
(default) - the module's state/objects have been isolated to the module (i.e interpreter)Py_mod_threadsafe
Py_MOD_NOT_THREADSAFE
(default) - the module's own state/code is not thread-safePy_MOD_THREADSAFE
- the module's state/code are thread-safe; external dependencies are covered byPy_mod_external_libs
Py_mod_external_libs
Py_MOD_EXTERN_NOT_THREADSAFE
- at least one external dependency is not thread-safe (may be used without the GIL held)Py_MOD_EXTERN_NOT_ISOLATED
- at least one external dependency is not isolated to the module object (i.e. interpreter)Py_MOD_EXTERN_ISOLATED
(default) - all external dependencies are isolated and using them is thread-safePy_mod_isolation
andPy_mod_threadsafe
, external dependencies are covered byPy_mod_external_libs
Py_mod_multiple_interpreters
slotPy_mod_gil
slotRegarding the defaults, for
Py_mod_isolation
, multi-phase init implies isolation (per PEP 489). ForPy_mod_external_libs
, we assume that the vast majority of modules will not have problematic dependencies.Equivalents to current usage:
CC @encukou
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
The text was updated successfully, but these errors were encountered: