Skip to content

gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 30, 2025
Merged

gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7f96223
gh-134160: Improve note on multi-phase init & subinterpreters
encukou May 27, 2025
3fc9447
Rewrite the whole section
encukou May 28, 2025
3b0514d
Apply suggestions from code review
encukou May 28, 2025
11f8675
Apply suggestions from code review
encukou May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions Doc/c-api/module.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,8 +302,13 @@ using :c:func:`PyModule_GetState`), or its contents (such as the module's
:attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`).

All modules created using multi-phase initialization are expected to support
:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules
are independent is typically enough to achieve this.
:ref:`sub-interpreters <sub-interpreter-support>`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:ref:`sub-interpreters <sub-interpreter-support>`.
:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly
signal a lack of support.

Typically, extensions ensure this in one of these ways:

- :ref:`isolating module instances <isolating-extensions-howto>`,
- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I just noticed that the referenced section does not mention opting out by setting the Py_mod_multiple_interpreters slot to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED. (I didn't think to update the docs when I added that value.) Fixing that doesn't need to be part of this PR, but it is relevant to this bullet (and the third bullet).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Please read this in a friendly voice; I can't find a way to reword this so it's clear and doesn't have a terrible sarcastic interpretation.)

Yeah. I'm not a good person to write that, as I never fully understood why a slot is better than explicitly raising an exception when you're about to clobber shared state.
It would be good to record that in the HOWTO, could you do it?

- limiting a module to the main interpreter using
:c:data:`Py_mod_multiple_interpreters`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit weird to present this as a way of supporting sub-interpreters, no?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. The original wording from PEP 489 would work a bit better:

All modules created using multi-phase initialization are expected to support
:ref:`sub-interpreters <sub-interpreter-support>` and multiple
:c:func:`Py_Initialize`/:c:func:`Py_Finalize` cycles correctly.
Typically, extensions ensure this in one of these ways:

or should we expand the “correctly“ to something like “work as expected or raise exceptions, rather than crash”?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would definitely not call this "support". So perhaps something like:

Modules created using multi-phase initialization are expected to handle
sub-interpreters in either of the following two ways:

* either they fully support multiple interpreters by
  :ref:`isolating module instances <isolating-extensions-howto>`;
* or they signal that multiple interpreters are not supported by
  :ref:`raising an error on repeated initialization <isolating-extensions-optout>`
  and/or limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from multiple interpreters.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess right thing would be to take step back and not make this so much about subinterpreters:

Multi-phase initialization allows creating multiple independent instances of the
module object.
Extensions are expected to handle this in one of the following ways:

* either by :ref:`isolating module instances <isolating-extensions-howto>`;
* and/or by :ref:`raising an error on repeated initialization <isolating-extensions-optout>`;
* and/or limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`.
The first two ways also prevent crashes when reusing the module after
Python runtime reinitialization (:c:func:`Py_Finalize` and :c:func:`Py_Initialize`)
and reimporting a module after it is removed from :py:attr:`sys.modules`.

Thanks for suggesting the effects on the users! It helps to make this less of a thou shalt commandment and more like something where you can consider the trade-offs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me wonder: why is there a third way if the second is better (and not more difficult to implement)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that it is critical for this PR, but I've been thinking about something related that might at least a little insight: gh-132861.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pitrou: Let's say there's subtle nuance that doesn't really belong on this page of the docs :)
It would be good to explain it in the HOWTO, of course.

AFAIK, the main interpreter is special; sometimes, being loadable once but in any interpreter isn't enough.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The extant prior paragraph covers much of this, so here's an alternative:

When using per-module state, module instances should be :ref:`isolated
<isolating-extensions-howto>` to ensure that they are free of unwanted
side-effects. If the module still uses global state, or it otherwise has not
yet been isolated, either or both of of the following two options should be used:

* :ref:`raising an error on repeated initialization <isolating-extensions-optout>`
* limiting a module to the main interpreter using
  :c:data:`Py_mod_multiple_interpreters`.

This way, users will not encounter crashes or undefined behavior when
trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`.
Ensuring a module is properly isolated or raising an explicit error also prevent
crashes when reusing the module after Python runtime reinitialization
(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`) or reimporting a module after
it has been removed from :py:attr:`sys.modules`.


To request multi-phase initialization, the initialization function
(PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty
Expand Down
4 changes: 3 additions & 1 deletion Doc/howto/isolating-extensions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ possible, consider explicit locking.
If it is necessary to use process-global state, the simplest way to
avoid issues with multiple interpreters is to explicitly prevent a
module from being loaded more than once per process—see
`Opt-Out: Limiting to One Module Object per Process`_.
:ref:`isolating-extensions-optout`.


Managing Per-Module State
Expand Down Expand Up @@ -207,6 +207,8 @@ An example of a module with per-module state is currently available as
example module initialization shown at the bottom of the file.


.. _isolating-extensions-optout:

Opt-Out: Limiting to One Module Object per Process
--------------------------------------------------

Expand Down
Loading