Merge remote-tracking branch 'upstream/main' into expose_codegen

Author: iritkatriel

Doc/c-api/call.rst

@@ -93,7 +93,7 @@ This is a pointer to a function with the following signature:
    and they must be unique.
    If there are no keyword arguments, then *kwnames* can instead be *NULL*.
 
-.. c:macro:: PY_VECTORCALL_ARGUMENTS_OFFSET
+.. data:: PY_VECTORCALL_ARGUMENTS_OFFSET
 
    If this flag is set in a vectorcall *nargsf* argument, the callee is allowed
    to temporarily change ``args[-1]``. In other words, *args* points to

Doc/c-api/typeobj.rst

@@ -1245,6 +1245,17 @@ and :c:type:`PyType_Type` effectively act as defaults.)
       **Inheritance:**
 
       This flag is not inherited.
+      However, subclasses will not be instantiable unless they provide a
+      non-NULL :c:member:`~PyTypeObject.tp_new` (which is only possible
+      via the C API).
+
+      .. note::
+
+         To disallow instantiating a class directly but allow instantiating
+         its subclasses (e.g. for an :term:`abstract base class`),
+         do not use this flag.
+         Instead, make :c:member:`~PyTypeObject.tp_new` only succeed for
+         subclasses.
 
       .. versionadded:: 3.10
 

Doc/faq/windows.rst

@@ -167,7 +167,7 @@ How can I embed Python into a Windows application?
 
 Embedding the Python interpreter in a Windows app can be summarized as follows:
 
-1. Do _not_ build Python into your .exe file directly.  On Windows, Python must
+1. Do **not** build Python into your .exe file directly.  On Windows, Python must
    be a DLL to handle importing modules that are themselves DLL's.  (This is the
    first key undocumented fact.)  Instead, link to :file:`python{NN}.dll`; it is
    typically installed in ``C:\Windows\System``.  *NN* is the Python version, a
@@ -191,7 +191,7 @@ Embedding the Python interpreter in a Windows app can be summarized as follows:
 2. If you use SWIG, it is easy to create a Python "extension module" that will
    make the app's data and methods available to Python.  SWIG will handle just
    about all the grungy details for you.  The result is C code that you link
-   *into* your .exe file (!)  You do _not_ have to create a DLL file, and this
+   *into* your .exe file (!)  You do **not** have to create a DLL file, and this
    also simplifies linking.
 
 3. SWIG will create an init function (a C function) whose name depends on the
@@ -218,10 +218,10 @@ Embedding the Python interpreter in a Windows app can be summarized as follows:
 5. There are two problems with Python's C API which will become apparent if you
    use a compiler other than MSVC, the compiler used to build pythonNN.dll.
 
-   Problem 1: The so-called "Very High Level" functions that take FILE *
+   Problem 1: The so-called "Very High Level" functions that take ``FILE *``
    arguments will not work in a multi-compiler environment because each
-   compiler's notion of a struct FILE will be different.  From an implementation
-   standpoint these are very _low_ level functions.
+   compiler's notion of a ``struct FILE`` will be different.  From an implementation
+   standpoint these are very low level functions.
 
    Problem 2: SWIG generates the following code when generating wrappers to void
    functions:

Doc/library/ctypes.rst

@@ -359,7 +359,7 @@ from within *IDLE* or *PythonWin*::
    >>> printf(b"%f bottles of beer\n", 42.5)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
-   ArgumentError: argument 2: exceptions.TypeError: Don't know how to convert parameter 2
+   ArgumentError: argument 2: TypeError: Don't know how to convert parameter 2
    >>>
 
 As has been mentioned before, all Python types except integers, strings, and
@@ -422,7 +422,7 @@ prototype for a C function), and tries to convert the arguments to valid types::
    >>> printf(b"%d %d %d", 1, 2, 3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
-   ArgumentError: argument 2: exceptions.TypeError: wrong type
+   ArgumentError: argument 2: TypeError: wrong type
    >>> printf(b"%s %d %f\n", b"X", 2, 3)
    X 2 3.000000
    13
@@ -487,7 +487,7 @@ single character Python bytes object into a C char::
    >>> strchr(b"abcdef", b"def")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
-   ArgumentError: argument 2: exceptions.TypeError: one character string expected
+   ArgumentError: argument 2: TypeError: one character string expected
    >>> print(strchr(b"abcdef", b"x"))
    None
    >>> strchr(b"abcdef", b"d")

Doc/library/enum.rst

@@ -242,8 +242,8 @@ Data Types
 
          Member values can be anything: :class:`int`, :class:`str`, etc..  If
          the exact value is unimportant you may use :class:`auto` instances and an
-         appropriate value will be chosen for you.  Care must be taken if you mix
-         :class:`auto` with other values.
+         appropriate value will be chosen for you.  See :class:`auto` for the
+         details.
 
    .. attribute:: Enum._ignore_
 
@@ -778,7 +778,16 @@ Utilities and Decorators
    For *Enum* and *IntEnum* that appropriate value will be the last value plus
    one; for *Flag* and *IntFlag* it will be the first power-of-two greater
    than the last value; for *StrEnum* it will be the lower-cased version of the
-   member's name.
+   member's name.  Care must be taken if mixing *auto()* with manually specified
+   values.
+
+   *auto* instances are only resolved when at the top level of an assignment:
+
+      * ``FIRST = auto()`` will work (auto() is replaced with ``1``);
+      * ``SECOND = auto(), -2`` will work (auto is replaced with ``2``, so ``2, -2`` is
+         used to create the ``SECOND`` enum member;
+      * ``THREE = [auto(), -3]`` will *not* work (``<auto instance>, -3`` is used to
+        create the ``THREE`` enum member)
 
    ``_generate_next_value_`` can be overridden to customize the values used by
    *auto*.

Doc/library/functions.rst

@@ -1419,7 +1419,7 @@ are always available.  They are listed here in alphabetical order.
       supported.
 
 
-.. function:: print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)
+.. function:: print(*objects, sep=' ', end='\n', file=None, flush=False)
 
    Print *objects* to the text stream *file*, separated by *sep* and followed
    by *end*.  *sep*, *end*, *file*, and *flush*, if present, must be given as keyword

Doc/library/socket.rst

@@ -189,8 +189,11 @@ created.  Socket addresses are represented as follows:
   ``(ifname, proto[, pkttype[, hatype[, addr]]])`` where:
 
   - *ifname* - String specifying the device name.
-  - *proto* - An in network-byte-order integer specifying the Ethernet
-    protocol number.
+  - *proto* - The Ethernet protocol number.
+    May be :data:`ETH_P_ALL` to capture all protocols,
+    one of the :ref:`ETHERTYPE_* constants <socket-ethernet-types>`
+    or any other Ethernet protocol number.
+    Value must be in network-byte-order.
   - *pkttype* - Optional integer specifying the packet type:
 
     - ``PACKET_HOST`` (the default) - Packet addressed to the local host.
@@ -508,6 +511,19 @@ Constants
    .. availability:: Linux >= 2.2.
 
 
+.. data:: ETH_P_ALL
+
+   :data:`!ETH_P_ALL` can be used in the :class:`~socket.socket`
+   constructor as *proto* for the :const:`AF_PACKET` family in order to
+   capture every packet, regardless of protocol.
+
+   For more information, see the :manpage:`packet(7)` manpage.
+
+   .. availability:: Linux.
+
+   .. versionadded:: 3.12
+
+
 .. data:: AF_RDS
           PF_RDS
           SOL_RDS
@@ -638,6 +654,22 @@ Constants
 
    .. versionadded:: 3.12
 
+.. _socket-ethernet-types:
+
+.. data:: ETHERTYPE_ARP
+          ETHERTYPE_IP
+          ETHERTYPE_IPV6
+          ETHERTYPE_VLAN
+
+   `IEEE 802.3 protocol number
+   <https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.txt>`_.
+   constants.
+
+   .. availability:: Linux, FreeBSD, macOS.
+
+   .. versionadded:: 3.12
+
+
 Functions
 ^^^^^^^^^
 

Doc/library/sqlite3.rst

@@ -1834,8 +1834,13 @@ The deprecated default adapters and converters consist of:
 Command-line interface
 ^^^^^^^^^^^^^^^^^^^^^^
 
-The :mod:`!sqlite3` module can be invoked as a script
+The :mod:`!sqlite3` module can be invoked as a script,
+using the interpreter's :option:`-m` switch,
 in order to provide a simple SQLite shell.
+The argument signature is as follows::
+
+   python -m sqlite3 [-h] [-v] [filename] [sql]
+
 Type ``.quit`` or CTRL-D to exit the shell.
 
 .. program:: python -m sqlite3 [-h] [-v] [filename] [sql]

Doc/reference/grammar.rst

@@ -12,7 +12,7 @@ and `PEG <https://en.wikipedia.org/wiki/Parsing_expression_grammar>`_.
 In particular, ``&`` followed by a symbol, token or parenthesized
 group indicates a positive lookahead (i.e., is required to match but
 not consumed), while ``!`` indicates a negative lookahead (i.e., is
-required _not_ to match).  We use the ``|`` separator to mean PEG's
+required *not* to match).  We use the ``|`` separator to mean PEG's
 "ordered choice" (written as ``/`` in traditional PEG grammars). See
 :pep:`617` for more details on the grammar's syntax.
 

Doc/reference/simple_stmts.rst

@@ -330,7 +330,7 @@ statement, of a variable or attribute annotation and an optional assignment stat
    annotated_assignment_stmt: `augtarget` ":" `expression`
                             : ["=" (`starred_expression` | `yield_expression`)]
 
-The difference from normal :ref:`assignment` is that only single target is allowed.
+The difference from normal :ref:`assignment` is that only a single target is allowed.
 
 For simple names as assignment targets, if in class or module scope,
 the annotations are evaluated and stored in a special class or module
@@ -365,8 +365,8 @@ target, then the interpreter evaluates the target except for the last
       IDEs.
 
 .. versionchanged:: 3.8
-   Now annotated assignments allow same expressions in the right hand side as
-   the regular assignments. Previously, some expressions (like un-parenthesized
+   Now annotated assignments allow the same expressions in the right hand side as
+   regular assignments. Previously, some expressions (like un-parenthesized
    tuple expressions) caused a syntax error.
 
 
@@ -756,7 +756,7 @@ commas) the two steps are carried out separately for each clause, just
 as though the clauses had been separated out into individual import
 statements.
 
-The details of the first step, finding and loading modules are described in
+The details of the first step, finding and loading modules, are described in
 greater detail in the section on the :ref:`import system <importsystem>`,
 which also describes the various types of packages and modules that can
 be imported, as well as all the hooks that can be used to customize

Doc/tutorial/errors.rst

@@ -496,7 +496,7 @@ Raising and Handling Multiple Unrelated Exceptions
 ==================================================
 
 There are situations where it is necessary to report several exceptions that
-have occurred. This it often the case in concurrency frameworks, when several
+have occurred. This is often the case in concurrency frameworks, when several
 tasks may have failed in parallel, but there are also other use cases where
 it is desirable to continue execution and collect multiple errors rather than
 raise the first exception.

Doc/whatsnew/3.12.rst

@@ -75,6 +75,36 @@ Important deprecations, removals or restrictions:
 Improved Error Messages
 =======================
 
+* Modules from the standard library are now potentially suggested as part of
+  the error messages displayed by the interpreter when a :exc:`NameError` is
+  raised to the top level. Contributed by Pablo Galindo in :gh:`98254`.
+
+    >>> sys.version_info
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in <module>
+    NameError: name 'sys' is not defined. Did you forget to import 'sys'?
+
+* Improve the error suggestion for :exc:`NameError` exceptions for instances.
+  Now if a :exc:`NameError` is raised in a method and the instance has an
+  attribute that's exactly equal to the name in the exception, the suggestion
+  will include ``self.<NAME>`` instead of the closest match in the method
+  scope. Contributed by Pablo Galindo in :gh:`99139`.
+
+    >>> class A:
+    ...    def __init__(self):
+    ...        self.blech = 1
+    ...
+    ...    def foo(self):
+    ...        somethin = blech
+
+    >>> A().foo()
+    Traceback (most recent call last):
+      File "<stdin>", line 1
+        somethin = blech
+                   ^^^^^
+    NameError: name 'blech' is not defined. Did you mean: 'self.blech'?
+
+
 * Improve the :exc:`SyntaxError` error message when the user types ``import x
   from y`` instead of ``from y import x``. Contributed by Pablo Galindo in :gh:`98931`.
 
@@ -85,6 +115,16 @@ Improved Error Messages
         ^^^^^^^^^^^^^^^^^^^^^^^
     SyntaxError: Did you mean to use 'from ... import ...' instead?
 
+* :exc:`ImportError` exceptions raised from failed ``from <module> import
+  <name>`` statements now include suggestions for the value of ``<name>`` based on the
+  available names in ``<module>``. Contributed by Pablo Galindo in :gh:`91058`.
+
+    >>> from collections import chainmap
+    Traceback (most recent call last):
+      File "<stdin>", line 1, in <module>
+    ImportError: cannot import name 'chainmap' from 'collections'. Did you mean: 'ChainMap'?
+
+
 New Features
 ============
 
@@ -656,6 +696,18 @@ New Features
   ``__dict__`` and weakrefs with less bookkeeping,
   using less memory and with faster access.
 
+* API for performing calls using
+  :ref:`the vectorcall protocol <vectorcall>` was added to the
+  :ref:`Limited API <stable>`:
+
+  * :c:func:`PyObject_Vectorcall`
+  * :c:func:`PyObject_VectorcallMethod`
+  * :const:`PY_VECTORCALL_ARGUMENTS_OFFSET`
+
+  This means that both the incoming and outgoing ends of the vector call
+  protocol are now available in the :ref:`Limited API <stable>`. (Contributed
+  by Wenzel Jakob in :gh:`98586`.)
+
 * Added two new public functions,
   :c:func:`PyEval_SetProfileAllThreads` and
   :c:func:`PyEval_SetTraceAllThreads`, that allow to set tracing and profiling

Grammar/python.gram

@@ -1254,8 +1254,8 @@ invalid_try_stmt:
     | a='try' ':' NEWLINE !INDENT {
         RAISE_INDENTATION_ERROR("expected an indented block after 'try' statement on line %d", a->lineno) }
     | 'try' ':' block !('except' | 'finally') { RAISE_SYNTAX_ERROR("expected 'except' or 'finally' block") }
-    | 'try' ':' block* ((except_block+ except_star_block) | (except_star_block+ except_block)) block* {
-        RAISE_SYNTAX_ERROR("cannot have both 'except' and 'except*' on the same 'try'") }
+    | a='try' ':' block* ((except_block+ except_star_block) | (except_star_block+ except_block)) block* {
+        RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot have both 'except' and 'except*' on the same 'try'") }
 invalid_except_stmt:
     | 'except' '*'? a=expression ',' expressions ['as' NAME ] ':' {
         RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types must be parenthesized") }

Include/internal/pycore_code.h

@@ -230,7 +230,7 @@ extern void _Py_Specialize_CompareOp(PyObject *lhs, PyObject *rhs,
                                      _Py_CODEUNIT *instr, int oparg);
 extern void _Py_Specialize_UnpackSequence(PyObject *seq, _Py_CODEUNIT *instr,
                                           int oparg);
-extern void _Py_Specialize_ForIter(PyObject *iter, _Py_CODEUNIT *instr);
+extern void _Py_Specialize_ForIter(PyObject *iter, _Py_CODEUNIT *instr, int oparg);
 
 /* Finalizer function for static codeobjects used in deepfreeze.py */
 extern void _PyStaticCode_Fini(PyCodeObject *co);

Include/internal/pycore_frame.h

@@ -61,6 +61,7 @@ typedef struct _PyInterpreterFrame {
     // over, or (in the case of a newly-created frame) a totally invalid value:
     _Py_CODEUNIT *prev_instr;
     int stacktop;     /* Offset of TOS from localsplus  */
+    uint16_t yield_offset;
     bool is_entry;  // Whether this is the "root" frame for the current _PyCFrame.
     char owner;
     /* Locals and stack */
@@ -110,6 +111,7 @@ _PyFrame_InitializeSpecials(
     frame->frame_obj = NULL;
     frame->prev_instr = _PyCode_CODE(code) - 1;
     frame->is_entry = false;
+    frame->yield_offset = 0;
     frame->owner = FRAME_OWNED_BY_THREAD;
 }