Update docs to reflect the fact that Iterable etc. are protocols (#4344)

Also update the discussion of ABCs.

Author: JukkaL

docs/source/class_basics.rst

@@ -117,16 +117,15 @@ annotations have no effect at runtime:
 Abstract base classes and multiple inheritance
 **********************************************
 
-Mypy uses Python abstract base classes for protocol types. There are
-several built-in abstract base classes types (for example,
-``Sequence``, ``Iterable`` and ``Iterator``). You can define abstract
-base classes using the ``abc.ABCMeta`` metaclass and the
-``abc.abstractmethod`` function decorator.
+Mypy supports Python abstract base classes (ABCs). Abstract classes
+have at least one abstract method or property that must be implemented
+by a subclass. You can define abstract base classes using the
+``abc.ABCMeta`` metaclass, and the ``abc.abstractmethod`` and
+``abc.abstractproperty`` function decorators. Example:
 
 .. code-block:: python
 
    from abc import ABCMeta, abstractmethod
-   import typing
 
    class A(metaclass=ABCMeta):
        @abstractmethod
@@ -140,32 +139,31 @@ base classes using the ``abc.ABCMeta`` metaclass and the
        def bar(self) -> str:
            return 'x'
 
-   a = A() # Error: A is abstract
-   b = B() # OK
+   a = A()  # Error: A is abstract
+   b = B()  # OK
 
-Unlike most Python code, abstract base classes are likely to play a
-significant role in many complex mypy programs.
+Note that mypy performs checking for unimplemented abstract methods
+even if you omit the ``ABCMeta`` metaclass. This can be useful if the
+metaclass would cause runtime metaclass conflicts.
 
 A class can inherit any number of classes, both abstract and
 concrete. As with normal overrides, a dynamically typed method can
-implement a statically typed abstract method defined in an abstract
-base class.
+implement a statically typed method defined in any base class,
+including an abstract method defined in an abstract base class.
+
+You can implement an abstract property using either a normal
+property or an instance variable.
 
 .. _protocol-types:
 
 Protocols and structural subtyping
 **********************************
 
-.. note::
-
-   Structural subtyping is experimental. Some things may not
-   work as expected. Mypy may pass unsafe code or it can reject
-   valid code.
-
 Mypy supports two ways of deciding whether two classes are compatible
 as types: nominal subtyping and structural subtyping. *Nominal*
 subtyping is strictly based on the class hierarchy. If class ``D``
-inherits class ``C``, it's also a subtype of ``C``. This form of
+inherits class ``C``, it's also a subtype of ``C``, and instances of
+``D`` can be used when ``C`` instances are expected. This form of
 subtyping is used by default in mypy, since it's easy to understand
 and produces clear and concise error messages, and since it matches
 how the native ``isinstance()`` check works -- based on class
@@ -174,16 +172,239 @@ a structural subtype of class ``C`` if the former has all attributes
 and methods of the latter, and with compatible types.
 
 Structural subtyping can be seen as a static equivalent of duck
-typing, which is well known to Python programmers. Mypy provides an
-opt-in support for structural subtyping via protocol classes described
+typing, which is well known to Python programmers. Mypy provides
+support for structural subtyping via protocol classes described
 below.  See `PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_ for
 the detailed specification of protocols and structural subtyping in
 Python.
 
+.. _predefined_protocols:
+
+Predefined protocols
+********************
+
+The ``typing`` module defines various protocol classes that correspond
+to common Python protocols, such as ``Iterable[T]``.  If a class
+defines a suitable ``__iter__`` method, mypy understands that it
+implements the iterable protocol and is compatible with ``Iterable[T]``.
+For example, ``IntList`` below is iterable, over ``int`` values:
+
+.. code-block:: python
+
+   from typing import Iterator, Iterable, Optional
+
+   class IntList:
+       def __init__(self, value: int, next: Optional[IntList]) -> None:
+           self.value = value
+           self.next = next
+
+       def __iter__(self) -> Iterator[int]:
+           current = self
+           while current:
+               yield current.value
+               current = current.next
+
+   def print_numbered(items: Iterable[int]) -> None:
+       for n, x in enumerate(items):
+           print(n + 1, x)
+
+   x = IntList(3, IntList(5, None))
+   print_numbered(x)  # OK
+   print_numbered([4, 5])  # Also OK
+
+The subsections below introduce all built-in protocols defined in
+``typing`` and the signatures of the corresponding methods you need to define
+to implement each protocol (the signatures can be left out, as always, but mypy
+won't type check unannotated methods).
+
+Iteration protocols
+...................
+
+The iteration protocols are useful in many contexts. For example, they allow
+iteration of objects in for loops.
+
+``Iterable[T]``
+---------------
+
+The :ref:`example above <predefined_protocols>` has a simple implementation of an
+``__iter__`` method.
+
+.. code-block:: python
+
+   def __iter__(self) -> Iterator[T]
+
+``Iterator[T]``
+---------------
+
+.. code-block:: python
+
+   def __next__(self) -> T
+   def __iter__(self) -> Iterator[T]
+
+Collection protocols
+....................
+
+Many of these are implemented by built-in container types such as
+``list`` and ``dict``, and these are also useful for user-defined
+collection objects.
+
+``Sized``
+---------
+
+This is a type for objects that support ``len(x)``.
+
+.. code-block:: python
+
+   def __len__(self) -> int
+
+``Container[T]``
+----------------
+
+This is a type for objects that support the ``in`` operator.
+
+.. code-block:: python
+
+   def __contains__(self, x: object) -> bool
+
+``Collection[T]``
+-----------------
+
+.. code-block:: python
+
+   def __len__(self) -> int
+   def __iter__(self) -> Iterator[T]
+   def __contains__(self, x: object) -> bool
+
+One-off protocols
+.................
+
+These protocols are typically only useful with a single standard
+library function or class.
+
+``Reversible[T]``
+-----------------
+
+This is a type for objects that support ``reversed(x)``.
+
+.. code-block:: python
+
+   def __reversed__(self) -> Iterator[T]
+
+``SupportsAbs[T]``
+------------------
+
+This is a type for objects that support ``abs(x)``. ``T`` is the type of
+value returned by ``abs(x)``.
+
+.. code-block:: python
+
+   def __abs__(self) -> T
+
+``SupportsBytes``
+-----------------
+
+This is a type for objects that support ``bytes(x)``.
+
+.. code-block:: python
+
+   def __bytes__(self) -> bytes
+
+``SupportsComplex``
+-------------------
+
+This is a type for objects that support ``complex(x)``.
+
+.. code-block:: python
+
+   def __complex__(self) -> complex
+
+``SupportsFloat``
+-----------------
+
+This is a type for objects that support ``float(x)``.
+
+.. code-block:: python
+
+   def __float__(self) -> float
+
+``SupportsInt``
+---------------
+
+This is a type for objects that support ``int(x)``.
+
+.. code-block:: python
+
+   def __int__(self) -> int
+
+``SupportsRound[T]``
+--------------------
+
+This is a type for objects that support ``round(x)``.
+
+.. code-block:: python
+
+   def __round__(self) -> T
+
+Async protocols
+...............
+
+These protocols can be useful in async code.
+
+``Awaitable[T]``
+----------------
+
+.. code-block:: python
+
+   def __await__(self) -> Generator[Any, None, T]
+
+``AsyncIterable[T]``
+--------------------
+
+.. code-block:: python
+
+   def __aiter__(self) -> AsyncIterator[T]
+
+``AsyncIterator[T]``
+--------------------
+
+.. code-block:: python
+
+   def __anext__(self) -> Awaitable[T]
+   def __aiter__(self) -> AsyncIterator[T]
+
+Context manager protocols
+.........................
+
+There are two protocols for context managers -- one for regular context
+managers and one for async ones. These allow defining objects that can
+be used in ``with`` and ``async with`` statements.
+
+``ContextManager[T]``
+---------------------
+
+.. code-block:: python
+
+   def __enter__(self) -> T
+   def __exit__(self,
+                exc_type: Optional[Type[BaseException]],
+                exc_value: Optional[BaseException],
+                traceback: Optional[TracebackType]) -> Optional[bool]
+
+``AsyncContextManager[T]``
+--------------------------
+
+.. code-block:: python
+
+   def __aenter__(self) -> Awaitable[T]
+   def __aexit__(self,
+                 exc_type: Optional[Type[BaseException]],
+                 exc_value: Optional[BaseException],
+                 traceback: Optional[TracebackType]) -> Awaitable[Optional[bool]]
+
 Simple user-defined protocols
 *****************************
 
-You can define a protocol class by inheriting the special
+You can define your own protocol class by inheriting the special
 ``typing_extensions.Protocol`` class:
 
 .. code-block:: python
@@ -216,12 +437,10 @@ similarly compatible with the protocol, as they support ``close()``.
    The ``Protocol`` base class is currently provided in the ``typing_extensions``
    package. Once structural subtyping is mature and
    `PEP 544 <https://www.python.org/dev/peps/pep-0544/>`_ has been accepted,
-   ``Protocol`` will be included in the ``typing`` module. Several library
-   types such as ``typing.Sized`` and ``typing.Iterable`` will also be changed
-   into protocols. They are currently treated as regular ABCs by mypy.
+   ``Protocol`` will be included in the ``typing`` module.
 
-Defining subprotocols
-*********************
+Defining subprotocols and subclassing protocols
+***********************************************
 
 You can also define subprotocols. Existing protocols can be extended
 and merged using multiple inheritance. Example:
@@ -249,7 +468,7 @@ and merged using multiple inheritance. Example:
 
 Note that inheriting from an existing protocol does not automatically
 turn the subclass into a protocol -- it just creates a regular
-(non-protocol) ABC that implements the given protocol (or
+(non-protocol) class or ABC that implements the given protocol (or
 protocols). The ``typing_extensions.Protocol`` base class must always
 be explicitly present if you are defining a protocol:
 
@@ -267,6 +486,13 @@ be explicitly present if you are defining a protocol:
    # Error: nominal subtyping used by default
    x: NewProtocol = Concrete()  # Error!
 
+You can also include default implementations of methods in
+protocols. If you explicitly subclass these protocols you can inherit
+these default implementations. Explicitly including a protocol as a
+base class is also a way of documenting that your class implements a
+particular protocol, and it forces mypy to verify that your class
+implementation is actually compatible with the protocol.
+
 .. note::
 
    You can use Python 3.6 variable annotations (`PEP 526
@@ -326,6 +552,9 @@ adds support for basic runtime structural checks:
    if isinstance(mug, Portable):
       use(mug.handles)  # Works statically and at runtime
 
+``isinstance()`` also works with the :ref:`predefined protocols <predefined_protocols>`
+in ``typing`` such as ``Iterable``.
+
 .. note::
    ``isinstance()`` with protocols is not completely safe at runtime.
    For example, signatures of methods are not checked. The runtime

docs/source/faq.rst

@@ -108,23 +108,24 @@ Mypy provides support for both `nominal subtyping
 <https://en.wikipedia.org/wiki/Nominative_type_system>`_ and
 `structural subtyping
 <https://en.wikipedia.org/wiki/Structural_type_system>`_.
-Support for structural subtyping is considered experimental.
 Some argue that structural subtyping is better suited for languages with duck
 typing such as Python. Mypy however primarily uses nominal subtyping,
-leaving structural subtyping opt-in. Here are some reasons why:
+leaving structural subtyping mostly opt-in (except for built-in protocols
+such as ``Iterable`` that always support structural subtyping). Here are some
+reasons why:
 
 1. It is easy to generate short and informative error messages when
    using a nominal type system. This is especially important when
    using type inference.
 
 2. Python provides built-in support for nominal ``isinstance()`` tests and
    they are widely used in programs. Only limited support for structural
-   ``isinstance()`` exists for ABCs in ``collections.abc`` and ``typing``
-   standard library modules.
+   ``isinstance()`` is available, and it's less type safe than
+   nominal type tests.
 
-3. Many programmers are already familiar with nominal subtyping and it
+3. Many programmers are already familiar with static, nominal subtyping and it
    has been successfully used in languages such as Java, C++ and
-   C#. Only few languages use structural subtyping.
+   C#. Fewer languages use structural subtyping.
 
 However, structural subtyping can also be useful. For example, a "public API"
 may be more flexible if it is typed with protocols. Also, using protocol types