Fix `len(T)` so it doesn't upcast `Literal[:int]` from to `int` if returned from `T.len()` #11830

jorenham · 2024-04-24T15:13:47Z

Previously len(T) always returned and int, even if T.__len__() -> Literal[42]. I.e. T.__len__() and len(T) did not invariably match.

This PR fixes this discrepancy, by making len() generic on the return type of __len__, i.e. as len[N: int]: (SupportsLen[N]) -> N.

JelleZijlstra · 2024-04-24T15:18:39Z

This is incorrect, len() always returns an int in CPython.

JelleZijlstra · 2024-04-24T15:19:17Z

>>> class X:
...     def __len__(self): return "x"
... 
>>> len(X())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object cannot be interpreted as an integer

JelleZijlstra · 2024-04-24T15:20:13Z

And similarly if you return a subclass of int:

>>> class X:
...     def __len__(self): return True
... 
>>> len(X())
1

jorenham · 2024-04-24T15:22:48Z

@JelleZijlstra I realize that, but Literal[42] is also an int, and len() is able to return that.

jorenham · 2024-04-24T15:27:18Z

Unfortunately, using a N = TypeVar("N", int, LiteralInt) isn't possible at the moment. But the current upcasting behaviour from Literal[:int] to int is also incorrect.

jorenham · 2024-04-24T15:29:23Z

>>> class X:
...     def __len__(self): return "x"
... 
>>> len(X())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object cannot be interpreted as an integer

With this change, this example would still be rejected.

JelleZijlstra · 2024-04-24T15:39:17Z

As you can see, this leads to lots of new obscure errors. I don't think it's worth it.

github-actions · 2024-04-24T15:42:12Z

Diff from mypy_primer, showing the effect of this PR on open source code:

vision (https://github.com/pytorch/vision)
+ torchvision/models/segmentation/lraspp.py:173: error: Argument 2 to "_lraspp_mobilenetv3" has incompatible type "int | None"; expected "int"  [arg-type]
+ torchvision/models/segmentation/fcn.py:168: error: Argument 2 to "_fcn_resnet" has incompatible type "int | None"; expected "int"  [arg-type]
+ torchvision/models/segmentation/fcn.py:227: error: Argument 2 to "_fcn_resnet" has incompatible type "int | None"; expected "int"  [arg-type]
+ torchvision/models/segmentation/deeplabv3.py:275: error: Argument 2 to "_deeplabv3_resnet" has incompatible type "int | None"; expected "int"  [arg-type]
+ torchvision/models/segmentation/deeplabv3.py:331: error: Argument 2 to "_deeplabv3_resnet" has incompatible type "int | None"; expected "int"  [arg-type]
+ torchvision/models/segmentation/deeplabv3.py:385: error: Argument 2 to "_deeplabv3_mobilenetv3" has incompatible type "int | None"; expected "int"  [arg-type]

aiortc (https://github.com/aiortc/aiortc)
+ src/aiortc/rtcsctptransport.py:1731: error: Argument 1 to "len" has incompatible type "bytes"; expected "_SupportsLen[bool]"  [arg-type]
+ src/aiortc/rtcsctptransport.py:1731: note: Following member(s) of "bytes" have conflicts:
+ src/aiortc/rtcsctptransport.py:1731: note:     Expected:
+ src/aiortc/rtcsctptransport.py:1731: note:         def __len__(self) -> bool
+ src/aiortc/rtcsctptransport.py:1731: note:     Got:
+ src/aiortc/rtcsctptransport.py:1731: note:         def __len__(self) -> int

pylox (https://github.com/sco1/pylox)
- pylox/builtins/py_builtins.py:149: error: Argument 1 to "len" has incompatible type "LoxInstance"; expected "Sized"  [arg-type]
+ pylox/builtins/py_builtins.py:149: error: Argument 1 to "len" has incompatible type "LoxInstance"; expected "_SupportsLen[Never]"  [arg-type]

aiohttp-devtools (https://github.com/aio-libs/aiohttp-devtools)
+ aiohttp_devtools/runserver/serve.py:215: error: Invalid index type "AppKey[set[tuple[WebSocketResponse, str]]]" for "Application"; expected type "AppKey[_SupportsLen[int]]"  [index]
+ aiohttp_devtools/runserver/serve.py:245: error: Invalid index type "AppKey[set[tuple[WebSocketResponse, str]]]" for "Application"; expected type "AppKey[_SupportsLen[int]]"  [index]
+ aiohttp_devtools/runserver/serve.py:254: error: Invalid index type "AppKey[set[tuple[WebSocketResponse, str]]]" for "Application"; expected type "AppKey[_SupportsLen[int]]"  [index]
+ aiohttp_devtools/runserver/watch.py:80: error: Invalid index type "AppKey[set[tuple[WebSocketResponse, str]]]" for "Application"; expected type "AppKey[_SupportsLen[int]]"  [index]
+ aiohttp_devtools/runserver/watch.py:86: error: Invalid index type "AppKey[set[tuple[WebSocketResponse, str]]]" for "Application"; expected type "AppKey[_SupportsLen[int]]"  [index]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/arrays/datetimes.py:523: error: Argument 1 to "len" has incompatible type "ndarray[Any, dtype[signedinteger[Any]]]"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/arrays/datetimes.py:525: error: Argument 1 to "len" has incompatible type "ndarray[Any, dtype[signedinteger[Any]]]"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/io/json/_json.py:1237: error: Argument 1 to "len" has incompatible type "Series"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/io/json/_json.py:1237: note: Following member(s) of "Series" have conflicts:
+ pandas/io/json/_json.py:1237: note:     Expected:
+ pandas/io/json/_json.py:1237: note:         def __len__(self) -> bool
+ pandas/io/json/_json.py:1237: note:     Got:
+ pandas/io/json/_json.py:1237: note:         def __len__(self) -> int
+ pandas/io/formats/format.py:784: error: Argument 1 to "len" has incompatible type "list[Any]"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/frame.py:12174: error: Argument 1 to "len" has incompatible type "Index"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/frame.py:12174: note: Following member(s) of "Index" have conflicts:
+ pandas/core/frame.py:12174: note:     Expected:
+ pandas/core/frame.py:12174: note:         def __len__(self) -> bool
+ pandas/core/frame.py:12174: note:     Got:
+ pandas/core/frame.py:12174: note:         def __len__(self) -> int
+ pandas/core/frame.py:12211: error: Argument 1 to "len" has incompatible type "Index"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/frame.py:12211: note: Following member(s) of "Index" have conflicts:
+ pandas/core/frame.py:12211: note:     Expected:
+ pandas/core/frame.py:12211: note:         def __len__(self) -> bool
+ pandas/core/frame.py:12211: note:     Got:
+ pandas/core/frame.py:12211: note:         def __len__(self) -> int
+ pandas/core/reshape/pivot.py:172: error: Argument 1 to "len" has incompatible type "Index"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/reshape/pivot.py:172: note: Following member(s) of "Index" have conflicts:
+ pandas/core/reshape/pivot.py:172: note:     Expected:
+ pandas/core/reshape/pivot.py:172: note:         def __len__(self) -> bool
+ pandas/core/reshape/pivot.py:172: note:     Got:
+ pandas/core/reshape/pivot.py:172: note:         def __len__(self) -> int
+ pandas/core/reshape/merge.py:1366: error: Argument 1 to "len" has incompatible type "ExtensionArray | ndarray[Any, Any]"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/indexes/base.py:3682: error: Argument 1 to "len" has incompatible type "Index"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/indexes/base.py:3682: note: Following member(s) of "Index" have conflicts:
+ pandas/core/indexes/base.py:3682: note:     Expected:
+ pandas/core/indexes/base.py:3682: note:         def __len__(self) -> bool
+ pandas/core/indexes/base.py:3682: note:     Got:
+ pandas/core/indexes/base.py:3682: note:         def __len__(self) -> int
+ pandas/core/indexes/base.py:3913: error: Argument 1 to "len" has incompatible type "ndarray[Any, Any]"; expected "_SupportsLen[bool]"  [arg-type]
+ pandas/core/groupby/grouper.py:879: error: Argument 1 to "len" has incompatible type "NDFrameT"; expected "_SupportsLen[bool]"  [arg-type]

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/tools/web/app.py:554: error: Unsupported operand types for >= ("int" and "None")  [operator]
+ mitmproxy/tools/web/app.py:554: note: Left operand is of type "int | None"

pylint (https://github.com/pycqa/pylint)
+ pylint/lint/run.py:96: error: Returning Any from function declared to return "int"  [no-any-return]
+ pylint/lint/run.py:97: error: Returning Any from function declared to return "int"  [no-any-return]

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/structured_configs/_implementations.py:2882: error: Argument 1 to "len" has incompatible type "str | int"; expected "Sized"  [arg-type]
+ src/hydra_zen/structured_configs/_implementations.py:2882: error: Argument 1 to "len" has incompatible type "str | int"; expected "_SupportsLen[int]"  [arg-type]
- src/hydra_zen/structured_configs/_implementations.py:2885: error: Argument 1 to "len" has incompatible type "str | int"; expected "Sized"  [arg-type]
+ src/hydra_zen/structured_configs/_implementations.py:2885: error: Argument 1 to "len" has incompatible type "str | int"; expected "_SupportsLen[int]"  [arg-type]

optuna (https://github.com/optuna/optuna)
+ optuna/study/_multi_objective.py:213: error: Unsupported operand types for < ("int" and "None")  [operator]
+ optuna/study/_multi_objective.py:213: note: Right operand is of type "int | None"
+ optuna/visualization/matplotlib/_pareto_front.py:195: error: Argument 1 to "len" has incompatible type "list[tuple[FrozenTrial, list[float]]]"; expected "_SupportsLen[bool]"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/python.py:1364: error: Need type annotation for "num_ids"  [var-annotated]
+ src/_pytest/python.py:1364: note: Error code "var-annotated" not covered by "type: ignore" comment

spark (https://github.com/apache/spark)
+ python/pyspark/core/context.py:786: error: Need type annotation for "size"  [var-annotated]
+ python/pyspark/pandas/series.py:4683: error: Need type annotation for "length"  [var-annotated]
+ python/pyspark/pandas/series.py:5462: error: Need type annotation for "length"  [var-annotated]

bandersnatch (https://github.com/pypa/bandersnatch)
+ src/bandersnatch_filter_plugins/allowlist_name.py: note: In member "pinned_version_exists" of class "AllowListRelease":
+ src/bandersnatch_filter_plugins/allowlist_name.py:210: error: Returning Any from function declared to return "bool"  [no-any-return]

ibis (https://github.com/ibis-project/ibis)
- ibis/formats/__init__.py:243: error: Argument 1 to "len" has incompatible type "T"; expected "Sized"  [arg-type]
+ ibis/formats/__init__.py:243: error: Argument 1 to "len" has incompatible type "T"; expected "_SupportsLen[Never]"  [arg-type]
- ibis/expr/types/pretty.py:239: error: Value of type variable "SupportsRichComparisonT" of "min" cannot be "int | None"  [type-var]
- ibis/expr/types/pretty.py:239: error: Incompatible types in assignment (expression has type "int | None", variable has type "int")  [assignment]
- ibis/expr/types/pretty.py:241: error: Incompatible types in assignment (expression has type "int | None", variable has type "int")  [assignment]
+ ibis/expr/api.py:312: error: Unsupported left operand type for != (Never)  [operator]
- ibis/expr/api.py:312: error: Argument 1 to "len" has incompatible type "Iterable[str] | None"; expected "Sized"  [arg-type]
+ ibis/expr/api.py:312: error: Argument 1 to "len" has incompatible type "Iterable[str] | None"; expected "_SupportsLen[Never]"  [arg-type]
- ibis/expr/api.py:312: error: Argument 1 to "len" has incompatible type "Iterable[str | DataType] | None"; expected "Sized"  [arg-type]
+ ibis/expr/api.py:312: error: Argument 1 to "len" has incompatible type "Iterable[str | DataType] | None"; expected "_SupportsLen[Never]"  [arg-type]
+ ibis/expr/api.py:505: error: Need type annotation for "provided_col"  [var-annotated]
- ibis/expr/api.py:505: error: Argument 1 to "len" has incompatible type "Iterable[str]"; expected "Sized"  [arg-type]
+ ibis/expr/api.py:505: error: Argument 1 to "len" has incompatible type "Iterable[str]"; expected "_SupportsLen[Never]"  [arg-type]
- ibis/backends/polars/__init__.py:321: error: Argument 1 to "len" has incompatible type "Iterable[str]"; expected "Sized"  [arg-type]
+ ibis/backends/polars/__init__.py:321: error: Argument 1 to "len" has incompatible type "Iterable[str]"; expected "_SupportsLen[Never]"  [arg-type]
- ibis/backends/polars/__init__.py:324: error: Argument 1 to "len" has incompatible type "Iterable[str]"; expected "Sized"  [arg-type]
+ ibis/backends/polars/__init__.py:324: error: Argument 1 to "len" has incompatible type "Iterable[str]"; expected "_SupportsLen[Never]"  [arg-type]

black (https://github.com/psf/black)
+ src/blib2to3/pytree.py:945:9: error: Returning Any from function declared to return "bool"  [no-any-return]

jax (https://github.com/google/jax)
+ jax/_src/ad_checkpoint.py:624: error: Argument 1 to "tuple" has incompatible type "bool | Sequence[bool]"; expected "Iterable[bool]"  [arg-type]
+ jax/_src/interpreters/ad.py:685: error: Argument 1 to "tuple" has incompatible type "bool | Sequence[bool]"; expected "Iterable[bool]"  [arg-type]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/runtime/secrets.py: note: In member "__len__" of class "AttrDict":
+ lib/streamlit/runtime/secrets.py:80:9: error: Returning Any from function declared to return "int"  [no-any-return]

jorenham · 2024-04-24T15:53:40Z

@JelleZijlstra Yea I tend to agree: It might be best to re-address this issue when something like a LiteralInt or Literal[int] type is introduced.

And for now, there's the simple workaround of using .__len__() instead of len().

jorenham and others added 3 commits April 24, 2024 16:55

test that len(T) and T.__len__() return the same type

e6f6b3b

prevent len(T) from upcasting literal ints returned from T.__len__()

899799e

[pre-commit.ci] auto fixes from pre-commit.com hooks

5096d68

workaround frozenset pyright error

1f4e827

This comment has been minimized.

Sign in to view

fix failing test on python 3.8

0ab5f90

jorenham closed this Apr 24, 2024

jorenham mentioned this pull request Dec 22, 2024

CI, TYP: Large-scale integration type-testing with mypy_primer numpy/numpy#28054

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix `len(T)` so it doesn't upcast `Literal[:int]` from to `int` if returned from `T.len()` #11830

Fix `len(T)` so it doesn't upcast `Literal[:int]` from to `int` if returned from `T.len()` #11830

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

JelleZijlstra commented Apr 24, 2024

Uh oh!

JelleZijlstra commented Apr 24, 2024

Uh oh!

JelleZijlstra commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

This comment has been minimized.

JelleZijlstra commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

Uh oh!

Uh oh!

Fix len(T) so it doesn't upcast Literal[:int] from to int if returned from T.__len__() #11830

Fix len(T) so it doesn't upcast Literal[:int] from to int if returned from T.__len__() #11830

Uh oh!

Conversation

jorenham commented Apr 24, 2024

Uh oh!

JelleZijlstra commented Apr 24, 2024

Uh oh!

JelleZijlstra commented Apr 24, 2024

Uh oh!

JelleZijlstra commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

This comment has been minimized.

JelleZijlstra commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

jorenham commented Apr 24, 2024

Uh oh!

Uh oh!

Fix `len(T)` so it doesn't upcast `Literal[:int]` from to `int` if returned from `T.len()` #11830

Fix `len(T)` so it doesn't upcast `Literal[:int]` from to `int` if returned from `T.len()` #11830