Skip to content

Commit cd9d1f6

Browse files
nipunn1313nipunn1313
authored
Convert V to ValueType (#306)
See protocolbuffers/protobuf#8182 which will make it into protobuf 3.20.0 Have to use a `V = Union[ValueType]` hax, because `V = ValueType` is interpreted as a value rather than alias `V: TypeAlias = ValueType` appears buggy within nested scopes. Fixes #169
1 parent 8225f8b commit cd9d1f6

File tree

11 files changed

+150
-128
lines changed

11 files changed

+150
-128
lines changed

‎CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
## Upcoming
22

33
- Require protobuf 3.18.1
4+
- Change `EnumTypeWrapper.V` to `EnumTypeWrapper.ValueType` per https://github.com/protocolbuffers/protobuf/pull/8182.
5+
Will allow for unquoted annotations starting with protobuf 3.20.0. `.V` will continue to work for the foreseeable
6+
future for backward compatibility.
47

58
## 3.0.0
69

‎README.md

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -87,32 +87,33 @@ enum MyEnum {
8787
BAR = 1;
8888
}
8989
```
90-
Will yield an [enum type wrapper](https://github.com/python/typeshed/blob/16ae4c61201cd8b96b8b22cdfb2ab9e89ba5bcf2/stubs/protobuf/google/protobuf/internal/enum_type_wrapper.pyi) whose methods type to `MyEnum.V` rather than `int`.
90+
Will yield an [enum type wrapper](https://github.com/python/typeshed/blob/16ae4c61201cd8b96b8b22cdfb2ab9e89ba5bcf2/stubs/protobuf/google/protobuf/internal/enum_type_wrapper.pyi) whose methods type to `MyEnum.ValueType` (a `NewType(int)` rather than `int`.
9191
This allows mypy to catch bugs where the wrong enum value is being used.
9292

9393
Calling code may be typed as follows.
9494

9595
In python >= 3.7
9696
```python
97-
# Need [PEP 563](https://www.python.org/dev/peps/pep-0563/) to postpone evaluation of annotations
98-
from __future__ import annotations # Not needed with python>=3.10
99-
def f(x: MyEnum.V):
97+
# May need [PEP 563](https://www.python.org/dev/peps/pep-0563/) to postpone evaluation of annotations
98+
# from __future__ import annotations # Not needed with python>=3.10 or protobuf>=3.20.0
99+
def f(x: MyEnum.ValueType):
100100
print(x)
101101
f(MyEnum.Value("FOO"))
102102
```
103103

104-
For usages of cast, the type of `x` must be quoted
105-
until [upstream protobuf](https://github.com/protocolbuffers/protobuf/pull/8182) includes `V`
104+
With protobuf <= 3.20.0, for usages of cast, the type of `x` must be quoted
105+
After protobuf >= 3.20.0 - `ValueType` exists in the python code and quotes aren't needed
106+
until [upstream protobuf](https://github.com/protocolbuffers/protobuf/pull/8182) includes `ValueType`
106107
```python
107-
cast('MyEnum.V', x)
108+
cast('MyEnum.ValueType', x)
108109
```
109110

110-
Similarly, for type aliases, you must either quote the type or hide it behind `TYPE_CHECKING`
111+
Similarly, for type aliases with protobuf < 3.20.0, you must either quote the type or hide it behind `TYPE_CHECKING`
111112
```python
112113
from typing import Tuple, TYPE_CHECKING
113-
FOO = Tuple['MyEnum.V', 'MyEnum.V']
114+
FOO = Tuple['MyEnum.ValueType', 'MyEnum.ValueType']
114115
if TYPE_CHECKING:
115-
FOO = Tuple[MyEnum.V, MyEnum.V]
116+
FOO = Tuple[MyEnum.ValueType, MyEnum.ValueType]
116117
```
117118

118119
#### Enum int impl details
@@ -123,18 +124,20 @@ mypy-protobuf autogenerates an instance of the EnumTypeWrapper as follows.
123124
class MyEnum(_MyEnum, metaclass=_MyEnumEnumTypeWrapper):
124125
pass
125126
class _MyEnum:
126-
V = typing.NewType('V', builtins.int)
127-
class _MyEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_MyEnum.V], builtins.type):
127+
ValueType = typing.NewType('ValueType', builtins.int)
128+
V = Union[ValueType]
129+
class _MyEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_MyEnum.ValueType], builtins.type):
128130
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor = ...
129-
FOO = MyEnum.V(0)
130-
BAR = MyEnum.V(1)
131-
FOO = MyEnum.V(0)
132-
BAR = MyEnum.V(1)
131+
FOO = MyEnum.ValueType(0)
132+
BAR = MyEnum.ValueType(1)
133+
FOO = MyEnum.ValueType(0)
134+
BAR = MyEnum.ValueType(1)
133135
```
134136

135-
`_MyEnumEnumTypeWrapper` extends the EnumTypeWrapper to take/return MyEnum.V rather than int
137+
`_MyEnumEnumTypeWrapper` extends the EnumTypeWrapper to take/return MyEnum.ValueType rather than int
136138
`MyEnum` is an instance of the `EnumTypeWrapper`.
137-
- Use `_MyEnum` and of metaclass is an implementation detail to make MyEnum.V a valid type w/o a circular dependency
139+
- Use `_MyEnum` and of metaclass is an implementation detail to make MyEnum.ValueType a valid type w/o a circular dependency
140+
- `V` is supported as an alias of `ValueType` for backward compatibility
138141

139142

140143

‎mypy_protobuf/main.py

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ def _add_enums(
118118
) -> None:
119119
for enum in enums:
120120
self.message_to_fd[prefix + enum.name] = _fd
121-
self.message_to_fd[prefix + enum.name + ".V"] = _fd
121+
self.message_to_fd[prefix + enum.name + ".ValueType"] = _fd
122122

123123
def _add_messages(
124124
messages: "RepeatedCompositeFieldContainer[d.DescriptorProto]",
@@ -332,7 +332,7 @@ def write_enums(
332332
class_name = (
333333
enum.name if enum.name not in PYTHON_RESERVED else "_r_" + enum.name
334334
)
335-
value_type_fq = prefix + class_name + ".V"
335+
value_type_fq = prefix + class_name + ".ValueType"
336336

337337
l(
338338
"class {}({}, metaclass={}):",
@@ -347,17 +347,21 @@ def write_enums(
347347
l("class {}:", "_" + enum.name)
348348
with self._indent():
349349
l(
350-
"V = {}('V', {})",
350+
"ValueType = {}('ValueType', {})",
351351
self._import("typing", "NewType"),
352352
self._builtin("int"),
353353
)
354+
# Ideally this would be `V: TypeAlias = ValueType`, but it appears
355+
# to be buggy in mypy in nested scopes
356+
# Workaround described here https://github.com/python/mypy/issues/7866
357+
l("V = {}[ValueType]", self._import("typing", "Union"))
354358
l(
355359
"class {}({}[{}], {}):",
356360
"_" + enum.name + "EnumTypeWrapper",
357361
self._import(
358362
"google.protobuf.internal.enum_type_wrapper", "_EnumTypeWrapper"
359363
),
360-
"_" + enum.name + ".V",
364+
"_" + enum.name + ".ValueType",
361365
self._builtin("type"),
362366
)
363367
with self._indent():
@@ -864,7 +868,7 @@ def python_type(
864868
d.FieldDescriptorProto.TYPE_STRING: lambda: self._import("typing", "Text"),
865869
d.FieldDescriptorProto.TYPE_BYTES: lambda: self._builtin("bytes"),
866870
d.FieldDescriptorProto.TYPE_ENUM: lambda: self._import_message(
867-
field.type_name + ".V"
871+
field.type_name + ".ValueType"
868872
),
869873
d.FieldDescriptorProto.TYPE_MESSAGE: lambda: self._import_message(
870874
field.type_name

‎test/generated/testproto/inner/inner_pb2.pyi

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@ DESCRIPTOR: google.protobuf.descriptor.FileDescriptor = ...
1313
class Inner(google.protobuf.message.Message):
1414
DESCRIPTOR: google.protobuf.descriptor.Descriptor = ...
1515
A_FIELD_NUMBER: builtins.int
16-
a: testproto.test3_pb2.OuterEnum.V = ...
16+
a: testproto.test3_pb2.OuterEnum.ValueType = ...
1717
def __init__(self,
1818
*,
19-
a : testproto.test3_pb2.OuterEnum.V = ...,
19+
a : testproto.test3_pb2.OuterEnum.ValueType = ...,
2020
) -> None: ...
2121
def ClearField(self, field_name: typing_extensions.Literal["a",b"a"]) -> None: ...
2222
global___Inner = Inner

‎test/generated/testproto/nested/nested_pb2.pyi

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,10 @@ DESCRIPTOR: google.protobuf.descriptor.FileDescriptor = ...
1515
class Nested(google.protobuf.message.Message):
1616
DESCRIPTOR: google.protobuf.descriptor.Descriptor = ...
1717
A_FIELD_NUMBER: builtins.int
18-
a: testproto.test3_pb2.OuterEnum.V = ...
18+
a: testproto.test3_pb2.OuterEnum.ValueType = ...
1919
def __init__(self,
2020
*,
21-
a : testproto.test3_pb2.OuterEnum.V = ...,
21+
a : testproto.test3_pb2.OuterEnum.ValueType = ...,
2222
) -> None: ...
2323
def ClearField(self, field_name: typing_extensions.Literal["a",b"a"]) -> None: ...
2424
global___Nested = Nested
@@ -28,47 +28,49 @@ class AnotherNested(google.protobuf.message.Message):
2828
class NestedEnum(_NestedEnum, metaclass=_NestedEnumEnumTypeWrapper):
2929
pass
3030
class _NestedEnum:
31-
V = typing.NewType('V', builtins.int)
32-
class _NestedEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_NestedEnum.V], builtins.type):
31+
ValueType = typing.NewType('ValueType', builtins.int)
32+
V = typing.Union[ValueType]
33+
class _NestedEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_NestedEnum.ValueType], builtins.type):
3334
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor = ...
34-
INVALID = AnotherNested.NestedEnum.V(0)
35-
ONE = AnotherNested.NestedEnum.V(1)
36-
TWO = AnotherNested.NestedEnum.V(2)
35+
INVALID = AnotherNested.NestedEnum.ValueType(0)
36+
ONE = AnotherNested.NestedEnum.ValueType(1)
37+
TWO = AnotherNested.NestedEnum.ValueType(2)
3738

38-
INVALID = AnotherNested.NestedEnum.V(0)
39-
ONE = AnotherNested.NestedEnum.V(1)
40-
TWO = AnotherNested.NestedEnum.V(2)
39+
INVALID = AnotherNested.NestedEnum.ValueType(0)
40+
ONE = AnotherNested.NestedEnum.ValueType(1)
41+
TWO = AnotherNested.NestedEnum.ValueType(2)
4142

4243
class NestedMessage(google.protobuf.message.Message):
4344
DESCRIPTOR: google.protobuf.descriptor.Descriptor = ...
4445
class NestedEnum2(_NestedEnum2, metaclass=_NestedEnum2EnumTypeWrapper):
4546
pass
4647
class _NestedEnum2:
47-
V = typing.NewType('V', builtins.int)
48-
class _NestedEnum2EnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_NestedEnum2.V], builtins.type):
48+
ValueType = typing.NewType('ValueType', builtins.int)
49+
V = typing.Union[ValueType]
50+
class _NestedEnum2EnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_NestedEnum2.ValueType], builtins.type):
4951
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor = ...
50-
UNDEFINED = AnotherNested.NestedMessage.NestedEnum2.V(0)
51-
NESTED_ENUM1 = AnotherNested.NestedMessage.NestedEnum2.V(1)
52-
NESTED_ENUM2 = AnotherNested.NestedMessage.NestedEnum2.V(2)
52+
UNDEFINED = AnotherNested.NestedMessage.NestedEnum2.ValueType(0)
53+
NESTED_ENUM1 = AnotherNested.NestedMessage.NestedEnum2.ValueType(1)
54+
NESTED_ENUM2 = AnotherNested.NestedMessage.NestedEnum2.ValueType(2)
5355

54-
UNDEFINED = AnotherNested.NestedMessage.NestedEnum2.V(0)
55-
NESTED_ENUM1 = AnotherNested.NestedMessage.NestedEnum2.V(1)
56-
NESTED_ENUM2 = AnotherNested.NestedMessage.NestedEnum2.V(2)
56+
UNDEFINED = AnotherNested.NestedMessage.NestedEnum2.ValueType(0)
57+
NESTED_ENUM1 = AnotherNested.NestedMessage.NestedEnum2.ValueType(1)
58+
NESTED_ENUM2 = AnotherNested.NestedMessage.NestedEnum2.ValueType(2)
5759

5860
S_FIELD_NUMBER: builtins.int
5961
B_FIELD_NUMBER: builtins.int
6062
NE_FIELD_NUMBER: builtins.int
6163
NE2_FIELD_NUMBER: builtins.int
6264
s: typing.Text = ...
6365
b: builtins.bool = ...
64-
ne: global___AnotherNested.NestedEnum.V = ...
65-
ne2: global___AnotherNested.NestedMessage.NestedEnum2.V = ...
66+
ne: global___AnotherNested.NestedEnum.ValueType = ...
67+
ne2: global___AnotherNested.NestedMessage.NestedEnum2.ValueType = ...
6668
def __init__(self,
6769
*,
6870
s : typing.Text = ...,
6971
b : builtins.bool = ...,
70-
ne : global___AnotherNested.NestedEnum.V = ...,
71-
ne2 : global___AnotherNested.NestedMessage.NestedEnum2.V = ...,
72+
ne : global___AnotherNested.NestedEnum.ValueType = ...,
73+
ne2 : global___AnotherNested.NestedMessage.NestedEnum2.ValueType = ...,
7274
) -> None: ...
7375
def ClearField(self, field_name: typing_extensions.Literal["b",b"b","ne",b"ne","ne2",b"ne2","s",b"s"]) -> None: ...
7476

‎test/generated/testproto/test3_pb2.pyi

Lines changed: 26 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,17 @@ DESCRIPTOR: google.protobuf.descriptor.FileDescriptor = ...
1515
class OuterEnum(_OuterEnum, metaclass=_OuterEnumEnumTypeWrapper):
1616
pass
1717
class _OuterEnum:
18-
V = typing.NewType('V', builtins.int)
19-
class _OuterEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_OuterEnum.V], builtins.type):
18+
ValueType = typing.NewType('ValueType', builtins.int)
19+
V = typing.Union[ValueType]
20+
class _OuterEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_OuterEnum.ValueType], builtins.type):
2021
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor = ...
21-
UNKNOWN = OuterEnum.V(0)
22-
FOO3 = OuterEnum.V(1)
23-
BAR3 = OuterEnum.V(2)
22+
UNKNOWN = OuterEnum.ValueType(0)
23+
FOO3 = OuterEnum.ValueType(1)
24+
BAR3 = OuterEnum.ValueType(2)
2425

25-
UNKNOWN = OuterEnum.V(0)
26-
FOO3 = OuterEnum.V(1)
27-
BAR3 = OuterEnum.V(2)
26+
UNKNOWN = OuterEnum.ValueType(0)
27+
FOO3 = OuterEnum.ValueType(1)
28+
BAR3 = OuterEnum.ValueType(2)
2829
global___OuterEnum = OuterEnum
2930

3031

@@ -44,14 +45,15 @@ class SimpleProto3(google.protobuf.message.Message):
4445
class InnerEnum(_InnerEnum, metaclass=_InnerEnumEnumTypeWrapper):
4546
pass
4647
class _InnerEnum:
47-
V = typing.NewType('V', builtins.int)
48-
class _InnerEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_InnerEnum.V], builtins.type):
48+
ValueType = typing.NewType('ValueType', builtins.int)
49+
V = typing.Union[ValueType]
50+
class _InnerEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_InnerEnum.ValueType], builtins.type):
4951
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor = ...
50-
INNER1 = SimpleProto3.InnerEnum.V(0)
51-
INNER2 = SimpleProto3.InnerEnum.V(1)
52+
INNER1 = SimpleProto3.InnerEnum.ValueType(0)
53+
INNER2 = SimpleProto3.InnerEnum.ValueType(1)
5254

53-
INNER1 = SimpleProto3.InnerEnum.V(0)
54-
INNER2 = SimpleProto3.InnerEnum.V(1)
55+
INNER1 = SimpleProto3.InnerEnum.ValueType(0)
56+
INNER2 = SimpleProto3.InnerEnum.ValueType(1)
5557

5658
class MapScalarEntry(google.protobuf.message.Message):
5759
DESCRIPTOR: google.protobuf.descriptor.Descriptor = ...
@@ -102,21 +104,21 @@ class SimpleProto3(google.protobuf.message.Message):
102104
a_string: typing.Text = ...
103105
@property
104106
def a_repeated_string(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[typing.Text]: ...
105-
a_outer_enum: global___OuterEnum.V = ...
107+
a_outer_enum: global___OuterEnum.ValueType = ...
106108
@property
107109
def outer_message(self) -> global___OuterMessage3: ...
108-
inner_enum: global___SimpleProto3.InnerEnum.V = ...
110+
inner_enum: global___SimpleProto3.InnerEnum.ValueType = ...
109111
a_oneof_1: typing.Text = ...
110112
a_oneof_2: typing.Text = ...
111113
@property
112114
def outer_message_in_oneof(self) -> global___OuterMessage3: ...
113-
outer_enum_in_oneof: global___OuterEnum.V = ...
114-
inner_enum_in_oneof: global___SimpleProto3.InnerEnum.V = ...
115+
outer_enum_in_oneof: global___OuterEnum.ValueType = ...
116+
inner_enum_in_oneof: global___SimpleProto3.InnerEnum.ValueType = ...
115117
b_oneof_1: typing.Text = ...
116118
b_oneof_2: typing.Text = ...
117119
@property
118120
def bool(self) -> global___OuterMessage3: ...
119-
OuterEnum: global___OuterEnum.V = ...
121+
OuterEnum: global___OuterEnum.ValueType = ...
120122
"""Test having fieldname match messagename"""
121123

122124
@property
@@ -132,18 +134,18 @@ class SimpleProto3(google.protobuf.message.Message):
132134
*,
133135
a_string : typing.Text = ...,
134136
a_repeated_string : typing.Optional[typing.Iterable[typing.Text]] = ...,
135-
a_outer_enum : global___OuterEnum.V = ...,
137+
a_outer_enum : global___OuterEnum.ValueType = ...,
136138
outer_message : typing.Optional[global___OuterMessage3] = ...,
137-
inner_enum : global___SimpleProto3.InnerEnum.V = ...,
139+
inner_enum : global___SimpleProto3.InnerEnum.ValueType = ...,
138140
a_oneof_1 : typing.Text = ...,
139141
a_oneof_2 : typing.Text = ...,
140142
outer_message_in_oneof : typing.Optional[global___OuterMessage3] = ...,
141-
outer_enum_in_oneof : global___OuterEnum.V = ...,
142-
inner_enum_in_oneof : global___SimpleProto3.InnerEnum.V = ...,
143+
outer_enum_in_oneof : global___OuterEnum.ValueType = ...,
144+
inner_enum_in_oneof : global___SimpleProto3.InnerEnum.ValueType = ...,
143145
b_oneof_1 : typing.Text = ...,
144146
b_oneof_2 : typing.Text = ...,
145147
bool : typing.Optional[global___OuterMessage3] = ...,
146-
OuterEnum : global___OuterEnum.V = ...,
148+
OuterEnum : global___OuterEnum.ValueType = ...,
147149
OuterMessage3 : typing.Optional[global___OuterMessage3] = ...,
148150
map_scalar : typing.Optional[typing.Mapping[builtins.int, typing.Text]] = ...,
149151
map_message : typing.Optional[typing.Mapping[builtins.int, global___OuterMessage3]] = ...,

‎test/generated/testproto/test_extensions3_pb2.pyi

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,9 @@ scalar_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor
2424

2525
repeated_scalar_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, google.protobuf.internal.containers.RepeatedScalarFieldContainer[typing.Text]] = ...
2626

27-
enum_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, testproto.test3_pb2.OuterEnum.V] = ...
27+
enum_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, testproto.test3_pb2.OuterEnum.ValueType] = ...
2828

29-
repeated_enum_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, google.protobuf.internal.containers.RepeatedScalarFieldContainer[testproto.test3_pb2.OuterEnum.V]] = ...
29+
repeated_enum_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, google.protobuf.internal.containers.RepeatedScalarFieldContainer[testproto.test3_pb2.OuterEnum.ValueType]] = ...
3030

3131
msg_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, testproto.test3_pb2.OuterMessage3] = ...
3232

0 commit comments

Comments
 (0)