Skip to content

request_timeout_ms should be reset when check_version fail #1963

Closed
@zhgjun

Description

@zhgjun

I have 3 kafka brokers, when i restart my client,it will chose one broker A to Probing broker version by check_version,but first try it failed for some reason. and it will try another broker B,and this time it get version succ. But, it cannot send requests to broker A any more. the log is :

`
2019-11-21T17:10:57.762+08:00 localhost test WARNING [pid:3011] [MainThread][tid:47912824863152] [conn.py:818 recv] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]> timed out after 1994.59195137 ms. Closing connection.
2019-11-21T17:10:57.762+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:724 close] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: Closing connection. [Error 7] RequestTimedOutError: Request timed out after 1994.59195137 ms
2019-11-21T17:10:57.762+08:00 localhost test WARNING [pid:3011] [MainThread][tid:47912824863152] [client_async.py:336 _conn_state_change] Node 2 connection failed -- refreshing metadata
2019-11-21T17:10:57.763+08:00 localhost test ERROR [pid:3011] [MainThread][tid:47912824863152] [future.py:79 _call_backs] Fetch to node 2 failed: [Error 7] RequestTimedOutError: Request timed out after 1994.59195137 ms
2019-11-21T17:10:57.765+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:360 connect] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: connecting to 192.168.0.2:6666 [('192.168.0.2', 6666) IPv4]
2019-11-21T17:10:57.776+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:584 _try_authenticate_plain] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: Authenticated as test via PLAIN
2019-11-21T17:10:57.777+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:415 connect] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: Connection complete.

2019-11-21T17:11:03.625+08:00 localhost test WARNING [pid:3011] [MainThread][tid:47912824863152] [client_async.py:649 _poll] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]> timed out after 1994.59195137 ms. Closing connection.
2019-11-21T17:11:03.626+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:724 close] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: Closing connection. [Error 7] RequestTimedOutError: Request timed out after 1994.59195137 ms
2019-11-21T17:11:03.626+08:00 localhost test WARNING [pid:3011] [MainThread][tid:47912824863152] [client_async.py:336 _conn_state_change] Node 2 connection failed -- refreshing metadata
2019-11-21T17:11:03.626+08:00 localhost test ERROR [pid:3011] [MainThread][tid:47912824863152] [future.py:79 _call_backs] Fetch to node 2 failed: [Error 7] RequestTimedOutError: Request timed out after 1994.59195137 ms
2019-11-21T17:11:03.628+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:360 connect] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: connecting to 192.168.0.2:6666 [('192.168.0.2', 6666) IPv4]
2019-11-21T17:11:03.647+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:584 _try_authenticate_plain] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: Authenticated as test via PLAIN
2019-11-21T17:11:03.647+08:00 localhost test INFO [pid:3011] [MainThread][tid:47912824863152] [conn.py:415 connect] <BrokerConnection node_id=2 host=192.168.0.2:6666 [IPv4 ('192.168.0.2', 6666)]>: Connection complete.

`
I set request_timeout_ms=60000, not 1994.59195137 ms. and i found when it check_version it will override_config.

override_config = {

when it check_version fail, it raise exception without reset the request_timeout_ms .

if not self.connect_blocking(timeout_at - time.time()):
. As a result the connection's request_timeout_ms update to 1994.59195137 ms, when broker need more time to handle requests , the client will failed for the requets timeout.

so we should reset the configs which changed in check_version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions