Skip to content

BrokerConnection | Error receiving network data closing socket #2111

Closed
@takwas

Description

@takwas

We have a relatively new setup that uses the Kafka protocal over Azure Event Hubs. Our usage of the KafkaConsumer client largely uses the default configuration, except for:

  • group_id - some string
  • auto_offset_reset - 'earliest'
  • enable_auto_commit - True
  • security_protocol - 'SASL_SSL'

Everything has run fine until recently -- and only on one occasion -- when we found this in the log. It appears the app would only resume upon a restart.

In light of the above premise, I'd like to ask:

  • Has anyone experienced this before?
  • How might one reproduce this?
  • What is the right way to resume from this error without having to restart the application?

[15:24:09] myapplication - INFO | Processing new message

[20:16:03] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead

[20:16:03] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead

[20:16:03] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.

[20:16:03] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.

[20:16:03] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)

[20:16:03] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)

[20:16:03] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication

[20:16:03] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication

[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:13] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead

[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:13] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead

[20:16:13] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.

[20:16:13] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.

[20:16:13] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)

[20:16:13] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)

[20:16:13] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication

[20:16:13] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication

[20:16:14] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:14] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:15] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:15] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0

[20:16:20] kafka.conn - ERROR | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Error receiving network data closing socket

Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/kafka/conn.py", line 1087, in _recv

data = self._sock.recv(self.config['sock_chunk_bytes'])

File "/usr/local/lib/python3.8/ssl.py", line 1226, in recv

return self.read(buflen)

File "/usr/local/lib/python3.8/ssl.py", line 1101, in read

return self._sslobj.read(len)

ConnectionResetError: [Errno 104] Connection reset by peer

[20:16:20] kafka.conn - ERROR | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Error receiving network data closing socket

Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/kafka/conn.py", line 1087, in _recv

data = self._sock.recv(self.config['sock_chunk_bytes'])

File "/usr/local/lib/python3.8/ssl.py", line 1226, in recv

return self.read(buflen)

File "/usr/local/lib/python3.8/ssl.py", line 1101, in read

return self._sslobj.read(len)

ConnectionResetError: [Errno 104] Connection reset by peer

[20:16:20] kafka.conn - INFO | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Closing connection. KafkaConnectionError: [Errno 104] Connection reset by peer

[20:16:20] kafka.conn - INFO | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Closing connection. KafkaConnectionError: [Errno 104] Connection reset by peer

[20:16:20] kafka.client - WARNING | Node coordinator-0 connection failed -- refreshing metadata

[20:16:20] kafka.client - WARNING | Node coordinator-0 connection failed -- refreshing metadata

[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: KafkaConnectionError: [Errno 104] Connection reset by peer.

[20:16:20] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: KafkaConnectionError: [Errno 104] Connection reset by peer.

[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - ERROR | Error sending OffsetCommitRequest_v2 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - ERROR | Error sending OffsetCommitRequest_v2 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]

overed coordinator coordinator-0 for group myapplication

[20:16:03] kafka.coordinator - INFO | Discovered coordinator coordinator

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions