Closed
Description
We have a relatively new setup that uses the Kafka protocal over Azure Event Hubs. Our usage of the KafkaConsumer
client largely uses the default configuration, except for:
- group_id - some string
- auto_offset_reset - 'earliest'
- enable_auto_commit - True
- security_protocol - 'SASL_SSL'
Everything has run fine until recently -- and only on one occasion -- when we found this in the log. It appears the app would only resume upon a restart.
In light of the above premise, I'd like to ask:
- Has anyone experienced this before?
- How might one reproduce this?
- What is the right way to resume from this error without having to restart the application?
[15:24:09] myapplication - INFO | Processing new message
[20:16:03] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead
[20:16:03] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead
[20:16:03] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.
[20:16:03] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.
[20:16:03] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)
[20:16:03] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)
[20:16:03] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication
[20:16:03] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication
[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:09] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:10] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:11] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:12] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:13] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead
[20:16:13] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:13] kafka.coordinator - WARNING | Heartbeat session expired, marking coordinator dead
[20:16:13] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.
[20:16:13] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: Heartbeat session expired.
[20:16:13] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)
[20:16:13] kafka.cluster - INFO | Group coordinator for myapplication is BrokerMetadata(nodeId='coordinator-0', host='my.kafka.host', port=9092, rack=None)
[20:16:13] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication
[20:16:13] kafka.coordinator - INFO | Discovered coordinator coordinator-0 for group myapplication
[20:16:14] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:14] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:15] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:15] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:16] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:17] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:18] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:19] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.coordinator.consumer - WARNING | Auto offset commit failed for group myapplication: NodeNotReadyError: coordinator-0
[20:16:20] kafka.conn - ERROR | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Error receiving network data closing socket
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/kafka/conn.py", line 1087, in _recv
data = self._sock.recv(self.config['sock_chunk_bytes'])
File "/usr/local/lib/python3.8/ssl.py", line 1226, in recv
return self.read(buflen)
File "/usr/local/lib/python3.8/ssl.py", line 1101, in read
return self._sslobj.read(len)
ConnectionResetError: [Errno 104] Connection reset by peer
[20:16:20] kafka.conn - ERROR | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Error receiving network data closing socket
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/kafka/conn.py", line 1087, in _recv
data = self._sock.recv(self.config['sock_chunk_bytes'])
File "/usr/local/lib/python3.8/ssl.py", line 1226, in recv
return self.read(buflen)
File "/usr/local/lib/python3.8/ssl.py", line 1101, in read
return self._sslobj.read(len)
ConnectionResetError: [Errno 104] Connection reset by peer
[20:16:20] kafka.conn - INFO | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Closing connection. KafkaConnectionError: [Errno 104] Connection reset by peer
[20:16:20] kafka.conn - INFO | <BrokerConnection node_id=coordinator-0 host=my.kafka.host:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Closing connection. KafkaConnectionError: [Errno 104] Connection reset by peer
[20:16:20] kafka.client - WARNING | Node coordinator-0 connection failed -- refreshing metadata
[20:16:20] kafka.client - WARNING | Node coordinator-0 connection failed -- refreshing metadata
[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: KafkaConnectionError: [Errno 104] Connection reset by peer.
[20:16:20] kafka.coordinator - WARNING | Marking the coordinator dead (node coordinator-0) for group myapplication: KafkaConnectionError: [Errno 104] Connection reset by peer.
[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - ERROR | Error sending OffsetCommitRequest_v2 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - ERROR | Error sending OffsetCommitRequest_v2 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
[20:16:20] kafka.coordinator - ERROR | Error sending HeartbeatRequest_v1 to node coordinator-0 [KafkaConnectionError: [Errno 104] Connection reset by peer]
overed coordinator coordinator-0 for group myapplication
[20:16:03] kafka.coordinator - INFO | Discovered coordinator coordinator
Metadata
Metadata
Assignees
Labels
No labels