Intermittent exception

[dvenza]

Quite often, I would say 70% of the time, I get this message:

Exception ignored in: <bound method ConsumerCoordinator.__del__ of <kafka.coordinator.consumer.ConsumerCoordinator object at 0x7ffe64f04978>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/kafka/coordinator/consumer.py", line 109, in __del__
  File "/usr/local/lib/python3.4/dist-packages/kafka/util.py", line 168, in __init__
TypeError: 'NoneType' object is not callable

When this piece of Python code exits:

consumer = KafkaConsumer(bootstrap_servers=args.kafka_address)

if args.list_logs:
     for topic in consumer.topics():
         if topic[0] != '_':
             print(topic)
     sys.exit(0)

Everything else works as expected and I'm using latest the pip package (1.1.0) on Python 3.

[dpkp]

Thanks for the note. You can safely ignore the error. We should clean this up.

[ivica-k]

I've noticed this issue happening Intermittently with:

library version: 2.0.1
Kafka version: 0.10.1.1

The full traceback

Exception ignored in: <function ConsumerCoordinator.__del__ at 0x7ff474cd9e60>
Traceback (most recent call last):

File "/var/task/kafka/coordinator/consumer.py", line 131, in __del__
  super(ConsumerCoordinator, self).__del__()
File "/var/task/kafka/coordinator/base.py", line 757, in __del__
  self._close_heartbeat_thread()
File "/var/task/kafka/coordinator/base.py", line 751, in _close_heartbeat_thread
  self._heartbeat_thread.close()
File "/var/task/kafka/coordinator/base.py", line 928, in close
  self.join(self.coordinator.config['heartbeat_interval_ms'] / 1000)
File "/var/lang/lib/python3.7/threading.py", line 1041, in join
  raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread

If it makes any difference, my code is doing this:

from kafka import KafkaConsumer

# ... SNIP ...

def consume():
	count = 0

	consumer = KafkaConsumer(
		TOPIC_NAME,
		bootstrap_servers=KAFKA_SERVERS,
		group_id=FUNCTION_NAME,
		consumer_timeout_ms=5000,
		auto_offset_reset="earliest",
		enable_auto_commit=False
	)

	for msg in consumer:
		x = msg.timestamp  # we have to access whatever attribute of the message in order to consume it
		count += 1

	return count

and it's running in AWS Lambda with 1.5GB of memory and a max timeout of 15 minutes. Max memory used is 10% of what is available, and the function is not timing out.