KafkaProducer - flush 'success' after 30 seconds but no message sent

[kcris]

Hi,
using python 3.7.1, kafka-python 1.4.4, kafka 2.1.0

I forgot to uncomment and set the actual ip for the kafka server in kafka's config/server.properties (listeners property), which leads to weird results while using kafka-python's producer.

I call send() and then flush(50), then close(50)
I noticed that it takes 30 seconds (consistently) for flush to return, with no error.
But no actual data is received by my broker (which is normal since the listener was incorrectly specified).

What is weird is that I get no exception, no error whatsoever from flush or send, even if I use a finite timeout. I suspect that 30 seconds is actually some kind of internal timeout (because mine is set to 50).

Bottom line: no data sent, but no error either in KafkaProducer's send / flush.

Thanks

[dpkp]

Errors are returned asynchronously via the future (aka promise) that is returned from send(). Can you check there?

…

On Wed, Dec 5, 2018, 6:12 AM kcris ***@***.*** wrote: Hi, using python 3.7.1, kafka-python 1.4.4, kafka 2.1.0 I forgot to uncomment and set the actual ip for the kafka server in kafka's config/server.properties (listeners property), which leads to weird results while using kafka-python's producer. I call send() and then flush(50), then close(50) I noticed that it takes 30 seconds (consistently) for flush to return, with no error. But no actual data is received by my broker (which is normal since the listener was incorrectly specified). What is weird is that I get no exception, no error whatsoever from flush or send, even if I use a finite timeout. I suspect that 30 seconds is actually some kind of internal timeout (because mine is set to 50). Bottom line: no data sent, but no error as well in KafkaProducer's send / flush. Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1667>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetHkdRJ3mFP-131E-F1BDBTeX9jWQks5u19RdgaJpZM4ZCzzf> .

[kcris]

Hi,
yes, I can confirm, checking the send's future... I get the KafkaTimeoutError.

The question remains though: why don't I get the same exception when calling flush?
Shouldn't they both (send and flush) offer the same semantics: wait until the data is sent (with success/failure) and if there's a failure, raise a KafkaTimeoutError?

according to the docs: no, we don't get the same semantics (my bad for not reading carefully earlier)

https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html
says that flush (unlike send's future) thinks that if a future is done (either success or failure!!) - everything is 'just fine'

So flush may fail to send something, and it will report no error whatsoever. Null, nada, zip, zilch.

Bottom line is: in case of an error, flush() is totally silent. There's no exception, no error, no nothing.
So it's impossible to check if something went wrong if you call flush().
To me, this makes flush() totally unsafe and unreliable.

However, using send(), you do get an error if something fails.

Thanks

[aliartiza75]

@kcris I am also facing the same issue

[jeffwidman]

Some of this behavior was changed in #1553 to make errors more obvious in the logs.