Verbose logging on messages signalling data loss in producer

[sibiryakov]

Recently, I've spend some human days trying to find where in our multi step pipeline we were having a data loss. It turned out one of our producers were generating batches too often resulting to their expiration. But there were no signs of any problems in logs. This PR attempts to fix this issue.

---

This change is 

[dpkp]

You can also achieve this in application code by adding error logging to your produce messages via an callback / errback. Would that approach work for you instead?

[sibiryakov]

Yes, I can @dpkp but this is awkward and non-obvious. Look from the user prospective: When user is starting to use the library it is assuming that Producer is reliable, and default settings are sufficient. Then he pushes too much small messages and it could take him days to debug where the problem is, depending on the infrastructure complexity. From my experience when messages are silently dropped in Kafka-based sequence of workers it is extremely hard to find where the problem is, and implies checking the whole sequence.

So my opinion that even there is a way to be notified of the problem, it will be rarely used and therefore result to a bad user experience.

[jeffwidman]

I used to be against this PR, but after thinking about it more (and trying to support many dev teams that aren't familiar with the intricacies of this library), I do agree that the default user experience is better with this.

New users will look for an errback/callback mechanism to let them handle retries or dumping the message into another database for further analysis. But for checking if an application is healthy, the first thing I see most devs do is check the logs and if no errors then they assume everything is fine...

[sibiryakov]

But for checking if an application is healthy, the first thing I see most devs do is check the logs and if no errors then they assume everything is fine...

And this is natural. I consider design poor if it requires code to be rewritten to diagnose that there is a data drop happens.

[dpkp]

+1 -- thanks!!