socket error at program exit when using MultiProcessConsumer

[alisaifee]

As of bb68965 I'm seeing the following errors when the process exits when using MultiProcessConsumer:

 Exception socket.error: error(2, 'No such file or directory') in <bound method MultiProcessConsumer.__del__ of <MultiProcessConsumer group=uuid, topic=test-topic, consumers=2>> ignored

After some investigation it seems like accessing any of the objects created via multiprocessing.Manager in the __del__ method can cause problems as the Manager registers a multiprocessing finalise hook which can get called before __del__.

For example, if you run the following piece of code, you'll see the same error:

from multiprocessing import Manager
class Test(object):
    def __init__(self):
        self.manager = Manager()
        self.thing = self.manager.list()
    def __del__(self):
        self.thing.append(1)
t = Test()

A few related observations:

• if auto_commit is False the error appears on process exit
• if auto_commit is True the error doesn't appear as __del__ is not executed
• if auto_commit is True and stop is called explicitly before the program exits, __del__ gets called on process exit and the exception occurs.

[alisaifee]

Since the invocation of __del__ isn't predictable and the main requirement is to ensure a commit when the program exits - it might be more reliable to use atexit.register. multiprocessing internally manages its finalise hooks via atexit which will run the exit hooks in the reverse order of registration (this should ensure that the multiprocessing Manager instance is still connected at the time when stop is called)

[dpkp]

agree -- atexit would probably be better

[dpkp]

fixed in master - thanks!