switch_db doesn't really switch the connection

[bsod90]

We're developing a multi tenant (SaaS) Django app, which relies on three main services on backend: postgresql, elastic search and mongodb.
We recently had to switch to the approach where we switch the DB context for every request or any other operation (command, celery task) dynamically. Everything worked fine, except the mongoengine...
Right now we're applying a stack of switch_db context managers (one for every model) in the first middleware for every request, where we also switch postgres and elastic.
The other two get switched fine, but mongo behaves weirdly.
Basically, every gunicorn worker remembers the first 'tenant' that was used and then just keeps using it for every subsequent request.
Here are two screenshots of the logs from two identical gunicorns, where I'm trying to fetch the same doc:
https://d.pr/i/qFcVQP
https://d.pr/i/R5ygRm

The one with 404 is the one where I fetched some other document as the other tenant before.
The log contains mode._meta right before I'm trying to fetch it from mongo.
You can see that db_alias is actually different, but I'm guessing the connection is cached somewhere else.
Right now the only workaround is to restart gunicorn on every request, which is not really feasible even in short term. I'm wondering if there's anything else we can do (any level of dirtiness is fine).

Below are some snippets on how things currently work:
Context manager to switch DB for all models:

@contextmanager
def switch_mongo_db(alias):
    with ExitStack() as stack:
        for model in all_models:
            stack.enter_context(switch_db(model, alias))
        yield


@contextmanager
def switch_tenant(tenant):
    old_tenant = settings.CURRENT_TENANT.set_tenant(tenant)
    connection.set_tenant(tenant)
    logger.info("Tenant switched to {}".format(tenant.schema_name))
    try:
        with switch_mongo_db(tenant.mongo_name):
            yield
    finally:
        settings.CURRENT_TENANT.set_tenant(old_tenant)
        if old_tenant:
            connection.set_tenant(old_tenant)
        else:
            connection.set_schema('public')
        logger.info("Tenant restored to {}".format(
            old_tenant.schema_name if old_tenant else None
        ))

Middleware:

class OnebarTenantMiddleware(BaseTenantMiddleware):

    def get_tenant(self, model, hostname, request):
        if hostname in settings.PUBLIC_DOMAINS:
            return model.objects.get(schema_name='public')
        subdomain = hostname.split('.')[0]
        return model.objects.get(domain_url=subdomain)

    def __call__(self, request):
        self.process_request(request)
        if settings.MULTITENANT_MODE:
            with switch_tenant(request.tenant):
                response = self.get_response(request)
        else:
            response = self.get_response(request)

        return response

That weird object in settings:

class CurrentTenantSettins(object):
    def __init__(self):
        self._tenant = None

    def __getattr__(self, key):
        if MULTITENANT_MODE:
            assert self._tenant, \
                'Current tenant must be set in MULTITENANT_MODE'
        elif not self._tenant:
            from cloud.models import DefaultCustomer
            self._tenant = DefaultCustomer()
        return getattr(self._tenant, key)

    def set_tenant(self, tenant):
        old_tenant = self._tenant
        if tenant:
            mongoengine.register_connection(
                alias=tenant.mongo_name,
                name=tenant.mongo_name,
                host=MONGODB_HOST,
                port=MONGODB_PORT,
                username=MONGODB_USER,
                password=MONGODB_PASSWORD
            )
            os.makedirs(tenant.parsers_root, exist_ok=True)
        else:
            disconnect(DEFAULT_CONNECTION_NAME)
        self._tenant = tenant
        return old_tenant


CURRENT_TENANT = CurrentTenantSettins()

P.S.
Storing things like this in settings is completely discouraged and we will rework it as soon as we get a change. We just desperately need something that works asap...

[bsod90]

I can also work on a fix for mongoengine as soon as I get some free time, but I would need some pointers on where to fix things and what's the preferred way.

[Fierozen]

Did anything ever happen with this issue? It's been almost 1 1/2 years.

[bagerard]

Better late than never but we've been working on fixing some of the issues around connect/disconnect functions (#2038). Pymongo's collection gets cached in Document class (as _collection) and many people had issues with this as it wasn't cleaned properly when calling disconnect().

The switch_db context manager seems to work fine but it could be that its not thread safe, maybe that's the problem here?

[jaisharma639]

Is this fixed now?

[DamianCasale]

There is certainly something odd still going on in this area.

Using a basic Flask based todo api PUT/GET with mongoengine==0.20.0 and two seperate mongoDB 3.6 databases, with a switch_db based on a flask request variable.

Any requests to "localhost" or "127.0.0.1" are passed to different database aliases. And the logs show it appears to be doing what it should be.

however with a basic test :
for i in {1..1000}; do curl http://localhost:5000/todo${i} -d "data=put to localhost ${i}" -X PUT; done
and
for i in {1..1000}; do curl http://127.0.0.1:5000/todo${i} -d "data=put to 127.0.0.1 ${i}" -X PUT; done
run simultaniously against the api.

I find both databases do not have 1000 entires e.g. one has 1004 while the other has 996
I find the majority of data has gone to the correct database, however some 127.0.0.1 data is actually in the localhost database and visa versa.

of course running for i in {1..1000}; do curl http://127.0.0.1:5000/todo${i}; done does give some cursor not founds.