Closed as not planned
Description
We run a multi.exec() to BLPOP from a lot of different lists. We do this several times per second. After some (random) time, the multi.exec() promise stops resolving or throwing. It likely got something to do with the redis command queue, but we are unable to reproduce the error in a test environment. It happens regularly in production, though.
Code:
multiBlpop (cb) {
const multi = this._redisClient.multi()
for (let i = 0; i < 100; i++) {
multi.blPop(this._listNames, this._multiBlpopTimeout)
}
const callbackifiedMultiExec = callbackify(multi.exec.bind(multi))
// If after 5s the multi call did not call back we have the node-redis stalling problem
// and need to call our callback so that the next cycle of blpops can be triggered.
// 5 seconds is a good one here, because these calls should actually take 100ms at most.
withTimeout(callbackifiedMultiExec, (err, replies) => {
if (err) {
Logger.logEvent({
event : 'RedisMultiBlpopErr',
err,
})
}
if (err && this._abortOnErr) {
return cb(err)
}
...
}
}Some example log output:
So the timeout then fires consistently after some time and no further blpops go through, even if called again. The returned error is most often not about "the queue is full", even though it happens. But why would there be a command queue not emptying out? When we restart the script that makes the redis connection, everything starts working again.
Any advice?
Environment:
- Node.js Version: 16.15.1
- Redis Server Version: 6.2.6
- Node Redis Version: 4.3.1
- Platform: Ubuntu 20.04.5 LTS