Adjusting disconnectedBehavior Option to Prevent Timeout During Redis Shutdown #2866 #2894
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… Shutdown We often need to shut down Redis for maintenance, such as version upgrades. During these times, requests do not fail immediately but instead experience timeouts, increasing application latency. This issue can be resolved by adjusting some options. Currently, our Redis client options are configured as follows: ```java ClientOptions options = ClientOptions.builder() .autoReconnect(true) .disconnectedBehavior(DisconnectedBehavior.REJECT_COMMANDS) ``` The DisconnectedBehavior.REJECT_COMMANDS option appears to cancel commands when the connection is lost. However, if autoReconnect is not set to false, commands in the CommandHandler.stack are not canceled but are placed into the disconnectedBuffer. Therefore, ongoing commands are not rejected if autoReconnect is true, even with the client option modified. For services heavily relying on Redis, latency is crucial. Additionally, we want to avoid writing custom code for reconnections by using the auto-reconnect feature. Adjusting the autoReconnect option can solve this issue immediately, but it would require significant changes to implement automatic reconnection. Proposal We propose that the condition for canceling commands in the CommandHandler.stack should be based solely on rejectCommandsWhileDisconnected and not combined with autoReconnect. I've submitted a PR with a simple code change. Please let me know your thoughts. === Additionally our applications, to maintain low latency and avoid queuing many requests in the buffer when the connection is down, we have configured the client to execute commands only when the connection is active. As a result, only a few requests in the CommandHandler.stack encounter timeouts, but this still negatively impacts the application.
|
Do you think you can also add a test that simulates the scenario you are fixing? |
tishun
added
for: team-attention
|
Thanks for the contribution! |
tishun
removed
for: team-attention
svc-squareup-copybara
pushed a commit
to cashapp/misk
that referenced
this pull request
Jun 3, 2025
| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [io.lettuce:lettuce-core](https://github.com/redis/lettuce) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `6.5.5.RELEASE` -> `6.7.0.RELEASE` | --- ### Release Notes <details> <summary>redis/lettuce (io.lettuce:lettuce-core)</summary> ### [`v6.7.0.RELEASE`](https://github.com/redis/lettuce/releases/tag/6.7.0.RELEASE) [Compare Source](redis/lettuce@6.6.0.RELEASE...6.7.0.RELEASE) The Lettuce team is pleased to announce the Lettuce 6.7.0 minor release! This release provides support for the newly introduced [Vector Sets](https://redis.io/docs/latest/develop/data-types/vector-sets/) data type which was released as part of Redis 8.0 and [helps Redis users with vector similarity](https://redis.io/blog/announcing-vector-sets-a-new-redis-data-type-for-vector-similarity/). Starting with 6.7 the `ConnectionPoolSupport` also provides a way to provide custom connection validations. The release comes with a bunch of smaller improvements and bugfixes. Lettuce 6.7.0 supports Redis 2.6+ up to Redis 8.x. In terms of Java runtime, Lettuce requires at least Java 8 and works with Java 24. The driver is tested against Redis 8.0, Redis 7.4 and Redis 7.2. Find the full changelog at the end of this document. Thanks to all contributors who made Lettuce 6.7.0.RELEASE possible. #### 📗 Links - Reference documentation: https://lettuce.io/core/6.7.0.RELEASE/reference/ - Javadoc: https://lettuce.io/core/6.7.0.RELEASE/api/ #### ⭐ New Features - Support Redis 8 vector sets [#​3296](redis/lettuce#3296) by [@​tishun](https://github.com/tishun) in redis/lettuce#3295 - Add custom connection validation to ConnectionPoolSupport [#​3081](redis/lettuce#3081) by [@​big-cir](https://github.com/big-cir) in redis/lettuce#3138 #### 🐞 Bug Fixes - Fix deadlock when an invalid URI is presented to DefaultClusterTopologyRefresh by [@​henry701](https://github.com/henry701) in redis/lettuce#3243 - Fix NPE in EntraIdIntegrationTests by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3254 - Deprecate DnsResolver in favor of AddressResolverGroup([#​1572](redis/lettuce#1572)) by [@​young0264](https://github.com/young0264) in redis/lettuce#3291 #### 💡 Other - Change native library default to epoll over io_uring by [@​thachlp](https://github.com/thachlp) in redis/lettuce#3278 - Use InfoPatterns enum for ReplicaTopologyProvider pattern management by [@​ori0o0p](https://github.com/ori0o0p) in redis/lettuce#3264 - Adjusting disconnectedBehavior Option to Prevent Timeout During Redis Shutdown [#​2866](redis/lettuce#2866) by [@​MagicalLas](https://github.com/MagicalLas) in redis/lettuce#2894 - Improve the performance of obtaining write connections through double-check locks. by [@​Chenrujie-85](https://github.com/Chenrujie-85) in redis/lettuce#3228 - Optimize string concatenation in NodeSelectionInvocationHandler.getNodeDescription() by [@​ori0o0p](https://github.com/ori0o0p) in redis/lettuce#3262 - DOC-4756 sorted set examples with join() by [@​andy-stark-redis](https://github.com/andy-stark-redis) in redis/lettuce#3184 - DOC-4757 list examples using join() by [@​andy-stark-redis](https://github.com/andy-stark-redis) in redis/lettuce#3185 - docs: add default threads count about NioEventLoopGroup by [@​brido4125](https://github.com/brido4125) in redis/lettuce#3221 - Guide on resolving native library conflicts by [@​thachlp](https://github.com/thachlp) in redis/lettuce#3309 #### ❤️ New Contributors - [@​big-cir](https://github.com/big-cir) made their first contribution in redis/lettuce#3138 - [@​Chenrujie-85](https://github.com/Chenrujie-85) made their first contribution in redis/lettuce#3228 - [@​brido4125](https://github.com/brido4125) made their first contribution in redis/lettuce#3221 - [@​MagicalLas](https://github.com/MagicalLas) made their first contribution in redis/lettuce#2894 - [@​young0264](https://github.com/young0264) made their first contribution in redis/lettuce#3291 - [@​henry701](https://github.com/henry701) made their first contribution in redis/lettuce#3243 **Full Changelog**: redis/lettuce@6.6.0.BETA4...6.7.0.RELEASE ### [`v6.6.0.RELEASE`](https://github.com/redis/lettuce/releases/tag/6.6.0.RELEASE) [Compare Source](redis/lettuce@6.5.5.RELEASE...6.6.0.RELEASE) The Lettuce team is pleased to announce the Lettuce 6.6.0 minor release! This release provides support for the newly introduced `HGETDEL`, `HGETEX` and `HSETEX` commands. Users of the driver are also now able to [use a command replay filter](https://redis.github.io/lettuce/advanced-usage/#controlling-replay-of-commands-in-at-lease-once-mode), that allows the user to control which (if any) commands are being retried when the reconnect feature is on. It is also possible to configure the driver to use the `HashIndexedQueue` as a backing data structure for the command queue, which speeds up the driver during disconnect / reconnect (needs to be configured using the ClientOptions). The driver comes with Microsoft EntraID authentication support. Last but not least the `STRALGO` command was replaced by the `LCS` command. Lettuce 6.6.0 supports Redis 2.6+ up to Redis 8.x. In terms of Java runtime, Lettuce requires at least Java 8 and works with Java 24. The driver is tested against Redis 8.0, Redis 7.4 and Redis 7.2. Find the full changelog at the end of this document. Thanks to all contributors who made Lettuce 6.6.0.RELEASE possible. #### 📗 Links - Reference documentation: https://lettuce.io/core/6.6.0.RELEASE/reference/ - Javadoc: https://lettuce.io/core/6.6.0.RELEASE/api/ #### ⭐ New Features - Hash Field Expiration (part II) ([#​3195](redis/lettuce#3195)) by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3204 - Introduce command replay filter to avoid command replaying after reconnect [#​1310](redis/lettuce#1310) by [@​tishun](https://github.com/tishun) in redis/lettuce#3118 - Deprecate the STRALGO command and implement the LCS in its place by [@​Dltmd202](https://github.com/Dltmd202) in redis/lettuce#3037 - fix: prevent blocking event loop thread by replacing ArrayDeque with HashIndexedQueue by [@​okg-cxf](https://github.com/okg-cxf) in redis/lettuce#2953 - Token based authentication integration with core extension by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3063 - Support for DefaultAzureCredential by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3230 - Add support up to max unsigned integer in Bitfield offset ([#​2964](redis/lettuce#2964)) by [@​psw0946](https://github.com/psw0946) in redis/lettuce#3099 - Improve code by adding some null checks by [@​tishun](https://github.com/tishun) in redis/lettuce#3115 - Introduce test matrix based on Redis server versions by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3145 - Add modules ACL support by [@​sazzad16](https://github.com/sazzad16) in redis/lettuce#3102 - Test modules CONFIG support by [@​sazzad16](https://github.com/sazzad16) in redis/lettuce#3103 - report block error when use with reactor mode [#​3168](redis/lettuce#3168) by [@​tishun](https://github.com/tishun) in redis/lettuce#3169 - Include command type in the timeout message by [@​arturaz](https://github.com/arturaz) in redis/lettuce#3167 - replace hardcoded GT and LT with CommandKeyword enum by [@​minwoo1999](https://github.com/minwoo1999) in redis/lettuce#3079 #### 🐞 Bug Fixes - Restore API that was accidently deleted when introducing the JSON feature by [@​tishun](https://github.com/tishun) in redis/lettuce#3065 - Propagate handshake failures to Handshake future by [@​mp911de](https://github.com/mp911de) in redis/lettuce#3058 - OpsForGeo producing "READONLY You can't write against a read only replica " on READS... by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3032 - Json commands not exposed in AsyncCluster [#​3048](redis/lettuce#3048) by [@​tishun](https://github.com/tishun) in redis/lettuce#3049 - WATCH during MULTI shouldn't fail transaction [#​3009](redis/lettuce#3009) by [@​tishun](https://github.com/tishun) in redis/lettuce#3027 - Fix: make sure FIFO order between write and notify channel active by [@​okg-cxf](https://github.com/okg-cxf) in redis/lettuce#2597 - UnsupportedOperationException from ListSubscriber during hrandfieldWithvalues [#​3122](redis/lettuce#3122) by [@​tishun](https://github.com/tishun) in redis/lettuce#3123 - Update CommonsPool2ConfigConverterUnitTests.java by [@​Rian-Ismael](https://github.com/Rian-Ismael) in redis/lettuce#3147 - Fix typo & add withSsl() in connecting to Entra ID enabled Redis doc by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3191 - Fix SimpleBatcher apparent deadlock [#​2196](redis/lettuce#2196) by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3148 - jsonArrpop fails with null return value ([#​3196](redis/lettuce#3196)) by [@​tishun](https://github.com/tishun) in redis/lettuce#3206 - json.arrpop forces index=-1 with root path ([#​3214](redis/lettuce#3214)) by [@​thachlp](https://github.com/thachlp) in redis/lettuce#3217 - Updates enableAdaptiveRefreshTrigger trigger assertion message by [@​ymiliaresis](https://github.com/ymiliaresis) in redis/lettuce#3216 #### 💡 Other - Add example configuration using SNI enabled TLS connection by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3045 - Disable docker image being used to call compose when running tests by [@​tishun](https://github.com/tishun) in redis/lettuce#3046 - Workflow for running benchmarks weekly by [@​tishun](https://github.com/tishun) in redis/lettuce#3052 - Fixing benchmark flow by [@​tishun](https://github.com/tishun) in redis/lettuce#3056 - Test failures not reported because step is skipped by [@​tishun](https://github.com/tishun) in redis/lettuce#3067 - Stale issues action bump by [@​tishun](https://github.com/tishun) in redis/lettuce#3182 - Migrate Lettuce test setup to use client-lib-test by [@​kiryazovi-redis](https://github.com/kiryazovi-redis) in redis/lettuce#3158 - JSON integration tests now do not use the test-containers framework by [@​tishun](https://github.com/tishun) in redis/lettuce#3203 - Test with 8.0-M05-pre by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3219 - Add sample examples to test redis.io build by [@​uglide](https://github.com/uglide) in redis/lettuce#3051 - DOC-4528 async hash examples by [@​andy-stark-redis](https://github.com/andy-stark-redis) in redis/lettuce#3069 - DOC-4531 set data type examples by [@​andy-stark-redis](https://github.com/andy-stark-redis) in redis/lettuce#3076 - DOC-4802 fix string example concurrency by [@​andy-stark-redis](https://github.com/andy-stark-redis) in redis/lettuce#3156 - Fix several typos on the advanced-usage page by [@​danicheg](https://github.com/danicheg) in redis/lettuce#3174 - docs: update Limitations section to reflect shaded JAR deprecation by [@​minwoo1999](https://github.com/minwoo1999) in redis/lettuce#3095 - Remove extra spaces in words in docs by [@​enjoy-binbin](https://github.com/enjoy-binbin) in redis/lettuce#3120 - Bump to v4 of checkout by [@​tishun](https://github.com/tishun) in redis/lettuce#3152 - Fix 'make test' test failures by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3157 - Readme doc on how to connect to Azure Managed Redis with Entra ID authentication by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3166 - refactor mget method improved readability and efficiency by [@​ori0o0p](https://github.com/ori0o0p) in redis/lettuce#3061 - Migrate JSON tests infra to use client-lilb-test by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3128 - Update the base project URLs in pom.xml by [@​danicheg](https://github.com/danicheg) in redis/lettuce#3151 - Update publish docs action to use latest versions of actions by [@​tishun](https://github.com/tishun) in redis/lettuce#3154 - Bump default client-libs-test container version by [@​ggivo](https://github.com/ggivo) in redis/lettuce#3165 - Bump org.slf4j:jcl-over-slf4j from 1.7.25 to 2.0.16 by [@​dependabot](https://github.com/dependabot) in redis/lettuce#2959 - Bump org.testcontainers:testcontainers from 1.20.1 to 1.20.4 by [@​dependabot](https://github.com/dependabot) in redis/lettuce#3082 - Bump io.micrometer:micrometer-bom from 1.12.4 to 1.14.2 by [@​dependabot](https://github.com/dependabot) in redis/lettuce#3096 - Bump io.netty.incubator:netty-incubator-transport-native-io_uring from 0.0.25.Final to 0.0.26.Final by [@​dependabot](https://github.com/dependabot) in redis/lettuce#3106 - Bump netty.version to 4.1.118.Final [#​3187](redis/lettuce#3187) by [@​tishun](https://github.com/tishun) in redis/lettuce#3189 #### ❤️ New Contributors - [@​Dltmd202](https://github.com/Dltmd202) made their first contribution in redis/lettuce#3037 - [@​andy-stark-redis](https://github.com/andy-stark-redis) made their first contribution in redis/lettuce#3069 - [@​okg-cxf](https://github.com/okg-cxf) made their first contribution in redis/lettuce#2953 - [@​minwoo1999](https://github.com/minwoo1999) made their first contribution in redis/lettuce#3079 - [@​psw0946](https://github.com/psw0946) made their first contribution in redis/lettuce#3099 - [@​ori0o0p](https://github.com/ori0o0p) made their first contribution in redis/lettuce#3061 - [@​enjoy-binbin](https://github.com/enjoy-binbin) made their first contribution in redis/lettuce#3120 - [@​sazzad16](https://github.com/sazzad16) made their first contribution in redis/lettuce#3102 - [@​danicheg](https://github.com/danicheg) made their first contribution in redis/lettuce#3151 - [@​Rian-Ismael](https://github.com/Rian-Ismael) made their first contribution in redis/lettuce#3147 - [@​kiryazovi-redis](https://github.com/kiryazovi-redis) made their first contribution in redis/lettuce#3158 - [@​ymiliaresis](https://github.com/ymiliaresis) made their first contribution in redis/lettuce#3216 - [@​arturaz](https://github.com/arturaz) made their first contribution in redis/lettuce#3167 **Full Changelog**: redis/lettuce@6.5.0.RELEASE...6.6.0.RELEASE </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: f67609a2c7de516500eacfa74f5c061609b8eff7
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The purpose of this PR is to adjust the disconnectedBehavior option to prevent timeouts when Redis is shutdown, addressing issue #2866.
Make sure that: