Feat/anthropic extended ttl #6205
Conversation
Added to `cacheBehaviorSchema` extra optional parameters. ``` useExtendedCacheTtlBeta: z.boolean().optional(), cacheTtl: z.enum(["5m", "1h"]).optional(), ```
👷 Deploy request for continuedev pending review.Visit the deploys page to approve it
|
dosubot
bot
added
the
size:L
|
All contributors have signed the CLA ✍️ ✅ |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
|
😱 Found 3 issues. Time to roll up your sleeves! 😱 |
|
I have read the CLA Document and I hereby sign the CLA |
|
Some details about how long session with big context looks with 5min cache and 1h cache and cost perspective: |
|
1h ttl |
dosubot
bot
added
size:XL
There was a problem hiding this comment.
This is a great PR as far as the code goes. I kind of want to step back though to better understand whether you think this could be a sensible default rather than a configuration option. I'm weary of too many options and if everyone would benefit from the way you are configuring your Anthropic models, maybe we should just ship that as the default (I repeated all this in a comment below)
| @@ -62,18 +62,40 @@ Anthropic currently does not offer any reranking models. | |||
|
|
|||
| Anthropic supports [prompt caching with Claude](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching), which allows Claude models to cache system messages and conversation history between requests to improve performance and reduce costs. | |||
|
|
|||
| > **NOTE:** As part of their `Beta` support [Extended caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching#1-hour-cache-duration) | |||
There was a problem hiding this comment.
The docs here feel a bit extensive since they take up most of this page now. I think we should try to make a collapsible block or make a dedicated page to prompt caching. If possible, I think the collapsible would be the better option
| @@ -927,6 +927,13 @@ export interface RequestOptions { | |||
| export interface CacheBehavior { | |||
| cacheSystemMessage?: boolean; | |||
| cacheConversation?: boolean; | |||
| useExtendedCacheTtlBeta?: boolean; | |||
There was a problem hiding this comment.
I'm coming at this review with the lens of "if we add it now, we'll have to support it forever (or go through a deliberate deprecation process)". I'm worried there are a large number of options here that aren't going to be relevant forever, or that they might not be the final form of this configuration.
It would be helpful to better understand whether all of the cacheUserMessages, cacheAssistantMessages, etc. are truly necessary for people to customize, or whether we just need to set a more sensible default. For example, I'd be curious what values you set here and whether you think we should just ship those as the defaults for everyone. Usage patterns in Continue are probably similar across a variety of users. Not that we couldn't eventually also allow this customization, but it might save a lot of maintenance (and give many users money back without needing to configure anything)
|
Agreed @chezsmithy ! Thanks for linking the PR here, that what I had in mind |
Description
Implements granular per-message-type caching for Anthropic models to improve token efficiency in Agent mode. Adds new
CacheBehavioroptions to specify how many of each message type to cache (user messages, tool results, assistant tool calls, etc.) instead of only caching the last 2 user messages.This is related to issue #6135
Checklist
Screenshots
N/A - Backend caching enhancement with no visual changes.
Tests
Added comprehensive test suite
core/llm/llms/Anthropic.enhanced-caching.test.tswith 6 test cases covering:shouldCacheMessagelogicAll tests pass and validate the new per-type caching functionality while maintaining backward compatibility.