Skip to content

Commit 7979f7f

Browse files
lgrammellgrammel
authored
feat (provider): support reasoning tokens, cached input tokens, total token in usage information (vercel#6140)
## Background Over the past year, additional token usage information such as cached input tokens and reasoning tokens were added by many providers. In addition, the meaning of total tokens is now provider-dependent, with some providers including reasoning tokens in the total token count. ## Summary Standardize sending reasoning tokens, cached prompt tokens, and total tokens from the providers.
1 parent 8919eff commit 7979f7f

File tree

74 files changed

+3177
-2047
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

74 files changed

+3177
-2047
lines changed

‎.changeset/silver-vans-march.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@ai-sdk/provider': major
3+
---
4+
5+
feat (provider): support reasoning tokens, cached input tokens, total token in usage information

‎content/providers/01-ai-sdk-providers/05-anthropic.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ const result = await generateText({
183183

184184
console.log(result.text);
185185
console.log(result.providerMetadata?.anthropic);
186-
// e.g. { cacheCreationInputTokens: 2118, cacheReadInputTokens: 0 }
186+
// e.g. { cacheCreationInputTokens: 2118 }
187187
```
188188

189189
You can also use cache control on system messages by providing multiple system messages at the head of your messages array:

‎examples/ai-core/src/generate-object/mock-error.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,11 @@ async function main() {
1616
modelId: 'model-1',
1717
},
1818
finishReason: 'stop',
19-
usage: { inputTokens: 10, outputTokens: 20 },
19+
usage: {
20+
inputTokens: 10,
21+
outputTokens: 20,
22+
totalTokens: 30,
23+
},
2024
}),
2125
}),
2226
schema: z.object({ content: z.string() }),

‎examples/ai-core/src/generate-object/mock-repair-add-close.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,11 @@ async function main() {
77
const result = await generateObject({
88
model: new MockLanguageModelV2({
99
doGenerate: async () => ({
10-
usage: { inputTokens: 10, outputTokens: 20 },
10+
usage: {
11+
inputTokens: 10,
12+
outputTokens: 20,
13+
totalTokens: 30,
14+
},
1115
warnings: [],
1216
finishReason: 'tool-calls',
1317
content: [

‎examples/ai-core/src/generate-object/mock.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ async function main() {
99
doGenerate: async () => ({
1010
content: [{ type: 'text', text: `{"content":"Hello, world!"}` }],
1111
finishReason: 'stop',
12-
usage: { inputTokens: 10, outputTokens: 20 },
12+
usage: {
13+
inputTokens: 10,
14+
outputTokens: 20,
15+
totalTokens: 30,
16+
},
1317
warnings: [],
1418
}),
1519
}),

‎examples/ai-core/src/generate-text/anthropic-cache-control.ts

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,13 @@ async function main() {
3535
});
3636

3737
console.log(result.text);
38-
console.log(result.providerMetadata?.anthropic);
39-
// e.g. { cacheCreationInputTokens: 2118, cacheReadInputTokens: 0 }
38+
console.log();
39+
40+
console.log('Cache read tokens:', result.usage.cachedInputTokens);
41+
console.log(
42+
'Cache write tokens:',
43+
result.providerMetadata?.anthropic?.cacheCreationInputTokens,
44+
);
4045
}
4146

4247
main().catch(console.error);

‎examples/ai-core/src/generate-text/mock-tool-call-repair-reask.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ async function main() {
99
model: new MockLanguageModelV2({
1010
doGenerate: async () => ({
1111
warnings: [],
12-
usage: { inputTokens: 10, outputTokens: 20 },
12+
usage: {
13+
inputTokens: 10,
14+
outputTokens: 20,
15+
totalTokens: 30,
16+
},
1317
finishReason: 'tool-calls',
1418
content: [
1519
{

‎examples/ai-core/src/generate-text/mock-tool-call-repair-structured-model.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ async function main() {
99
model: new MockLanguageModelV2({
1010
doGenerate: async () => ({
1111
warnings: [],
12-
usage: { inputTokens: 10, outputTokens: 20 },
12+
usage: {
13+
inputTokens: 10,
14+
outputTokens: 20,
15+
totalTokens: 30,
16+
},
1317
finishReason: 'tool-calls',
1418
content: [
1519
{

‎examples/ai-core/src/generate-text/mock.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,11 @@ async function main() {
88
doGenerate: async () => ({
99
content: [{ type: 'text', text: `Hello, world!` }],
1010
finishReason: 'stop',
11-
usage: { inputTokens: 10, outputTokens: 20 },
11+
usage: {
12+
inputTokens: 10,
13+
outputTokens: 20,
14+
totalTokens: 30,
15+
},
1216
warnings: [],
1317
}),
1418
}),

‎examples/ai-core/src/generate-text/openai-reasoning.ts

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,15 @@ import { generateText } from 'ai';
33
import 'dotenv/config';
44

55
async function main() {
6-
const { text, usage, providerMetadata } = await generateText({
6+
const { text, usage } = await generateText({
77
model: openai('o3-mini'),
88
prompt: 'How many "r"s are in the word "strawberry"?',
99
temperature: 0.5, // should get ignored (warning)
1010
maxOutputTokens: 1000, // mapped to max_completion_tokens
1111
});
1212

1313
console.log(text);
14-
console.log();
15-
console.log('Usage:', {
16-
...usage,
17-
reasoningTokens: providerMetadata?.openai?.reasoningTokens,
18-
});
14+
console.log('Usage:', usage);
1915
}
2016

2117
main().catch(console.error);

‎examples/ai-core/src/stream-object/mock.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,11 @@ async function main() {
1818
type: 'finish',
1919
finishReason: 'stop',
2020
logprobs: undefined,
21-
usage: { inputTokens: 3, outputTokens: 10 },
21+
usage: {
22+
inputTokens: 3,
23+
outputTokens: 10,
24+
totalTokens: 13,
25+
},
2226
},
2327
]),
2428
}),

‎examples/ai-core/src/stream-object/openai-token-usage.ts

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,12 @@ async function main() {
1717
});
1818

1919
// your custom function to record usage:
20-
function recordUsage({
21-
promptTokens,
22-
completionTokens,
23-
totalTokens,
24-
}: LanguageModelUsage) {
25-
console.log('Prompt tokens:', promptTokens);
26-
console.log('Completion tokens:', completionTokens);
27-
console.log('Total tokens:', totalTokens);
20+
function recordUsage(usage: LanguageModelUsage) {
21+
console.log('Input tokens:', usage.inputTokens);
22+
console.log('Cached input tokens:', usage.cachedInputTokens);
23+
console.log('Reasoning tokens:', usage.reasoningTokens);
24+
console.log('Output tokens:', usage.outputTokens);
25+
console.log('Total tokens:', usage.totalTokens);
2826
}
2927

3028
// use as promise:

‎examples/ai-core/src/stream-text/mock.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,11 @@ async function main() {
1414
type: 'finish',
1515
finishReason: 'stop',
1616
logprobs: undefined,
17-
usage: { inputTokens: 3, outputTokens: 10 },
17+
usage: {
18+
inputTokens: 3,
19+
outputTokens: 10,
20+
totalTokens: 13,
21+
},
1822
},
1923
]),
2024
}),

‎examples/ai-core/src/stream-text/openai-reasoning.ts

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,7 @@ async function main() {
1515
}
1616

1717
console.log();
18-
console.log('Usage:', {
19-
...(await result.usage),
20-
reasoningTokens: (await result.providerMetadata)?.openai?.reasoningTokens,
21-
});
18+
console.log('Usage:', await result.usage);
2219
console.log('Warnings:', await result.warnings);
2320
}
2421

‎examples/ai-core/src/stream-text/smooth-stream-chinese.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,11 @@ async function main() {
1616
type: 'finish',
1717
finishReason: 'stop',
1818
logprobs: undefined,
19-
usage: { inputTokens: 3, outputTokens: 10 },
19+
usage: {
20+
inputTokens: 3,
21+
outputTokens: 10,
22+
totalTokens: 13,
23+
},
2024
},
2125
],
2226
chunkDelayInMs: 400,

‎examples/ai-core/src/stream-text/smooth-stream-japanese.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,11 @@ async function main() {
1616
type: 'finish',
1717
finishReason: 'stop',
1818
logprobs: undefined,
19-
usage: { inputTokens: 3, outputTokens: 10 },
19+
usage: {
20+
inputTokens: 3,
21+
outputTokens: 10,
22+
totalTokens: 13,
23+
},
2024
},
2125
],
2226
chunkDelayInMs: 400,

‎packages/ai/core/generate-object/__snapshots__/stream-object.test.ts.snap

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,10 @@ exports[`streamObject > output = "object" > options.onFinish > should be called
1818
"timestamp": 1970-01-01T00:00:00.000Z,
1919
},
2020
"usage": {
21-
"completionTokens": 10,
22-
"promptTokens": 3,
21+
"cachedInputTokens": undefined,
22+
"inputTokens": 3,
23+
"outputTokens": 10,
24+
"reasoningTokens": undefined,
2325
"totalTokens": 13,
2426
},
2527
"warnings": undefined,
@@ -38,8 +40,10 @@ exports[`streamObject > output = "object" > options.onFinish > should be called
3840
"timestamp": 1970-01-01T00:00:00.000Z,
3941
},
4042
"usage": {
41-
"completionTokens": 10,
42-
"promptTokens": 3,
43+
"cachedInputTokens": undefined,
44+
"inputTokens": 3,
45+
"outputTokens": 10,
46+
"reasoningTokens": undefined,
4347
"totalTokens": 13,
4448
},
4549
"warnings": undefined,
@@ -99,9 +103,11 @@ exports[`streamObject > output = "object" > result.fullStream > should send full
99103
},
100104
"type": "finish",
101105
"usage": {
102-
"completionTokens": 10,
103-
"promptTokens": 2,
104-
"totalTokens": 12,
106+
"cachedInputTokens": undefined,
107+
"inputTokens": 3,
108+
"outputTokens": 10,
109+
"reasoningTokens": undefined,
110+
"totalTokens": 13,
105111
},
106112
},
107113
]
@@ -119,8 +125,9 @@ exports[`streamObject > telemetry > should not record telemetry inputs / outputs
119125
"ai.settings.maxRetries": 2,
120126
"ai.settings.output": "object",
121127
"ai.settings.temperature": 0,
122-
"ai.usage.completionTokens": 10,
123-
"ai.usage.promptTokens": 3,
128+
"ai.usage.inputTokens": 3,
129+
"ai.usage.outputTokens": 10,
130+
"ai.usage.totalTokens": 13,
124131
"operation.name": "ai.streamObject",
125132
},
126133
"events": [],
@@ -138,8 +145,9 @@ exports[`streamObject > telemetry > should not record telemetry inputs / outputs
138145
"ai.settings.maxRetries": 2,
139146
"ai.settings.temperature": 0,
140147
"ai.stream.msToFirstChunk": 0,
141-
"ai.usage.completionTokens": 10,
142-
"ai.usage.promptTokens": 3,
148+
"ai.usage.inputTokens": 3,
149+
"ai.usage.outputTokens": 10,
150+
"ai.usage.totalTokens": 13,
143151
"gen_ai.request.model": "mock-model-id",
144152
"gen_ai.request.temperature": 0,
145153
"gen_ai.response.finish_reasons": [
@@ -189,8 +197,9 @@ exports[`streamObject > telemetry > should record telemetry data when enabled 1`
189197
"ai.telemetry.functionId": "test-function-id",
190198
"ai.telemetry.metadata.test1": "value1",
191199
"ai.telemetry.metadata.test2": false,
192-
"ai.usage.completionTokens": 10,
193-
"ai.usage.promptTokens": 3,
200+
"ai.usage.inputTokens": 3,
201+
"ai.usage.outputTokens": 10,
202+
"ai.usage.totalTokens": 13,
194203
"operation.name": "ai.streamObject test-function-id",
195204
"resource.name": "test-function-id",
196205
},
@@ -220,8 +229,9 @@ exports[`streamObject > telemetry > should record telemetry data when enabled 1`
220229
"ai.telemetry.functionId": "test-function-id",
221230
"ai.telemetry.metadata.test1": "value1",
222231
"ai.telemetry.metadata.test2": false,
223-
"ai.usage.completionTokens": 10,
224-
"ai.usage.promptTokens": 3,
232+
"ai.usage.inputTokens": 3,
233+
"ai.usage.outputTokens": 10,
234+
"ai.usage.totalTokens": 13,
225235
"gen_ai.request.frequency_penalty": 0.3,
226236
"gen_ai.request.model": "mock-model-id",
227237
"gen_ai.request.presence_penalty": 0.4,

‎packages/ai/core/generate-object/generate-object.test.ts

Lines changed: 17 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,13 @@ import { generateObject } from './generate-object';
1010

1111
const dummyResponseValues = {
1212
finishReason: 'stop' as const,
13-
usage: { inputTokens: 10, outputTokens: 20 },
13+
usage: {
14+
inputTokens: 10,
15+
outputTokens: 20,
16+
totalTokens: 30,
17+
reasoningTokens: undefined,
18+
cachedInputTokens: undefined,
19+
},
1420
response: { id: 'id-1', timestamp: new Date(123), modelId: 'm-1' },
1521
warnings: [],
1622
};
@@ -388,9 +394,9 @@ describe('output = "object"', () => {
388394
...dummyResponseValues,
389395
content: [{ type: 'text', text: '{ "content": "Hello, world!" }' }],
390396
providerMetadata: {
391-
anthropic: {
392-
cacheCreationInputTokens: 10,
393-
cacheReadInputTokens: 20,
397+
exampleProvider: {
398+
a: 10,
399+
b: 20,
394400
},
395401
},
396402
}),
@@ -400,9 +406,9 @@ describe('output = "object"', () => {
400406
});
401407

402408
expect(result.providerMetadata).toStrictEqual({
403-
anthropic: {
404-
cacheCreationInputTokens: 10,
405-
cacheReadInputTokens: 20,
409+
exampleProvider: {
410+
a: 10,
411+
b: 20,
406412
},
407413
});
408414
});
@@ -573,9 +579,11 @@ describe('output = "object"', () => {
573579
modelId: 'm-1',
574580
},
575581
usage: {
576-
completionTokens: 20,
577-
promptTokens: 10,
582+
inputTokens: 10,
583+
outputTokens: 20,
578584
totalTokens: 30,
585+
reasoningTokens: undefined,
586+
cachedInputTokens: undefined,
579587
},
580588
finishReason: 'stop',
581589
});

0 commit comments

Comments
 (0)