feat (provider): add providerMetadata to ImageModelV2 interface (vercel#5977)

## Background

Compare vercel#5698

## Summary

Additional provider-specific options to the image model provider
interface. They are passed through to the provider from the AI SDK and
enable provider-specific functionality that can be fully encapsulated in
the provider.

Unlike other models, ImageModel request return an array of images, and
provider can return image-specific metadata for each. So far, this pull
request passing through the revised prompt used for each image. In order
to make that possible, I introduced a new type
`ImageModelV2ProviderMetadata` which is the same as
`SharedV2ProviderMetadata` plus it guarantees the presence of the
`.images` key

```js
export type ImageModelV2ProviderMetadata = Record<
  string,
  {
    images: JSONArray;
  } & JSONValue
>;
```

That also makes it possible to deeply merge providerMetadata from
multiple responses effectively.

## Verification

I updated the `examples/ai-core/src/generate-image/openai.ts` example to
verify that the code is working.

Author: gr2m

.changeset/sour-bananas-remain.md

@@ -0,0 +1,25 @@
+---
+'@ai-sdk/provider': patch
+'@ai-sdk/openai': patch
+'ai': patch
+---
+
+feat (provider): add providerMetadata to ImageModelV2 interface (#5977)
+
+The `experimental_generateImage` method from the `ai` package now returnes revised prompts for OpenAI's image models.
+
+```js
+const prompt = 'Santa Claus driving a Cadillac';
+
+const { providerMetadata } = await experimental_generateImage({
+  model: openai.image('dall-e-3'),
+  prompt,
+});
+
+const revisedPrompt = providerMetadata.openai.images[0]?.revisedPrompt;
+
+console.log({
+  prompt,
+  revisedPrompt,
+});
+```

content/docs/03-ai-sdk-core/35-image-generation.mdx

@@ -182,6 +182,28 @@ const { image, warnings } = await generateImage({
 });
 ```
 
+### Additional provider-specific meta data
+
+Some providers expose additional meta data for the result overall or per image.
+
+```tsx
+const prompt = 'Santa Claus driving a Cadillac';
+
+const { image, providerMetaData } = await generateImage({
+  model: openai.image('dall-e-3'),
+  prompt,
+});
+
+const revisedPrompt = providerMetaData.openai.images[0]?.revisedPrompt;
+
+console.log({
+  prompt,
+  revisedPrompt,
+});
+```
+
+The outer key of the returned `providerMetaData` is the provider name. The inner values are the metadata. An `images` key is always present in the metadata and is an array with the same length as the top level `images` key.
+
 ### Error Handling
 
 When `generateImage` cannot generate a valid image, it throws a [`AI_NoImageGeneratedError`](/docs/reference/ai-sdk-errors/ai-no-image-generated-error).

content/docs/07-reference/01-ai-sdk-core/10-generate-image.mdx

@@ -165,6 +165,13 @@ console.log(images);
       description:
         'Warnings from the model provider (e.g. unsupported settings).',
     },
+    {
+      name: 'providerMetadata',
+      type: 'ImageModelV2ProviderMetadata',
+      isOptional: true,
+      description:
+        'Optional metadata from the provider. The outer key is the provider name. The inner values are the metadata. An `images` key is always present in the metadata and is an array with the same length as the top level `images` key. Details depend on the provider.',
+    },
     {
       name: 'responses',
       type: 'Array<ImageModelResponseMetadata>',

content/providers/01-ai-sdk-providers/02-openai.mdx

@@ -909,7 +909,7 @@ const model = openai.image('dall-e-3');
 You can pass optional `providerOptions` to the image model. These are prone to change by OpenAI and are model dependent. For example, the `gpt-image-1` model supports the `quality` option:
 
 ```ts
-const { image } = await generateImage({
+const { image, providerMetadata } = await generateImage({
   model: openai.image('gpt-image-1'),
   prompt: 'A salamander at sunrise in a forest pond in the Seychelles.',
   providerOptions: {
@@ -920,6 +920,8 @@ const { image } = await generateImage({
 
 For more on `generateImage()` see [Image Generation](/docs/ai-sdk-core/image-generation).
 
+OpenAI's image models may return a revised prompt for each image. It can be access at `providerMetadata.openai.images[0]?.revisedPrompt`.
+
 For more information on the available OpenAI image model options, see the [OpenAI API reference](https://platform.openai.com/docs/api-reference/images/create).
 
 ## Transcription Models

examples/ai-core/src/generate-image/openai.ts

@@ -4,12 +4,21 @@ import { presentImages } from '../lib/present-image';
 import 'dotenv/config';
 
 async function main() {
-  const { image } = await generateImage({
+  const prompt = 'Santa Claus driving a Cadillac';
+  const result = await generateImage({
     model: openai.image('dall-e-3'),
-    prompt: 'Santa Claus driving a Cadillac',
+    prompt,
   });
 
-  await presentImages([image]);
+  // @ts-expect-error
+  const revisedPrompt = result.providerMetadata.openai.images[0]?.revisedPrompt;
+
+  console.log({
+    prompt,
+    revisedPrompt,
+  });
+
+  await presentImages([result.image]);
 }
 
 main().catch(console.error);

packages/ai/core/generate-image/generate-image-result.ts

@@ -1,5 +1,8 @@
 import { GeneratedFile } from '../generate-text';
-import { ImageGenerationWarning } from '../types/image-model';
+import {
+  ImageGenerationWarning,
+  ImageModelProviderMetadata,
+} from '../types/image-model';
 import { ImageModelResponseMetadata } from '../types/image-model-response-metadata';
 
 /**
@@ -26,4 +29,10 @@ Warnings for the call, e.g. unsupported settings.
 Response metadata from the provider. There may be multiple responses if we made multiple calls to the model.
    */
   readonly responses: Array<ImageModelResponseMetadata>;
+
+  /**
+   * Provider-specific metadata. They are passed through from the provider to the AI SDK and enable provider-specific
+   * results that can be fully encapsulated in the provider.
+   */
+  readonly providerMetadata: ImageModelProviderMetadata;
 }

packages/ai/core/generate-image/generate-image.test.ts

@@ -1,4 +1,8 @@
-import { ImageModelV2, ImageModelV2CallWarning } from '@ai-sdk/provider';
+import {
+  ImageModelV2,
+  ImageModelV2CallWarning,
+  ImageModelV2ProviderMetadata,
+} from '@ai-sdk/provider';
 import { MockImageModelV2 } from '../test/mock-image-model-v2';
 import { generateImage } from './generate-image';
 import {
@@ -20,10 +24,16 @@ const createMockResponse = (options: {
   warnings?: ImageModelV2CallWarning[];
   timestamp?: Date;
   modelId?: string;
+  providerMetaData?: ImageModelV2ProviderMetadata;
   headers?: Record<string, string>;
 }) => ({
   images: options.images,
   warnings: options.warnings ?? [],
+  providerMetadata: options.providerMetaData ?? {
+    testProvider: {
+      images: options.images.map(() => null),
+    },
+  },
   response: {
     timestamp: options.timestamp ?? new Date(),
     modelId: options.modelId ?? 'test-model-id',
@@ -382,4 +392,30 @@ describe('generateImage', () => {
       },
     ]);
   });
+
+  it('should return provider metadata', async () => {
+    const result = await generateImage({
+      model: new MockImageModelV2({
+        doGenerate: async () =>
+          createMockResponse({
+            images: [pngBase64, pngBase64],
+            timestamp: testDate,
+            modelId: 'test-model',
+            providerMetaData: {
+              testProvider: {
+                images: [{ revisedPrompt: 'test-revised-prompt' }, null],
+              },
+            },
+            headers: {},
+          }),
+      }),
+      prompt,
+    });
+
+    expect(result.providerMetadata).toStrictEqual({
+      testProvider: {
+        images: [{ revisedPrompt: 'test-revised-prompt' }, null],
+      },
+    });
+  });
 });

packages/ai/core/generate-image/generate-image.ts

@@ -1,10 +1,11 @@
-import { ImageModelV2, JSONValue } from '@ai-sdk/provider';
+import { ImageModelV2, ImageModelV2ProviderMetadata } from '@ai-sdk/provider';
 import { NoImageGeneratedError } from '../../errors/no-image-generated-error';
 import {
   DefaultGeneratedFile,
   GeneratedFile,
 } from '../generate-text/generated-file';
 import { prepareRetries } from '../prompt/prepare-retries';
+import { ProviderMetadata } from '../types';
 import { ImageGenerationWarning } from '../types/image-model';
 import { ImageModelResponseMetadata } from '../types/image-model-response-metadata';
 import { GenerateImageResult } from './generate-image-result';
@@ -144,6 +145,7 @@ Only applicable for HTTP-based providers.
   const images: Array<DefaultGeneratedFile> = [];
   const warnings: Array<ImageGenerationWarning> = [];
   const responses: Array<ImageModelResponseMetadata> = [];
+  const providerMetadata: ImageModelV2ProviderMetadata = {};
   for (const result of results) {
     images.push(
       ...result.images.map(
@@ -159,29 +161,49 @@ Only applicable for HTTP-based providers.
       ),
     );
     warnings.push(...result.warnings);
+
+    if (result.providerMetadata) {
+      for (const [providerName, metadata] of Object.entries<{
+        images: unknown;
+      }>(result.providerMetadata)) {
+        providerMetadata[providerName] ??= { images: [] };
+        providerMetadata[providerName].images.push(
+          ...result.providerMetadata[providerName].images,
+        );
+      }
+    }
+
     responses.push(result.response);
   }
 
   if (!images.length) {
     throw new NoImageGeneratedError({ responses });
   }
 
-  return new DefaultGenerateImageResult({ images, warnings, responses });
+  return new DefaultGenerateImageResult({
+    images,
+    warnings,
+    responses,
+    providerMetadata,
+  });
 }
 
 class DefaultGenerateImageResult implements GenerateImageResult {
   readonly images: Array<GeneratedFile>;
   readonly warnings: Array<ImageGenerationWarning>;
   readonly responses: Array<ImageModelResponseMetadata>;
+  readonly providerMetadata: ImageModelV2ProviderMetadata;
 
   constructor(options: {
     images: Array<GeneratedFile>;
     warnings: Array<ImageGenerationWarning>;
     responses: Array<ImageModelResponseMetadata>;
+    providerMetadata: ImageModelV2ProviderMetadata;
   }) {
     this.images = options.images;
     this.warnings = options.warnings;
     this.responses = options.responses;
+    this.providerMetadata = options.providerMetadata;
   }
 
   get image() {

packages/ai/core/types/image-model.ts

@@ -1,4 +1,8 @@
-import { ImageModelV2, ImageModelV2CallWarning } from '@ai-sdk/provider';
+import {
+  ImageModelV2,
+  ImageModelV2CallWarning,
+  ImageModelV2ProviderMetadata,
+} from '@ai-sdk/provider';
 
 /**
 Image model that is used by the AI SDK Core functions.
@@ -10,3 +14,8 @@ Warning from the model provider for this call. The call will proceed, but e.g.
 some settings might not be supported, which can lead to suboptimal results.
   */
 export type ImageGenerationWarning = ImageModelV2CallWarning;
+
+/**
+Metadata from the model provider for this call
+  */
+export type ImageModelProviderMetadata = ImageModelV2ProviderMetadata;

packages/ai/core/types/index.ts

@@ -2,6 +2,7 @@ export type { Embedding, EmbeddingModel } from './embedding-model';
 export type {
   ImageModel,
   ImageGenerationWarning as ImageModelCallWarning,
+  ImageModelProviderMetadata,
 } from './image-model';
 export type { ImageModelResponseMetadata } from './image-model-response-metadata';
 export type { JSONValue } from './json-value';

packages/openai/src/openai-image-model.test.ts

@@ -253,4 +253,29 @@ describe('doGenerate', () => {
     const requestBody = await server.calls[server.calls.length - 1].requestBody;
     expect(requestBody).toHaveProperty('response_format', 'b64_json');
   });
+
+  it('should return image meta data', async () => {
+    prepareJsonResponse();
+
+    const result = await model.doGenerate({
+      prompt,
+      n: 1,
+      size: '1024x1024',
+      aspectRatio: undefined,
+      seed: undefined,
+      providerOptions: { openai: { style: 'vivid' } },
+    });
+
+    expect(result.providerMetadata).toStrictEqual({
+      openai: {
+        images: [
+          {
+            revisedPrompt:
+              'A charming visual illustration of a baby sea otter swimming joyously.',
+          },
+          null,
+        ],
+      },
+    });
+  });
 });

packages/openai/src/openai-image-model.ts

@@ -99,12 +99,25 @@ export class OpenAIImageModel implements ImageModelV2 {
         modelId: this.modelId,
         headers: responseHeaders,
       },
+      providerMetadata: {
+        openai: {
+          images: response.data.map(item =>
+            item.revised_prompt
+              ? {
+                  revisedPrompt: item.revised_prompt,
+                }
+              : null,
+          ),
+        },
+      },
     };
   }
 }
 
 // minimal version of the schema, focussed on what is needed for the implementation
 // this approach limits breakages when the API changes and increases efficiency
 const openaiImageResponseSchema = z.object({
-  data: z.array(z.object({ b64_json: z.string() })),
+  data: z.array(
+    z.object({ b64_json: z.string(), revised_prompt: z.string().optional() }),
+  ),
 });

packages/provider/src/image-model/v2/image-model-v2-call-options.ts

@@ -39,9 +39,9 @@ The outer record is keyed by the provider name, and the inner
 record is keyed by the provider-specific metadata key.
 ```ts
 {
-"openai": {
-"style": "vivid"
-}
+  "openai": {
+    "style": "vivid"
+  }
 }
 ```
  */