feat: Support response_format and structured JSON responses.

[actow]

• I have searched the existing issues

Is your feature request related to a problem? Please describe it

There is not a away to force the model to return structured json at the API level.

Describe the solution

The response_format parameter is supported by certain models, such as Groq's llama3 (8b & 70b), Fireworks AI's llama3 70b and OpenAI gpt 3.5 and 4 etc.

https://console.groq.com/docs/api-reference#chat-create
https://platform.openai.com/docs/api-reference/audio/createSpeech#audio-createspeech-response_format
https://readme.fireworks.ai/docs/structured-response-formatting

If will be great if Jan provide a UI to set that.

Teachability, documentation, adoption, migration strategy

https://console.groq.com/docs/api-reference#chat-create
https://platform.openai.com/docs/api-reference/audio/createSpeech#audio-createspeech-response_format
https://readme.fireworks.ai/docs/structured-response-formatting

What is the motivation / use case for changing the behavior?

The JSON structure response is very useful to experiment the models' capability to extend beyond a normal chat bot.

[dan-menlo]

@nguyenhoangthuan99 I'm marking this for Sprint 20, but it's possible that this is not a simple upstream of llama.cpp capability.

• Can you take a look and assess
• If it's too big, we will postpone to Sprint 21

[dan-menlo]

Also: linked to menloresearch/cortex.cpp#295?