Support `reponse_format: {"type": "json_object"}` without any constrained schema

[lhoestq]

In other inference APIs, response_format={"type": "json_object"} restricts the model output to be a valid JSON object without enforcing a schema.

Right now this is not supported:

Failed to deserialize the JSON body into the target type: response_format: missing field `value` at line 1 column 168

I ended up with this error while using lotus-ai which uses the litellm library with response_format={ "type": "json_object" }

To reproduce:

from huggingface_hub import InferenceClient

c = InferenceClient("meta-llama/Llama-3.3-70B-Instruct")
c.chat_completion([{"role": "user", "content": "Give me a dummy json of a person"}], response_format={"type": "json_object"})

Original issue: huggingface/huggingface_hub#2744

[drbh]

Hi @lhoestq I just took a look at how this may be implemented in TGI and it seems that outlines does not support arbitrary JSON so we would need to add different grammar backend that supports this feature.

I believe XGrammar supports this functionality, and an issues has been opened recently here: #2900. Its unclear to me at the moment what would need to change to support XGrammar but I will dive a bit deeper and share updates here

[sadra-barikbin]

@drbh , couldn't we use outlines CFGGuide?

from outlines.fsm.guide import CFGGuide
from outlines.grammars import json
from transformers import AutoTokenizer

tok=AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
guide = CFGGuide(json,tok)