Always pass add_generation_prompt=True to apply_chat_template

[albertvillanova]

Always pass add_generation_prompt=True to apply_chat_template.

This PR passes add_generation_prompt=True to apply_chat_template, regardless of whether tools are provided or not, specifically to:

• VLLMModel
• TransformersModel

Note that before this PR, we always passed add_generation_prompt=True regardless of tools to:

• MLXModel

The goal is to serve as a basis to continue the discussion on how we want to handle prompt formatting. See previous discussion: #993 (comment)

Based on my own (non-expert) understanding of the transformers documentation, setting add_generation_prompt=True appends a generation prompt, which prepares the input so the model knows it should generate a response from the assistant. This seems to apply broadly in conversational contexts, which is the case when working with agents, regardless of whether tool calls are used. I would say:

• You should use add_generation_prompt=True when:
  • You're preparing an input that should lead to the assistant's next message
  • You want the model to generate a reply, whether that's plain text or a tool call
• The link to tools:
  • add_generation_prompt=True does not require tools=...
  • But if you are using tools, then add_generation_prompt=True is required

In a discussion with @molbap, he mentioned that in conversational contexts, regardless of whether tools are passed to model.generate, it is generally recommended to include the generation prompt. This is because the model's output is conditioned on special tokens (like <|assistant|>, <|bos|>, or <|eos|> defined in its config). Omitting them can often lead to suboptimal generation results.

Fix #993.

[albertvillanova]

Please note that the required fixes in the tests suggest that the fix may indeed be required:

• Input: "Hello!"

  • Output:
    • Before: "assistant\nHello"
    • Now: "Hello! I'm here"
      • The "assistance" tag is not added, which makes the conversational flow more natural

• Input: "Hello!"

  • Output:
    • Before: "I am"
      • It was continuing the "user" message, instead of creating the assistant response
    • Now: "."

• Input: "What is this?"

  • Output:
    • Before: "I am a"
      • It was continuing the "user" message, instead of creating the assistant response
    • Now: "This is a very"
      • This is the assistant response, as expected

In conclusion, passing add_generation_prompt=True ensures the chat model generates an assistant response instead of continuing a user's message.

[aymeric-roucher]

Thanks a lot @albertvillanova ! 🤗