Unstructured data grpo training

[yuyuhua918]

I am currently using my own dataset to perform GRPO in a vertical field. It is an unstructured training task, and the goal is to make it an assistant for scientific research projects. But now it keeps showing gradient overflow. What is the reason? Is it related to my hardware? Four v10032G, can only support sdpa, fp16

[shirinyamani]

Is your dataset following the format which grpo trainer accepts (standard or conversational)?
dataset format

[yuyuhua918]

Is your dataset following the format which grpo trainer accepts (standard or conversational)? dataset format

It is in conversational format and contains system, user prompt and completion. Is this suitable for grpo？

[yuyuhua918]

Is your dataset following the format which grpo trainer accepts (standard or conversational)? dataset format

Also I use another llm api to set the accuracy function, Is this appropriate?

[shirinyamani]

Is your dataset following the format which grpo trainer accepts (standard or conversational)? dataset format

It is in conversational format and contains system, user prompt and completion. Is this suitable for grpo？

I believe Yes! As long as it follows the Conversational format, should be fine for training with grpo!

[shirinyamani]

Is your dataset following the format which grpo trainer accepts (standard or conversational)? dataset format

Also I use another llm api to set the accuracy function, Is this appropriate?

I can interpret your sentence in two ways;

1. Calling a different LLM API to compute the accuracy func for each of the outputs?
2. Using another LLM to create the accuracy_func more specialized to your target than the standard?

Could you please help me figuring which one you mean? if none, can you elaborate on it please?

[yuyuhua918]

Is your dataset following the format which grpo trainer accepts (standard or conversational)? dataset format

Also I use another llm api to set the accuracy function, Is this appropriate?

I can interpret your sentence in two ways;

1. Calling a different LLM API to compute the accuracy func for each of the outputs?
2. Using another LLM to create the accuracy_func more specialized to your target than the standard?

Could you please help me figuring which one you mean? if none, can you elaborate on it please?

Thank you for your reply. I use the same LLM API to evaluate the accuracy score of all the answers of the model. I think the regular function may not be suitable for evaluating the quality of the model's answers in my task. Is this reasonable? Or do I need to find a suitable rule to define the accuracy function?

[qgallouedec]

regular function may not be suitable for evaluating the quality of the model's answers in my task. Is this reasonable?

without knowing the nature of your task, it's completely impossible to say. What's more, you seem to be asking questions that are more scientific than technical, and considering that the literature on this subject is still very young, I doubt anyone will be able to give you a definitive answer. The best you can do is try and share your results.