[GPG][new trainer] Add support to new GPG method

[lerogo]

Method description

• short description: In this work, we revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimizes the original RL objective, thus obviating the need for surrogate loss functions. By eliminating the critic and reference models, avoiding KL divergence constraints, and addressing the advantage and gradient estimation bias, our approach significantly simplifies the training process compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. Extensive experiments demonstrate that our method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks.
• paper link: https://arxiv.org/abs/2504.02546

Open source status

• The method implementation is available
• The model weights are available
• The training datasets are available

Provide useful links for the implementation

• source-code link: https://github.com/AMAP-ML/GPG
• model weights link:
  • https://huggingface.co/GD-ML/Qwen2.5-Math-7B-GPG
• authors:
  • GitHub: @cxxgtxy @lerogo @undyingjoker
  • Scholar: Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, Yong Wang (AMAP, Alibaba Group)