🤧 LD-DPO support

[AIR-hl]

What does this PR do?

This PR adds LD-DPO implementations to TRL. This paper has been accepted by CoRR 2024

• Liu W, Bai Y, Han C, et al. Length Desensitization in Directed Preference Optimization[J]. CoRR, 2024.

This paper proposed a novel method aims to desensitize DPO to data length by decoupling explicit length preference, which is relatively insignificant, from the other implicit preferences, thereby enabling more effective learning of the intrinsic preferences

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[kashif]

congrats! can you also kindly add the method in the dpo documentation with some description?

[AIR-hl]

congrats! can you also kindly add the method in the dpo documentation with some description?

done

[kashif]

also you can add a small test too?

[AIR-hl]

also you can add a small test too?

Sorry, i dont know how to test if it is effective. T_T

[kashif]

no problem, I can do it

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Thanks, LGTM!