Benchmark for Devstral? #4058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

R-Dson opened this issue May 22, 2025 · 1 comment

R-Dson commented May 22, 2025

Issue

Will there be a benchmark done of Mistral's new model Devstral? This seems like the exact type of model to test and see on the Aider leaderboard.

Version and model info

No response

van-sprundel commented May 22, 2025

I tried running benchmarks myself using the bench docs but it takes a lot longer than I anticipated (Running on an M3).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment