Skip to content

Add max batch size check #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 11, 2023
Merged

Add max batch size check #47

merged 2 commits into from
Oct 11, 2023

Conversation

azahed98
Copy link
Contributor

This PR raises a ValueError should the user specify a max size above what we support for the model

Things to change in the future:

  1. The default batch size is 32, which is above max batch size for code llamas. They need a different default batch size.
  2. We need to instead store min, max, and default batch size for each model in mongo, which can then be upaded separately and used in the CLI and finetune processor

@azahed98 azahed98 requested review from clam004 and justusc October 11, 2023 20:16
@azahed98 azahed98 merged commit 558ac89 into main Oct 11, 2023
Comment on lines +84 to +86
raise ValueError(
f"The max batch size for model {model} is {max_batch_size}. Received batch size {batch_size}."
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clam004 put in some changes that allow the user to confirm changes to the input. I think for max batch size we can clamp it and set adjusted_inputs = True as we did with other parameters.

@orangetin orangetin deleted the hard_max_bs branch April 5, 2024 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants