Add resume for adapter_v2, enable continued finetuning for adapter #1354
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
altria-zewei-wang
requested review from
awaelchli,
carmocca and
lantiga
as code owners
rasbt
approved these changes
Apr 25, 2024
|
Thanks for looking into this. Sorry, I haven't spent much time on thinking through the ramifications here. But would the simple resuming from the https://github.com/Lightning-AI/litgpt/blob/main/litgpt/finetune/full.py#L43 |
|
I was specifically looking into testing finetuning with adapter and loras in my paper, and that my gpu cuts off after certain time limit. I figure adding this feature can help anyone who is in similar situation as me. |
Borda
approved these changes
Apr 3, 2025
t-vi
approved these changes
Apr 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi all!
I was checking #238 to add a function to finish resume finetuning for adapter. It would search the largest number of step point it saved from out_dir and update the state_dict.
Current Problem: I updated the step_count but find out to keep the iteration count from last time would have to read in the metrics in the log folder. The problem is that I don't know how to retrieve the corresponding version in the log file without adding an input of the version of the metrics.csv (currently not implemented).
Let me know what you think! Thanks for your repo!