Add parse blocks to allow setting values per block #2032
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This could allow setting values per block. Handles a lot of variations.
I tried to implement this but it required a lot of changes. So providing the function to parse the block values.
For example would handle all these options for values:
Then create a list of values for each block. Would be similar to behavior in networks.lora but be more flexible in what sort of values it handles. Could work for block alpha, dim/rank, dropout (neuron, rank, module), block LR, LoRA+ LR.
Some issues with integration is changing a single value to a list of values, but also handling non-block values like embedders, which would require more indices or a different way to handle it.
Some of the math for the functions may not be accurate and we can add further tests for those cases.
Plotting the blocks: