FSDP2 with Ghost Clipping and Fast Gradient Clipping prototyping #761
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
facebook-github-bot
pushed a commit
that referenced
this pull request
May 14, 2025
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
6a340cd to
d875c8c
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
d875c8c to
0aa8971
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 4, 2025
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 4, 2025
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
0aa8971 to
c8fe1fe
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 4, 2025
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
c8fe1fe to
ae71b3a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 6, 2025
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
ae71b3a to
f6413b3
Compare
facebook-github-bot
pushed a commit
that referenced
this pull request
Jun 6, 2025
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
f6413b3 to
0ea701a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
aparna-aketi
added a commit
that referenced
this pull request
Jun 10, 2025
Summary: Pull Request resolved: #761 Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
aparna-aketi
force-pushed
the
export-D70533184
branch
from
0ea701a to
9e5dcd2
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
aparna-aketi
added a commit
that referenced
this pull request
Jun 10, 2025
Summary: Pull Request resolved: #761 Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
aparna-aketi
force-pushed
the
export-D70533184
branch
from
9e5dcd2 to
aff0cbf
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
aparna-aketi
added a commit
that referenced
this pull request
Jun 10, 2025
Summary: Pull Request resolved: #761 Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
aparna-aketi
force-pushed
the
export-D70533184
branch
from
aff0cbf to
61afca7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
aparna-aketi
added a commit
that referenced
this pull request
Jun 10, 2025
Summary: Pull Request resolved: #761 Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
aparna-aketi
force-pushed
the
export-D70533184
branch
from
61afca7 to
090e923
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
aparna-aketi
added a commit
that referenced
this pull request
Jun 10, 2025
Summary: Pull Request resolved: #761 Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 3. No freezing/unfreezing of parameters in between the training. Design Doc: [Opacus Ghost Clipping and FSDP2](https://docs.google.com/document/d/1MHqIMKBAXhkUZYQ9kkHCmUs3iLq_G5Q25uS1u3g7Asw/edit?tab=t.0#heading=h.eqambyjwzqsu) Differential Revision: D70533184
aparna-aketi
force-pushed
the
export-D70533184
branch
from
090e923 to
1aaa372
Compare
Summary: Integrating FSDP2 with Opacus First Prototype: 1. FSDP is supported only if all the layers with trainable parameters are supported by ghost clipping or fast gradient clipping. 2. No freezing/unfreezing of parameters in between the training. Differential Revision: D70533184
facebook-github-bot
force-pushed
the
export-D70533184
branch
from
1aaa372 to
89baef1
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70533184 |
|
This pull request has been merged in a478ad5. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Integrating FSDP2 with Opacus
First Prototype:
Design Doc: Opacus Ghost Clipping and FSDP2
Differential Revision: D70533184