Skip to content

[WIP] SD3.5 IP-Adapter Pipeline Integration #9987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Dec 20, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
0af910b
Initial pipeline for SD3.5-Large-IP-Adapter
guiyrt Nov 21, 2024
5567438
Added support for single IPAdapter on SD3.5 pipeline
guiyrt Dec 6, 2024
50d09d9
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 6, 2024
0ef36dd
Fixed typo and reverted removal of skip_layers in SD3Transformer2DModel
guiyrt Dec 7, 2024
de8909a
Added new SD3IPAdapterMixin loader
guiyrt Dec 9, 2024
ab0d904
ip_adapter image embeds now considers num_images_per_prompt
guiyrt Dec 9, 2024
d868ddb
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 9, 2024
5aed1d3
Removed usage of einops
guiyrt Dec 9, 2024
4383175
Reverted joint_attention_kwargs default for consistency
guiyrt Dec 9, 2024
461ab73
Corrected einops removal
guiyrt Dec 9, 2024
8323240
Quality and style checks
guiyrt Dec 9, 2024
89c4e63
Quality and style checks
guiyrt Dec 9, 2024
27d574f
Handle None joint_attention_kwargs in JointTransformerBlock
guiyrt Dec 9, 2024
0a48648
Fix test_components_function
hlky Dec 9, 2024
10d0a06
Remove from img2img/inpaint for now
hlky Dec 9, 2024
c78c4fd
Fixed loading ip_adapter state dict
guiyrt Dec 10, 2024
0f6c607
Simpler image encoding
guiyrt Dec 10, 2024
53fd40d
Style check
guiyrt Dec 10, 2024
8039599
Better checks for image prompt considering ip_adapter scale
guiyrt Dec 10, 2024
7333bfc
Minor change correcting checking for ip_adapter embeds
guiyrt Dec 10, 2024
a87895e
Removing old check of ip_adapter scale
guiyrt Dec 10, 2024
4ba374a
Refactor of image_proj (testing)
guiyrt Dec 10, 2024
819dd3e
Revert "Removing old check of ip_adapter scale"
guiyrt Dec 10, 2024
262a3bb
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 10, 2024
ea32e13
Corrected property check
guiyrt Dec 10, 2024
f60751f
Corrected forward() of IPAdapterTimeImageProjectionBlock
guiyrt Dec 11, 2024
b0aa5cb
IPAdapterTimeImageProjectionBlock now uses original attention impleme…
guiyrt Dec 12, 2024
b3dc69a
Clean-up and make style
guiyrt Dec 12, 2024
84aa4a3
Minor changes in code structure
guiyrt Dec 13, 2024
34793fb
make style && make quality
guiyrt Dec 13, 2024
27fe083
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 13, 2024
68169f8
Updated dosctrings and doc entries
guiyrt Dec 16, 2024
d824451
Merge branch 'main' into sd3.5_IPAdapter
hlky Dec 16, 2024
24e6880
make
hlky Dec 16, 2024
43d2e77
More docs and small refactors
guiyrt Dec 17, 2024
05f49e6
Merge remote-tracking branch 'origin' into sd3.5_IPAdapter
guiyrt Dec 17, 2024
44e3847
Fix in loading state dict
guiyrt Dec 18, 2024
178e513
Enabled cpu offload
guiyrt Dec 18, 2024
7899c6a
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 18, 2024
8daca65
Renaming from transformers_sd3 to transformer_sd3
guiyrt Dec 18, 2024
7c918db
Missing rename
guiyrt Dec 18, 2024
99a6d59
Updated docs for SD3 pipeline
guiyrt Dec 18, 2024
3916298
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 18, 2024
02a6d90
Update docs/source/en/api/pipelines/stable_diffusion/stable_diffusion…
guiyrt Dec 18, 2024
64ab7f9
Minor doc correction
guiyrt Dec 18, 2024
b254aa3
Updated img source to hf/documentation-images
guiyrt Dec 18, 2024
5c28161
image_proj is now called from SD3Transformer2DModel
guiyrt Dec 19, 2024
1313501
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 19, 2024
b882e1b
ip_adapter_image_embeds go through joint_attention_kwargs
guiyrt Dec 19, 2024
988447f
Warning for sequential cpu offloading with image_encoder
guiyrt Dec 19, 2024
66c4866
Merge branch 'main' into sd3.5_IPAdapter
guiyrt Dec 19, 2024
98f4521
make style quality
guiyrt Dec 19, 2024
5eef7f1
Merge branch 'main' into sd3.5_IPAdapter
yiyixuxu Dec 19, 2024
18cd8e4
Update src/diffusers/models/attention.py
yiyixuxu Dec 19, 2024
65b477f
Update src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_dif…
yiyixuxu Dec 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixed typo and reverted removal of skip_layers in SD3Transformer2DModel
  • Loading branch information
guiyrt committed Dec 7, 2024
commit 0ef36dd247eac3f5a4ae959d99b3ff234cba72e3
11 changes: 8 additions & 3 deletions src/diffusers/models/transformers/transformer_sd3.py
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,7 @@ def forward(
block_controlnet_hidden_states: List = None,
joint_attention_kwargs: Optional[Dict[str, Any]] = None,
return_dict: bool = True,
skip_layers: Optional[List[int]] = None,
) -> Union[torch.FloatTensor, Transformer2DModelOutput]:
"""
The [`SD3Transformer2DModel`] forward method.
Expand All @@ -363,6 +364,8 @@ def forward(
return_dict (`bool`, *optional*, defaults to `True`):
Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain
tuple.
skip_layers (`list` of `int`, *optional*):
A list of layer indices to skip during the forward pass.

Returns:
If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a
Expand Down Expand Up @@ -390,7 +393,10 @@ def forward(
encoder_hidden_states = self.context_embedder(encoder_hidden_states)

for index_block, block in enumerate(self.transformer_blocks):
if self.training and self.gradient_checkpointing:
# Skip specified layers
is_skip = True if skip_layers is not None and index_block in skip_layers else False

if torch.is_grad_enabled() and self.gradient_checkpointing and not is_skip:

def create_custom_forward(module, return_dict=None):
def custom_forward(*inputs):
Expand All @@ -410,8 +416,7 @@ def custom_forward(*inputs):
joint_attention_kwargs,
**ckpt_kwargs,
)

else:
elif not is_skip:
encoder_hidden_states, hidden_states = block(
hidden_states=hidden_states, encoder_hidden_states=encoder_hidden_states, temb=temb,
joint_attention_kwargs=joint_attention_kwargs,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -855,7 +855,7 @@ def set_ip_adapter_scale(self, scale):
only conditioned by the text prompt. Lowering this value encourages the model to produce more diverse images, but they
may not be as aligned with the image prompt.
"""
for attn_processor in self.transformes.attn_processors.values():
for attn_processor in self.transformer.attn_processors.values():
if isinstance(attn_processor, IPAdapterJointAttnProcessor2_0):
attn_processor.scale = scale

Expand Down