Skip to content

Commit 50d09d9

Browse files
guiyrtguiyrt
committed
Merge branch 'main' into sd3.5_IPAdapter
2 parents 5567438 + 18f9b99 commit 50d09d9

File tree

193 files changed

+12070
-2095
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

193 files changed

+12070
-2095
lines changed

‎.github/workflows/nightly_tests.yml

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,64 @@ jobs:
347347
pip install slack_sdk tabulate
348348
python utils/log_reports.py >> $GITHUB_STEP_SUMMARY
349349
350+
run_nightly_quantization_tests:
351+
name: Torch quantization nightly tests
352+
strategy:
353+
fail-fast: false
354+
max-parallel: 2
355+
matrix:
356+
config:
357+
- backend: "bitsandbytes"
358+
test_location: "bnb"
359+
runs-on:
360+
group: aws-g6e-xlarge-plus
361+
container:
362+
image: diffusers/diffusers-pytorch-cuda
363+
options: --shm-size "20gb" --ipc host --gpus 0
364+
steps:
365+
- name: Checkout diffusers
366+
uses: actions/checkout@v3
367+
with:
368+
fetch-depth: 2
369+
- name: NVIDIA-SMI
370+
run: nvidia-smi
371+
- name: Install dependencies
372+
run: |
373+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
374+
python -m uv pip install -e [quality,test]
375+
python -m uv pip install -U ${{ matrix.config.backend }}
376+
python -m uv pip install pytest-reportlog
377+
- name: Environment
378+
run: |
379+
python utils/print_env.py
380+
- name: ${{ matrix.config.backend }} quantization tests on GPU
381+
env:
382+
HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }}
383+
# https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms
384+
CUBLAS_WORKSPACE_CONFIG: :16:8
385+
BIG_GPU_MEMORY: 40
386+
run: |
387+
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
388+
--make-reports=tests_${{ matrix.config.backend }}_torch_cuda \
389+
--report-log=tests_${{ matrix.config.backend }}_torch_cuda.log \
390+
tests/quantization/${{ matrix.config.test_location }}
391+
- name: Failure short reports
392+
if: ${{ failure() }}
393+
run: |
394+
cat reports/tests_${{ matrix.config.backend }}_torch_cuda_stats.txt
395+
cat reports/tests_${{ matrix.config.backend }}_torch_cuda_failures_short.txt
396+
- name: Test suite reports artifacts
397+
if: ${{ always() }}
398+
uses: actions/upload-artifact@v4
399+
with:
400+
name: torch_cuda_${{ matrix.config.backend }}_reports
401+
path: reports
402+
- name: Generate Report and Notify Channel
403+
if: always()
404+
run: |
405+
pip install slack_sdk tabulate
406+
python utils/log_reports.py >> $GITHUB_STEP_SUMMARY
407+
350408
# M1 runner currently not well supported
351409
# TODO: (Dhruv) add these back when we setup better testing for Apple Silicon
352410
# run_nightly_tests_apple_m1:

‎README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -112,9 +112,9 @@ Check out the [Quickstart](https://huggingface.co/docs/diffusers/quicktour) to l
112112
| **Documentation** | **What can I learn?** |
113113
|---------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
114114
| [Tutorial](https://huggingface.co/docs/diffusers/tutorials/tutorial_overview) | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model. |
115-
| [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading_overview) | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. |
116-
| [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/pipeline_overview) | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. |
117-
| [Optimization](https://huggingface.co/docs/diffusers/optimization/opt_overview) | Guides for how to optimize your diffusion model to run faster and consume less memory. |
115+
| [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading) | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. |
116+
| [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/overview_techniques) | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. |
117+
| [Optimization](https://huggingface.co/docs/diffusers/optimization/fp16) | Guides for how to optimize your diffusion model to run faster and consume less memory. |
118118
| [Training](https://huggingface.co/docs/diffusers/training/overview) | Guides for how to train a diffusion model for different tasks with different training techniques. |
119119
## Contribution
120120

‎docs/source/en/api/pipelines/cogvideox.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,15 +30,17 @@ Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers.m
3030
This pipeline was contributed by [zRzRzRzRzRzRzR](https://github.com/zRzRzRzRzRzRzR). The original codebase can be found [here](https://huggingface.co/THUDM). The original weights can be found under [hf.co/THUDM](https://huggingface.co/THUDM).
3131

3232
There are three official CogVideoX checkpoints for text-to-video and video-to-video.
33+
3334
| checkpoints | recommended inference dtype |
34-
|---|---|
35+
|:---:|:---:|
3536
| [`THUDM/CogVideoX-2b`](https://huggingface.co/THUDM/CogVideoX-2b) | torch.float16 |
3637
| [`THUDM/CogVideoX-5b`](https://huggingface.co/THUDM/CogVideoX-5b) | torch.bfloat16 |
3738
| [`THUDM/CogVideoX1.5-5b`](https://huggingface.co/THUDM/CogVideoX1.5-5b) | torch.bfloat16 |
3839

3940
There are two official CogVideoX checkpoints available for image-to-video.
41+
4042
| checkpoints | recommended inference dtype |
41-
|---|---|
43+
|:---:|:---:|
4244
| [`THUDM/CogVideoX-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-5b-I2V) | torch.bfloat16 |
4345
| [`THUDM/CogVideoX-1.5-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-1.5-5b-I2V) | torch.bfloat16 |
4446

@@ -48,8 +50,9 @@ For the CogVideoX 1.5 series:
4850
- Both T2V and I2V models support generation with 81 and 161 frames and work best at this value. Exporting videos at 16 FPS is recommended.
4951

5052
There are two official CogVideoX checkpoints that support pose controllable generation (by the [Alibaba-PAI](https://huggingface.co/alibaba-pai) team).
53+
5154
| checkpoints | recommended inference dtype |
52-
|---|---|
55+
|:---:|:---:|
5356
| [`alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose`](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose) | torch.bfloat16 |
5457
| [`alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose`](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose) | torch.bfloat16 |
5558

‎docs/source/en/api/pipelines/flux.md

Lines changed: 161 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,20 @@ Flux can be quite expensive to run on consumer hardware devices. However, you ca
2222

2323
</Tip>
2424

25-
Flux comes in two variants:
25+
Flux comes in the following variants:
2626

27-
* Timestep-distilled (`black-forest-labs/FLUX.1-schnell`)
28-
* Guidance-distilled (`black-forest-labs/FLUX.1-dev`)
27+
| model type | model id |
28+
|:----------:|:--------:|
29+
| Timestep-distilled | [`black-forest-labs/FLUX.1-schnell`](https://huggingface.co/black-forest-labs/FLUX.1-schnell) |
30+
| Guidance-distilled | [`black-forest-labs/FLUX.1-dev`](https://huggingface.co/black-forest-labs/FLUX.1-dev) |
31+
| Fill Inpainting/Outpainting (Guidance-distilled) | [`black-forest-labs/FLUX.1-Fill-dev`](https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev) |
32+
| Canny Control (Guidance-distilled) | [`black-forest-labs/FLUX.1-Canny-dev`](https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev) |
33+
| Depth Control (Guidance-distilled) | [`black-forest-labs/FLUX.1-Depth-dev`](https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev) |
34+
| Canny Control (LoRA) | [`black-forest-labs/FLUX.1-Canny-dev-lora`](https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev-lora) |
35+
| Depth Control (LoRA) | [`black-forest-labs/FLUX.1-Depth-dev-lora`](https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev-lora) |
36+
| Redux (Adapter) | [`black-forest-labs/FLUX.1-Redux-dev`](https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev) |
2937

30-
Both checkpoints have slightly difference usage which we detail below.
38+
All checkpoints have different usage which we detail below.
3139

3240
### Timestep-distilled
3341

@@ -77,7 +85,132 @@ out = pipe(
7785
out.save("image.png")
7886
```
7987

88+
### Fill Inpainting/Outpainting
89+
90+
* Flux Fill pipeline does not require `strength` as an input like regular inpainting pipelines.
91+
* It supports both inpainting and outpainting.
92+
93+
```python
94+
import torch
95+
from diffusers import FluxFillPipeline
96+
from diffusers.utils import load_image
97+
98+
image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup.png")
99+
mask = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup_mask.png")
100+
101+
repo_id = "black-forest-labs/FLUX.1-Fill-dev"
102+
pipe = FluxFillPipeline.from_pretrained(repo_id, torch_dtype=torch.bfloat16).to("cuda")
103+
104+
image = pipe(
105+
prompt="a white paper cup",
106+
image=image,
107+
mask_image=mask,
108+
height=1632,
109+
width=1232,
110+
max_sequence_length=512,
111+
generator=torch.Generator("cpu").manual_seed(0)
112+
).images[0]
113+
image.save(f"output.png")
114+
```
115+
116+
### Canny Control
117+
118+
**Note:** `black-forest-labs/Flux.1-Canny-dev` is _not_ a [`ControlNetModel`] model. ControlNet models are a separate component from the UNet/Transformer whose residuals are added to the actual underlying model. Canny Control is an alternate architecture that achieves effectively the same results as a ControlNet model would, by using channel-wise concatenation with input control condition and ensuring the transformer learns structure control by following the condition as closely as possible.
119+
120+
```python
121+
# !pip install -U controlnet-aux
122+
import torch
123+
from controlnet_aux import CannyDetector
124+
from diffusers import FluxControlPipeline
125+
from diffusers.utils import load_image
126+
127+
pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", torch_dtype=torch.bfloat16).to("cuda")
128+
129+
prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
130+
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")
131+
132+
processor = CannyDetector()
133+
control_image = processor(control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024)
134+
135+
image = pipe(
136+
prompt=prompt,
137+
control_image=control_image,
138+
height=1024,
139+
width=1024,
140+
num_inference_steps=50,
141+
guidance_scale=30.0,
142+
).images[0]
143+
image.save("output.png")
144+
```
145+
146+
### Depth Control
147+
148+
**Note:** `black-forest-labs/Flux.1-Depth-dev` is _not_ a ControlNet model. [`ControlNetModel`] models are a separate component from the UNet/Transformer whose residuals are added to the actual underlying model. Depth Control is an alternate architecture that achieves effectively the same results as a ControlNet model would, by using channel-wise concatenation with input control condition and ensuring the transformer learns structure control by following the condition as closely as possible.
149+
150+
```python
151+
# !pip install git+https://github.com/huggingface/image_gen_aux
152+
import torch
153+
from diffusers import FluxControlPipeline, FluxTransformer2DModel
154+
from diffusers.utils import load_image
155+
from image_gen_aux import DepthPreprocessor
156+
157+
pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-Depth-dev", torch_dtype=torch.bfloat16).to("cuda")
158+
159+
prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
160+
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")
161+
162+
processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
163+
control_image = processor(control_image)[0].convert("RGB")
164+
165+
image = pipe(
166+
prompt=prompt,
167+
control_image=control_image,
168+
height=1024,
169+
width=1024,
170+
num_inference_steps=30,
171+
guidance_scale=10.0,
172+
generator=torch.Generator().manual_seed(42),
173+
).images[0]
174+
image.save("output.png")
175+
```
176+
177+
### Redux
178+
179+
* Flux Redux pipeline is an adapter for FLUX.1 base models. It can be used with both flux-dev and flux-schnell, for image-to-image generation.
180+
* You can first use the `FluxPriorReduxPipeline` to get the `prompt_embeds` and `pooled_prompt_embeds`, and then feed them into the `FluxPipeline` for image-to-image generation.
181+
* When use `FluxPriorReduxPipeline` with a base pipeline, you can set `text_encoder=None` and `text_encoder_2=None` in the base pipeline, in order to save VRAM.
182+
183+
```python
184+
import torch
185+
from diffusers import FluxPriorReduxPipeline, FluxPipeline
186+
from diffusers.utils import load_image
187+
device = "cuda"
188+
dtype = torch.bfloat16
189+
190+
191+
repo_redux = "black-forest-labs/FLUX.1-Redux-dev"
192+
repo_base = "black-forest-labs/FLUX.1-dev"
193+
pipe_prior_redux = FluxPriorReduxPipeline.from_pretrained(repo_redux, torch_dtype=dtype).to(device)
194+
pipe = FluxPipeline.from_pretrained(
195+
repo_base,
196+
text_encoder=None,
197+
text_encoder_2=None,
198+
torch_dtype=torch.bfloat16
199+
).to(device)
200+
201+
image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy/img5.png")
202+
pipe_prior_output = pipe_prior_redux(image)
203+
images = pipe(
204+
guidance_scale=2.5,
205+
num_inference_steps=50,
206+
generator=torch.Generator("cpu").manual_seed(0),
207+
**pipe_prior_output,
208+
).images
209+
images[0].save("flux-redux.png")
210+
```
211+
80212
## Running FP16 inference
213+
81214
Flux can generate high-quality images with FP16 (i.e. to accelerate inference on Turing/Volta GPUs) but produces different outputs compared to FP32/BF16. The issue is that some activations in the text encoders have to be clipped when running in FP16, which affects the overall image. Forcing text encoders to run with FP32 inference thus removes this output difference. See [here](https://github.com/huggingface/diffusers/pull/9097#issuecomment-2272292516) for details.
82215

83216
FP16 inference code:
@@ -188,3 +321,27 @@ image.save("flux-fp8-dev.png")
188321
[[autodoc]] FluxControlNetImg2ImgPipeline
189322
- all
190323
- __call__
324+
325+
## FluxControlPipeline
326+
327+
[[autodoc]] FluxControlPipeline
328+
- all
329+
- __call__
330+
331+
## FluxControlImg2ImgPipeline
332+
333+
[[autodoc]] FluxControlImg2ImgPipeline
334+
- all
335+
- __call__
336+
337+
## FluxPriorReduxPipeline
338+
339+
[[autodoc]] FluxPriorReduxPipeline
340+
- all
341+
- __call__
342+
343+
## FluxFillPipeline
344+
345+
[[autodoc]] FluxFillPipeline
346+
- all
347+
- __call__

‎docs/source/en/api/pipelines/pag.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,10 @@ Since RegEx is supported as a way for matching layer identifiers, it is crucial
9696
- all
9797
- __call__
9898

99+
## StableDiffusion3PAGImg2ImgPipeline
100+
[[autodoc]] StableDiffusion3PAGImg2ImgPipeline
101+
- all
102+
- __call__
99103

100104
## PixArtSigmaPAGPipeline
101105
[[autodoc]] PixArtSigmaPAGPipeline

‎docs/source/en/conceptual/evaluation.md

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,7 @@ Then we load the [v1-5 checkpoint](https://huggingface.co/stable-diffusion-v1-5/
181181

182182
```python
183183
model_ckpt_1_5 = "stable-diffusion-v1-5/stable-diffusion-v1-5"
184-
sd_pipeline_1_5 = StableDiffusionPipeline.from_pretrained(model_ckpt_1_5, torch_dtype=weight_dtype).to(device)
184+
sd_pipeline_1_5 = StableDiffusionPipeline.from_pretrained(model_ckpt_1_5, torch_dtype=torch.float16).to("cuda")
185185

186186
images_1_5 = sd_pipeline_1_5(prompts, num_images_per_prompt=1, generator=generator, output_type="np").images
187187
```
@@ -280,7 +280,7 @@ from diffusers import StableDiffusionInstructPix2PixPipeline
280280

281281
instruct_pix2pix_pipeline = StableDiffusionInstructPix2PixPipeline.from_pretrained(
282282
"timbrooks/instruct-pix2pix", torch_dtype=torch.float16
283-
).to(device)
283+
).to("cuda")
284284
```
285285

286286
Now, we perform the edits:
@@ -326,9 +326,9 @@ from transformers import (
326326

327327
clip_id = "openai/clip-vit-large-patch14"
328328
tokenizer = CLIPTokenizer.from_pretrained(clip_id)
329-
text_encoder = CLIPTextModelWithProjection.from_pretrained(clip_id).to(device)
329+
text_encoder = CLIPTextModelWithProjection.from_pretrained(clip_id).to("cuda")
330330
image_processor = CLIPImageProcessor.from_pretrained(clip_id)
331-
image_encoder = CLIPVisionModelWithProjection.from_pretrained(clip_id).to(device)
331+
image_encoder = CLIPVisionModelWithProjection.from_pretrained(clip_id).to("cuda")
332332
```
333333

334334
Notice that we are using a particular CLIP checkpoint, i.e., `openai/clip-vit-large-patch14`. This is because the Stable Diffusion pre-training was performed with this CLIP variant. For more details, refer to the [documentation](https://huggingface.co/docs/transformers/model_doc/clip).
@@ -350,7 +350,7 @@ class DirectionalSimilarity(nn.Module):
350350

351351
def preprocess_image(self, image):
352352
image = self.image_processor(image, return_tensors="pt")["pixel_values"]
353-
return {"pixel_values": image.to(device)}
353+
return {"pixel_values": image.to("cuda")}
354354

355355
def tokenize_text(self, text):
356356
inputs = self.tokenizer(
@@ -360,7 +360,7 @@ class DirectionalSimilarity(nn.Module):
360360
truncation=True,
361361
return_tensors="pt",
362362
)
363-
return {"input_ids": inputs.input_ids.to(device)}
363+
return {"input_ids": inputs.input_ids.to("cuda")}
364364

365365
def encode_image(self, image):
366366
preprocessed_image = self.preprocess_image(image)
@@ -459,6 +459,7 @@ with ZipFile(local_filepath, "r") as zipper:
459459
```python
460460
from PIL import Image
461461
import os
462+
import numpy as np
462463

463464
dataset_path = "sample-imagenet-images"
464465
image_paths = sorted([os.path.join(dataset_path, x) for x in os.listdir(dataset_path)])
@@ -477,6 +478,7 @@ Now that the images are loaded, let's apply some lightweight pre-processing on t
477478

478479
```python
479480
from torchvision.transforms import functional as F
481+
import torch
480482

481483

482484
def preprocess_image(image):
@@ -498,6 +500,10 @@ dit_pipeline = DiTPipeline.from_pretrained("facebook/DiT-XL-2-256", torch_dtype=
498500
dit_pipeline.scheduler = DPMSolverMultistepScheduler.from_config(dit_pipeline.scheduler.config)
499501
dit_pipeline = dit_pipeline.to("cuda")
500502

503+
seed = 0
504+
generator = torch.manual_seed(seed)
505+
506+
501507
words = [
502508
"cassette player",
503509
"chainsaw",

0 commit comments

Comments
 (0)