Skip to content

Add SkyReels V2: Infinite-Length Film Generative Model #11518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 161 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
161 commits
Select commit Hold shift + click to select a range
4dd739f
Add SkyReels-V2 pipelines for text-to-video, image-to-video, and diff…
tolgacangoz May 7, 2025
899f41c
Merge branch 'main' into skyreels-v2
tolgacangoz May 7, 2025
607b5ba
up
tolgacangoz May 7, 2025
3ccf201
second draft
tolgacangoz May 8, 2025
959ca1f
Merge branch 'main' into skyreels-v2
tolgacangoz May 8, 2025
37ca14f
up
tolgacangoz May 8, 2025
d80b505
Merge branch 'main' into skyreels-v2
tolgacangoz May 8, 2025
95d0621
3rd draft
tolgacangoz May 8, 2025
6f8a945
4th draft
tolgacangoz May 9, 2025
e781084
upup
tolgacangoz May 9, 2025
4806660
style
tolgacangoz May 9, 2025
0986e81
up
tolgacangoz May 9, 2025
6a300f5
up
tolgacangoz May 9, 2025
45e1680
fix fn name
tolgacangoz May 9, 2025
c8a0c14
update import structure for SkyReelsV2
tolgacangoz May 9, 2025
47306b6
add SkyreelsV2 pipeline classes with backend requirements
tolgacangoz May 9, 2025
c5b8da9
up
tolgacangoz May 10, 2025
5835eaa
up
tolgacangoz May 10, 2025
9d2880e
add draft transformer_skyreels_v2.py with a custom WanModel and atten…
tolgacangoz May 10, 2025
2c0586e
up
tolgacangoz May 10, 2025
52590ea
split i2v and t2v pipes for diffusion forcing
tolgacangoz May 10, 2025
c50fcad
Refactors SkyReelsV2 attention and normalizations
tolgacangoz May 10, 2025
d0c71fd
Add SkyReelsV2 image and time-text embeddings
tolgacangoz May 11, 2025
f318efa
up
tolgacangoz May 11, 2025
9688a82
Refactors the `SkyReelsV2Transformer3DModel` by removing unused metho…
tolgacangoz May 11, 2025
825c2c1
Refactors `SkyReelsV2TransformerBlock` to integrate its `forward()` m…
tolgacangoz May 12, 2025
d848500
Refactors `SkyReelsV2AttnProcessor2_0` to enhance the `forward()` met…
tolgacangoz May 12, 2025
2f5a4e2
Refactors `SkyReelsV2Transformer3DModel` to enhance the `forward()` m…
tolgacangoz May 12, 2025
e5870dd
Refactors `SkyReelsV2Transformer3DModel` to improve the `forward()` m…
tolgacangoz May 12, 2025
d54e3e1
Refactors `SkyReelsV2Transformer3DModel` forward pass
tolgacangoz May 13, 2025
10d7480
Add DF inference template.
tolgacangoz May 13, 2025
fc68bf3
style
tolgacangoz May 13, 2025
1cb6a9e
Refactor `SkyReelsV2DiffusionForcingPipeline` to remove image process…
tolgacangoz May 13, 2025
ded93bc
Enhance `SkyReelsV2DiffusionForcingImageToVideoPipeline` by refining …
tolgacangoz May 13, 2025
c9483b2
Remove unused dtype handling in `SkyReelsV2DiffusionForcingPipeline` …
tolgacangoz May 13, 2025
f7fed01
up
tolgacangoz May 13, 2025
0e7b21d
up
tolgacangoz May 13, 2025
b3698d7
Update references
tolgacangoz May 13, 2025
7e0f0f5
Add `generate_timestep_matrix` method to `SkyReelsV2DiffusionForcingP…
tolgacangoz May 14, 2025
47080c2
Merge branch 'main' into skyreels-v2
tolgacangoz May 14, 2025
8c23208
Remove training-related code
tolgacangoz May 14, 2025
1f8e268
Add gradient checkpointing support in `SkyReelsV2Transformer3DModel` …
tolgacangoz May 14, 2025
d853521
Refactor `SkyReelsV2TransformerBlock` and remove unused `Head` class.…
tolgacangoz May 14, 2025
2b79584
Remove unused parameter `y` and associated documentation from `SkyRee…
tolgacangoz May 14, 2025
600ced3
Update context length calculation in `SkyReelsV2AttnProcessor2_0` to …
tolgacangoz May 14, 2025
586fe56
Fix comparison logic in `SkyReelsV2AttnProcessor2_0` to correctly det…
tolgacangoz May 14, 2025
afcaf6e
Remove unused `flex_attention` variable from `transformer_skyreels_v2…
tolgacangoz May 14, 2025
465df8c
Updates SkyReelsV2 pipeline defaults and docs
tolgacangoz May 15, 2025
cad2d38
Remove `enable_teacache` functionality from `SkyReelsV2Transformer3DM…
tolgacangoz May 15, 2025
1fcdf98
Refactor `SkyReelsV2Transformer3DModel` to use configuration paramete…
tolgacangoz May 15, 2025
6d57725
Remove unused import of `numpy` and clean up whitespace in `transform…
tolgacangoz May 15, 2025
c4cec04
Refactor `SkyReelsV2DiffusionForcingPipeline` to improve error handli…
tolgacangoz May 15, 2025
6a85ba1
Refactor `SkyReelsV2DiffusionForcingPipeline` to enhance sample sched…
tolgacangoz May 16, 2025
76af29b
update template for df_i2v
tolgacangoz May 16, 2025
81206ce
style
tolgacangoz May 16, 2025
906b6f5
Refactor `SkyReelsV2DiffusionForcingPipeline` to improve the handling…
tolgacangoz May 16, 2025
e2391b6
Add newly released `SkyReelsV2DiffusionForcingVideoToVideoPipeline` t…
tolgacangoz May 16, 2025
245534f
up df_i2v
tolgacangoz May 16, 2025
aaa8a8b
Refactor `SkyReelsV2DiffusionForcingPipeline` to improve the handling…
tolgacangoz May 16, 2025
ca3f7bd
Integrate video decoding in pipeline
tolgacangoz May 18, 2025
b4e26fd
up
tolgacangoz May 18, 2025
c3bcd1d
Fix variable name typo in `SkyReelsV2DiffusionForcingPipeline` from `…
tolgacangoz May 18, 2025
c9bea14
Fix variable name from `casual_block_size` to `causal_block_size` for…
tolgacangoz May 18, 2025
00fdeb0
Update `_no_split_modules` in `SkyReelsV2Transformer3DModel` and adju…
tolgacangoz May 18, 2025
cf91fb4
Refactor type hint for `device` parameter in `_prepare_blockwise_caus…
tolgacangoz May 18, 2025
256fa6d
Refactor `SkyReelsV2DiffusionForcingPipeline` to streamline the setti…
tolgacangoz May 19, 2025
a74252c
Add `flag_df` parameter to `SkyReelsV2Transformer3DModel` for improve…
tolgacangoz May 19, 2025
771fb05
Refactor `SkyReelsV2DiffusionForcingPipeline` to enhance clarity and …
tolgacangoz May 19, 2025
8e61893
Merge branch 'main' into skyreels-v2
tolgacangoz May 19, 2025
bccad55
Add script for converting SkyReelsV2 models to Diffusers format
tolgacangoz May 20, 2025
59c1e88
down
tolgacangoz May 20, 2025
02f038d
Update documentation in `SkyReelsV2DiffusionForcingPipeline` to clari…
tolgacangoz May 20, 2025
32ca01a
up
tolgacangoz May 20, 2025
02ffe0c
Refactor model directory path handling in `convert_transformer` funct…
tolgacangoz May 20, 2025
a215677
fix "inject_sample_info": true,
tolgacangoz May 20, 2025
1e4c501
temp fix
tolgacangoz May 20, 2025
322ce0c
up
tolgacangoz May 20, 2025
b7d54d6
fix `qk_norm`
tolgacangoz May 20, 2025
be77ad8
Refactor `convert_skyreelsv2_to_diffusers.py` to use `SkyreelsV2Image…
tolgacangoz May 20, 2025
6f8ffb2
for vae
tolgacangoz May 20, 2025
4576f6e
for t5
tolgacangoz May 20, 2025
10174ca
up
tolgacangoz May 20, 2025
9223f2d
temp fix
tolgacangoz May 20, 2025
a1aadd3
up
tolgacangoz May 20, 2025
f369cc4
Remove assertion for 1D timesteps in `get_timestep_embedding` functio…
tolgacangoz May 21, 2025
eb32376
Refactor timestep handling in `SkyReelsV2DiffusionForcingPipeline` to…
tolgacangoz May 21, 2025
671b37e
Enhance `get_timestep_embedding` to support 2D tensor inputs, allowin…
tolgacangoz May 21, 2025
6f8bf30
Fix unflattening of timestep projection in `SkyReelsV2Transformer3DMo…
tolgacangoz May 21, 2025
c71d3aa
Update dtype handling in `SkyReelsV2Transformer3DModel` to ensure con…
tolgacangoz May 21, 2025
1afa337
Refactor tensor reshaping in `SkyReelsV2Transformer3DModel` to utiliz…
tolgacangoz May 21, 2025
c74675c
Refactor timestep preparation in `SkyReelsV2DiffusionForcingPipeline`…
tolgacangoz May 21, 2025
602cff7
fix: multi-dimentional indexing
tolgacangoz May 21, 2025
237e468
Comment out tensor unsqueezing in `SkyReelsV2DiffusionForcingPipeline…
tolgacangoz May 21, 2025
40c456d
Update dtype handling in `SkyReelsV2DiffusionForcingPipeline` to use …
tolgacangoz May 21, 2025
9ed88da
fix dype
tolgacangoz May 21, 2025
6a3c7bf
fix
tolgacangoz May 21, 2025
5652aa0
Refactor sample scheduler initialization in `SkyReelsV2DiffusionForci…
tolgacangoz May 21, 2025
e529fea
Adds shift parameter to scheduler timestep setting
tolgacangoz May 21, 2025
b3ffeca
Fix slicing of latents in `SkyReelsV2DiffusionForcingPipeline` to ens…
tolgacangoz May 21, 2025
4479afc
Fix tensor slicing in `SkyReelsV2DiffusionForcingPipeline` to ensure …
tolgacangoz May 21, 2025
e4f6743
Update progress bar total in `SkyReelsV2DiffusionForcingPipeline` to …
tolgacangoz May 23, 2025
7420446
Refactor error handling and tensor processing in `SkyReelsV2Diffusion…
tolgacangoz May 23, 2025
2d59ebd
Refactor tensor processing and noise application in `SkyReelsV2Diffus…
tolgacangoz May 23, 2025
8af4a9f
Refactor variable naming and tensor handling in `SkyReelsV2DiffusionF…
tolgacangoz May 23, 2025
57a2bf9
style
tolgacangoz May 23, 2025
ae6adbe
fix number of frames for long video generation
tolgacangoz May 24, 2025
9afb214
up
tolgacangoz May 24, 2025
f1483ad
fix: `latents` initialization for long video generation in processing…
tolgacangoz May 24, 2025
a16c31b
update templates
tolgacangoz May 24, 2025
3b7b63b
Enhance `convert_skyreelsv2_to_diffusers.py` by adding support for lo…
tolgacangoz May 24, 2025
5e1126d
Update model configuration in `convert_skyreelsv2_to_diffusers.py` to…
tolgacangoz May 24, 2025
820d415
Refactor `set_ar_attention` method in `SkyReelsV2Transformer3DModel` …
tolgacangoz May 24, 2025
528e0d7
up
tolgacangoz May 25, 2025
6c4301c
up
tolgacangoz May 25, 2025
7d5328f
upp
tolgacangoz May 25, 2025
00849fd
fix file name
tolgacangoz May 25, 2025
8e34d89
Update `SkyReelsV2Transformer3DModel` to conditionally apply `causal_…
tolgacangoz May 25, 2025
493a08c
Merge branch 'main' into skyreels-v2
tolgacangoz May 25, 2025
a6f0d11
style
tolgacangoz May 25, 2025
cc0660c
Fix class name casing for SkyReelsV2 components in multiple files to …
tolgacangoz May 25, 2025
14d8d7a
cleaning
tolgacangoz May 25, 2025
85a1f90
cleansing
tolgacangoz May 25, 2025
5264ac9
Refactor `get_timestep_embedding` to move modifications into `SkyReel…
tolgacangoz May 26, 2025
81acfae
Remove unnecessary line break in `get_timestep_embedding` function fo…
tolgacangoz May 26, 2025
11baa00
Remove `skyreels_v2` entry from `_import_structure` and update its in…
tolgacangoz May 26, 2025
2906c37
cleansing
tolgacangoz May 26, 2025
a38eaab
Refactor attention processing in `SkyReelsV2AttnProcessor2_0` to alwa…
tolgacangoz May 26, 2025
150ea56
Enhance example usage in `pipeline_skyreels_v2_diffusion_forcing.py` …
tolgacangoz May 26, 2025
ad7d4c4
Refactor import structure in `__init__.py` for SkyReelsV2 components …
tolgacangoz May 26, 2025
ed7843a
Merge branch 'main' into skyreels-v2
tolgacangoz May 26, 2025
f1ee024
Update `guidance_scale` parameter in `SkyReelsV2DiffusionForcingPipel…
tolgacangoz May 26, 2025
421e0dc
Update `guidance_scale` parameter in example documentation and class …
tolgacangoz May 26, 2025
4b688c4
Update `causal_block_size` parameter in `SkyReelsV2DiffusionForcingPi…
tolgacangoz May 26, 2025
c6b5391
up
tolgacangoz May 26, 2025
3bf1e4a
Fix dtype conversion for `timestep_proj` in `SkyReelsV2Transformer3DM…
tolgacangoz May 26, 2025
f48363c
Optimize causal mask generation by replacing repeated tensor with `re…
tolgacangoz May 26, 2025
920d956
style
tolgacangoz May 26, 2025
cedee34
Merge branch 'main' into skyreels-v2
tolgacangoz May 26, 2025
db9cda9
Enhance example documentation in `SkyReelsV2DiffusionForcingPipeline`…
tolgacangoz May 27, 2025
ff6eeea
Refactor sample scheduler creation in `SkyReelsV2DiffusionForcingPipe…
tolgacangoz May 27, 2025
82db3ab
Merge branch 'main' into skyreels-v2
tolgacangoz May 27, 2025
c0abccc
Enhance error handling and documentation in `SkyReelsV2DiffusionForci…
tolgacangoz May 27, 2025
35061d0
Update documentation and progress bar handling in `SkyReelsV2Diffusio…
tolgacangoz May 27, 2025
cede08c
Refine progress bar calculation in `SkyReelsV2DiffusionForcingPipelin…
tolgacangoz May 27, 2025
5bc9a1b
Update import statements in `SkyReelsV2DiffusionForcingPipeline` docu…
tolgacangoz May 27, 2025
0cdfb99
Merge branch 'main' into skyreels-v2
tolgacangoz May 28, 2025
5c658c9
Refactor progress bar handling in `SkyReelsV2DiffusionForcingPipeline…
tolgacangoz May 28, 2025
b30a426
update templates for i2v, v2v
tolgacangoz May 28, 2025
238d07d
Add `retrieve_latents` function to streamline latent retrieval in `Sk…
tolgacangoz May 28, 2025
d3bd638
Add `retrieve_latents` function to both i2v and v2v pipelines for con…
tolgacangoz May 28, 2025
2aab1de
Remove redundant ValueError for `overlap_history` in `SkyReelsV2Diffu…
tolgacangoz May 28, 2025
8ab5bb1
Update default video dimensions and flow matching scheduler parameter…
tolgacangoz May 28, 2025
323ec66
Refactor `SkyReelsV2DiffusionForcingPipeline` to support Image-to-Vid…
tolgacangoz May 28, 2025
ce804ad
Improve organization for image-last_image condition.
tolgacangoz May 28, 2025
ff97206
Refactor `SkyReelsV2DiffusionForcingImageToVideoPipeline` to improve …
tolgacangoz May 28, 2025
5d702cf
style
tolgacangoz May 28, 2025
b6536ed
Merge branch 'main' into skyreels-v2
tolgacangoz May 28, 2025
9d35809
style
tolgacangoz May 28, 2025
0f915f6
Add example usage of PIL for image input in `SkyReelsV2DiffusionForci…
tolgacangoz May 28, 2025
9a6746b
Refactor `SkyReelsV2DiffusionForcingPipeline` to `SkyReelsV2Diffusion…
tolgacangoz May 28, 2025
b879963
Refactor `SkyReelsV2DiffusionForcingImageToVideoPipeline` by removing…
tolgacangoz May 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
448 changes: 448 additions & 0 deletions scripts/convert_skyreelsv2_to_diffusers.py

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions src/diffusers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@
"SD3ControlNetModel",
"SD3MultiControlNetModel",
"SD3Transformer2DModel",
"SkyReelsV2Transformer3DModel",
"SparseControlNetModel",
"StableAudioDiTModel",
"StableCascadeUNet",
Expand Down Expand Up @@ -272,6 +273,7 @@
"FlowMatchEulerDiscreteScheduler",
"FlowMatchHeunDiscreteScheduler",
"FlowMatchLCMScheduler",
"FlowMatchUniPCMultistepScheduler",
"HeunDiscreteScheduler",
"IPNDMScheduler",
"KarrasVeScheduler",
Expand Down Expand Up @@ -446,6 +448,10 @@
"SemanticStableDiffusionPipeline",
"ShapEImg2ImgPipeline",
"ShapEPipeline",
"SkyReelsV2DiffusionForcingImageToVideoPipeline",
"SkyReelsV2DiffusionForcingPipeline",
"SkyReelsV2ImageToVideoPipeline",
"SkyReelsV2Pipeline",
"StableAudioPipeline",
"StableAudioProjectionModel",
"StableCascadeCombinedPipeline",
Expand Down Expand Up @@ -804,6 +810,7 @@
SD3ControlNetModel,
SD3MultiControlNetModel,
SD3Transformer2DModel,
SkyReelsV2Transformer3DModel,
SparseControlNetModel,
StableAudioDiTModel,
T2IAdapter,
Expand Down Expand Up @@ -876,6 +883,7 @@
FlowMatchEulerDiscreteScheduler,
FlowMatchHeunDiscreteScheduler,
FlowMatchLCMScheduler,
FlowMatchUniPCMultistepScheduler,
HeunDiscreteScheduler,
IPNDMScheduler,
KarrasVeScheduler,
Expand Down Expand Up @@ -1031,6 +1039,10 @@
SemanticStableDiffusionPipeline,
ShapEImg2ImgPipeline,
ShapEPipeline,
SkyReelsV2DiffusionForcingImageToVideoPipeline,
SkyReelsV2DiffusionForcingPipeline,
SkyReelsV2ImageToVideoPipeline,
SkyReelsV2Pipeline,
StableAudioPipeline,
StableAudioProjectionModel,
StableCascadeCombinedPipeline,
Expand Down
2 changes: 2 additions & 0 deletions src/diffusers/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
_import_structure["transformers.transformer_mochi"] = ["MochiTransformer3DModel"]
_import_structure["transformers.transformer_omnigen"] = ["OmniGenTransformer2DModel"]
_import_structure["transformers.transformer_sd3"] = ["SD3Transformer2DModel"]
_import_structure["transformers.transformer_skyreels_v2"] = ["SkyReelsV2Transformer3DModel"]
_import_structure["transformers.transformer_temporal"] = ["TransformerTemporalModel"]
_import_structure["transformers.transformer_wan"] = ["WanTransformer3DModel"]
_import_structure["unets.unet_1d"] = ["UNet1DModel"]
Expand Down Expand Up @@ -173,6 +174,7 @@
PriorTransformer,
SanaTransformer2DModel,
SD3Transformer2DModel,
SkyReelsV2Transformer3DModel,
StableAudioDiTModel,
T5FilmDecoder,
Transformer2DModel,
Expand Down
1 change: 1 addition & 0 deletions src/diffusers/models/transformers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,6 @@
from .transformer_mochi import MochiTransformer3DModel
from .transformer_omnigen import OmniGenTransformer2DModel
from .transformer_sd3 import SD3Transformer2DModel
from .transformer_skyreels_v2 import SkyReelsV2Transformer3DModel
from .transformer_temporal import TransformerTemporalModel
from .transformer_wan import WanTransformer3DModel
Loading