Add OmniGen: A Unified Image Generation Model Pipeline

[ighoshsubho]

Model/Pipeline/Scheduler description

Adding support for OmniGen, a unified image generation model that can handle multiple tasks including text-to-image, image editing, subject-driven generation, and various computer vision tasks within a single framework.

Key features,

• Unified architecture handling multiple generation tasks without additional moduls like adapters or controlnet
• Supports arbitrarily interleaved text and image inputs as conditions
• Competitive text-to-image generation compared to existing models
• Built-in support for image editing, visual-conditional generation, and subject-driven generation

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Paper: OmniGen: Unified Image Generation
Official Implementation: https://github.com/VectorSpaceLab/OmniGen

I would like to implement this pipeline in diffusers

[bghira]

i've implemented training for this and it's very similar interface they've used vs Diffusers. adapting it to use the Mixins instead should be straightforward.

[ighoshsubho]

i've implemented training for this and it's very similar interface they've used vs Diffusers. adapting it to use the Mixins instead should be straightforward.

can we add a community example for this in diffusers, would be helpful.

[sayakpaul]

Cc: @staoxiao from the OmniGen team.

[bghira]

yes and OmniGen already has a training example script that mirrors the type provided by the Diffusers project which should make this a trivial adaptation. @ighoshsubho if you start that effort i can contribute.

[staoxiao]

It would be very helpful if OmniGen could be called within diffusers. I’ve been wanting to do this but haven’t started yet. It would be fantastic if you @ighoshsubho @bghira could help implement it. Feel free to ask me if you have any questions.

[bghira]

@staoxiao i don't know if it's too much to ask but i have a pull request where i attempted to fit OmniGen's training requirements into simpletuner. it has very high loss so i'm sure i missed something with the padding. should I open an issue on your repository to follow-up with this?

[staoxiao]

Hi, @bghira, I'd be pleased to see that more tools can support OmniGen. You can share the code with me in a way that's convenient for you: opening an issue on OmniGen, or showing me your PR on simpletuner.

[a-r-r-o-w]

Bumping this to keep the activity going. @ighoshsubho Looking forward to your PR if you're interested in contributing this!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.