Add OmniGen by staoxiao · Pull Request #10148 · huggingface/diffusers

staoxiao · 2024-12-08T09:17:20Z

What does this PR do?

Add a new pipeline along with corresponding tests and documentation.

Fixes #9873

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

@sayakpaul

pull the latest code

hlky · 2024-12-08T09:40:18Z

Hi @staoxiao! Thanks for your contribution 🤗 Could you run make style?

cc @stevhliu can you take a look at the docs?

cc @asomoza can you help test the pipelines?

HuggingFaceDocBuilderDev · 2024-12-08T09:40:48Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

staoxiao · 2024-12-08T11:26:21Z

Hi, @hlky , I ran 'make style' and modified my code, but it seems there are some errors in the other original files that I haven't changed.

hlky · 2024-12-08T11:34:26Z

@staoxiao This is due to ruff version, try pip install ruff==0.1.5 or pip install -e ".[dev]" in Diffusers

stevhliu

Thank you for contributing such a cool pipeline and complete docs!

docs/source/en/api/models/omnigen_transformer.md

docs/source/en/api/pipelines/omnigen.md

docs/source/en/using-diffusers/omnigen.md

docs/source/en/using-diffusers/multimodal2img.md

docs/source/en/using-diffusers/omnigen.md

Co-authored-by: Steven Liu <[email protected]>

yiyixuxu

oh thanks for your PR!
super excited to have this in diffusers. My main feedbacks is that we cannot import the Phi3Model from transformers and use it as a block in diffusers. I left some comment on how to rewrite and fit into diffusers code. Let us know if you need any help!

src/diffusers/models/embeddings.py

src/diffusers/models/transformers/transformer_omnigen.py

Co-authored-by: hlky <[email protected]>

update to latest version

a-r-r-o-w

Thanks @staoxiao! Apologies for the delay in reviews

Just some nits and refactors required to follow latest design choices in diffusers. A good reference would be the HunyuanVideo PR: #10136

One other remaining requirement is the minimal transformer modeling test. This would be a good example. Happy to help make any of the changes 🤗

src/diffusers/models/attention.py

src/diffusers/models/attention_processor.py

src/diffusers/models/transformers/transformer_omnigen.py

src/diffusers/pipelines/omnigen/pipeline_omnigen.py

staoxiao · 2025-02-08T11:46:28Z

Thanks for all suggestions and I have updated the code!

nitinmukesh · 2025-02-09T10:44:59Z

Working good so far. Just observed 1 issue.

With use_input_image_size_as_output as True

In Diffuser version [EDIT: checked HF space the issue is there as well], if I don't select an image and do prompt infer only, it throws an error. Solution is there (in case someone else face this problem NoneType), add use_input_image_size_as_output conditionally

if input_images is not None and len(input_images) > 0:
    inference_params["use_input_image_size_as_output"] = use_input_image_size_as_output

The woman in <|image_1|> and boy in <|image_2|> are holding hand and walking on street.

staoxiao · 2025-02-09T16:03:12Z

@nitinmukesh, use_input_image_size_as_output means setting the output image size to be consistent with the input image size. When using text-to-image generation, there is no need to set this parameter. To avoid this issue, I have added a check for this parameter in the check_inputs function.

nitinmukesh · 2025-02-09T16:09:48Z

Thank you @staoxiao .
I will modify at my end and set it to False for text2image generation.

yiyixuxu

thanks!

yiyixuxu · 2025-02-10T19:02:55Z

@staoxiao can you run make style and make fix-copies? so that the quality test would pass on our CI

@a-r-r-o-w, can you take a look again to see if all your comments are addressed? we can merge after that:)

a-r-r-o-w

Thanks @staoxiao for addressing the reviews! The PR looks good to merge. There are some things that we could refactor further to make consistent with diffusers-style implementation, but it is mostly just nitpicking -- we can do a follow-up to address this.

Really sorry for the long wait!

There seems to be some tests that are failing. Could you look at them? Happy to help fix them if you're busy

failing tests

https://github.com/huggingface/diffusers/actions/runs/13254834397/job/37014475117?pr=10148

FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_callback_cfg - AttributeError: 'OmniGenPipeline' object has no attribute 'num_timesteps'
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_callback_inputs - assert tensor(18.2633) == 0
 +  where tensor(18.2633) = <built-in method sum of Tensor object at 0x7fed4b365490>()
 +    where <built-in method sum of Tensor object at 0x7fed4b365490> = tensor([[[[0.2224, 1.5052],\n          [1.5705, 0.7162]],\n\n         [[0.5437, 0.5603],\n          [0.5271, 1.9422]],\n\n         [[1.4330, 0.6505],\n          [0.3135, 0.6654]],\n\n         [[1.2489, 2.0913],\n          [3.4383, 0.8350]]]]).sum
 +      where tensor([[[[0.2224, 1.5052],\n          [1.5705, 0.7162]],\n\n         [[0.5437, 0.5603],\n          [0.5271, 1.9422]],\n\n         [[1.4330, 0.6505],\n          [0.3135, 0.6654]],\n\n         [[1.2489, 2.0913],\n          [3.4383, 0.8350]]]]) = <built-in method abs of Tensor object at 0x7fed4b364540>()
 +        where <built-in method abs of Tensor object at 0x7fed4b364540> = tensor([[[[-0.2224, -1.5052],\n          [ 1.5705,  0.7162]],\n\n         [[ 0.5437, -0.5603],\n          [ 0.5271,  1.9422]],\n\n         [[-1.4330,  0.6505],\n          [ 0.3135, -0.6654]],\n\n         [[ 1.2489,  2.0913],\n          [-3.4383, -0.8350]]]]).abs
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_inference_batch_single_identical - Failed: Timeout >60.0s
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_loading_with_variants - OSError: Error no file named diffusion_pytorch_model.fp16.bin found in directory /tmp/tmp1u_zu527/transformer.
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_num_images_per_prompt - RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 4 for tensor number 1 in the list.
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_save_load_dduf - Failed: Timeout >60.0s
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_save_load_local - OSError: Error no file named diffusion_pytorch_model.bin found in directory /tmp/tmpwmjdsdca/transformer.
FAILED tests/pipelines/omnigen/test_pipeline_omnigen.py::OmniGenPipelineFastTests::test_save_load_optional_components - OSError: Error no file named diffusion_pytorch_model.bin found in directory /tmp/tmpxrfny9bf/transformer.

staoxiao · 2025-02-11T13:04:30Z

Hi, @a-r-r-o-w , I have fixed some of the failing tests. However, the remaining failing tests need your assistance to resolve, for example, OmniGen does not have an fp16 version for the function test_loading_with_variants(OmniGen cannot support fp16).

a-r-r-o-w · 2025-02-11T20:45:12Z

@staoxiao Seems like that might be a local environment error, as it is passing for my environment and on our CI.

Merging since the currently failing tests look unrelated. Thanks Shitao! :)

nitinmukesh · 2025-02-12T06:34:07Z

Thank you @staoxiao and @a-r-r-o-w .

tin2tin · 2025-02-12T11:07:41Z

Yes, thank you @staoxiao and @a-r-r-o-w I've quickly added OmniGen to a free Blender add-on, I'm sometimes working on. And posted about it on Reddit. I hope it's okay, and it'll give your hard work some exposure. Here's the video I did in full res:

OmniGen.mp4

sayakpaul · 2025-02-12T11:10:36Z

Keep doing it @tin2tin! We appreciate the good work.

staoxiao added 18 commits November 30, 2024 18:01

OmniGen model.py

36eee40

update OmniGenTransformerModel

bbe2b98

omnigen pipeline

b839590

omnigen pipeline

0d04194

update omnigen_pipeline

85abe5e

test case for omnigen

db92c69

update omnigenpipeline

308766c

update docs

4c5e8c5

update docs

d9f80fc

offload_transformer

c78d1f4

enable_transformer_block_cpu_offload

236f14b

update docs

6b52547

reformat

4fef9c8

reformat

f2fc182

reformat

5f3148d

Merge pull request #1 from huggingface/main

cdd500e

pull the latest code

update docs

178d377

update docs

08c05f9

make style

286990d

make style

3bb092b

stevhliu reviewed Dec 9, 2024

View reviewed changes

staoxiao and others added 4 commits December 10, 2024 14:23

Update docs/source/en/api/models/omnigen_transformer.md

5925cb9

Co-authored-by: Steven Liu <[email protected]>

Update docs/source/en/using-diffusers/omnigen.md

56aa821

Co-authored-by: Steven Liu <[email protected]>

Update docs/source/en/using-diffusers/omnigen.md

1e33ca8

Co-authored-by: Steven Liu <[email protected]>

update docs

c81a84d

yiyixuxu reviewed Dec 10, 2024

View reviewed changes

src/diffusers/models/embeddings.py Outdated Show resolved Hide resolved

src/diffusers/models/transformers/transformer_omnigen.py Outdated Show resolved Hide resolved

staoxiao and others added 7 commits February 8, 2025 12:33

Update tests/pipelines/omnigen/test_pipeline_omnigen.py

52a6f9e

Co-authored-by: hlky <[email protected]>

Update tests/pipelines/omnigen/test_pipeline_omnigen.py

aeea57a

Co-authored-by: hlky <[email protected]>

Update src/diffusers/pipelines/omnigen/pipeline_omnigen.py

6b1177b

Co-authored-by: hlky <[email protected]>

Update src/diffusers/pipelines/omnigen/pipeline_omnigen.py

7003a80

Co-authored-by: hlky <[email protected]>

Update src/diffusers/pipelines/omnigen/pipeline_omnigen.py

792c3e6

Co-authored-by: hlky <[email protected]>

Merge pull request #2 from huggingface/main

b0c6267

update to latest version

consistent attention processor

f5e3f0b

a-r-r-o-w reviewed Feb 8, 2025

View reviewed changes

staoxiao added 2 commits February 8, 2025 19:32

updata

3541ab8

update

4e9850a

check_inputs

f91cfcf

yiyixuxu approved these changes Feb 10, 2025

View reviewed changes

yiyixuxu added the close-to-merge label Feb 10, 2025

make style

711dded

a-r-r-o-w approved these changes Feb 11, 2025

View reviewed changes

staoxiao added 2 commits February 11, 2025 20:54

update testpipeline

565e51c

update testpipeline

29ad6ae

a-r-r-o-w merged commit 798e171 into huggingface:main Feb 11, 2025
10 of 12 checks passed

a-r-r-o-w mentioned this pull request Feb 11, 2025

Refactor OmniGen #10771

Merged

nitinmukesh mentioned this pull request Jun 16, 2025

Thank you for the new release VectorSpaceLab/OmniGen2#1

Closed

Conversation

staoxiao commented Dec 8, 2024

What does this PR do?

Before submitting

Uh oh!

hlky commented Dec 8, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Dec 8, 2024

Uh oh!

staoxiao commented Dec 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hlky commented Dec 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

staoxiao commented Feb 8, 2025

Uh oh!

nitinmukesh commented Feb 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

staoxiao commented Feb 9, 2025

Uh oh!

nitinmukesh commented Feb 9, 2025

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Feb 10, 2025

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

staoxiao commented Feb 11, 2025

Uh oh!

a-r-r-o-w commented Feb 11, 2025

Uh oh!

Uh oh!

nitinmukesh commented Feb 12, 2025

Uh oh!

tin2tin commented Feb 12, 2025

Uh oh!

sayakpaul commented Feb 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

staoxiao commented Dec 8, 2024 •

edited

Loading

hlky commented Dec 8, 2024 •

edited

Loading

nitinmukesh commented Feb 9, 2025 •

edited

Loading