Added Support for WAN Image Generation#974
Conversation
|
I added support for video generation models starting with Wan2.1 models will later extend to more models if this works, currently I added support for WanPipeline in supported model architectures in Image_diffusion plugin but it gives an error where its trying to find /workspace/jobs/None which doesn't exist |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was an issue with running diffusion models, could you update and check if that issue is resolved now? The issue was caused by our new system with the workspace under orgs |
I figure this is still a WIP but I would also suggest not adding things to the existing image_diffusion plugin and creating a new video diffusion plugin. |
deep1401
left a comment
deep1401
left a comment
There was a problem hiding this comment.
Added some comments, we might have to restructure the PR and break it down into 2 and test these
api/transformerlab/plugins/video_diffusion_plugin/video_diffusion_worker.py
Outdated
Show resolved
Hide resolved
deep1401
left a comment
deep1401
left a comment
There was a problem hiding this comment.
Sorry I forgot to add this earlier but lets use this as a reference model for Wan support. This feels small enough to run on our hardware and diffusers required version is 0.33.0.dev so just updating to current diffusers prod version should work
https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers
ParamThakkar123
changed the title
deep1401
changed the title
deep1401
left a comment
deep1401
left a comment
There was a problem hiding this comment.
Okay I tested this and it works for me. There was a small bug which I pushed and fixed
But I just have some comments on changes made in the plugin, please let me know if you think they're not apt!
After test:
|
|
||
| def is_zimage_model(model: str) -> bool: | ||
| """Return True if the model architecture is ZImagePipeline.""" | ||
| def is_diffusion_pipeline_model(model: str) -> bool: |
There was a problem hiding this comment.
Why do we need to check this? Isnt this done by the router itself?
| name = (model or "").lower() | ||
| return "z-image" in name or "zimage" in name | ||
| for arch in architectures: | ||
| if arch in DIFFUSION_PIPELINE_MODELS: |
There was a problem hiding this comment.
Shouldnt this just be based on a dictionary which determines pipeline based on the model_architecture param or something?
deep1401
left a comment
deep1401
left a comment
There was a problem hiding this comment.
Changing state of my previous review
dadmobile
left a comment
dadmobile
left a comment
There was a problem hiding this comment.
Ruff error:
Error: api/transformerlab/plugins/image_diffusion/main.py:1166:32: F821 Undefined name run_video_diffusion_genration
|
it was a typo 😅 |
dadmobile
left a comment
dadmobile
left a comment
There was a problem hiding this comment.
- Need to update plugin version
- I can't get this to run on azure. Any ideas?
Error during image generation: AutoPipeline can't find a pipeline linked to WanPipeline for wan
Error in Async Job: 500: Image generation failed: AutoPipeline can't find a pipeline linked to WanPipeline for wan
Traceback (most recent call last):
File "/home/azureuser/.transformerlab/orgs/3c33c85b-628a-4ca8-93d3-b657cb7973b2/workspace/plugins/image_diffusion/main.py", line 1009, in diffusion_generate_job
pipe = get_pipeline(
^^^^^^^^^^^^^
File "/home/azureuser/.transformerlab/orgs/3c33c85b-628a-4ca8-93d3-b657cb7973b2/workspace/plugins/image_diffusion/main.py", line 411, in get_pipeline
pipe = AutoPipelineForText2Image.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/.transformerlab/orgs/3c33c85b-628a-4ca8-93d3-b657cb7973b2/workspace/plugins/image_diffusion/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/.transformerlab/orgs/3c33c85b-628a-4ca8-93d3-b657cb7973b2/workspace/plugins/image_diffusion/venv/lib/python3.11/site-packages/diffusers/pipelines/auto_pipeline.py", line 482, in from_pretrained
text_2_image_cls = _get_task_class(AUTO_TEXT2IMAGE_PIPELINES_MAPPING, orig_class_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/azureuser/.transformerlab/orgs/3c33c85b-628a-4ca8-93d3-b657cb7973b2/workspace/plugins/image_diffusion/venv/lib/python3.11/site-packages/diffusers/pipelines/auto_pipeline.py", line 311, in _get_task_class
raise ValueError(f"AutoPipeline can't find a pipeline linked to {pipeline_class_name} for {model_name}")
ValueError: AutoPipeline can't find a pipeline linked to WanPipeline for wan
During handling of the above exception, another exception occurred:
dadmobile
left a comment
dadmobile
left a comment
There was a problem hiding this comment.
Bumped version to get it to reinstall and now it works!
|
Caution Review failedThe pull request is closed. 📝 WalkthroughWalkthroughThis PR introduces video diffusion support to the image diffusion plugin, adds WanPipeline architecture support throughout the system, enhances image retrieval with robust multi-file handling, and bumps the plugin version. Setup script adds ftfy dependency installation. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Main as main.py<br/>(diffusion plugin)
participant Loader as Pipeline<br/>Loader
participant VideoGen as run_video_<br/>diffusion_...
participant Storage as File<br/>Storage
Client->>Main: DiffusionRequest (video-style model)
Main->>Main: is_diffusion_pipeline_model()
alt Detected as Video Pipeline
Main->>Loader: get_pipeline(model, controlnet_id, ...)
Loader->>Loader: Detect DIFFUSION_PIPELINE_MODELS<br/>architecture
Loader->>Main: Return DiffusionPipeline
Main->>VideoGen: run_video_diffusion_generation_for_images()
VideoGen->>VideoGen: Setup output directory<br/>Select device
VideoGen->>Loader: Load pipeline with dtype
VideoGen->>VideoGen: Extract & generate frames
VideoGen->>Storage: Save frames as<br/>sequential PNGs
VideoGen->>Main: Return image_paths, metadata
Main->>Client: Return generation result<br/>(bypass multi-GPU path)
else Standard Pipeline
Main->>Main: Use existing loaders<br/>and multi-GPU path
Main->>Client: Return generation result
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
✨ Finishing Touches
🧪 Generate unit tests (beta)
Comment |
4 participants
Summary by CodeRabbit
New Features
Bug Fixes & Improvements
Chores