[Diffusion] Support Diffusers backend - Run any model supported by Diffusers #14112

adarshxs · 2025-11-28T22:34:10Z

Diffusers Backend Support for `SGLang Diffusion`

Summary

This PR adds a new diffusers backend to SGLang's multimodal generation module, allowing users to run any model supported by the Hugging Face diffusers library through SGLang's infrastructure even if SGLang doesn't have native support for that specific model.

Motivation

SGLang has excellent optimized pipelines for specific models (Flux, Wan, HunyuanVideo, etc.), but users often want to use less common diffusion models that don't yet have native SGLang implementations in their pipelines/workflows. This PR enables a fallback mechanism that wraps vanilla diffusers pipelines, providing:

Immediate compatibility with hundreds of diffusers models
Graceful fallback when native support isn't available
Explicit backend selection via --backend diffusers

Usage

CLI

# Explicit diffusers backend
sglang generate --model-path briaai/BRIA-3.2 --prompt "A curious raccoon" --backend diffusers

# Auto fallback (uses diffusers if no native support)
sglang generate --model-path some-new-model/model-name --prompt "Hello world"

# Image editing models
sglang generate --model-path Qwen/Qwen-Image-Edit --prompt "Convert to 3D style" --image-path input.jpg --backend diffusers

Server Mode

sglang serve --model-path briaai/BRIA-3.2 --backend diffusers --port 3000

Python API

from sglang.multimodal_gen import DiffGenerator

generator = DiffGenerator.from_pretrained("briaai/BRIA-3.2", backend="diffusers")
images = generator.generate(["A beautiful sunset"])

Changes

New Files

runtime/pipelines/diffusers_pipeline.py - Main pipeline wrapper containing:
- DiffusersExecutionStage: Pipeline stage that wraps diffusers execution
- DiffusersPipeline: ComposedPipelineBase implementation for diffusers models
configs/pipeline_configs/diffusers_generic.py - Generic pipeline config for diffusers backend
configs/sample/diffusers_generic.py - Generic sampling params for diffusers backend

Modified Files

runtime/server_args.py - Added Backend enum (AUTO, SGLANG, DIFFUSERS) and --backend CLI argument
registry.py - Updated get_model_info() to:
- Return DiffusersPipeline when backend=DIFFUSERS
- Fallback to diffusers when backend=AUTO and no native support found

Testing

Tested with:

briaai/BRIA-3.2 (custom BriaPipeline - not supported in sglang)
Qwen/Qwen-Image-Edit (image editing)
Standard Stable Diffusion models (SDXL 1.0,
Video generation models (Wan, Skyreels etc.)

Notes

Models with custom pipeline classes may require --trust-remote-code
Some newer pipelines require the latest diffusers version (pip install --upgrade diffusers or install from source)

gemini-code-assist · 2025-11-28T22:34:36Z

Summary of Changes

Hello @adarshxs, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances SGLang's multimodal generation capabilities by integrating a diffusers backend. This strategic addition broadens the range of supported models, allowing users to leverage the vast ecosystem of Hugging Face diffusers models, including those without native SGLang optimizations. It provides a robust fallback mechanism and explicit control over backend selection, ensuring greater flexibility and accessibility for various diffusion models within the SGLang framework.

Highlights

Diffusers Backend Integration: Introduced a new diffusers backend to SGLang's multimodal generation module, enabling compatibility with any model supported by the Hugging Face diffusers library.
Flexible Model Support: Allows users to run models without native SGLang implementations, providing immediate compatibility and a graceful fallback mechanism.
Backend Selection: Added explicit backend selection via a --backend CLI argument with options for auto (prefer native, fallback to diffusers), sglang (native optimized), and diffusers (vanilla diffusers pipeline).
Generic Configuration: New generic pipeline and sampling parameter configurations (DiffusersGenericPipelineConfig, DiffusersGenericSamplingParams) were added to support the diffusers backend with minimal SGLang-specific setup.
Extended API Parameters: The CLI and OpenAI-compatible API now support passing diffusers-specific keyword arguments via --diffusers-kwargs (CLI) or diffusers_kwargs (API) for fine-grained control.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a diffusers backend, which is an excellent feature for expanding model compatibility. The implementation is thorough, with a robust DiffusersPipeline that handles various output formats and loading scenarios. The fallback logic in the model registry is well-designed. The changes also extend the CLI and OpenAI-compatible API to pass through diffusers-specific arguments, which adds a lot of flexibility. My review includes a few suggestions for improving code clarity, fixing a potential tensor shape inconsistency, and addressing some minor stylistic issues. Overall, this is a high-quality contribution.

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py

python/sglang/multimodal_gen/runtime/entrypoints/utils.py

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py

adarshxs · 2025-11-29T21:53:27Z

needed some changes with ref to: #14129

… diffbackend

sayakpaul

Thanks a lot for spearheading this!

sayakpaul · 2025-11-30T23:17:29Z

python/sglang/multimodal_gen/configs/pipeline_configs/diffusers_generic.py

+This module provides a minimal pipeline configuration that works with the diffusers backend.
+Since diffusers handles its own model loading and configuration, this config is intentionally minimal.


This is cool!

Some bits I wanted to mention:

We can speed up the pipeline (that includes a bunch of models internally) loading process by specifying device_map="cuda" (cuda is just an example here) to from_pretrained(). This warms up the CUDA caching allocator to avoid small allocations of tensors.

We can load shards parallely into the individual pipeline models. Check out: https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading#parallel-loading

sayakpaul · 2025-11-30T23:26:55Z

python/sglang/multimodal_gen/configs/pipeline_configs/diffusers_generic.py

+
+from dataclasses import dataclass, field
+
+from sglang.multimodal_gen.configs.models import DiTConfig, EncoderConfig, VAEConfig


Question (out of curiosity):

Is EncoderConfig designed to handle multiple encoders a pipeline might be using? For example, Flux.1 uses two text encoders, Flux.2 uses one. Then there are some pipelines, that make use of an image encoder as well (in case they are doing some image-guided tasks).

EncoderConfig is designed to guide the generation of a single text encoder. We use one config for each encoder

sayakpaul · 2025-11-30T23:31:00Z

python/sglang/multimodal_gen/configs/pipeline_configs/diffusers_generic.py

+    text_encoder_configs: tuple[EncoderConfig, ...] = field(
+        default_factory=lambda: (EncoderConfig(),)
+    )
+    text_encoder_precisions: tuple[str, ...] = field(default_factory=lambda: ("fp16",))


Just a note that some pipelines that use models like Gemma cannot use FP16 because those models don't support inference in FP16. By "don't support", I mean one can obviously run inference, but the results will be all garbled up.

sayakpaul · 2025-11-30T23:32:41Z

python/sglang/multimodal_gen/configs/pipeline_configs/diffusers_generic.py

+    text_encoder_precisions: tuple[str, ...] = field(default_factory=lambda: ("fp16",))
+
+    # VAE settings
+    vae_tiling: bool = False  # diffusers handles this


Not sure if we support it here as well but "slicing" is also pretty popular:
https://github.com/huggingface/diffusers/blob/152f7ca357c066c4af3d1a58cdf17662ef5a2f87/src/diffusers/models/autoencoders/vae.py#L914

sayakpaul · 2025-11-30T23:40:13Z

python/sglang/multimodal_gen/runtime/entrypoints/openai/image_api.py

    background: Optional[str],
    image_path: Optional[str] = None,
+    num_inference_steps: Optional[int] = None,
+    guidance_scale: Optional[float] = None,


For some pipelines like QwenImage, we have true_cfg_scale as well:
https://github.com/huggingface/diffusers/blob/152f7ca357c066c4af3d1a58cdf17662ef5a2f87/src/diffusers/pipelines/qwenimage/pipeline_qwenimage_edit.py#L553

This is to distinguish between guidance distillation (in which case, guidance_scale would mean an embedded scale) and CFG. This is a bit of a future-proof thing since QwenImage doesn't have a guidance-distilled checkpoint yet (this decision was taken by the authors themselves).

sayakpaul · 2025-11-30T23:42:36Z

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py

+class DiffusersExecutionStage(PipelineStage):
+    """Pipeline stage that wraps diffusers pipeline execution."""
+
+    def __init__(self, diffusers_pipe: Any):


(nit): should diffusers_pipe be of type DiffusionPipeline?

sayakpaul · 2025-11-30T23:44:43Z

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py

+
+        return output
+
+    def _build_pipeline_kwargs(self, batch: Req, server_args: ServerArgs) -> dict:


Question: how do we handle the __call__ argument mismatches if they occur? For example, if a user specifies fps or num_frames for an image pipeline, how do we convey it to the user that those arguments will be ignored?

sayakpaul · 2025-11-30T23:46:42Z

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py

+        self.diffusers_pipe = self._load_diffusers_pipeline(model_path, server_args)
+        self._detect_pipeline_type()
+
+    def _load_diffusers_pipeline(self, model_path: str, server_args: ServerArgs) -> Any:


(nit): should we also let the users provide a quantization_config?
https://huggingface.co/docs/diffusers/main/en/quantization/overview#pipeline-level-quantization

adarshxs added 13 commits November 29, 2025 01:48

diffusers backend

2479679

diff generator fix

61e3775

import fix

9c44086

fix

eacb216

error tracing

532322e

fix

82a62d6

fix

e094f17

remove folder

71ace6e

fix import

e2bc01d

fix

1109d5b

fix

7951da7

fix

3b0a3fa

fix

549049a

adarshxs requested a review from mickqian as a code owner November 28, 2025 22:34

github-actions bot added the diffusion SGLang Diffusion label Nov 28, 2025

gemini-code-assist bot reviewed Nov 28, 2025

View reviewed changes

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py Show resolved Hide resolved

python/sglang/multimodal_gen/runtime/entrypoints/utils.py Show resolved Hide resolved

python/sglang/multimodal_gen/runtime/pipelines/diffusers_pipeline.py Outdated Show resolved Hide resolved

lint

12884ab

adarshxs force-pushed the diffbackend branch from 81c1af9 to 12884ab Compare November 29, 2025 08:41

adarshxs and others added 8 commits November 30, 2025 02:08

fix

c5b9a5e

fix

195d7a0

fix conflicts

7e2b0ce

fix conflicts

b95411e

fix

96e8cc2

fix

c866217

fix lint

8b32655

remove outputs

9c51ee0

adarshxs added 2 commits November 30, 2025 03:29

fix

b508984

Merge branch 'diffbackend' of https://github.com/adarshxs/sglang into…

88e823c

… diffbackend

sayakpaul reviewed Nov 30, 2025

View reviewed changes

		This module provides a minimal pipeline configuration that works with the diffusers backend.
		Since diffusers handles its own model loading and configuration, this config is intentionally minimal.


		from dataclasses import dataclass, field

		from sglang.multimodal_gen.configs.models import DiTConfig, EncoderConfig, VAEConfig


		return output

		def _build_pipeline_kwargs(self, batch: Req, server_args: ServerArgs) -> dict:

[Diffusion] Support Diffusers backend - Run any model supported by Diffusers #14112

Are you sure you want to change the base?

[Diffusion] Support Diffusers backend - Run any model supported by Diffusers #14112

Uh oh!

Conversation

adarshxs commented Nov 28, 2025

Diffusers Backend Support for SGLang Diffusion

Summary

Motivation

Usage

CLI

Server Mode

Python API

Changes

New Files

Modified Files

Testing

Notes

Uh oh!

gemini-code-assist bot commented Nov 28, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adarshxs commented Nov 29, 2025

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Diffusers Backend Support for `SGLang Diffusion`