@sayakpaul @patrickvonplaten What do you think about this pipeline design?

Thanks for the WIP PR. Could you help understand the adapter a bit better? Like what does it do, etc.? The results don't look great to me either. So, a bit more context would be helpful.

@sayakpaul

IP-Adapter, an effective and lightweight adapter to achieve image
prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter
is decoupled cross-attention mechanism that separates cross-attention layers for text features and
image features.

We can use an image as a prompt. We can combine IP-Adapter with other methods like ControlNet.

Decoupled cross-attention: Separate text prompt cross attention and Image prompt cross attention.

You can also check demo notebook. https://github.com/tencent-ailab/IP-Adapter/blob/main/ip_adapter_demo.ipynb

Seems cool but the additional image encoder and the adapter modules are heavy IMO, no? The image encoder is 3.69 GB and the adapter is 700 MBs. Given that do you think the gains are reasonable?

We have #4388 too.

In BLIP-Diffusion, the vision encoder is 1.16GB and the qformer is 1.98GB for SDv1.5.
https://huggingface.co/ayushtues/blipdiffusion

In IP-Adapter, the image encoder is 2.53 GB and the adapter is 44 MBs for SDv1.5, 3.69 GB and 770MBs for SDXL.

Both can combine with other methods like ControlNet.

One advantage is there are trained weights for SDXL.

We also compare our IP-Adapter with other methods including Versatile Diffusion, BLIP Diffusion [31], Uni-ControlNet, T2I-Adapter, ControlNet Shuffle, and ControlNet Reference-only. The comparison results are shown in Figure 9. Compared with other existing methods, our method can generate superior results in both image quality and alignment with multimodal prompts.

Alright! I have a better understanding now. Flexibility is controlling the generation process is definitely desirable. I think it also helps us achieve zero-shot subject-driven generation (like BLIP Diffusion)?

Also, the example you showed in the OP refers to image variation, right?

Regardless, the result looks okayish to me.

@patrickvonplaten what are your thoughts?

Alright! I have a better understanding now. Flexibility is controlling the generation process is definitely desirable. I think it also helps us achieve zero-shot subject-driven generation (like BLIP Diffusion)?

I think so too!

Also, the example you showed in the OP refers to image variation, right?

Exactly.

I am leaning towards having the pipeline as a part of the core library. Let's see what @patrickvonplaten has to share. Given the pipeline can do image variations, zero-shot subject-driven generation, can be combined with ControlNets, etc. -- I do see potential use-cases.

it's a little confusing to have a load_ip_adapter step, since every IP-adapter weighted is matched to the given image encoder and projector, why not mege it into init process? @okotaku

@ultranity Because we should set IPAttnProcessor first.

1. load pipeline
2. set IPAttnProcessor
3. load the weights of IPAttnProcessor

This flow is similar to LoRA.

This PR seems to be a bit in limbo, can someone take it over? cc @DN6 @yiyixuxu maybe?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Has there been any progress on this? IP Adapter works great for me and I'd love to see it supported in diffusers

@andypotato could you provide some example results?

@sayakpaul sure, here are some examples I created with IP Adapter using their "ip-adapter-plus-face_sd15.bin" checkpoint.

For convenience, these were created using A1111, ReV Animated 1.22 and the ControlNet extension to load the IP Adapter models. Although I can also create similar images using diffusers and their own repo at https://github.com/tencent-ailab/IP-Adapter to generate similar images.

First image is the reference image:

These are the generated images, the likeness is really quite stunning.

Wow, that's quite impressive indeed.

@apolinario has made us aware that IP Adapters are becoming extremely popular in the community. So, I'd be in favor of supporting it in diffusers too. @patrickvonplaten what are your thoughts?

Wow, that's quite impressive indeed.

@apolinario has made us aware that IP Adapters are becoming extremely popular in the community. So, I'd be in favor of supporting it in diffusers too. @patrickvonplaten what are your thoughts?

Agree let's add it to main!

cc @DN6 @yiyixuxu can you review here as well?

@okotaku I went into your PR and addresses #4944 (comment). I hope that's okay?

it's a little confusing to have a load_ip_adapter step, since every IP-adapter weighted is matched to the given image encoder and projector, why not mege it into init process? @okotaku

IIUC the encoder and the projection modules are not changed for every IP module being loaded. So, not sure why having a separate load_ip_adapter() method isn't beneficial. @ultranity

@yiyixuxu summary of the changes I have done so far this morning:

• Delegated IP Adapter loading logic to a separate mixin class (thanks for the idea!)
• Rejig how we handle passing the inputs through image_projection so that we don't have to touch the vanilla unet for an SD.
• Some minor refactorings that I think make the code cleaner.

The code from #4944 (comment) works and produces expected results. However, I updated it to better reflect the loading of the image encoder from the official repo:

from transformers import CLIPVisionModelWithProjection
from diffusers import StableDiffusionPipeline
import torch
from diffusers.utils import load_image

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter", 
    subfolder="models/image_encoder",
    torch_dtype=torch.float16,
).to("cuda")

pipeline = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    image_encoder=image_encoder,
    torch_dtype=torch.float16
).to("cuda")

image = load_image("https://user-images.githubusercontent.com/24734142/266492875-2d50d223-8475-44f0-a7c6-08b51cb53572.png")

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")


images = pipeline(
    prompt='best quality, high quality', 
    image_prompt=image,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", 
    num_inference_steps=50,
).images

images[0].save("sayak_test_out.png")

Okay for me either way.

If we are attaching ad-hoc weights to the unet I think that is still okay here. I don't see a lot of side-effects of doing it.

I'd still like an opinion from @patrickvonplaten here.

@patrickvonplaten I am trying to address your concerns around save_pretrained(). This is my testing script:

from transformers import CLIPVisionModelWithProjection
from diffusers import StableDiffusionPipeline
from diffusers.models.attention_processor import AttnProcessor2_0
import torch
import numpy as np
import tempfile

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter", 
    subfolder="models/image_encoder",
    torch_dtype=torch.float16,
).to("cuda")

pipeline = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    image_encoder=image_encoder,
    torch_dtype=torch.float16
).to("cuda")
output = pipeline("hey", num_inference_steps=10, output_type="np").images

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")

ckpt_path = "ip_adapter_loaded_pipeline"

with tempfile.TemporaryDirectory() as tmpdirname:
    pipeline.save_pretrained(tmpdirname)
    pipeline.unet.set_attn_processor(AttnProcessor2_0())
    loaded_pipeline = StableDiffusionPipeline.from_pretrained(tmpdirname, torch_dtype=torch.float16).to("cuda")

output_loaded = loaded_pipeline("hey", num_inference_steps=10, output_type="np").images

print(np.allclose(output[0, :3, :3, -1], output_loaded[0, :3, :3, -1], atol=1e-4, rtol=1e-4))

I am facing this:

Some weights of the model checkpoint were not used when initializing UNet2DConditionModel: 
 ['down_blocks.0.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, down_blocks.0.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, down_blocks.0.attentions.1.transformer_blocks.0.attn2.processor.to_k_ip.weight, down_blocks.0.attentions.1.transformer_blocks.0.attn2.processor.to_v_ip.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, down_blocks.1.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, down_blocks.1.attentions.1.transformer_blocks.0.attn2.processor.to_k_ip.weight, down_blocks.1.attentions.1.transformer_blocks.0.attn2.processor.to_v_ip.weight, down_blocks.2.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, down_blocks.2.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, down_blocks.2.attentions.1.transformer_blocks.0.attn2.processor.to_k_ip.weight, down_blocks.2.attentions.1.transformer_blocks.0.attn2.processor.to_v_ip.weight, mid_block.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, mid_block.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.1.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.1.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.1.attentions.1.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.1.attentions.1.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.1.attentions.2.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.1.attentions.2.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.2.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.2.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.2.attentions.1.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.2.attentions.1.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.2.attentions.2.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.2.attentions.2.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.3.attentions.0.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.3.attentions.0.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.3.attentions.1.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.3.attentions.1.transformer_blocks.0.attn2.processor.to_v_ip.weight, up_blocks.3.attentions.2.transformer_blocks.0.attn2.processor.to_k_ip.weight, up_blocks.3.attentions.2.transformer_blocks.0.attn2.processor.to_v_ip.weight']

However, the assertion passes. Since that is the case, I think that's fine for now. Later, we can work on an unload_ip_adapter() method and document about it so that the users are aware of these things.

WDYT?

@yiyixuxu what's your take?

Chatted with @okotaku over Slack. We're okay closing this one in favor of #5713 (where @okotaku is already a co-author of the commits). Thanks very much for laying out the initial design, truly appreciated.