Dual-Interrelated Diffusion Model for Few-Shot Anomaly Image Generation(CVPR2025)

Ying Jin, Jinlong Peng^2#, Qingdong He^2#, Teng Hu³, Hao Chen¹, Haoxuan Wang¹,

Jiafu Wu², wenbing zhu¹, Mingmin Chi^1*, Jun Liu², Yabiao Wang^2,4

(#Equal contribution,*Corresponding author)

¹Fudan University, ²Youtu Lab, Tencent, ³Shanghai Jiao Tong University, ⁴Zhe Jiang University

Abstract

The performance of anomaly inspection in industrial manufacturing is constrained by the scarcity of anomaly data. To overcome this challenge, researchers have started employing anomaly generation approaches to augment the anomaly dataset. However, existing anomaly generation methods suffer from limited diversity in the generated anomalies and struggle to achieve a seamless blending of this anomaly with the original image. Moreover, the generated mask is usually not aligned with the generated anomaly. In this paper, we overcome these challenges from a new perspective, simultaneously generating a pair of the overall image and the corresponding anomaly part. We propose DualAnoDiff, a novel diffusion-based few-shot anomaly image generation model, which can generate diverse and realistic anomaly images by using a dual-interrelated diffusion model, where one of them is employed to generate the whole image while the other one generates the anomaly part. Moreover, we extract background and shape information to mitigate the distortion and blurriness phenomenon in few-shot image generation. Extensive experiments demonstrate the superiority of our proposed model over state-of-the-art methods in terms of diversity, realism and the accuracy of mask. Overall, our approach significantly improves the performance of downstream anomaly inspection tasks, including anomaly detection, anomaly localization, and anomaly classification tasks. Code will be made available.

✨Overview

Getting Started

Download stable-diffusion-v1-5.

git clone https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5

Environment Setup and Run.

pip -r requirments.txt
cd dual-interrelated_diff # or bcm-dual-interrelated_diff
sh run_mvtec_split.py
# run_mvtec_split.py includes operations for training, inference, and mask generation. Since our method involves training the model for a single category, it is necessary to modify the name in the run_mvtec_split.py file, which represents the category to be generated from Mvtec.

To see the usage and instructions for U2-Net, please refer to: U-2-Net

Detail steps

0. Environment:

accelerate==0.24.1
clip
Cython==0.29.35
matplotlib==3.8.0
numpy==1.24.3
open-clip-torch==2.23.0
opencv-python==4.7.0.72
opencv-python-headless==.7.0.72
pandas==2.0.3
Pillow==9.4.0
pytorch-lightning==1.5.0
PyYAML==6.0
scikit-image==0.22.0
scikit-learn==1.3.2
scipy==1.10.1
setuptools==65.6.3
tensorboard==.15.0
timm==0.4.12
torch==2.0.1+cu118
torchaudio==2.0.2+cu118
torchmetrics==0.6.0
torchvision==0.15.2+cu118
transformers==4.30.2

1. Download stable-diffusion-v1-5.

git clone https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5

2. Train

After running, the model weights will be saved in "all_generate". For different amounts of training data, you can choose the most appropriate steps by visualizing the generated results.

cd dual-interrelated_diff
sh train.sh

3. Infer

You can modify the "guidance_scale" to observe the generated results that are closer in color to the training data.

python inference_mvtec_split.py hazelnut hole

4. Get Mask

Referencing U-2-Net, use this model to segment the generated fg files.

Data and checkpoints

	url
Checkpoints for anomaly generation	url
Checkpoints for anomaly generation with BCM	url
1-shot generate images	url

Result

Comparison in Anomaly Generation

Anomaly Generation Quality

Anomaly Generation for Anomaly Detection and Localization

Comparison with Anomaly Detection Models

Citation

@article{jin2024dualanodiff,
  title={DualAnoDiff: Dual-Interrelated Diffusion Model for Few-Shot Anomaly Image Generation},
  author={Jin, Ying and Peng, Jinlong and He, Qingdong and Hu, Teng and Chen, Hao and Wu, Jiafu and Zhu, Wenbing and Chi, Mingmin and Liu, Jun and Wang, Yabiao and others},
  journal={arXiv preprint arXiv:2408.13509},
  year={2024}
}