Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking

Pengxiang Li^1†, Shilin Yan^2†♠, Joey Tsai³, Renrui Zhang⁴, Ruichuan An⁵,
Ziyu Guo⁴, Xiaowei Gao^6‡

¹DLUT  ²Alibaba  ³Tsinghua  ⁴CUHK  ⁵PKU  ⁶ICL

^†Equal contribution   ^♠Project leader   ^‡Corresponding author

---

A-CFG is an adaptive version of Classifier-Free Guidance for diffusion-based language models. Instead of a static unconditional input, A-CFG dynamically re-masks low-confidence tokens at every denoising step, focusing guidance precisely where the model is uncertain.

✨ Key Features

• Plug-and-play guidance module for any masked diffusion language model (e.g. LLaDA, Dream).
• Token-level confidence heuristics with a single hyper-parameter ρ (remask ratio).

---

🚀 Quick Start

This project builds on LLaDA. See their README for more details on the base model setup.

LLaDA Inference

The LLaDA-8B-Base and LLaDA-8B-Instruct are uploaded in Huggingface. Please first install transformers==4.38.2 and employ the transformers to load.

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('GSAI-ML/LLaDA-8B-Base', trust_remote_code=True)
model = AutoModel.from_pretrained('GSAI-ML/LLaDA-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16)

📄 Cite

@article{li2025adaptive,
  title={Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking},
  author={Li, Pengxiang and Yan, Shilin and Tsai, Joey and Zhang, Renrui and An, Ruichuan and Guo, Ziyu and Gao, Xiaowei},
  journal={arXiv preprint arXiv:2505.20199},
  year={2025}
}