OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

Cheng Luo¹ · Jianghui Wang¹ · Bing Li¹ · Siyang Song^2,3 · Bernard Ghanem¹

¹ IVUL Lab, KAUST · ² University of Exeter · ³ Awakening AI

📢 News

• Sep, 2025 — 🎉 Our paper OmniResponse has been accepted to NeurIPS 2025!

📝 TODO

• ✅ Release the OmniResponse paper
• ✅ Release the dataset
• ▢ Release the inference code
• ▢ Release the rendering code and pretrained models
• ▢ Provide model checkpoints for various LLM backbones
• ▢ Release the training code

📋 Table of Contents

🛠️ Installation

👨‍🏫 Getting Started

🧰 Data Preparation

🔧 External Tool Preparation

🚀 Training

🎯 Evaluation

🧪 Customized Inference

🖊️ Citation

If this work helps in your research, please cite the following papers:

@article{luo2025omniresponse,
  title   = {OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions},
  author  = {Luo, Cheng and Wang, Jianghui and Li, Bing and Song, Siyang and Ghanem, Bernard},
  journal = {arXiv preprint arXiv:2505.21724},
  year    = {2025}
}