Interspeech 2023 Tutorial

• Interspeech 23 Resource Efficient and Cross-Modal Learning Toward Foundation Modeling Tutorial - Video

• ICASSP 22 Tutorial Neural Model Reprogramming and Prompting for Speech Modeling - Video | Slide

• ICASSP 23 Tutorial Parameter-Efficient Learning (PEL) for Speech and NLP: Adapters, Prompts, and Reprogramming - Slide

Part 1. Overview of Resource Efficient Learning, Dr. Huck Yang

9:00

1.1. Parameter-Efficient Learning

• Background of Frozen Model Adaptation
• Neural Adapter, Reprogramming, Prompting, and Low-Rank Adaptation (LoRA)

Title	Authors	Code	Year
Differentially Private Adapters for Parameter Efficient Acoustic Modeling	C.-W. Ho et al.	code	Interspeech 2023
Parameter-Efficient Learning for Text-to-Speech Accent Adaptation	L.-J. Yang et al.	code	Interspeech 2023
A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model	S. Radhakrishnan et al.	code	Interspeech 2023

1.2. Memory-Efficient Learning

• Reduce to GPU / TPU Memory During the Training (e.g., the Memory of Activation)
• Model Serialization
• Efficient On-Device Learning via Feature Reprogramming (CVPR 2022)
• Ladder-Side Tuning (NeurIPS 2022)

1.3 How to Estimate which Layer or which Model to Tune?

• Universal Approximation Theory (IEEE TIP 1993)
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning (ICML 2021)
• Latent Space Alignment in "Reprogramming Acoustic Models for Time Series Classification" (ICML 2021)

Title	Authors	Code	Year
How to Estimate Model Transferability of Pre-Trained Speech Models?	Z.-C. Chen et al.	code	Interspeech 2023

1.4 Advanced Low-Rank Adaptation (LoRA) Techniques

• Cross-Modal Merging
• Low-Rank Adaptation (LoRA) for Foundation Modeling and Pre-Training

1.5 Community Service

• Special Session in ICASSP 2024: In-Context Learning for Speech and Language Processing
• [email protected]

Break: Hand-on Session 1 (5 min)

• How to Train Your Whisper with Neural Adapter and LoRA

Part 2: Trustworthy AI and Cross-Modal Learning in the Era of Foundation Models, Dr. Pin-Yu Chen

11:00 to 11:45

Part 3: Multimodal Pre-Training for Automatic Speech Recognition and Vision Sharing, Dr. Shalini Ghosh

11:45 to 12:20

Spotlight Invited Talk, "Prompting LLM for ASR," by Dr. Chunyang Wu, Meta AI

12:20 to 12:30

QA and Plenary Discussion

12:40 to 12:45