One World Seminar on Mathematics of Machine Learning

Abstract: Diffusion models have emerged as an expressive generative policy class for decision making, capable of representing rich distributions over actions and trajectories. Yet deploying these models in real-world sequential tasks raises two fundamental control questions: how to align a pretrained generative policy with task-specific rewards, and how to schedule inference when sample generation introduces non-negligible latency. In this talk, we present two principled approaches to these challenges. The first part develops a training-free reward steering framework based on the classical Doob $h$-transform. The method adapts a pre-trained model at inference time by modifying its sampling dynamics toward reward-preferred distributions, without requiring reward differentiability. We establish convergence guarantees for a prototypical version of the method and propose practical implementations that incur only modest additional sampling cost while achieving state-of-the-art empirical performance. The second part shifts from reward steering to meta-control of diffusion policies. Because diffusion policies generate actions through iterative denoising, inference latency becomes a central issue in real-time decision making. This makes inference timing critical for both task performance and computational efficiency. We formulate latency-aware inference scheduling as a semi-Markov decision process, in which a meta-controller decides when to invoke the diffusion policy while accounting for delayed execution and action buffering. We then develop a learning algorithm with theoretical guarantees. Together, these works suggest a broader view of diffusion models for decision making: they are not merely expressive policy parameterizations, but controllable components inside sequential decision-making systems.