Yi Wan

Founding research scientist, Pokee AI. Ex Meta AI.

My research focuses on fundamental challenges in reinforcement-learning-based intelligent systems that are crucial yet not well understood.

I obtained my Ph.D. in Computing Science at the University of Alberta, supervised by Professor Rich Sutton. I earned my Bachelor's degree in Electrical and Computer Engineering (ECE) from Shanghai Jiao Tong University (SJTU), where I worked in Professor Kai Yu's group. After that, I earned my Master's degree, also in ECE, from the University of Michigan.

Email: [email protected]

Google Scholar

Publications

An Empirical Study of Deep Reinforcement Learning in Continuing Tasks

Yi Wan*, Dmytro Korenkevych, Zheqing Zhu (2025),

Under Review.

Paper.

On Convergence of Average-Reward Q-learning in Weakly Communicating MDPs.

Yi Wan*, Huizhen Yu*, Richard S. Sutton (2024),

Under Review.

Paper.

Asynchronous Stochastic Approximation and Average-Reward Reinforcement Learning

Huizhen Yu, Yi Wan, Richard S. Sutton (2024),

Under Review.

Paper.

Reward Centering.

Abhishek Naik, Yi Wan, Manor Tomar, Richard S. Sutton (2024),

RLC.

Paper.

Loosely Consistent Emphatic Temporal-Difference Learning.

Jiamin He, Fengdi Che, Yi Wan, Rupam Mahmood (2023),

UAI

Paper.

The Emphatic Approach to Average-Reward Policy Evaluation.

Jiamin He, Yi Wan, Rupam Mahmood (2022),

In NeurIPS 2022 Workshop on DeepRL.

Paper.

Toward Discovering Options that Achieve Faster Planning.

Yi Wan, Richard S. Sutton (2022),

Abstract accepted by the Multi-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM).

Paper, Code.

Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods.

Yi Wan,* Ali Rahimi-Kalahroudi,* Janarthanan Rajendran, Ida Momennejad, Sarath Chandar, and Harm van Seijen (2022),

ICML.

Paper, Code, Talk, Slides, Poster.

Average-Reward Learning and Planning with Options.

Yi Wan, Abhishek Naik, and Richard S. Sutton (2021),

NeurIPS.

Paper, Talk, Slides, Poster.

Average-Reward Off-Policy Policy Evaluation with Function Approximation.

Shangtong Zhang,* Yi Wan,* Richard S. Sutton, and Shimon Whiteson (2021),

ICML.

Paper, Code, Talk, Slides.

Learning and Planning in Average-Reward Markov Decision Processes.

Yi Wan,* Abhishek Naik,* and Richard S. Sutton (2021),

ICML.

Paper, Code, Talk, Slides, Poster.

Planning with Expectation Models for Control.

Katya Kudashkina, Yi Wan, Abhishek Naik, and Richard S. Sutton (2021),

preprint.

Paper.

Off-policy Maximum Entropy Reinforcement Learning: Soft Actor-Critic with Advantage Weighted Mixture Policy (SAC-AWMP).

Zhimin Hou,* Kuangen Zhang,* Yi Wan, Dongyu Li, Chenglong Fu, Haoyong Yu (2020),

preprint.

Paper.

Planning with Expectation Models.

Yi Wan,* Muhammad Zaheer,* Adam White, Martha White, and Richard S. Sutton (2019),

IJCAI.

Paper, Talk, Slides.

Model-based Reinforcement Learning with Non-linear Expectation Models and Stochastic Environments.

Yi Wan,* Zaheer Abbas,* Martha White, and Richard S. Sutton (2018),

The Joint IJCAI/ECAI/AAMAS/ICML Conference Workshop on Prediction and Generative Modeling in Reinforcement Learning.

Paper, Slides.

Code

Pearl: A Production-ready Reinforcement Learning AI Agent Library

Zheqing Zhu, Rodrigo de Salvo Braz, Jalaj Bhandari, Daniel Jiang, Yi Wan, Yonathan Efroni, Ruiyang Xu, Liyuan Wang, Hongbo Guo, Alex Nikulkov, Dmytro Korenkevych, Urun Dogan, Frank Cheng, Zheng Wu, Wanqiao Xu (2023)

AlphaEx: A Python Toolkit for Managing a Large Number of Experiments

Yi Wan and Daniel Plop (2019)

Services

Moderator of the reinforcement learning mail list since 2023.

Journal Reviewer: TMLR (2022, 2023)

Conference Reviewer: NeurIPS (2021, 2022, 2023), ICML (2022), ICLR (2020, 2021, 2022), CoLLAs (2022, 2023), AAAI (2023)

Workshop Reviewer: Decision Aware RL workshop in ICML (2022), RL4RealLife workshop in ICML (2021), optimization for machine learning workshop in NeurIPS (2022).

Organizer: Deployable RL workshop at RLC 2024, Designing an RL system toward AGI social at ICML (2022), Continuing (Non-Episodic) RL problems social at ICML (2021).

Volunteer: ICML (2022) session moderator

Teaching

Reinforcement Learning II (COMPT 609) 2020, 2021, 2022 Teaching Assistant. Guest lecture: A Second Tutorial on Tabular TD(λ), Slides

Reinforcement Learning I (COMPT 366) 2018 Teaching Assistant

Skiing

Winter is long, skiing is fun.

Marmot Basin, Jasper, Canada

Photo from a video filmed by Shangtong Zhang

Blackcomb Glacier Ice Cave, Whistler, Canada