End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations

ICML 2024 (Spotlight)

Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, Qing li

Website	Arxiv

---

Abstract

Neuro-symbolic reinforcement learning (NS-RL) has emerged as a promising paradigm for explainable decision-making, characterized by the interpretability of symbolic policies. NS-RL entails structured state representations for tasks with visual observations, but previous methods are unable to refine the structured states with rewards due to a lack of efficiency. Accessibility also remains to be an issue, as extensive domain knowledge is required to interpret symbolic policies. In this paper, we present a framework for learning structured states and symbolic policies jointly, whose key idea is to distill vision foundation models into a scalable perception module and refines it during policy learning. Moreover, we design a pipeline to generate language explanations for policies and decisions using large language models. In experiments on nine Atari tasks, we verify the efficacy of our approach, and we also present explanations for policies and decisions.

The INSIGHT framework.

Results

Here is the segmentation videos before and after policy learing on Freeway:

The video before.

The video after.

---

Usage

Installation

# core dependencies
conda env create -f environment.yml
conda activate insight
bash ./scripts/install.sh
# download
bash ./scripts/download_ckpt.sh

Train INSIGHT

To generate dataset, use

bash scripts/dataset_generate.sh

To train cnn, use

bash scripts/train_cnn.sh

Or you can use a build-in dataset directly

To train policy, use

bash scripts/train_policy_atari.sh

To train metadrive, use

bash scripts/train_policy_metadrive.sh

Here is a report for INSIGHT:

Report

Citation

If you find our code implementation helpful for your own research or work, please cite our paper.

@article{luo2024insight,
  title={End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations},
  author={Luo, Lirui and Zhang, Guoxi and Xu, Hongming and Yang, Yaodong and Fang, Cong and Li, Qing},
  journal={ICML},
  year={2024}
}

Contact

For any queries, please raise an issue or contact Qing Li.

License

This project is open sourced under MIT License.