Spectral RL: Spectral Representations for Reinforcement Learning

Spectral-RL is a comprehensive reinforcement learning library focused on leveraging the power of spectral representations for RL.

🎯 Key Features

• State-of-the-art Algorithms: Implementation of 4 major spectral representation learning methods;
• Multi-environment Support: Compatible with MuJoCo, DMControl, and MetaWorld tasks;
• Modular Design: Easy-to-use, extensible framework built on popular model-free RL algorithms;
• Research-Ready: Comprehensive evaluation tools and reproducible experiments.

📚 Supported Algorithms

Algorithm	Paper
LVRep	Latent Variable Representation
CTRL	Contrastive Representation
μLVRep	Multi-step Latent Variable Representation
Speder	Spectral Decomposition Representation
Diff-SR	Diffusion Spectral Representation

🚀 Installation

Prerequisites

• Python >= 3.7
• PyTorch >= 1.8.0
• CUDA (optional, for GPU acceleration)

Quick Start

# Clone the repository
git clone https://github.com/spectral-rl/spectral-rl.git
cd spectral-rl

# Install in development mode
pip install -e .

🔧 Usage

Directory Structure

examples/
├── config/           # Configuration files for experiments
├── main_state_dmc.py # DMControl with proprioceptive states
├── main_state_mujoco.py     # Gym-MuJoCo environments
└── main_visual.py    # MetaWorld and visual DMControl

Running Experiments

Basic Usage

python examples/main_state_dmc.py algo=<algorithm_name> task=<task_name>

📝 Citation

If you use this library in your research, please cite the relevant papers:

BibTeX

@inproceedings{ren2023latent,
  title={Latent Variable Representation for Reinforcement Learning},
  author={Ren, Tongzheng and Xiao, Chenjun and Zhang, Tianjun and Li, Na and Wang, Zhaoran and Schuurmans, Dale and Dai, Bo},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023}
}

@inproceedings{zhang2022making,
  title={Making linear mdps practical via contrastive representation learning},
  author={Zhang, Tianjun and Ren, Tongzheng and Yang, Mengjiao and Gonzalez, Joseph and Schuurmans, Dale and Dai, Bo},
  booktitle={International Conference on Machine Learning},
  pages={26447--26466},
  year={2022},
  organization={PMLR}
}

@inproceedings{zhang2024provable,
  title={Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning},
  author={Zhang, Hongming and Ren, Tongzheng and Xiao, Chenjun and Schuurmans, Dale and Dai, Bo},
  booktitle={International Conference on Machine Learning},
  pages={59759--59782},
  year={2024},
  organization={PMLR}
}

@inproceedings{ren2023spectral,
  title={Spectral Decomposition Representation for Reinforcement Learning},
  author={Ren, Tongzheng and Zhang, Tianjun and Lee, Lisa and Gonzalez, Joseph E and Schuurmans, Dale and Dai, Bo},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023}
}

@article{shribak2024diffusion,
  title={Diffusion Spectral Representation for Reinforcement Learning},
  author={Shribak, Dmitry and Gao, Chen-Xiao and Li, Yitong and Xiao, Chenjun and Dai, Bo},
  journal={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
  year={2024}
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.