TwinRL-VLA: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation

Qinwen Xu^1,*   Jiaming Liu^1,*,†   Rui Zhou^4,*   Shaojun Shi^1,*   Nuowei Han^1,*   Zhuoyang Liu¹   Chenyang Gu¹   Shuo Gu²   Yang Yue³   Gao Huang³   Wenzhao Zheng³   Sirui Han⁴   Peng Jia²   Shanghang Zhang^1,📧

¹Peking University   ²Simplexity Robotics   ³Tsinghua University   ⁴Hong Kong University of Science and Technology

_{^*Equal Contribution  ^†Project Lead  ^📧Corresponding Author}

---

Twin-RL is a digital twin-real-world collaborative RL framework designed to scale and guide exploration for VLA models.

📋 Table of Contents

• Repository Structure
• Environment Setup
• Offline Training
• Acknowledgments
• Citation

🗓 TODO

• Release digital twin assets & twin-generated datasets
• Release offline training code
• Release real-world RL training code (coming soon)

📁 Repository Structure

Twin-RL/
├── examples/
│   ├── train_offline.py                # Entry point for offline training
│   └── experiments/*/config.py         # Task configurations
├── scripts/
│   ├── dataset_process_scripts/        # Data conversion & preprocessing
│   └── visualization_scripts/          # Trajectory & camera video visualization
├── octo/                               # Foundational VLA architecture
├── third_party/
│   ├── agentlace/                      # Data & network communication
│   └── dlimp/                          # Dataloading & processing utilities
└── docs/
    ├── TwinRL_Scripts_Usage_Guide.md   # Scripts usage guide
    └── Offline_Training_Guide.md       # Offline training walkthrough

📚 Documentation & Guides:

• Scripts Usage Guide — Data processing and visualization examples for all scripts in scripts/.
• Offline Training Guide — Complete walkthrough for the offline training phase.
• Digital Twin Assets & Dataset — Twin assets, demonstration trajectories, and download instructions.

📦 Digital Twin Assets & Dataset

We release the high-fidelity digital twin assets and twin-generated trajectories used in this project, covering four manipulation tasks: Pick-and-Place, Insert-Hexagon-Block, Insert-Triple-Column-Block, and Erase-Whiteboard.

👉 Download from Google Drive

For detailed contents and task descriptions, see the Digital Twin Assets & Dataset Guide.

🛠 Environment Setup

1. Clone & Create Conda Environment

git clone https://github.com/zhourui9813/Twin-RL.git
cd Twin-RL

conda create -n twin-rl python=3.10
conda activate twin-rl

2. Install Dependencies

pip install -r requirements.txt

⚠️ Note: On-Demand JAX Installation

Because CUDA versions vary across machines, jax is intentionally omitted from requirements.txt. Install it manually based on your CUDA environment:

# Example: CUDA 11
pip install --upgrade "jax[cuda11_pip]==0.4.20" \
  -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

Replace cuda11_pip with the version that matches your CUDA setup.

3. Install Octo

cd octo && pip install -e . && cd ..

4. Install Third-Party Dependencies

cd third_party/agentlace && pip install -e .
cd ../dlimp && pip install -e .
cd ../..

🚀 Offline Training

Before conducting online real-world RL, the model requires an offline training phase using digital twin data.

We have prepared a comprehensive guide covering everything from downloading pre-trained weights and dataset preprocessing to launching your training experiments.

👉 Offline Training Guide

🙏 Acknowledgments

We thank the authors of Octo, HIL-SERL, ConRFT, Agentlace, Dlimp for sharing their codebase, which provided a solid foundation for our work.

📄 Citation

If you find our work helpful, please consider citing our paper:

@article{xu2026twinrl,
  title={TwinRL-VLA: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation},
  author={Xu, Qinwen and Liu, Jiaming and Zhou, Rui and Shi, Shaojun and Han, Nuowei and Liu, Zhuoyang and Gu, Chenyang and Gu, Shuo and Yue, Yang and Huang, Gao and others},
  journal={arXiv preprint arXiv:2602.09023},
  year={2026}
}