ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes

Zeyuan Chen*^1,2, Qiyang Yan*^1,2, Yuanpei Chen*^1,3
Tianhao Wu†^1,2, Jiyao Zhang†^1,2, Zihan Ding⁴, Jinzhou Li^1,2, Yaodong Yang^1,3, Hao Dong✉^1,2

¹CFCS, School of Computer Science, Peking University
²PKU-AgiBot Lab, ³PKU-PsiBot Lab, ⁴Princeton University

*: Equal Contribution, †: Project Lead, ✉: Corresponding Author

Conference on Robot Learning (CoRL) 2025 — Oral

[Website] • [arXiv] • [Video]

---

Dexterous grasping in cluttered scenes presents significant challenges due to diverse object geometries, occlusions, and potential collisions. Existing methods primarily focus on single-object grasping or grasp-pose prediction without interaction, which are insufficient for complex, cluttered scenes. Recent vision-language-action models offer a potential solution but require extensive real-world demonstrations, making them costly and difficult to scale. To address these limitations, we revisit the sim-to-real transfer pipeline and develop key techniques that enable zero-shot deployment in reality while maintaining robust generalization.

We propose ClutterDexGrasp, a two-stage teacher-student framework for closed-loop target-oriented dexterous grasping in cluttered scenes. The framework features a teacher policy trained in simulation using clutter density curriculum learning, incorporating both a novel geometry- and spatially-embedded scene representation and a comprehensive safety curriculum, enabling general, dynamic, and safe grasping behaviors. Through imitation learning, we distill the teacher's knowledge into a student 3D diffusion policy (DP3) that operates on partial point cloud observations.

To the best of our knowledge, this represents the first zero-shot sim-to-real closed-loop system for target-oriented dexterous grasping in cluttered scenes, demonstrating robust performance across diverse objects and layouts.

---

Table of Contents

1. Installation
2. Usage
3. Citation

---

Installation

---

Usage

---

Citation

Our paper is available on arXiv.
If you find our work useful, please consider citing:

@inproceedings{chen2025clutterdexgrasp,
  title     = {ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes},
  author    = {Chen, Zeyuan and Yan, Qiyang and Chen, Yuanpei and Wu, Tianhao and Zhang, Jiyao and Ding, Zihan and Li, Jinzhou and Yang, Yaodong and Dong, Hao},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2025}
}