Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesis

by Jingyu Gong, Chong Zhang, Fengqi Liu, Ke Fan, Qianyu Zhou, Xin Tan, Zhizhong Zhang*, Yuan Xie

Introduction

This repository provides the implementation of our AAAI2026 paper Diffusion Implicit Policy for Unpaired Scene-aware Motion Synthesis.

Preparation

Installation

Please follow these instructions to set up your environment.

cd DIP
conda env create -f environment.yml
conda activate dip
python -m spacy download en_core_web_sm
pip install git+https://github.com/openai/CLIP.git

Body Models

Please download the SMPL-X body model and place it in the ./body_models/ folder.

Dataset

To help you get started more quickly, we have provided processed data here. The processed motion data is available here with password: z8mh. The processed Replica scene data is available here with password: hjqe. The processed PROX scene data is availabe here with password: bxx2. Random cluttered scenes generated by DIMOS is available here with password: 9s88. ShapeNet data is available here with password: u3bi.

(Recommanded) If you want to preprocess the data yourself or modify the data processing procedure, please download the original data from the following website.

We train our models on AMASS. Then, we evaluate our method on clutterd scenes from DIMOS+ShapeNet, PROX+PROX-S, and Replica. All datasets downloaded from the links should be placed under the project's dataset/ folder.

For textual annotation, please download the Babel and HumanML3D and place them in the ./dataset/amass/ folder.

Dataset Structure

project-folder/
└── dataset/
    ├── amass/
    ├── HumanML3D/
    ├── processed_datasets/
    ├── dimos_data/
    │   ├── replica/
    │   ├── proxs/
    │   ├── shapenet_real/
    │   └── scenes/
    │       └── random_scene_test/
    └── PROX_data/
        └── proxs/

Usage

Data Preprocessing

Please process the scene and training data separately. Run this script to preprocess the motion data:

sh shell_scripts/data_process_scripts/process_mdm_data.sh

Training

First, train the base diffusion model with the following script:

bash shell_scripts/train_scripts/train_action2motion.sh

Then, use the following script to train the ControlNet:

sh shell_scripts/train_scripts/train_action2motion_control.sh

Generation

You can run the following command for motion generation in scenes from DIMOS:

sh shell_scripts/generate_scripts/generate_for_eval.sh $ACTION

where ACTION is one of walk, sit, or lie.

We also provide a convenient script for motion generation in scenes from PROX and Replica with the following command:

sh shell_scripts/generate_scripts/generate_scene2motion.sh

Evaluation

You can evalute the generated motions in scenes from DIMOS using following command:

sh shell_scripts/evaluate_scripts/eval_metric.sh $ACTION

Acknowledgement

This code is based on MDM, OmniControl, SMPL-X, COINS, and DIMOS. If you find them useful, please consider citing them in your work.