MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

[SIGGRAPH 2024 CONFERENCE PROCEEDINGS]

👉 MotionCtrl for [SVD], for [VideoCrafter], for [AnimateDiff]

   

🤗   🤗

---

---

🔥🔥 We release the codes, models MotionCtrl on AnimateDiff.

Official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.

MotionCtrl can Independently control complex camera motion and object motion of generated videos, with only a unified model.

Results of MotionCtrl+AnimateDiff

⚙️ Environment

conda create -n motionctrl python=3.10.6
conda activate motionctrl
pip install -r requirements.txt

💫 Inference

1. Preparing Models

• motionctrl_animatediff_cmcm.ckpt -> ./models/MotionCtrl.
• motionctrl_animatediff_omcm.ckpt -> ./models/MotionCtrl.
• v3_sd15_mm.ckpt.ckpt -> models/Motion_Module
• stable-diffusion-v1-5 -> models/StableDiffusion/stable-diffusion-v1-5
• (Options) LoRAs described in download_basescripts/

2. Running scripts:

   • CMCM: sh scripts/test_cmcm.sh
   • OMCM: sh scripts/test_omcm.sh

🔥🔥 Training

1. Preparing Pre-Trained Models

• v3_sd15_mm.ckpt.ckpt -> models/Motion_Module
• stable-diffusion-v1-5 -> models/StableDiffusion/stable-diffusion-v1-5

2. Preparing Dataset

• RealEstate10K

  1. Following https://github.com/cashiwamochi/RealEstate10K_Downloader to download and process the videos.
  2. Corresponding Captions and List are provided in GoogleDrive.

• WebVid with Object Trajectories

  1. Preparing ParticleSfM. Our experiments is running on CentOS 8.5 and we provide a detailed install note here.

  2. Moving prepare_webvideo_len32.py and run_particlesfm_obj_traj.py to ParticleSfM project.

  3. Step 1: Prepare sub-videos with lenth of 32 and size of 256 x 256.

     ## start_idx and end_idx is used to process a subset of the dataset in different machines parallelly
     
     python prepare_webvideo_len32.py --start_idx 0 --end_idx 1000
     

  4. Step 2: Get object trajectories

       root_dir="WebVid/train_256_32"
       start_idx=0
       end_idx=1000
     
       CUDA_VISIBLE_DEVICES=0 python run_particlesfm_obj_traj.py \
       --root_dir $root_dir \
       --start_idx $start_idx \
       --end_idx $end_idx \
     

• You can customize object Trajectories with our provided HandyTrajDrawer.

3. Training

   • Train Camera Motion Control: sh scripts/train_cmcm.sh

   • Train Object Motion Control:

     • Dense: sh scripts/train_omcm_dense.sh
     • Sparse: sh scripts/train_omcm_sparse.sh

📚 Citation

If you make use of our work, please cite our paper.

@inproceedings{wang2024motionctrl,
  title={Motionctrl: A unified and flexible motion controller for video generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Li, Yaowei and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Ying},
  booktitle={ACM SIGGRAPH 2024 Conference Papers},
  pages={1--11},
  year={2024}
}

🤗 Acknowledgment

The current version of MotionCtrl is built on AnimateDiff. We appreciate the authors for sharing their awesome codebase.

❓ Contact

For any question, feel free to email [email protected] or [email protected].