ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs

Install

1. Our code builds on MoSca. Please follow MoSca installation to setup the environment.

2. Install additional dependencies

pip install -r requirements_vidar.txt

DyCheck training

We provide training on full and half resolution DyCheck scenes. In our benchmark we evaluate the methods on masked dynamic regions. Masks can be found here. We follow MoSca directory structure and place the files in data/iphone and data/iphone_full_res.

To speed up the training, consider using pregenerated sampled camera poses and corresponding masks found here.

The pipeline can be run as:

bash pipeline.sh

The steps involved:

1. MoSca reconstruction.
2. LoRA training on the input video.
3. Camera sampling (skipped if downloaded).
4. Sampled views generation.
5. Mask generation (skipped if downloaded).
   • For the best masks consider using Track Anything or similar on the sampled views.
6. Sampled views enhancement.
7. Pseudo-multi-view reconstruction.

Evalation

To run evaluation, use the following:

python vidar_evaluate.py --scene apple --input_dir data/iphone_full_res/apple --pred_dir data/iphone_full_res/apple/logs/iphone_fit_vidar/tto_test --output_dir output/vidar/apple

Citation

@inproceedings{nazarczuk2025vidar,
  title={{ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs}},
  author={Nazarczuk, Michal and Catley-Chandar, Sibi and Tanay, Thomas and Zhang, Zhensong and Slabaugh, Gregory and Pérez-Pellitero, Eduardo},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}