ZeroMSF: Zero-shot Monocular Scene Flow Estimation in the Wild

🏆 CVPR 2025 Oral & Best Paper Award Candidate

   

Yiqing Liang^1,2     Abhishek Badki^1,*     Hang Su^1,*     James Tompkin²     Orazio Gallo¹

* indicates equal contribution

^{1   2}

ZeroMSF is a model for monocular scene flow that jointly estimates geometry and motion in a zero-shot feedforward fashion. It leverages a joint geometry-motion estimation architecture and a scale-adaptive optimization strategy. The model exhibits strong generalization abilities, benefiting from a diverse training set compiled from six data sources.

Installation

Testing environment:

• Hardware: Single NVIDIA RTX 3090 GPU
• System: Ubuntu 24.04, CUDA 12.6

Pull code with all submodules:

git clone --recurse-submodules [email protected]:NVlabs/zero-msf.git

Install with conda:

# conda environment
conda create -y --name zero_msf python=3.10
conda activate zero_msf

# install pytorch
# here showing pytorch 2.6 and cuda 12.6
# see https://pytorch.org/get-started/locally/ for other versions
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu126
python -c "import torch; print(torch.cuda.is_available())" # make sure can see CUDA by printing True

# install most dependencies
pip install -r requirements.txt

# (optional) install curope
conda install -y -c conda-forge libstdcxx-ng
cd zmsf/mast3r/dust3r/croco/models/curope
python setup.py build_ext --inplace 
cd -

Demo

First, download model checkpoint from Google Drive and put it under checkpoints/.

Input sample should be provided as a folder containing two images. If more than two images are found within the input folder, only the first two are used. Some samples can be downloaded from Google Drive.

• Run demo:

model_config=zmsf_shift+dynamicreplica+vkitti+kubric+pointodyssey+spring_adap_241106

python demo.py \
     --model zmsf/configs/${model_config}.yaml \
     --data PATH_INPUT_FOLDER \
     --output PATH_OUTPUT_FOLDER

Estimated point clouds and scene flow can be found under PATH_OUTPUT_FOLDER.

• Visualize results with viser:

python visualize_viser.py \
     --steps 50 \
     --data PATH_OUTPUT_FOLDER

Citation

@InProceedings{liang2025zeroshot,
    author    = {Liang, Yiqing and Badki, Abhishek and Su, Hang and Tompkin, James and Gallo, Orazio},
    title     = {Zero-Shot Monocular Scene Flow Estimation in the Wild},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {21031-21044}
}