Junpeng Jing, Ye Mao, Anlan Qiu, Krystian Mikolajczyk
An update of the previous project BiDAStereo, for stereo videos:
☑ Support BiDAStabilizer for image-based network, RAFTStereo and IGEVStereo, for video stereo matching.
☑ Support added video datasets: KITTI Depth, Infinigen SV, Southkensington SV.
☑ Support new metrics and losses.
☑ Support demo code for arbitrary stereo video as input.
Please see Link.
Installation with PyTorch3D, PyTorch 1.12.1 & cuda 11.3
git clone https://github.com/tomtomtommi/bidavideo
cd bidavideo
export PYTHONPATH=`(cd ../ && pwd)`:`pwd`:$PYTHONPATH
conda create -n bidavideo python=3.8
conda activate bidavideo
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
pip install -r requirements.txt
Demo code can be run using the following script:
sh demo.sh
In this script, you can specify which model (RAFTStereo, IGEVStereo with/without BiDAStabilizer; BiDAStereo) you want to use for demo.
Before running, download the checkpoints on google drive.
Copy the checkpoints to ./checkpoints/
To test on your own data, modify --path ./demo_video/. More arguments can be found and modified in demo.py
Download the following datasets and put in ./data/datasets:
The folder stucture of the datasets should look like:
/data/datasets/
├── Driving
├── FlyingThings3D
├── Monkaa
├── sintel_stereo
├── training
├── camdata_left
├── disparities
├── ...
├── clean_left
├── clean_right
├── dynamic_replica_data
├── train
├── test
├── real
├── InfinigenStereo
├── train
├── test
├── val
├── SouthKensington
├── Indoor
├── video001
├── ...
├── Outdoor
├── video001
├── ...
├── kitti_depth
├── gt_depth
├── 2011_09_26_drive_0001_sync
├── 2011_09_26_drive_0002_sync
├── ...
├── 2011_10_03_drive_0047_sync
├── input
│ ├── 2011_09_26
│ ├── 2011_09_28
│ ├── ...
│ ├── 2011_10_03
Download the checkpoints on google drive.
Copy the checkpoints to ./checkpoints/.
To evaluate BiDAStabilizer:
sh evaluate_bidastabilizer.sh
To evaluate BiDAStereo:
sh evaluate_bidastereo.sh
The results are evaluated on an A6000 48GB GPU.
To train BiDAStabilizer:
sh train_bidastabilizer.sh
--restore_ckpt is the fixed weights of image-based models. --ckpt_path is the logging path.
To train BiDAStereo:
sh train_bidastereo.sh
You can decrease image_size and / or sample_len if you don't have enough GPU memory.
The code is under MIT license.
SouthKensington SV dataset is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). Under this license, if you want to modify SK-SV or generate new data from SK-SV dataset (e.g., super-resolution, denoising, defocus), the releasing of your new data should be licensed under the same CC BY-NC-SA 4.0.
If you use BiDAStabilizer or BiDAStereo in your research, please use the following BibTeX entry.
@article{jing2024match,
title={Match stereo videos via bidirectional alignment},
author={Jing, Junpeng and Mao, Ye and Qiu, Anlan and Mikolajczyk, Krystian},
journal={arXiv preprint arXiv:2409.20283},
year={2024}
}
@inproceedings{jing2024match-stereo-videos,
title={Match-stereo-videos: Bidirectional alignment for consistent dynamic stereo matching},
author={Jing, Junpeng and Mao, Ye and Mikolajczyk, Krystian},
booktitle={European Conference on Computer Vision},
pages={415--432},
year={2024},
organization={Springer}
}