NeRF-VO: Real-Time Sparse Visual Odometry With Neural Radiance Fields

Jens Naumann · Binbin Xu · Stefan Leutenegger · Xingxing Zuo

IEEE Robotics and Automation Letters 2024

Paper | Video | Project Page

Table of Contents

1. Installation
2. Datasets
3. Execution
4. Citation
5. Contact

Installation

Tested on Ubuntu 20.04.

Clone repository

git clone https://github.com/jens-nau/NeRF-VO.git
git submodule update --init --recursive

Create conda environment

conda create -n nerf_vo python=3.8

PyTorch and PyTorch Scatter

pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118 
# Don't worry about the warning of the missing pkgs, we will install them later
pip install torch-scatter -f https://data.pyg.org/whl/torch-2.0.0+cu118.html

Requirements

pip install -r requirements.txt

Build dependencies

Build Tiny CUDA Neural Networks

cd nerf_vo/thirdparty/tiny_cuda_nn/bindings/torch
python setup.py install

Build Lietorch

cd ../../../../../
cd nerf_vo/thirdparty/lietorch
python setup.py install

Build DPVO

cd ../../../
cd nerf_vo/thirdparty/dpvo
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty
pip install .

Download pre-trained models

DPVO

cd ../../../
wget https://www.dropbox.com/s/nap0u8zslspdwm4/models.zip -P nerf_vo/build/dpvo/

Omnidata Depth Estimator

gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O nerf_vo/build/omnidata_models/

Omnidata Normal Estimator

gdown '1wNxVO4vVbDEMEpnAi_jwQObf2MFodcBR&confirm=t' -O nerf_vo/build/omnidata_models/

DROID-SLAM

gdown '1PpqVt1H4maBa_GbPJp4NwxRsd9jk-elh&confirm=t' -O nerf_vo/build/droid_slam/
# If gdown does not work, please manulaly download it from https://drive.google.com/file/d/1PpqVt1H4maBa_GbPJp4NwxRsd9jk-elh

Build NeRF-SLAM

Build Instant-NGP

cmake nerf_vo/thirdparty/nerf_slam/thirdparty/instant-ngp -B nerf_vo/build/instant_ngp
cmake --build nerf_vo/build/instant_ngp --config RelWithDebInfo -j8

Build GTSAM

cmake nerf_vo/thirdparty/nerf_slam/thirdparty/gtsam -DGTSAM_BUILD_PYTHON=1 -B nerf_vo/build/gtsam
cmake --build nerf_vo/build/gtsam --config RelWithDebInfo -j8

• Fix GTSAM build error by modifying nerf_vo/build/gtsam/python/linear.cpp (line 400):

.def(py::init<const gtsam::KeyVector&, const std::vector<gtsam::Matrix>&, const std::vector<gtsam::Vector>&, double>(), py::arg("js"), py::arg("Gs"), py::arg("gs"), py::arg("f"))

cd nerf_vo/build/gtsam
make python-install

Build NeRF-SLAM

cd ../../../
cd nerf_vo/thirdparty/nerf_slam
python setup.py install

Datasets

Replica

bash scripts/download_replica.sh 

ScanNet

python scripts/download_scannet.py

7-Scenes

bash scripts/download_7_scenes.sh

TUM-RGBD

bash scripts/download_tum_rgbd.sh 

Execution

cd ../../../
python run.py

Citation

If you find our code or paper useful, please cite:

@article{naumann2024nerfvo,
  author={Naumann, Jens and Xu, Binbin and Leutenegger, Stefan and Zuo, Xingxing},
  journal={IEEE Robotics and Automation Letters}, 
  title={NeRF-VO: Real-Time Sparse Visual Odometry With Neural Radiance Fields}, 
  year={2024},
}

Contact

Contact Jens Naumann and Xingxing Zuo for questions and comments.

Acknolegement

The implementation of NeRF-VO borrows many modules from nerfstudio, DPVO, DROID-SLAM, OmiDepth, Instant-NGP , NeRF-SLAM, etc., and we sincerely thank the contributors of them for their contributions to the community.