S2R-ViT for multi-agent cooperative perception: Bridging the gap from simulation to reality

This is the official implementation of ICRA 2042 paper. "S2R-ViT for multi-agent cooperative perception: Bridging the gap from simulation to reality". Jinlong Li, Runsheng Xu, Xinyu Liu, Baolu Li, Qin Zou, Jiaqi Ma, Hongkai Yu.

IEEE International Conference on Robotics and Automation (ICRA) 2024!

Project Page

S2R-UViT: Simulation-to-Reality Uncertainty-aware Vision Transformer

Data Download

We conduct experiments on two public benchmark datasets (OPV2V, V2V4Real) for the V2V cooperative perception task You can download these data from OPV2V and V2V4Real.

Getting Started

Environment Setup

S2R-ViT's codebase is build upon V2V4Real. this codebase supports both the simulation and real-world data and more perception tasks. To set up the codebase environment(following the V2V4Real), do the following steps:

1. Create conda environment (python >= 3.7)

conda create -n v2v4real python=3.7
conda activate v2v4real

2. Pytorch Installation (>= 1.12.0 Required)

Take pytorch 1.12.0 as an example:

conda install pytorch==1.12.0 torchvision==0.13.0 cudatoolkit=11.3 -c pytorch -c conda-forge

3. spconv 2.x Installation

pip install spconv-cu113

4. Install other dependencies

pip install -r requirements.txt
python setup.py develop

5.Install bbx nms calculation cuda version

python opencood/utils/setup.py build_ext --inplace

Train your model

OpenCOOD uses yaml file to configure all the parameters for training. To train your own model from scratch or a continued checkpoint, run the following commonds:

CUDA_VISIBLE_DEVICES=0 python opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER} --half]

Arguments Explanation:

• hypes_yaml: the path of the training configuration file, e.g. opencood/hypes_yaml/point_pillar_fax.yaml, meaning you want to train CoBEVT with pointpillar backbone. See Tutorial 1: Config System to learn more about the rules of the yaml files.
• model_dir (optional) : the path of the checkpoints. This is used to fine-tune the trained models. When the model_dir is given, the trainer will discard the hypes_yaml and load the config.yaml in the checkpoint folder.
• half (optional): If set, the model will be trained with half precision. It cannot be set with multi-gpu training togetger.
• S2R-ViT: training the S2R-UViT model: set the yaml file opencood\hypes_yaml\point_pillar_S2Rformer.yaml

To train on domain adaptation manner, run the following command:

CUDA_VISIBLE_DEVICES=0 python opencood/tools/train_da.py --hypes_yaml ${CONFIG_FILE}  --model_target  ${CHECKPOINT}   --model_source  ${CHECKPOINT}

• S2R-ViT: the path of the training configuration file, you can set the DA_trainin and da_training in the train_params of yaml file in the opencood\hypes_yaml\point_pillar_S2Rformer.yaml;
• S2R-ViT: model_target: the path to target domain model checkpoint (our final testing model for real data), at the beginning, you can set here as the checkpoint of the source data trained model like model_source;
• S2R-ViT: model_source the path to source domain model checkpoint (trained on simulation data (OPV2V));

Note that: opencood/tools/train_da.py is only for Adversarial Training of Domain Adaptation; S2R-UViT model can be directly used in the 3D object detection task

Test the model

Before you run the following command, first make sure the validation_dir in config.yaml under your checkpoint folder refers to the testing dataset path, e.g. v2v4real/test.

python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} [--show_vis] [--show_sequence]

Citation

@inproceedings{li2024s2r,
  title={S2r-vit for multi-agent cooperative perception: Bridging the gap from simulation to reality},
  author={Li, Jinlong and Xu, Runsheng and Liu, Xinyu and Li, Baolu and Zou, Qin and Ma, Jiaqi and Yu, Hongkai},
  booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={16374--16380},
  year={2024},
  organization={IEEE}
}

Acknowledgment

The codebase is build upon OPV2V and V2V4Real, which is the first Open Cooperative Detection framework for autonomous driving.