Skip to content

HoAdrian/ICRA2025_lidar_edit

BranchesTags

Repository files navigation

LIDAR-EDIT: Synthetic Generation of Faithful and Controllable Lidar Data for Autonomous Driving via Editing Real-World Scenes

Authors: Shing-Hei Ho, Bao Thach, Minghan Zhu

Accepted at ICRA 2025

Project Website

Interactive demo

Our data and model checkpoints

Arxiv link

File structure

Lidar_generation/
├── configs/
│   ├── nuscenes_config.py (configuration containing parameters for voxelization and transformer models' hyperparameters)
├── models/
│   ├── quantizer.py (for VQ-VAE)
│   ├── transformers.py (encoder, decoder, bidirectional transformer for MaskGIT)
│   ├── vqvae_transformers.py (VQ-VAE and MaskGIT)
├── train_transformer_models/
│   ├── train_vqvae_transformer.py
│   ├── test_vqvae_transformer.py
│   ├── train_maskgit_transformer.py
│   ├── test_maskgit_transformer.py
├── datasets/
│   ├── data_utils.py (general utilities for data processing and visualization)
│   ├── data_utils_nuscenes.py (utilities for NuScenes dataset)
│   ├── dataset_nuscenes.py (contains datasets for training, inpainting evaluation, foreground object extraction for point cloud completion and foreground object insertion)
│   ├── dataset.py (voxelizer and a general dataset class shared by all types of datasets in dataset_nuscenes.py)
├── evaluation/
│   ├── histogram_evaluate_allocentric_compare.py (evaluate in-painting performance via statistical metrics)
│   └── save_pc_for_feat_diff.py (save the in-painted backgrounds from different baselines)
├── actor_insertion/
│   ├── collect_foreground_objects.py (collect partial foreground vehicle (car,bus,truck) point clouds for point cloud completion)
│   ├── generate_insert_obj_dict_baselines.py (insert dense foreground vehicles for training (perturbed poses) and evaluation (inserted at original poses) of 3D detection models)
├── installation
│   ├── ultralidar.sh
│   ├── mmdet3d.sh
├── other_repo
│   ├── AnchorFormer
│   │      └── ...
│   └── OpenPCDet_minghan
│           └── ...
├── removal.sh
├── evaluation.sh
└── insertion.sh
  • other_repo

    • AnchorFormer: Some scripts are customized to run point cloud completion on our extracted vehicles from NuScenes.

    • OpenPCDet_minghan: 3D detection model's dataset creation, training and evaluation. We use VoxelNext trained on single-sweep lidar with intensity values (reflectance) set to zero.

Installation

Setting up NuScenes dataset file structure

└── nuscenes  
    ├── Usual nuscenes folders (i.e. samples, sweep)
    │
    ├── lidarseg
    │   └── v1.0-{mini, test, trainval} <- Contains the .bin files; a .bin file 
    │                                      contains the labels of the points in a 
    │                                      point cloud (note that v1.0-test does not 
    │                                      have any .bin files associated with it)
    │
    ├── panoptic
    │   └── v1.0-{mini, test, trainval} <- Contains the *_panoptic.npz files; a .npz file 
    │                                      contains the panoptic labels of the points in a 
    │                                      point cloud (note that v1.0-test does not 
    │                                      have any .npz files associated with it) 
    └── v1.0-{mini, test, trainval}
        ├── Usual files (e.g. attribute.json, calibrated_sensor.json etc.)
        ├── lidarseg.json  <- contains the mapping of each .bin file to the token
        ├── panoptic.json  <- contains the mapping of each .npz file to the token       
        └── category.json  <- contains the categories of the labels (note that the 
                              category.json from nuScenes v1.0 is overwritten)
  • It is recommended to put NuScenes data at /mnt and create a symbolic link to it at arbitrary location (e.g. /home/shinghei/OpenPCDet/data) to use it.
  • The bash scripts (e.g. removal.sh) in this repo have a variable dataset_path, which is where I put the dataset. "version" specifies whether I am using v1.0-trainval or v1.0-mini. v1.0-mini is for quick testing and debugging. Use v1.0-trainval. You can customize dataset_path yourself.

OpenPCDet file structure

  • OpenPCDet assume the following file structure for NuScenes data.
OpenPCDet
├── data
│   ├── nuscenes
│   │   │── v1.0-trainval (or v1.0-mini if you use mini)
│   │   │   │── samples
│   │   │   │── sweeps
│   │   │   │── maps
│   │   │   │── v1.0-trainval

Assume that we are at the root directory of this repo.

1. Background in-painting training and evaluation

  • Train VQ-VAE and MaskGIT for background in-painting by running train_vqvae_transformer.py and train_maskgit_transformer.py
  • Visualize outputs from VQ-VAE and MaskGIT by running test_vqvae_transformer.py and test_maskgit_transformer.py
conda activate ultralidar
./removal.sh
  • evaluation
conda activate ultralidar
./evaluation.sh

2. Object Library Construction

Collect partial foreground objects' point clouds

  • You can use our pre-collected foreground object vehicles' partial point clouds and their dense point clouds here (TODO). If you do so, skip this section.
  • Collect foreground vehicles' point clouds from NuScenes lidar scans
conda activate ultralidar

# run this line that is in insertion.sh:
python actor_insertion/collect_foreground_objects.py --trainval_data_path=$dataset_path --data_version=$version --pc_path="./foreground_object_pointclouds" --visualize_dense=0 --save_as_pcd=1

# REMEMBER to remove all previously collected foreground objects first. otherwise, there would be 
# inconsistencies between the dictionaries and the point clouds we save.
  • After running the code above, ./foreground_object_pointclouds should have the following structure:
foreground_object_pointclouds
├── car
│    └── point clouds of cars ... each pointcloud has a name of the form "sample_<i>.pcd"
├── bus
│    └── poitn clouds of buses ... each pointcloud has a name of the form "sample_<i>.pcd"
├── truck
│    └── point clouds of trucks ... each pointcloud has a name of the form "sample_<i>.pcd"
├── allocentric_dict.pickle
├── sample_dict.pickle

# - let's say we have the following class_names for vehicles: {"car", "truck", "bus"}
# - each point cloud we collected is uniquely identified by (class_name, pc_name) e.g. ("car", "sample_14.pcd")
# - allocentric_dict.pickle: a dictionary mapping from class_name to 2D list. Each row of the 2D list has the same length, each entry i of each row corresponds to the same object point cloud and the entry is a property of that object (e.g. pc_name, bounding box, allocentric angle, etc.)
# - sample_dict.pickle: a dictionary mapping from (class_name, pc_name) to the 2D list same as above.

Point cloud completion

  • Run AnchorFormer: run point cloud completion and save the resultant dense point clouds of collected foreground vehicles
    • Before running the following commands, make sure to rename some variables in:
      • AnchorFormer/datasets/customDataset.py: rename the variable pc_root in the _get_file_list method as the path to foreground_object_pointclouds
      • AnchorFormer/tools/runner.py: rename the variable save_root in the test method as the path to foreground_object_pointclouds
cd AnchorFormer # go to where the AnchorFormer directory is
conda activate pointTr
CUDA_VISIBLE_DEVICES=0 python main.py --ckpts anchorformer_dict.pth --config ./cfgs/custom_models/AnchorFormer.yaml --exp_name test_ckpt --test
  • The directory dense_nusc storing the corresponding dense point clouds should appear in the directory foreground_object_pointclouds

3. Generate synthetic LiDAR scenes via LiDAR Editing

Preparing synthetic dataset

  • You can uncomment certain lines in actor_insertion/nuscenes_utils.py and actor_insertion/generate_insert_obj_dict_baselines.py to visualize the original, inpainted and edited point clouds.
  • Run the following code in ./insertion.sh to collect synthetic LiDAR scenes:
    • Modify the following variables:
      • root in insertion.sh: where you save the data
      • For generating synthetic LiDAR scenes based on the NuScenes validation set:
        • set split="val" in insertion.sh and use the method insertion_vehicles_driver in generate_insert_obj_dict_baselines
      • For generating synthetic LiDAR scenes based on the NuScenes training set:
        • set split="train" and use the method insertion_vehicles_driver_perturbed in generate_insert_obj_dict_baselines
python actor_insertion/generate_insert_obj_dict_baselines.py --trainval_data_path=$dataset_path --data_version=$version --pc_path="./foreground_object_pointclouds" --dense=$dense --vqvae_path="$vqvae_path/epoch_$vqvae_epoch" --maskgit_path="$maskgit_path/epoch_$maskgit_epoch" --save_lidar_path=$root --split=$split --intensity_model_path="$intensity_model_path/epoch_$intensity_vqvae_epoch"

Data format

  • The generated data should be saved inside <path to OpenPCDet repo>/data/nuscenes/v1.0-trainval/ours. In my code, the path is /home/shinghei/lidar_generation/OpenPCDet_minghan/data/nuscenes/v1.0-trainval/ours.
  • The file structure of ours is
ours
├── v1.0-trainval
│    ├── lidar_point_clouds
│           ├── a bunch of .pcd files which are synthetic LiDAR point clouds .......
│    ├── token2sample.pickle
  • token2sample.pickle is a dictionary mapping from the lidar sample token to
(full_path_to_synthetic_lidar, bounding_boxes_in_synthetic_lidar, list_of_annotation_tokens_in_synthetic_lidar, sample_records, list_of_annotation_info_in_synthetic_lidar)
  • It is generated according to the following procedure in actor_insertion/insertion_utils.py:
points_xyz = new_scene_points_xyz
bounding_boxes = new_bboxes
lidar_sample_token = dataset.lidar_sample_token

sample_records = dataset.obj_properties[12]
cs_record, sensor_record, pose_record = sample_records["cs_record"], sample_records["sensor_record"], sample_records["pose_record"]

############# save point cloud 
pc_name = f'{args.split}_{lidar_sample_token}.bin'
os.makedirs(os.path.join(save_lidar_path, "lidar_point_clouds"), exist_ok=True)
lidar_full_path = os.path.join(save_lidar_path, "lidar_point_clouds", pc_name)
#assert(not os.path.exists(lidar_full_path))
new_points_xyz.astype(np.float32).tofile(lidar_full_path)

############## Save the data needed to build the new database
token2sample_dict[lidar_sample_token] = (lidar_full_path, bounding_boxes+other_foreground_boxes, new_obj_ann_token_list+other_foreground_obj_ann_token_list, sample_records, new_ann_info_list+other_foreground_obj_ann_info_list)
  • each lidar_sample_token is data_token in the get_path_infos method of datasets/dataset_nuscenes.py:
for sample in nusc.sample:
        #### This is a sample token from visualization of eval of voxel nexr in OpenPCNet
        # if sample['token']!='5d773ca713f54023b34cd4718a5ee293':
        #     continue
        scene_token = sample['scene_token']
        data_token = sample['data']['LIDAR_TOP']

4. 3D object detection training and evaluation

  • run_openpcdet.sh contains example commands I ran

Synthetic and NuScenes Datasets

  • Files customized for synthetic data
dataset class: <path to OpenPcDet>/pcdet/datasets/nuscenes/custom_nuscenes_dataset.py
dataset configuration: <path to OpenPcDet>/tools/cfgs/dataset_configs/custom_nuscenes_dataset.yaml
voxelnext configuration: <path to OpenPcDet>/tools/cfgs/nuscenes_models/custom_cbgs_voxel0075_voxelnext.yaml
  • Files customized for NuScenes data
dataset class: <path to OpenPcDet>/pcdet/datasets/nuscenes/nuscenes_dataset.py
dataset configuration: <path to OpenPcDet>/tools/cfgs/dataset_configs/nuscenes_dataset.yaml
voxelnext configuration: <path to OpenPcDet>/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml

Data processing for OpenPCDet framework

  • Before training or evaluation, you need to generate processed data for the OpenPCDet framework and model to consume. For example, you may want to obtain:

    • a concatenation of processed sythetic and NuScenes data
    • processed synthetic data
    • processed NuScenes data

  • We assume that the synthetic data is in <path to OpenPCDet repo>/data/nuscenes/v1.0-trainval/ours. Rename the directory as ours if necessary.

  • You may need to modify the following variables before you process training and evaluation data for each experiment:

    • In <path to OpenPCDet>/pcdet/datasets/nuscenes/nuscenes_utils.py:
      • root: the path to the directory where our synthetic dataset ours is located
    • In <path to OpenPCDet>/pcdet/datasets/nuscenes/custom_nuscenes_dataset.py:
      • my_root: same as root
      • TRAIN_WITH_MY_AUGMENT: True for concatenating synethtic dataset with the original NuScenes dataset. Otherwise, only get the synthetic data.
    • For example, root = "/home/shinghei/lidar_generation/OpenPCDet_minghan/data/nuscenes/v1.0-trainval"

  • To generate processed synthetic data or concatneation of synthetic and NuScenes data, run these commands at <path to OpenPCDet>

VERSION=v1.0-trainval

############ export the cloned nuscenes-devkit repo path
export PYTHONPATH=/home/shinghei/lidar_generation/nuscenes-devkit/python-sdk:$PYTHONPATH

python -m pcdet.datasets.nuscenes.custom_nuscenes_dataset --func create_nuscenes_infos --cfg_file tools/cfgs/dataset_configs/custom_nuscenes_dataset.yaml --version $VERSION

  • Some files and directories should be generated at <path to OpenPCDet>/data/nuscenes/v1.0-trainval containing the processed data. When generating another processed data, you should move the existing processed data to another directory to avoid overwritting them.

  • To generate processed NuScenes data, run these commands at the <path to OpenPCDet>

VERSION=v1.0-trainval

############ export the cloned nuscenes-devkit repo path
export PYTHONPATH=/home/shinghei/lidar_generation/nuscenes-devkit/python-sdk:$PYTHONPATH

python -m pcdet.datasets.nuscenes.nuscenes_dataset --func create_nuscenes_infos --cfg_file tools/cfgs/dataset_configs/nuscenes_dataset.yaml --version $VERSION
  • The code above use token2sample_dict; for each lidar scan (1-1 correspondence to lidar sample token), the full path to its corresponding synthetic lidar scan is not a relative path. If you are using some pre-generated synthetic data not generated by yourself, make sure each full path is correct. If necessary, please modify the full path when processing data in fill_trainval_infos in nuscenes_utils.py. Use ctrl-F to search for "#### TODO: adjust the lidar path if necessary:" in that file, and uncomment and modify that part accordingly

Training

CONFIG_FILE=<voxel next configuration file>
CHECKPOINT_FILE=<the path to a model checkpoint>

##### if you want to initialize the model as the checkpoint model, reset the optimizer state and learning rate scheduler state and then train
python tools/train.py --pretrained_model ${CHECKPOINT_FILE} --cfg_file ${CONFIG_FILE}

##### if you want to train the model from scratch
python tools/train.py --cfg_file ${CONFIG_FILE}

##### if you want to resume model training from the checkpoint model (load model weight, resume optimizer state, learning rate scheduler state and current epoch)
python tools/train.py --ckpt ${CHECKPOINT_FILE} --cfg_file ${CONFIG_FILE}
  • You should see training log, tensorboard logging and model weights saved inside directories in <path to OpenPCDet>/output.

Evaluation

CONFIG_FILE=<voxel next configuration file>
CHECKPOINT_FILE=<the path to a model checkpoint>

python tools/test.py --cfg_file ${CONFIG_FILE} --batch_size 1 --ckpt ${CHECKPOINT_FILE}
  • You should see the evaluated performance printed on the terminal and possibly some visualization saved inside directories in <path to OpenPCDet>/output.

License

MIT License

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

8 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages