Sayan Deb Sarkar1 . Ondrej Miksik2 . Marc Pollefeys2, 3 . Dániel Béla Baráth3, 4 . Iro Armeni1
Computer Vision And Pattern Recognition (CVPR) 2025
1Stanford University · 2Microsoft Spatial AI Lab · 3ETH Zürich · 4HUN-REN SZTAKI
Multi-modal 3D object understanding has gained significant attention, yet current approaches often rely on rigid object-level modality alignment or assume complete data availability across all modalities. We present CrossOver, a novel framework for cross-modal 3D scene understanding via flexible, scene-level modality alignment. Unlike traditional methods that require paired data for every object instance, CrossOver learns a unified, modality-agnostic embedding space for scenes by aligning modalities - RGB images, point clouds, CAD models, floorplans, and text descriptions - without explicit object semantics. Leveraging dimensionality-specific encoders, a multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver supports robust scene retrieval and object localization, even with missing modalities. Evaluations on ScanNet and 3RScan datasets show its superior performance across diverse metrics, highlighting CrossOver’s adaptability for real-world applications in 3D scene understanding.
- Flexible Scene-Level Alignment 🌐 - Aligns RGB, point clouds, CAD, floorplans, and text at the scene level— no perfect data needed!
- Emergent Cross-Modal Behaviors 🤯 - Learns unseen modality pairs (e.g., floorplan ↔ text) without explicit pairwise training.
- Real-World Applications 🌍 AR/VR, robotics, construction—handles temporal changes (e.g., object rearrangement) effortlessly.
Table of Contents
-
Version 1.0 - CrossOver is now stronger than ever. We recommend updating to this version; changes include:
- More powerful pre-trained checkpoints; now available on Huggingface 👉 here.
- Support for 2 additional datasets - ARKitScenes & MultiScan
-
[2025-03] CrossOver is accepted to CVPR 2025 as Highlight. 🔥
-
[2025-02] Version 0.1 - We release CrossOver on arXiv with codebase + pre-trained checkpoints. Checkout our paper and website.
The code has been tested on:
Ubuntu: 22.04 LTS
Python: 3.9.20
CUDA: 12.1
GPU: GeForce RTX 4090/RTX 3090Clone the repo and setup as follows:
$ git clone [email protected]:GradientSpaces/CrossOver.git
$ cd CrossOver
$ conda env create -f req.yml
$ conda activate crossoverFurther installation for MinkowskiEngine, Pointnet2_PyTorch and GPU kNN (for I2P-MAE setup). Setup as follows:
$ git clone --recursive "https://github.com/EthenJ/MinkowskiEngine"
$ conda install openblas-devel -c anaconda
# Minkowski Engine
$ cd MinkowskiEngine/ && python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --force_cuda --blas=openblas
# Pointnet2_PyTorch
$ cd .. && git clone https://github.com/erikwijmans/Pointnet2_PyTorch.git
$ pip install pointnet2_ops_lib/.
# GPU kNN
$ cd .. && pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whlSince we use CUDA 12.1, we use the above
MinkowskiEnginefork; for other CUDA drivers, please refer to the official repo.
See DATA.MD for detailed instructions on data download, preparation and preprocessing. We list the data used in the current version of CrossOver in the table below:
| Dataset Name | Object Modality | Scene Modality | Object Temporal Information | Scene Temporal Information |
|---|---|---|---|---|
| Scannet | [point, rgb, cad, referral] |
[point, rgb, floorplan, referral] |
❌ | ✅ |
| 3RScan | [point, rgb, referral] |
[point, rgb, referral] |
✅ | ✅ |
| ARKitScenes | [point, rgb, referral] |
[point, rgb, referral] |
❌ | ✅ |
| MultiScan | [point, rgb, referral] |
[point, rgb, referral] |
❌ | ✅ |
To run our scene retrieval demo, you only need to download generated embedding data; no need for any data preprocessing. Running the instance retrieval demo for 3RScan, MultiScan and ARKitScenes requires generating 'gt-projection-seg.npz', containing framewise 2D instance segmentation, as described in our dataset preprocessing instructions.
This demo script allows users to process a custom object and run cross-modal retrieval to find the closest matched object within a target scene . Detailed usage can be found inside the script. Example usage below:
$ python demo/demo_instance_retrieval.pyVarious configurable parameters:
--query_path: Path to query object (point cloud, image, or text)--query_modality: Query modality - Options:point,rgb,referral--scan_id: Scene ID to search in--target_modality: Target modality to match against - Options:point,rgb,referral,cad--dataset: Dataset name - Options:scannet,scan3r,arkitscenes,multiscan--data_dir: Path to dataset directory--process_dir: Path to preprocessed features directory (for gt-projection-seg.npz)--ckpt: Path to pre-trained instance crossover model checkpoint (details here)--top_k: Number of top results to return
This demo script allows users to process a custom scene and retrieve the closest match from the supported datasets using different modalities. Detailed usage can be found inside the script. Example usage below:
$ python demo/demo_scene_retrieval.pyVarious configurable parameters:
--query_path: Path to the query scene file (eg:./example_data/dining_room/scene_cropped.ply).--database_path: Path to the precomputed embeddings of the database scenes downloaded before (eg:./release_data/embed_scannet.pt).--query_modality: Modality of the query scene, Options:point,rgb,floorplan,referral--database_modality: Modality used for retrieval. Same options as above.--ckpt: Path to the pre-trained scene crossover model checkpoint (details here), example_path:./checkpoints/scene_crossover_scannet+scan3r.pth/.
For embedding and pre-trained model download, refer to generated embedding data and checkpoints sections.
Tip
We also provide scripts for inference on a single scan of the supported datasets. Details in Single Inference section in TRAIN.md.
See TRAIN.md for the inventory of available checkpoints and detailed instructions on training and inference/evaluation with pre-trained checkpoints. The checkpoint inventory is listed below:
We provide all available checkpoints on huggingface 👉 here. Detailed descriptions in the table below:
| Description | Checkpoint Link |
|---|---|
| Instance Baseline trained on 3RScan | 3RScan |
| Instance Baseline trained on ScanNet | ScanNet |
| Instance Baseline trained on ScanNet + 3RScan | ScanNet+3RScan |
| Description | Checkpoint Link |
|---|---|
| Instance CrossOver trained on 3RScan | 3RScan |
| Instance CrossOver trained on ScanNet | ScanNet |
| Instance CrossOver trained on ScanNet + 3RScan | ScanNet+3RScan |
| Instance CrossOver trained on ScanNet + 3RScan + ARKitScenes + MultiScan | ScanNet+3RScan+ARKitScenes+MultiScan |
| Description | Checkpoint Link |
|---|---|
| Unified CrossOver trained on ScanNet + 3RScan | ScanNet+3RScan |
| Unified CrossOver trained on ScanNet + 3RScan + ARKitScenes + MultiScan | ScanNet+3RScan+ARKitScenes+MultiScan |
- Release evaluation on temporal instance matching
- Release inference on single scan cross-modal object retrieval
If you have any questions regarding this project, please use the github issue tracker or contact Sayan Deb Sarkar ([email protected]).
We thank the authors from 3D-VisTa, SceneVerse and SceneGraphLoc for open-sourcing their codebases.
@InProceedings{sarkar2025crossover,
author = {Sarkar, Sayan Deb and Miksik, Ondrej and Pollefeys, Marc and Barath, Daniel and Armeni, Iro},
title = {CrossOver: 3D Scene Cross-Modal Alignment},
booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
month = {June},
year = {2025},
pages = {8985-8994}
}