S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs

Yuzhou Ji¹   Qijian Tian¹   He Zhu¹   Xiaoqi Jiang⁴   Guangzhi Cao⁴
Lizhuang Ma¹   Yuan Xie²   Xin Tan²³

¹Shanghai Jiao Tong University     ²East China Normal University
³Shanghai Artificial Intelligence Laboratory     ⁴Chery Automobile

⭐️ Overview

This release organizes the S2D pipeline as three stages:

• stage1_3dgs_init.py: initialize 3DGS from sparse input views and render novel-view artifacts
• stage2_artifact_fix.py: refine the stage-1 novel renders with the S2D artifact fixer
• stage3_3dgs_mixed_opt.py: continue mixed optimization with original input views and refined pseudo views

The repository depends on external components:

• Pi3: used for point-cloud guidance rendering
• gaussian-splatting: used for 3DGS optimization and rendering

📈 Repository Layout

S2D_Base/
├── stage1_3dgs_init.py
├── stage2_artifact_fix.py
├── stage3_3dgs_mixed_opt.py
├── train_s2d_artifact.py
├── requirements-train.txt
├── pi3_extensions/
│   ├── geometry_numpy.py
│   └── render_scene_pcd.py
├── s2d_artifact/
│   ├── __init__.py
│   ├── refiner.py
│   ├── autoencoder.py
│   └── mv_unet.py
├── s2d_training/
│   ├── args.py
│   ├── data.py
│   └── losses.py
├── training_configs/
│   └── multigpu.yaml
└── third_party/
    ├── Pi3
    └── gaussian-splatting

🚀 Quick Start

Clone the repo and initialize submodules:

git clone https://github.com/George-Attano/S2D_Base.git
cd S2D_Base
git submodule update --init --recursive

If you prefer to use your own external checkouts, the scripts also support environment overrides:

export S2D_GS_ROOT=/path/to/gaussian-splatting

For artifact fixer, create the env by:

conda create -n S2D python=3.10
pip install -r requirements-artifact.txt

Envs for Pi3 and 3DGS should be separately configured.

Stage 0: Preparing PC Guidance

Copy the following two files into the root of third_party/Pi3:

• pi3_extensions/geometry_numpy.py
• pi3_extensions/render_scene_pcd.py

Switch to the Pi3 virtual environment and run:

cd third_party/Pi3
python render_scene_pcd.py \
  --image-dir /path/to/images_ori \
  --output-dir /path/to/rendered_guidance \
  --weight-path /path/to/pi3.model.safetensors \
  --scene-path /path/to/tmp_output_scene.glb \
  --device cuda:0 \
  --keep-every 50

Supported training-view selection modes:

• --keep-every 50
• --keep-indices 0,50,130,160,210,308
• --keep-count 6

Stage 1: Sparse-View 3DGS Initialization

stage1_3dgs_init.py is a top-level S2D entry script. It automatically imports third_party/gaussian-splatting unless S2D_GS_ROOT is set.

Usage:

python stage1_3dgs_init.py \
  -s /path/to/colmap_or_3dgs_ready_scene \
  -m /path/to/output/scene_name \
  --sample_iteration 4000 \
  --stage1_meta_dir /path/to/output/stage1_meta \
  --stage1_novel_render_dir /path/to/output/stage1_novel_renders \
  --keep-every 50

Outputs:

• train_indices.txt
• novel_indices.txt
• image_names.txt
• chkpnt_stage1_<sample_iteration>.pth
• rendered novel-view artifacts in --stage1_novel_render_dir

Stage 2: S2D Artifact Fix

Usage (base-model-path can be local dir):

python stage2_artifact_fix.py \
  --base-model-path "nvidia/difix_ref" \
  --stage2-checkpoint s2d_artifact/S2D_pretrained_lora_75001.pkl \
  --stage1_meta_dir /path/to/output/stage1_meta \
  --input_image_dir /path/to/images_ori \
  --stage1_novel_render_dir /path/to/output/stage1_novel_renders \
  --point_guidance_dir /path/to/rendered_guidance \
  --fixed_dir /path/to/output/stage2_fixed \
  --fixed_mask_dir /path/to/output/stage2_masks \
  --device cuda

Outputs:

• refined pseudo views in --fixed_dir
• guidance masks in --fixed_mask_dir

Our finetuned weights at s2d_artifact/S2D_pretrained_lora_75001.pkl, used by default.

Stage 3: Mixed Optimization

Usage:

python stage3_3dgs_mixed_opt.py \
  -s /path/to/colmap_or_3dgs_ready_scene \
  -m /path/to/output/scene_name \
  --start_checkpoint /path/to/output/scene_name/chkpnt_stage1_4000.pth \
  --stage1_meta_dir /path/to/output/stage1_meta \
  --fixed_dir /path/to/output/stage2_fixed \
  --fixed_mask_dir /path/to/output/stage2_masks

🛠️ Training Artifact Fixer

train_s2d_artifact.py trains the S2D stage-2 artifact refiner with LoRA, which is used in stage2_artifact_fix.py.

Note: The DINO mixing is replaced with fixed mixing. There are no significant difference in quality while the configuration is simpler.

You should switch to S2D env. Expected dataset layout:

dataset_root/
├── train_A/
├── train_B/
├── train_P/
├── test_A/
├── test_B/
└── test_P/

Here:

• *_A stores the corrupted images with artifacts
• *_B stores the paired target clean images
• *_P stores the paired point-guidance images

Filenames must be aligned across the three folders. If filenames end with frame indices, the training code will sample the stage-2 reference image from nearby frames in the same *_B split.

Point guidance images can be created similar to stage 0. Artifact images can be rendered under sparse reconstruction and we also provide a minimal example script of adding random rotation to gaussians before rendering for more artifacts, see s2d_training/eg_3dgs_rotation_aug.py.

Before launching, update training_configs/multigpu.yaml so num_processes matches the number of GPUs you actually use.

Example command:

accelerate launch --config_file training_configs/multigpu.yaml train_s2d_artifact.py \
  --base-model-path "nvidia/difix_ref" \
  --output-dir /path/to/output/s2d_stage2 \
  --dataset-folder /path/to/dataset_root \
  --train-image-prep resize_512 \
  --test-image-prep resize_512 \
  --train-batch-size 4 \
  --num-training-epochs 20 \
  --max-train-steps 80000 \
  --checkpointing-steps 500 \
  --viz-freq 100 \
  --eval-freq 100 \
  --lambda-lpips 1.0 \
  --lambda-l2 6.0 \
  --lambda-ssim 4.0 \
  --enable-xformers-memory-efficient-attention

Training outputs:

• checkpoints in output_dir/checkpoints/
• validation visualizations in output_dir/eval/
• model_final.pkl as the final stage-2 checkpoint

Training notes:

• Set --output-dir and --dataset-folder to your actual paths.
• Set --train-batch-size according to available GPU memory.
• --checkpointing-steps controls how often checkpoints are saved.
• --viz-freq controls how often validation visualizations are dumped.
• If --max-train-steps is set to -1, it is derived automatically from num_training_epochs.
• The saved .pkl checkpoints are compatible with stage2_artifact_fix.py --stage2-checkpoint.

🙏 Acknowledgements

Our work is largely built upon the following projects:

• Pi3
• img2img-turbo
• Difix3D+
• DL3DV-10K and FCGS
• 3D Gaussian Splatting

📜 Citation

    @misc{ji2026s2d,
      title={S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs}, 
      author={Yuzhou Ji and Qijian Tian and He Zhu and Xiaoqi Jiang and Guangzhi Cao and Lizhuang Ma and Yuan Xie and Xin Tan},
      year={2026},
      eprint={2603.10893},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.10893}, 
    }