Skip to content
ZhaoJingjing713Public

About

[CVPR2026] Long-horizon, spatially consistent video generation enabled by persistent 3D scene point clouds and dynamic-static disentanglement.

Resources

Stars

238 stars

Watchers

11 watching

Forks

Latest commit

 

History

12 Commits

Folders and files

Repository files navigation

Spatia: Video Generation with Updatable Spatial Memory
CVPR 2026

Long-horizon, spatially consistent video generation enabled by persistent 3D scene point clouds and dynamic-static disentanglement.

1The University of Sydney   2Microsoft Research   3HKUST   4University of Waterloo
*Equal Contribution

arXiv   Project Page   Hugging Face


Clone This Repo

This repository has multiple branches. The page branch contains the project website and many large asset files, so for inference usage you should clone only the main branch:

git clone --single-branch --branch main https://github.com/ZhaoJingjing713/Spatia.git
cd Spatia

If you have already cloned the full repository and want to avoid fetching other branches later, keep working on main and avoid checking out page.


Overview

This release provides the inference pipeline for Spatia, built on top of:

  • Wan2.2-TI2V-5B as the base video generator
  • a control / VACE checkpoint for spatial guidance
  • a LoRA checkpoint for autoregressive long-horizon generation

Main entry point:

python inference.py

At a high level, the pipeline:

  1. Reconstructs and updates a 3D scene point cloud with MapAnything.
  2. Renders point-guided control videos from target camera trajectories.
  3. Generates spatially consistent clips with Wan2.2.
  4. Reuses history frames and matched reference views for long-horizon rollout.

Repository Layout

.
├── inference.py
├── install.sh
├── download.sh
├── requirements.txt
├── test_cases/
├── utils/
└── wan/

Key files:

  • inference.py: end-to-end inference entry
  • install.sh: environment setup script
  • download.sh: base Wan2.2 checkpoint download helper
  • test_cases/: example prompts, intrinsics, and camera trajectories

Installation

1. Create the environment

First create and activate a Conda environment with Python 3.12:

conda create -n spatia python=3.12 -y
conda activate spatia

Then run the install script. It supports both cuda and rocm.

bash install.sh cuda

or

bash install.sh rocm

You can also specify a custom build directory:

bash install.sh cuda ./env

The script installs:

  • PyTorch
  • FlashAttention
  • DiffSynth-Studio
  • PyTorch3D
  • MapAnything
  • Python dependencies from requirements.txt

2. Download the base Wan2.2 model

Download Wan2.2-TI2V-5B to model_weights/Wan2.2-TI2V-5B:

bash download.sh

Equivalent manual command:

pip install -U "huggingface_hub[cli]<1.0.0"
hf download Wan-AI/Wan2.2-TI2V-5B --local-dir ./model_weights/Wan2.2-TI2V-5B

3. Download Spatia weights

Download the Spatia checkpoints from Hugging Face:

hf download Jinjing713/Spatia --local-dir ./checkpoints/Spatia

You need:

  • one control / VACE checkpoint
  • one LoRA checkpoint

Then point --vace_path and --lora_path to the downloaded files.

Example:

python inference.py \
  --vace_path ./checkpoints/Spatia/step-8500.safetensors \
  --lora_path ./checkpoints/Spatia/lora_weights_10000.safetensors

If the filenames in your local download differ, just update the paths accordingly.


Input Format

Each run requires:

  • one starting image via --img_path
  • one camera trajectory file via --camera_w2c_path
  • one intrinsics file via --camera_intrinsics_path
  • one prompt or prompt file

w2c.txt

w2c.txt stores one camera extrinsic per line in JSON-style list format.

  • Coordinate convention: OpenCV camera coordinates
  • Matrix type: world-to-camera
  • Supported shapes per line:
    • 3x4
    • homogeneous 4x4

Examples:

[[r11, r12, r13, t1], [r21, r22, r23, t2], [r31, r32, r33, t3]]
[[r11, r12, r13, t1], [r21, r22, r23, t2], [r31, r32, r33, t3], [0, 0, 0, 1]]

intrinsics.txt

intrinsics.txt stores normalized intrinsics:

[fx fy cx cy]

Here:

  • fx, cx are normalized by image width
  • fy, cy are normalized by image height

Pixel-space intrinsics are reconstructed internally using runtime --width and --height.

--mask_path

--mask_path specifies foreground object masks for dynamic/static disentanglement and controllable generation.

The foreground mask marks dynamic or editable foreground regions in the input image or video. It is used during spatial memory construction so the model can better separate:

  • static scene structure
  • foreground dynamic content

Supported formats:

  • one or more binary mask images
  • one .npy file with shape [N, H, W]

Typical usage:

  • a single foreground mask for the first frame
  • frame-aligned masks for a full sequence

Notes:

  • Image masks should be binary foreground masks.
  • For .npy, each slice should be one binary mask frame.
  • If the masks are frame-aligned, use --per_frame_mask.

prompt.txt

prompt.txt can contain:

  • a single prompt line, reused for all rounds
  • multiple prompt lines, one prompt per round

Quick Start

Minimal example:

python inference.py \
  --img_path path/to/input.jpg \
  --camera_w2c_path test_cases/case_2/w2c.txt \
  --camera_intrinsics_path test_cases/case_2/intrinsics.txt \
  --prompt_path test_cases/case_2/prompt.txt \
  --save_path test_cases/case_2/output.mp4 \
  --vace_path ./checkpoints/Spatia/step-8500.safetensors \
  --lora_path ./checkpoints/Spatia/lora_weights_10000.safetensors

CLI Arguments

Core paths

Argument Default Meaning
--img_path test_cases/case_2/img.jpg Starting image for the first frame.
--camera_w2c_path test_cases/case_2/w2c.txt Camera trajectory in OpenCV world-to-camera format.
--camera_intrinsics_path test_cases/case_2/intrinsics.txt Normalized camera intrinsics file.
--save_path test_cases/case_2/output.mp4 Final output video path.
--work_dir test_cases/case_2/output/ Directory for intermediate reconstruction and rendering assets.
--vace_path project-specific default Control / VACE checkpoint path.
--control_path "" Alias of --vace_path.
--lora_path project-specific default LoRA checkpoint path.

Prompt and masks

Argument Default Meaning
--prompt "" Inline text prompt. Used if --prompt_path is empty.
--prompt_path test_cases/case_2/prompt.txt Prompt file path.
--mask_path [] Foreground mask input for dynamic/static disentanglement. Supports binary image masks or one .npy file of shape [N, H, W].
--per_frame_mask True Treat provided masks as frame-aligned masks instead of one shared mask.

Resolution and rollout

Argument Default Meaning
--width 1248 Output video width.
--height 704 Output video height.
--max_frames 194 Maximum number of camera poses to use.
--first_round_frames 121 Number of frames generated in the first round.
--round_frames 81 Number of frames generated in later rounds.
--hist_frames 9 Number of history frames reused across rounds.
--fps 24 Output video FPS.

Reference-frame selection

Argument Default Meaning
--map_fps 4 Sampling FPS used for MapAnything reconstruction.
--ref_fps 2 Sampling FPS used for target reference matching.
--ref_hist_fps 6 Sampling FPS used for history-frame reference matching.
--ref_num 7 Number of matched reference frames used per round.

Diffusion settings

Argument Default Meaning
--num_inference_steps 40 Number of denoising steps per generated clip.
--cfg_scale 3.5 Classifier-free guidance scale.
--sigma_shift 5.0 Scheduler sigma shift.
--sampler uni_pc Sampler type.
--vace_scale 1.0 Control branch strength.
--seed 20917 Random seed.

Subprocess and reconstruction

Argument Default Meaning
--map_python sys.executable Python executable used for MapAnything subprocesses.
--render_python sys.executable Python executable used for point rendering subprocesses.
--map_device cuda Device for MapAnything.
--render_device cuda Device for point rendering.
--map_conf_percentile 0.0 Confidence filtering percentile for reconstructed points.
--map_voxel_size 0.01 Voxel size for MapAnything downsampling.
--render_voxel_size 0.01 Initial voxel size for point rendering.
--render_voxel_size_step 0.005 Voxel size increment per round.
--render_batchsize 64 Batch size for point rendering.
--point_retrieval_batch_size 10000000 Batch size for frustum-based point matching.
--force_rebuild_intermediate False Recompute intermediate outputs even if cached results exist.
--verbose_subprocess False Print subprocess logs instead of running quietly.

Test Cases

The repository includes three example camera/prompt setups:

  • test_cases/case_1
  • test_cases/case_2
  • test_cases/case_3

Each case includes:

  • prompt.txt
  • w2c.txt
  • intrinsics.txt

case_3 additionally uses:

  • mask.png

The starting image is not included. Before running the examples below, place an input image in each case directory, for example:

test_cases/case_1/img.jpg
test_cases/case_2/img.jpg
test_cases/case_3/img.jpg

Case 1

python inference.py \
  --img_path test_cases/case_1/img.jpg \
  --camera_w2c_path test_cases/case_1/w2c.txt \
  --camera_intrinsics_path test_cases/case_1/intrinsics.txt \
  --prompt_path test_cases/case_1/prompt.txt \
  --save_path test_cases/case_1/output.mp4 \
  --work_dir test_cases/case_1/output \
  --vace_path checkpoints/Spatia/control_weight_8500.safetensors \
  --lora_path checkpoints/Spatia/lora_weights_10000.safetensors

Case 2

python inference.py \
  --img_path test_cases/case_2/img.jpg \
  --camera_w2c_path test_cases/case_2/w2c.txt \
  --camera_intrinsics_path test_cases/case_2/intrinsics.txt \
  --prompt_path test_cases/case_2/prompt.txt \
  --save_path test_cases/case_2/output.mp4 \
  --work_dir test_cases/case_2/output \
  --vace_path checkpoints/Spatia/control_weight_8500.safetensors \
  --lora_path checkpoints/Spatia/lora_weights_10000.safetensors

Case 3

python inference.py \
  --img_path test_cases/case_3/img.jpg \
  --camera_w2c_path test_cases/case_3/w2c.txt \
  --camera_intrinsics_path test_cases/case_3/intrinsics.txt \
  --prompt_path test_cases/case_3/prompt.txt \
  --mask_path test_cases/case_3/mask.png \
  --save_path test_cases/case_3/output.mp4 \
  --work_dir test_cases/case_3/output \
  --vace_path checkpoints/Spatia/control_weight_8500.safetensors \
  --lora_path checkpoints/Spatia/lora_weights_10000.safetensors

Notes

  • model_weights/Wan2.2-TI2V-5B must exist before running inference.
  • --vace_path and --lora_path must point to valid Spatia checkpoints.
  • Intermediate reconstruction and rendering results are cached in --work_dir.
  • If prompts, masks, or camera files change, use --force_rebuild_intermediate for a clean rerun.

Citation

If you find this project useful, please cite the paper.

@inproceedings{zhao2026spatia,
  title={Spatia: Video Generation with Updatable Spatial Memory},
  author={Zhao, Jinjing and Wei, Fangyun and Liu, Zhening and Zhang, Hongyang and Xu, Chang and Lu, Yan},
  booktitle={Proceedings of the IEEE/cvf conference on computer vision and pattern recognition},
  year={2026}
}

About

[CVPR2026] Long-horizon, spatially consistent video generation enabled by persistent 3D scene point clouds and dynamic-static disentanglement.

Resources

Stars

238 stars

Watchers

11 watching

Forks

Releases

Packages

Contributors

Languages