SpaCeFormer.

An atlas of 7,361 scenes,
604K masks, 3M captions.

We construct SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation training corpus. Using training-free multi-view mask clustering and structured VLM prompting, we generate 604K instance masks with 3M diverse captions across 7,361 indoor scenes — without any human annotation.

Mask quality vs. ground truth · 312 ScanNet train scenes

Auto-generated training masks benchmarked against GT instance annotations at IoU > 0.5. Compared to Mosaic3D's SAM2-based 2D-to-3D lifting, SpaCeFormer-3M achieves 21× higher recall (54.3% vs. 2.5%) — multi-view aggregation produces substantially more complete, geometry-consistent instances.

Plates · Instance galleries from the corpus

Each 3D instance mask is paired with 5 diverse captions generated from multiple viewpoints, capturing shape, material, texture, and spatial context.

Scene 0270 — Hotel Room

Entertainment Center

1. The wooden entertainment center features a warm, honey-toned finish and provides storage and display space for electronics.
2. A medium-sized, rectangular unit with a dark brown hue sits adjacent to a desk, offering a dedicated space for a television.
3. This cabinet-like structure offers ample storage with its multiple drawers, suitable for organizing media components and accessories.
4. Crafted from wood, the built-in unit's design integrates seamlessly into the room's decor, providing a sturdy base for a television.
5. The brown wooden cabinet, with its integrated shelf, stands near a desk, offering a functional and aesthetically pleasing media storage solution.

Metal Rack

1. Dark metal rack with a tiered design, likely used for drying dishes or storing small items.
2. The black, angular rack stands near a light-colored cabinet, its open structure creating visual contrast.
3. A medium-sized, sturdy rack provides a surface for air-drying, positioned beside a wooden cabinet.
4. Constructed from dark metal bars, the rack's design allows for ventilation and easy access to items.
5. The angled rack, with its rectangular openings, sits adjacent to a cabinet, offering a practical storage solution.

Wooden Table

1. The wooden table features a warm, brown finish and a smooth, polished surface, ideal for supporting lamps and other decor.
2. A medium-sized table with a scalloped edge sits adjacent to a cabinet, its shape complementing the room's traditional style.
3. This small table offers a surface for placing items, providing a convenient spot for a phone or remote control.
4. Crafted from wood, the table's design incorporates a single drawer and decorative cutouts, blending functionality with aesthetic appeal.
5. Positioned near a chair, the brown wooden table provides a stable base for a lamp, creating a cozy reading nook.

Bed Frame

1. The bedframe is constructed from wood, providing structural support for the mattress and bedding.
2. A dark brown bed frame with a rectangular headboard sits against a wall, adjacent to a bedside table.
3. This medium-sized bed frame offers a comfortable place to rest, featuring a flat surface for pillows.
4. The wooden headboard's design complements the hotel room's decor, creating a cohesive aesthetic.
5. The bed frame's upper edge supports a white pillow, positioned near a lamp and artwork.

Scene 0601 — Living Room

Whiteboard

1. The red metal whiteboard serves as a surface for writing and displaying information, positioned against a dark red wall.
2. A rectangular whiteboard with a red frame stands upright, offering a large, blank space for notes and diagrams.
3. Medium-sized whiteboard, suitable for brainstorming or presentations, provides a writable surface near laundry machines.
4. The whiteboard's flat surface and sturdy frame allow for easy cleaning and repeated use in a communal space.
5. Red-painted metal whiteboard, mounted on the wall, offers a functional space for communication and collaboration.

Side Table

1. The small side table features a smooth, wooden tabletop and a dark gray metal stem, providing a surface for drinks or books.
2. A round table with a warm brown wood top sits adjacent to a patterned armchair, creating a cozy seating area.
3. This medium-sized table offers a convenient surface for placing items, easily accessible for resting a cup or phone.
4. The table's simple, cylindrical design complements the surrounding furniture, blending seamlessly into the waiting room decor.
5. Positioned next to a chair, the table's wooden surface and dark metal leg provide a functional and stylish accent.

Upholstered Chair

1. Upholstered chair with a textured, patterned fabric in shades of red, orange, and brown offers a comfortable seating option.
2. The chair's rounded shape and reddish-brown hue complement the surrounding decor, positioned near a small table.
3. A medium-sized chair provides a place to sit and rest, its sturdy frame suggesting durability and frequent use.
4. The chair's design features a cutout back and a patterned upholstery, blending seamlessly into the waiting area's aesthetic.
5. Located near a side table, the chair's fabric texture and reddish-brown color create a welcoming and functional seating arrangement.

Washing Machine

1. The large, white washing machine features a stainless steel drum and digital control panel, designed for laundry tasks.
2. A white appliance with a rounded door sits adjacent to other machines, displaying a modern, streamlined shape.
3. This medium-sized washing machine offers a convenient space for loading clothes, facilitating household chores.
4. The appliance's durable white plastic exterior integrates seamlessly into the laundry room's utilitarian design.
5. Positioned between other washers, the white machine's front-loading door and digital display indicate its operational status.

Scene 0656 — Bedroom

Nightstand

1. A small, dark wood nightstand provides storage near the bed, featuring brass-toned hardware and a traditional design.
2. The reddish-brown bedside table sits adjacent to the bed, its rectangular shape complementing the room's layout.
3. This medium-sized wooden cabinet offers a surface for lamps and books, easily accessible from the bed.
4. Crafted from wood with a visible grain, the nightstand's design blends classic style with functional storage.
5. The dark brown nightstand, positioned next to the bed, provides a convenient spot for personal items and a lamp.

Pillow

1. The pillow's woven fabric offers a soft, comfortable surface for resting.
2. A rectangular pillow with a muted brown and black checkered pattern rests against the bed.
3. Medium-sized pillow, suitable for supporting the head during sleep or relaxation.
4. The pillow's design complements the bedroom's decor, providing a cozy accent.
5. Positioned against the headboard, the pillow offers support and adds visual texture to the bed.

Writing Desk

1. Dark brown wooden table with ornate carved details, providing a surface for small items.
2. Rectangular table with a glossy finish, positioned near a curtain and a chair.
3. Medium-sized table offering a flat surface for holding stationery and decorative objects.
4. The table's design features cabriole legs and a drawer, blending into the room's decor.
5. A dark wood table sits adjacent to a chair, providing a functional space for writing or display.

Desk Chair

1. The chair's dark fabric upholstery provides a comfortable seating surface, designed for supporting a person.
2. A medium-sized chair with a curved backrest sits adjacent to a dark wooden desk, displaying a classic design.
3. This chair offers a place to sit; its slender frame allows for easy movement around the room.
4. The chair's metal legs and dark fabric seat create a simple, functional design, blending into the room's decor.
5. Positioned near a desk, the chair's dark color contrasts with the lighter carpet, offering a spot for focused work.

Window + curve attention,
3D RoPE, learned queries.

Why space-filling curves alone break.

Most 3D transformers serialize point clouds along Morton (Z-order) curves — space-filling curves that map 3D coordinates to 1D indices while preserving locality on average. The advantage: every attention block sees a fixed number of tokens, so compute is predictable.

“Two points adjacent in 3D can land far apart in the serialized order, breaking spatial coherence within an attention window.” — SpaCeFormer · §4.1

For dense prediction tasks like instance segmentation, where you need sharp boundaries, this fragmentation is a real problem. SpaCeFormer interleaves curve attention with spatial window attention — fixed geometric extent (H×W×D voxels) with variable token count — to keep geometrically adjacent points in the same attention group. Shifted partitions across layers (à la Swin) restore connectivity beyond window boundaries.

−28.6% mean pairwise distance
vs. Morton alone

O(N·L) complexity, fixed
L=1024 per patch

Fig. 01 RoPE-Enhanced Instance Segmentation Decoder. Learned queries are iteratively refined through cross-attention with 3D RoPE-encoded point features and self-attention, directly predicting instance masks, CLIP features, and foreground scores.

Attention block lineage.

SpaCeFormer is a hierarchical sparse-voxel U-Net interleaving Space attention (3D windows, fixed metric extent) at shallow high-resolution stages with Curve attention (Morton/Hilbert serialized patches, length 1024) at deeper low-resolution stages. Both flavors share fused QKV + 3D RoPE CUDA kernels on top of sparse convolution shortcuts.

ViT

CvT

Point Transformer

SpaCeFormer · Ours

State of the art from 3D alone,
at interactive speed.

ScanNet++ · zero-shot, 100 classes

From 3D input alone, SpaCeFormer reaches 22.9 mAP and surpasses the prior state of the art — OpenTrack3D (20.6 mAP), which fuses multi-view 2D streams with YOLO-World + SAM2 proposals — while running over 2,000× faster at 0.30 s per scene.

ScanNet200 · zero-shot, 200 classes

Under the matched setting (proposal-free, 3D-only, no GT 3D annotations), SpaCeFormer is 2.8× over the next-best method at 11.1 mAP, running in 0.12 s per scene. Higher-scoring methods rely on GT-trained Mask3D proposals or multi-view 2D streams with YOLO/SAM proposals.

Interactive latency & ablations

Every prediction runs at interactive rates — 0.12–0.30 s per scene, orders of magnitude faster than proposal-based pipelines. Ablations confirm each component contributes: space-curve attention, 3D RoPE, and learned queries each improve open-vocabulary mAP, and the final model reaches strong class-agnostic mask quality (22.5 AP, 45.7 AP₅₀, 64.4 AP₂₅) alongside 11.1 mAP over 200 categories.

Qualitative results.

SpaCeFormer predictions on ScanNet200 validation scenes. Each color represents a distinct predicted instance with its open-vocabulary label.

scene0011_00

scene0019_00

scene0025_00

scene0435_03

scene0462_00

scene0474_00

Weights, example code,
and a one-command demo.

Pretrained checkpoints and runnable examples live on the Hugging Face model page. The demo folder ships a viser viewer, a headless CLI, and a Gradio app. Inference requires WarpConvNet's compiled CUDA extension.

Load a scene, build a checkpoint, run inference.

import numpy as np
import torch
from plyfile import PlyData
from huggingface_hub import hf_hub_download
from warpconvnet.geometry.types.points import Points
from warpconvnet.models.spaceformer import build_spaceformer, load_spaceformer_checkpoint

device = torch.device("cuda")

# 1 — Load a colored point cloud from a .ply file
ply = PlyData.read("my_scene.ply")["vertex"]
coord = np.column_stack([ply["x"], ply["y"], ply["z"]]).astype(np.float32)   # [N, 3] meters
color = np.column_stack([ply["red"], ply["green"], ply["blue"]]).astype(np.float32)  # [N, 3] in [0, 255]

# 2 — Convert RGB color to input features, normalized to [-1, 1]
feat = color / 127.5 - 1.0                                                    # [N, 3]

# 3 — Move to GPU and wrap in a WarpConvNet Points object
coord_t = torch.from_numpy(coord).to(device)
feat_t = torch.from_numpy(feat).to(device)
offset = torch.tensor([0, coord_t.shape[0]], dtype=torch.int32, device=device)  # single scene
pc = Points(batched_coordinates=coord_t, batched_features=feat_t, offsets=offset)

# 4 — Build the released model, load weights, and run
ckpt = hf_hub_download("chrischoy/SpaCeFormer", "spaceformer_512_siglip2_ssccc.ckpt")
net = build_spaceformer(device=device)
load_spaceformer_checkpoint(net, ckpt)

out = net({"pc": pc})   # {"logit": [B,Q,2], "mask": List[[N,Q]], "clip_feat": [B,Q,1152]}

Launch the interactive demo.

pip install -r demo/requirements.txt

# Interactive viser viewer (open http://localhost:8080)
python demo/demo_viser.py --port 8080

# Run on your own scene with custom class names
python demo/demo_viser.py --ply my_scene.ply \
    --class-names "chair" "table" "lamp" "other"

# Alternatives: headless CLI (demo/inference.py) or Gradio app (demo/app.py)