Skip to content

Hoist camera model dispatch per row in image warps - #4935

Merged
sarlinpe merged 5 commits into
mainfrom
sarlinpe/warp-row-dispatch
Nov 2, 2026
Merged

sarlinpe merged 5 commits into
mainfrom
sarlinpe/warp-row-dispatch

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Supersedes #4930.

Problem

WarpImageBetweenCameras and WarpImageWithHomographyBetweenCameras call Camera::CamFromImg and Camera::ImgFromCam on every pixel. Both go through the 19-way runtime switch (model_id) in sensor/models/runtime.h (~24M dispatches for a 12 MP image), which prevents the compiler from inlining the concrete camera model across a scanline, hoisting row-invariant terms, constant-folding unit depth (w = 1), and vectorizing the inner loop.

Fix

Hoist the CAMERA_MODEL_SWITCH_CASES dispatch out of the inner x loop into two scanline helpers (CamFromImgRow and ImgFromCamRow) in src/colmap/image/warp.cc, and unify both warp functions into a single WarpBetweenCamerasImpl (+30 net lines in warp.cc):

  • Optimizes both WarpImageBetweenCameras and WarpImageWithHomographyBetweenCameras, while deduplicating their setup, scanline loop, interpolation dispatch, and rescaling.
  • Accelerates both CamFromImg (target) and ImgFromCam (source) for all camera models without leaking camera math or parameter indices outside sensor/models/.
  • Kept local to warp.cc rather than sensor/models/runtime.h: other CamFromImg / ImgFromCam loops in the codebase (P3PEstimator::Residuals, HomographyMatrixRayEstimator::Residuals, UndistortCamera, UndistortReconstruction) operate on 100–10k points (<0.5–3 ms) or 3D rays with w != 1, whereas scanline generation (x + 0.5 with scalar y, unit depth w = 1, and separate input/output row buffers) is significantly faster than generic in-place point-array batching.

Comparison with #4930

#4930 special-cases PinholeCameraModel in WarpImageBetweenCameras by indexing params[0..3] directly. Compared to #4930, this PR:

  • Keeps all camera parameter layout and projection math inside sensor/models/.
  • Works for all 19 camera models on both source and target cameras (not only PINHOLE targets).
  • Covers WarpImageWithHomographyBetweenCameras (stereo rectification) as well.
  • Delivers larger speedups even on PINHOLE targets by also eliminating per-pixel dispatch and w = 1 division in ImgFromCam.

Benchmark

4032×3024 RGB image (12.2 MP), Release build (g++ 15.2 -O3 -DNDEBUG) on Intel Xeon @ 2.20 GHz, best of 5 runs after warmup; all outputs are bit-identical to main:

Function Cameras (source -> target) Interpolation main #4930 This PR
WarpImageBetweenCameras OPENCV -> PINHOLE Bilinear 1175.9 ms 908.7 ms (−22.7%) 723.9 ms (−38.4%)
WarpImageBetweenCameras OPENCV -> PINHOLE Nearest 726.3 ms 447.4 ms (−38.4%) 274.4 ms (−62.2%)
WarpImageBetweenCameras SIMPLE_RADIAL -> PINHOLE Bilinear 1109.1 ms 844.7 ms (−23.8%) 660.3 ms (−40.5%)
WarpImageBetweenCameras OPENCV -> SIMPLE_PINHOLE Bilinear 1217.4 ms 1180.9 ms (−3.0%) 669.5 ms (−45.0%)
WarpImageWithHomographyBetweenCameras OPENCV -> PINHOLE Bilinear 1159.3 ms 1159.3 ms (0.0%) 767.6 ms (−33.8%)

Tests

  • Warp.CameraModelsMatchPerPixelProjection (src/colmap/image/warp_test.cc) checks bit-exact output against per-pixel Camera::CamFromImg and Camera::ImgFromCam across OPENCV -> PINHOLE, SIMPLE_DIVISION -> FOV (partial ImgFromCam failure), and SIMPLE_RADIAL -> EQUIRECTANGULAR (partial CamFromImg failure) for both interpolation modes, both direct and intermediate-resolution warp paths, and both WarpImageBetweenCameras and WarpImageWithHomographyBetweenCameras.
  • image/warp_test, image/undistortion_test, image/line_test, and controllers/undistorters_test pass.

@sarlinpe
sarlinpe requested a review from ahojnnes October 31, 2026 21:20

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two follow-ups, neither blocking: the new test's coverage, and an invariant worth a comment.

Comment thread src/colmap/image/warp_test.cc Resolved
Comment thread src/colmap/image/warp.cc Resolved

@ahojnnes ahojnnes left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great improvement!

@sarlinpe
sarlinpe enabled auto-merge (squash) November 1, 2026 20:52
@sarlinpe
sarlinpe disabled auto-merge November 1, 2026 20:52
@sarlinpe
sarlinpe enabled auto-merge (squash) November 1, 2026 20:52
@sarlinpe sarlinpe added the full-ci Run Windows and remaining pycolmap CI label Nov 1, 2026
@sarlinpe
sarlinpe merged commit 105817b into main Nov 2, 2026
18 of 21 checks passed
@sarlinpe
sarlinpe deleted the sarlinpe/warp-row-dispatch branch November 2, 2026 01:06
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 3, 2026
Integrate thirteen official upstream commits after the private GPU cleanup
checkpoint. Preserve blocking host-sync selection and its CUDA 12.0/12.1
flag boundary while adopting upstream's generalized compute-capability
helper and LoMa/FlexGlue callers. Keep both capability and policy tests;
remove the duplicate colmap_util_cuda link from automatic composition.

Includes upstream rig-frame covisibility, thin bitmap thumbnails, visibility
coordinate conversion, row-wise warp dispatch, benchmark fixes, FlexGlue,
review guidance, and the Windows non-GUI SIFT-test build fix.

Validation at this private checkpoint: merge conflicts resolved in cuda.cc,
cuda.h, cuda_test.cc; source search found no remaining old BF16-helper call;
git diff --check passed. Native merged-source qualification and independent
exact-diff review remain pending. Prior GPU tests qualify their pinned source,
not this entire merged product. No reconstruction or performance claim.

AI-assisted integration. Existing published history is preserved.

## Upstream commits

- `e52aa8ee12ef58eb1cd16ada2be9316fa1c02cd3` Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- `8007d1a51ae0c0af87fff3ebd204cf4698df7565` Preserve float weight ordering in maximum spanning trees (colmap#4928)
- `dd84102ef5e75fc17e20da281d2fc2fee5aca6a9` Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- `721e185ddb6205b85fddc39ccf0774567f6be6ed` Add FlexGlue feature matcher (colmap#4908)
- `d0a411f2a9af139b5ea8118d64e5da5b70c6a890` FlexGlue: cache the dense features on the GPU (colmap#4915)
- `105817b6299cf22144fbbf39f95689681352476d` Hoist camera model dispatch per row in image warps (colmap#4935)
- `27ffad4388ce5647df3ea9e547bf8e1ea155fbcd` Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- `e8f3522b936882cbceae074a19fca58fe378d323` Resolve benchmark paths before changing subprocess directories (colmap#4931)
- `f664d5d219230c1f1ebf9c2c7ecdcafc451b95ce` Clamp visibility coordinates before unsigned conversion (colmap#4943)
- `138119b7640d46f1f51fe329694a9ddae3ae0908` Keep thin bitmap thumbnails nonempty (colmap#4927)
- `b16ed5f97fa405ff2462745f9509386a9ebe6ece` Count rig frame covisibility once per 3D point (colmap#4893)
- `93dbbcb7f3019da30bd4a2c72753421e6cb85b45` Add colmap-review agent skill for code review (colmap#4934)
- `2a6f6955ed73a3d15b2a75edede5750e7c7b3304` Fix sift_test build on MSVC without the GUI (colmap#4955)
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 4, 2026
Integrate thirteen official upstream commits after the private GPU cleanup
checkpoint. Preserve blocking host-sync selection and its CUDA 12.0/12.1
flag boundary while adopting upstream's generalized compute-capability
helper and LoMa/FlexGlue callers. Keep both capability and policy tests;
remove the duplicate colmap_util_cuda link from automatic composition.

Includes upstream rig-frame covisibility, thin bitmap thumbnails, visibility
coordinate conversion, row-wise warp dispatch, benchmark fixes, FlexGlue,
review guidance, and the Windows non-GUI SIFT-test build fix.

Validation at this private checkpoint: merge conflicts resolved in cuda.cc,
cuda.h, cuda_test.cc; source search found no remaining old BF16-helper call;
git diff --check passed. Native merged-source qualification and independent
exact-diff review remain pending. Prior GPU tests qualify their pinned source,
not this entire merged product. No reconstruction or performance claim.

AI-assisted integration. Existing published history is preserved.

## Upstream commits

- `e52aa8ee12ef58eb1cd16ada2be9316fa1c02cd3` Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- `8007d1a51ae0c0af87fff3ebd204cf4698df7565` Preserve float weight ordering in maximum spanning trees (colmap#4928)
- `dd84102ef5e75fc17e20da281d2fc2fee5aca6a9` Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- `721e185ddb6205b85fddc39ccf0774567f6be6ed` Add FlexGlue feature matcher (colmap#4908)
- `d0a411f2a9af139b5ea8118d64e5da5b70c6a890` FlexGlue: cache the dense features on the GPU (colmap#4915)
- `105817b6299cf22144fbbf39f95689681352476d` Hoist camera model dispatch per row in image warps (colmap#4935)
- `27ffad4388ce5647df3ea9e547bf8e1ea155fbcd` Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- `e8f3522b936882cbceae074a19fca58fe378d323` Resolve benchmark paths before changing subprocess directories (colmap#4931)
- `f664d5d219230c1f1ebf9c2c7ecdcafc451b95ce` Clamp visibility coordinates before unsigned conversion (colmap#4943)
- `138119b7640d46f1f51fe329694a9ddae3ae0908` Keep thin bitmap thumbnails nonempty (colmap#4927)
- `b16ed5f97fa405ff2462745f9509386a9ebe6ece` Count rig frame covisibility once per 3D point (colmap#4893)
- `93dbbcb7f3019da30bd4a2c72753421e6cb85b45` Add colmap-review agent skill for code review (colmap#4934)
- `2a6f6955ed73a3d15b2a75edede5750e7c7b3304` Fix sift_test build on MSVC without the GUI (colmap#4955)
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 7, 2026
Refresh the experimental alternative with fork main while retaining its previous head as a parent. The resulting tree is exactly main plus the isolated shared-current-cost-cache deletion: one CUDA file, +1/-25.

## Change

Read the existing cost map directly during the ordered sampled-cost fold. Preserve current plane propagation, source packing, scheduling and fallback mathematics. The cache originates in custom a37c1c3, MVS: Reuse sampled current costs (GPU perf). The earlier alternative is retained through 8481f89; its original experiment was 8b1412f.

Historical Fountain11 photo+geometry at 3072 recorded 76.85 -> 78.82 seconds, one observation per arm, on the older revision. Retain this inconclusive/adverse observation with its earlier revision; current performance has not been measured. No performance experiment was reopened.

## Validation

- Exact candidate CUDA owner compiled with CUDA 13.2 for compute_86/sm_86. In a bounded retained-support context on RTX 3060, both matched targets passed all 13 selected cases: propagation, ordered source folding, checkpoint-HMM and paired-sweep preservation.
- Four complete inputs and four outputs matched across executables: 8,492,128 bytes per arm. Maps were finite with positive depth and checked ordered graph IDs. Complete witnesses use checkpoint_hmm=false; separate checkpoint tests cover both settings without full cross-executable true-mode witnesses.
- Direct owners, link maps and actual loaded providers were checked; managed jobs ended and released their bookings. Later host resize/warp changes have separate owner qualification. This is not a whole-branch build, reconstruction-quality, portability or current-performance result.

## Upstream commits carried forward

- e52aa8e Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- 8007d1a Preserve float weight ordering in maximum spanning trees (colmap#4928)
- dd84102 Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- 721e185 Add FlexGlue feature matcher (colmap#4908)
- d0a411f FlexGlue: cache the dense features on the GPU (colmap#4915)
- 105817b Hoist camera model dispatch per row in image warps (colmap#4935)
- 27ffad4 Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- e8f3522 Resolve benchmark paths before changing subprocess directories (colmap#4931)
- f664d5d Clamp visibility coordinates before unsigned conversion (colmap#4943)
- 138119b Keep thin bitmap thumbnails nonempty (colmap#4927)
- b16ed5f Count rig frame covisibility once per 3D point (colmap#4893)
- 93dbbcb Add colmap-review agent skill for code review (colmap#4934)
- 2a6f695 Fix sift_test build on MSVC without the GUI (colmap#4955)
- 78cc729 Fix default prior_position_loss_scale to sqrt(chi2_95_3dof) (colmap#4936)
- 8653c6b Estimate two-view relative rotation covariance (colmap#4950)
- e1317dd FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)
- 3db6eb0 Extract ConvertPosePriorsToENU and skip pose priors without position (colmap#4937)
- 77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
- 30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)
- e6644e7 Fix -Wunused-result warning in ThreadPool::AddTask (colmap#4958)
- 5b6da88 Clear descriptor-index results for empty queries (colmap#4960)
- d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953)
- 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823)
- fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants