Skip to content

Support covariance whitening in CeresRotationAverager - #4952

Merged
sarlinpe merged 1 commit into
sarlinpe/pose-graph-rotation-covariancesfrom
sarlinpe/ceres-rotation-averager-whitening
Nov 4, 2026
Merged

sarlinpe merged 1 commit into
sarlinpe/pose-graph-rotation-covariancesfrom
sarlinpe/ceres-rotation-averager-whitening

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Supports whitening relative-rotation residuals in CeresRotationAverager for both fixed-size camera pairs and dynamic multi-sensor rig blocks:

  • Adds DynamicCovarianceWeightedCostFunction<kNumResiduals> in estimators/cost_functions/utils.h, the runtime parameter-block counterpart of CovarianceWeightedCostFunction reusing internal::ComputeLeftSqrtInformation.
  • Documents the right-perturbation convention of RelativeRotationCostFunctor (hat_sensor2_from_sensor1 = sensor2_from_sensor1_prior * Exp(r)).
  • Adds an optional cam2_from_cam1_cov argument to CeresRotationAverager::AddRelativeRotationResidual (and CeresRotationAverager.add_relative_rotation_residual in pycolmap) to whiten individual relative-rotation residuals with DynamicCovarianceWeightedCostFunction<3>.

Tests

  • DynamicCovarianceWeightedCostFunction.MatchesCovarianceWeighted and CovarianceWeightedCostFunction.RejectsNonPositiveDefiniteCovariance in estimators/cost_functions/utils_test.cc.
  • CeresRotationAverager.WhitensAddedRelativeRotationResidual in rotation_averaging_ceres_test.cc (both fixed 2-block frames and dynamic rig sensors).
  • Python binding test in src/pycolmap/estimators/rotation_averaging_ceres_test.py.

PR stack (3/5 splitting #4823, plus #4912):

  1. Estimate two-view relative rotation covariance #4950
  2. Store and estimate rotation covariances on PoseGraph #4951
  3. Support covariance whitening in CeresRotationAverager #4952 (this PR)
  4. Add COVARIANCE reweighting for Ceres rotation averaging #4953
  5. Enable covariance-weighted rotation averaging in global pipelines #4823
  6. Reuse the pose graph of GlobalPipeline in the component mappers #4912

@ahojnnes ahojnnes added the full-ci Run Windows and remaining pycolmap CI label Nov 3, 2026
sarlinpe added a commit that referenced this pull request Nov 3, 2026
## Summary

Estimates the 3×3 marginal covariance of a two-view relative rotation
from its inlier correspondences:

- For a calibrated or uncalibrated relative pose $`(R, t)`$ with $`\|t\|
= 1`$, `EstimateTwoViewRotationCovariance` evaluates the Jacobian $`J`$
of the tangent Sampson error over `geometry.inlier_matches` w.r.t. the
right perturbation $`R\,\mathrm{Exp}(\delta\theta)`$ and the tangent
perturbation of $`t \in S^2`$. This yields the 5-DoF information matrix

$$
\Lambda = \frac{1}{\sigma^2} J^\top J = \begin{pmatrix} \Lambda_{RR} &
\Lambda_{Rt} \\ \Lambda_{tR} & \Lambda_{tt} \end{pmatrix},
\qquad
\Sigma_R = \left(\Lambda_{RR} - \Lambda_{Rt} \Lambda_{tt}^{-1}
\Lambda_{tR}\right)^{-1} ,
$$

where the translation direction is marginalized with the Schur
complement and $`\sigma`$ is the 2D observation noise
(`point2D_stddev_px`, 1 px by default).
- Pure-rotation (panoramic) pairs use a rotation-only model on the 2D
tangent planes orthogonal to the second camera's rays.
- The standard deviation along poorly or unconstrained axes (e.g.,
geometrically degenerate configurations) is capped to
`max_rotation_sigma_deg` (60°), with a `1e-5` relative variance floor so
single-precision storage preserves positive-definiteness. Returns
`std::nullopt` if the relative pose is missing, has too few valid
inliers, or if the eigendecomposition of the information fails.
- The covariance is expressed in camera 1's local tangent frame,
matching `RelativeRotationCostFunctor`.
- Exposes `ExtractMatchedCamRaysWithJac` in
`estimators/two_view_geometry.h`.
- Adds pycolmap bindings: `TwoViewRotationCovarianceOptions` and
`estimate_two_view_rotation_covariance`.

## Tests

- `TwoViewRotationCovarianceTests.MonteCarloCalibration`: 1000-trial
Monte Carlo NEES ($`\delta\theta^\top \Sigma_R^{-1} \delta\theta`$)
checks against the rotation MLE across wide-baseline, narrow-baseline,
forward-motion, planar, one-sided, and pure-rotation configurations.
- `TwoViewRotationCovarianceOptions.Check`,
`EstimateTwoViewRotationCovariance.Nominal`,
`UnconstrainedAxisIsCapped`, and `MissingPoseOrTooFewInliers`.
- `ExtractMatchedCamRaysWithJac.Nominal`.
- Python binding tests in
`src/pycolmap/estimators/two_view_rotation_covariance_test.py`.

PR stack (1/5 splitting #4823, plus #4912):
1. **#4950** (this PR)
2. #4951
3. #4952
4. #4953
5. #4823
6. #4912
@sarlinpe
sarlinpe force-pushed the sarlinpe/ceres-rotation-averager-whitening branch 2 times, most recently from ad83ea5 to c340028 Compare November 3, 2026 14:13
@sarlinpe
sarlinpe force-pushed the sarlinpe/ceres-rotation-averager-whitening branch from c340028 to 9e109ab Compare November 3, 2026 16:45
@sarlinpe
sarlinpe enabled stacked auto-merge (squash) November 3, 2026 16:47
@sarlinpe
sarlinpe merged commit 30a1a81 into main Nov 4, 2026
16 checks passed
@sarlinpe
sarlinpe deleted the sarlinpe/ceres-rotation-averager-whitening branch November 4, 2026 00:54
sarlinpe added a commit that referenced this pull request Nov 4, 2026
## Summary

Stores optional relative-rotation covariances on `PoseGraph::Edge` and
estimates them in parallel from a `DatabaseCache`:

- Adds `std::optional<Eigen::Matrix3f> cam2_from_cam1_rotation_cov` to
`PoseGraph::Edge` (single precision to keep edges lightweight),
expressed in camera 1's local tangent frame. `PoseGraph::Edge::Invert()`
conjugates it accordingly.
- Adds `EstimatePoseGraphRotationCovariances`, which populates
`cam2_from_cam1_rotation_cov` in parallel over all valid pose graph
edges that do not already have one, evaluated at each edge's relative
pose from its correspondence-graph inliers. Edges whose covariance
cannot be estimated are left unset. The database schema does not change.
- Adds pycolmap bindings: `PoseGraphEdge.cam2_from_cam1_rotation_cov`
and `estimate_pose_graph_rotation_covariances`.

## Tests

- `PoseGraph.InvertEdgeCovariance`: checks covariance conjugation and
round-trip under `Edge::Invert()`.
- `EstimatePoseGraphRotationCovariances.Nominal`, plus single-precision
inversion positive-definiteness in
`EstimateTwoViewRotationCovariance.UnconstrainedAxisIsCapped` and option
validation in `TwoViewRotationCovarianceOptions.Check`.
- Python binding tests in `pose_graph_test.py` and
`two_view_rotation_covariance_test.py`.

PR stack (2/5 splitting #4823, plus #4912):
1. #4950
2. **#4951** (this PR)
3. #4952
4. #4953
5. #4823
6. #4912
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 4, 2026
## Change

Integrate upstream pose-graph rotation covariances and optional covariance
whitening in Ceres rotation averaging. Preserve the fork's existing native
and Python covariance regression tests.

## Upstream commits

77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)

## Validation

Linux GCC 12.4: four maintained native suites passed all 52 cases without
failures or skips. The incremental build compiled 18 C++ objects, updated
three archives and linked four tests; no CUDA compilation or CMake rerun.
Source, dependency, selected object and runtime identities are retained.

On this x86-64 configuration, a deterministic real-header probe measured
PoseGraph::Edge at 64 to 104 bytes (alignment 8); PoseGraph itself remains
48 bytes. This records the optional covariance field's ABI storage cost,
not heap, RSS or whole-scene memory use. One reading per header version.

A focused real-binding Python module passed 23 maintained tests across
three unchanged test files; one test skipped because pyceres is absent.
The module used 19 freshly compiled binding/entry objects, nine qualified
unchanged binding objects and one link. Its 80 selected native archive
members were checked against the current build objects and dependencies.
Actual default enum, RANSAC and option-property checks passed. This is
focused binding qualification, not a full installed pycolmap package build.

The standard averager still creates unweighted residuals; covariance
whitening requires the explicit API. No Windows build or reconstruction
performance claim is made for this merge.

## AI disclosure

AI-assisted integration and validation with independent code review.
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 4, 2026
## Change

Integrate upstream pose-graph rotation covariances and optional covariance
whitening in Ceres rotation averaging. Preserve the fork's existing native
and Python covariance regression tests.

## Upstream commits

77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)

## Validation

Linux GCC 12.4: four maintained native suites passed all 52 cases without
failures or skips. The incremental build compiled 18 C++ objects, updated
three archives and linked four tests; no CUDA compilation or CMake rerun.
Source, dependency, selected object and runtime identities are retained.

On this x86-64 configuration, a deterministic real-header probe measured
PoseGraph::Edge at 64 to 104 bytes (alignment 8); PoseGraph itself remains
48 bytes. This records the optional covariance field's ABI storage cost,
not heap, RSS or whole-scene memory use. One reading per header version.

A focused real-binding Python module passed 23 maintained tests across
three unchanged test files; one test skipped because pyceres is absent.
The module used 19 freshly compiled binding/entry objects, nine qualified
unchanged binding objects and one link. Its 80 selected native archive
members were checked against the current build objects and dependencies.
Actual default enum, RANSAC and option-property checks passed. This is
focused binding qualification, not a full installed pycolmap package build.

The standard averager still creates unweighted residuals; covariance
whitening requires the explicit API. No Windows build or reconstruction
performance claim is made for this merge.

## AI disclosure

AI-assisted integration and validation with independent code review.
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 7, 2026
Refresh the experimental alternative with fork main while retaining its previous head as a parent. The resulting tree is exactly main plus the isolated shared-current-cost-cache deletion: one CUDA file, +1/-25.

## Change

Read the existing cost map directly during the ordered sampled-cost fold. Preserve current plane propagation, source packing, scheduling and fallback mathematics. The cache originates in custom a37c1c3, MVS: Reuse sampled current costs (GPU perf). The earlier alternative is retained through 8481f89; its original experiment was 8b1412f.

Historical Fountain11 photo+geometry at 3072 recorded 76.85 -> 78.82 seconds, one observation per arm, on the older revision. Retain this inconclusive/adverse observation with its earlier revision; current performance has not been measured. No performance experiment was reopened.

## Validation

- Exact candidate CUDA owner compiled with CUDA 13.2 for compute_86/sm_86. In a bounded retained-support context on RTX 3060, both matched targets passed all 13 selected cases: propagation, ordered source folding, checkpoint-HMM and paired-sweep preservation.
- Four complete inputs and four outputs matched across executables: 8,492,128 bytes per arm. Maps were finite with positive depth and checked ordered graph IDs. Complete witnesses use checkpoint_hmm=false; separate checkpoint tests cover both settings without full cross-executable true-mode witnesses.
- Direct owners, link maps and actual loaded providers were checked; managed jobs ended and released their bookings. Later host resize/warp changes have separate owner qualification. This is not a whole-branch build, reconstruction-quality, portability or current-performance result.

## Upstream commits carried forward

- e52aa8e Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- 8007d1a Preserve float weight ordering in maximum spanning trees (colmap#4928)
- dd84102 Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- 721e185 Add FlexGlue feature matcher (colmap#4908)
- d0a411f FlexGlue: cache the dense features on the GPU (colmap#4915)
- 105817b Hoist camera model dispatch per row in image warps (colmap#4935)
- 27ffad4 Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- e8f3522 Resolve benchmark paths before changing subprocess directories (colmap#4931)
- f664d5d Clamp visibility coordinates before unsigned conversion (colmap#4943)
- 138119b Keep thin bitmap thumbnails nonempty (colmap#4927)
- b16ed5f Count rig frame covisibility once per 3D point (colmap#4893)
- 93dbbcb Add colmap-review agent skill for code review (colmap#4934)
- 2a6f695 Fix sift_test build on MSVC without the GUI (colmap#4955)
- 78cc729 Fix default prior_position_loss_scale to sqrt(chi2_95_3dof) (colmap#4936)
- 8653c6b Estimate two-view relative rotation covariance (colmap#4950)
- e1317dd FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)
- 3db6eb0 Extract ConvertPosePriorsToENU and skip pose priors without position (colmap#4937)
- 77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
- 30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)
- e6644e7 Fix -Wunused-result warning in ThreadPool::AddTask (colmap#4958)
- 5b6da88 Clear descriptor-index results for empty queries (colmap#4960)
- d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953)
- 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823)
- fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants