Skip to content

Add COVARIANCE reweighting for Ceres rotation averaging - #4953

Merged
sarlinpe merged 10 commits into
mainfrom
sarlinpe/rotation-averaging-covariance-reweighting
Nov 6, 2026
Merged

sarlinpe merged 10 commits into
mainfrom
sarlinpe/rotation-averaging-covariance-reweighting

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds RotationAveragingReweighting::COVARIANCE to whiten relative-rotation constraints in the Ceres rotation averaging backend using per-edge rotation covariances:

  • CeresRotationAverager whitens each residual, $r_{ij} \mapsto \Sigma_{R,ij}^{-1/2} r_{ij}$, where $\Sigma_{R,ij}$ is regularized with an isotropic floor, $\Sigma_{R,ij} + \sigma_{\text{floor}}^2 I_3$ (CeresRotationAveragerOptions::covariance_sigma_floor_deg, 0.02° by default). The pose graph keeps the raw covariances.
  • The robust loss acts on the whitened, unitless residual with scale CeresRotationAveragerOptions::covariance_loss_scale (3.0 standard deviations by default), in place of loss_function_scale (in degrees).
  • Splits Ceres rotation averaging preparation into CheckPoseGraph, PrepareWeights, and InitializeRotations (reusing internal::TryComputeLeftSqrtInformation and DynamicCovarianceWeightedCostFunction::FromLeftSqrtInformation) so both RotationEstimator and CeresRotationAverager validate connectivity, sensor rotations, loss scales, covariance_sigma_floor_deg, and that every valid edge has a finite, symmetric positive-definite floored covariance before MST initialization modifies the reconstruction, without repeating connected-components or per-edge Cholesky factorizations.
  • Adds use_rotation_covariance to InitializeFromMaximumSpanningTree, preferring edges with the smallest $\mathrm{tr}(\Sigma_R)$ via weights $-\log \mathrm{tr}(\Sigma_R)$. RotationEstimator runs MST initialization once before the warm-start.
  • Keeps the Huber warm-start on angular residuals with UNIFORM weights so confidently wrong relative rotations (e.g., from symmetric structures) do not dominate early iterations.
  • COVARIANCE is unsupported by L1_IRLS (throws), and is ignored with a warning when CERES falls back to L1_IRLS for gravity priors.
  • Binds RotationAveragingReweighting.COVARIANCE, covariance_loss_scale, and covariance_sigma_floor_deg in pycolmap.

Tests

  • CeresRotationAverager.CovarianceWhitening, CovarianceDownweightsUncertainEdges, CovarianceReweightingWithRigs, and RotationEstimator.CeresChecksInputsBeforeInitialization in rotation_averaging_ceres_test.cc.
  • RotationAveraging.InitializeFromMaximumSpanningTreeWithCovariance and RotationAveraging.CovarianceReweighting in rotation_averaging_test.cc.
  • Python binding tests in motion_averaging_test.py and rotation_averaging_ceres_test.py.

Stack created with GitHub Stacks CLI • Give Feedback 💬

@sarlinpe
sarlinpe added this pull request to stack #4954 November 3, 2026 01:25
@sarlinpe
sarlinpe force-pushed the sarlinpe/rotation-averaging-covariance-reweighting branch from c2d42ad to 10de34e Compare November 3, 2026 08:26
sarlinpe added a commit that referenced this pull request Nov 3, 2026
## Summary

Estimates the 3×3 marginal covariance of a two-view relative rotation
from its inlier correspondences:

- For a calibrated or uncalibrated relative pose $`(R, t)`$ with $`\|t\|
= 1`$, `EstimateTwoViewRotationCovariance` evaluates the Jacobian $`J`$
of the tangent Sampson error over `geometry.inlier_matches` w.r.t. the
right perturbation $`R\,\mathrm{Exp}(\delta\theta)`$ and the tangent
perturbation of $`t \in S^2`$. This yields the 5-DoF information matrix

$$
\Lambda = \frac{1}{\sigma^2} J^\top J = \begin{pmatrix} \Lambda_{RR} &
\Lambda_{Rt} \\ \Lambda_{tR} & \Lambda_{tt} \end{pmatrix},
\qquad
\Sigma_R = \left(\Lambda_{RR} - \Lambda_{Rt} \Lambda_{tt}^{-1}
\Lambda_{tR}\right)^{-1} ,
$$

where the translation direction is marginalized with the Schur
complement and $`\sigma`$ is the 2D observation noise
(`point2D_stddev_px`, 1 px by default).
- Pure-rotation (panoramic) pairs use a rotation-only model on the 2D
tangent planes orthogonal to the second camera's rays.
- The standard deviation along poorly or unconstrained axes (e.g.,
geometrically degenerate configurations) is capped to
`max_rotation_sigma_deg` (60°), with a `1e-5` relative variance floor so
single-precision storage preserves positive-definiteness. Returns
`std::nullopt` if the relative pose is missing, has too few valid
inliers, or if the eigendecomposition of the information fails.
- The covariance is expressed in camera 1's local tangent frame,
matching `RelativeRotationCostFunctor`.
- Exposes `ExtractMatchedCamRaysWithJac` in
`estimators/two_view_geometry.h`.
- Adds pycolmap bindings: `TwoViewRotationCovarianceOptions` and
`estimate_two_view_rotation_covariance`.

## Tests

- `TwoViewRotationCovarianceTests.MonteCarloCalibration`: 1000-trial
Monte Carlo NEES ($`\delta\theta^\top \Sigma_R^{-1} \delta\theta`$)
checks against the rotation MLE across wide-baseline, narrow-baseline,
forward-motion, planar, one-sided, and pure-rotation configurations.
- `TwoViewRotationCovarianceOptions.Check`,
`EstimateTwoViewRotationCovariance.Nominal`,
`UnconstrainedAxisIsCapped`, and `MissingPoseOrTooFewInliers`.
- `ExtractMatchedCamRaysWithJac.Nominal`.
- Python binding tests in
`src/pycolmap/estimators/two_view_rotation_covariance_test.py`.

PR stack (1/5 splitting #4823, plus #4912):
1. **#4950** (this PR)
2. #4951
3. #4952
4. #4953
5. #4823
6. #4912
@sarlinpe
sarlinpe force-pushed the sarlinpe/rotation-averaging-covariance-reweighting branch 2 times, most recently from 390846b to e0baf64 Compare November 3, 2026 14:13
@sarlinpe
sarlinpe force-pushed the sarlinpe/rotation-averaging-covariance-reweighting branch from e0baf64 to b93a193 Compare November 3, 2026 16:45
@sarlinpe
sarlinpe requested review from B1ueber2y and ahojnnes and removed request for ahojnnes November 3, 2026 16:47
Base automatically changed from sarlinpe/ceres-rotation-averager-whitening to main November 4, 2026 00:54
sarlinpe added a commit that referenced this pull request Nov 4, 2026
## Summary

Stores optional relative-rotation covariances on `PoseGraph::Edge` and
estimates them in parallel from a `DatabaseCache`:

- Adds `std::optional<Eigen::Matrix3f> cam2_from_cam1_rotation_cov` to
`PoseGraph::Edge` (single precision to keep edges lightweight),
expressed in camera 1's local tangent frame. `PoseGraph::Edge::Invert()`
conjugates it accordingly.
- Adds `EstimatePoseGraphRotationCovariances`, which populates
`cam2_from_cam1_rotation_cov` in parallel over all valid pose graph
edges that do not already have one, evaluated at each edge's relative
pose from its correspondence-graph inliers. Edges whose covariance
cannot be estimated are left unset. The database schema does not change.
- Adds pycolmap bindings: `PoseGraphEdge.cam2_from_cam1_rotation_cov`
and `estimate_pose_graph_rotation_covariances`.

## Tests

- `PoseGraph.InvertEdgeCovariance`: checks covariance conjugation and
round-trip under `Edge::Invert()`.
- `EstimatePoseGraphRotationCovariances.Nominal`, plus single-precision
inversion positive-definiteness in
`EstimateTwoViewRotationCovariance.UnconstrainedAxisIsCapped` and option
validation in `TwoViewRotationCovarianceOptions.Check`.
- Python binding tests in `pose_graph_test.py` and
`two_view_rotation_covariance_test.py`.

PR stack (2/5 splitting #4823, plus #4912):
1. #4950
2. **#4951** (this PR)
3. #4952
4. #4953
5. #4823
6. #4912
@sarlinpe
sarlinpe force-pushed the sarlinpe/rotation-averaging-covariance-reweighting branch from b93a193 to 3a717c1 Compare November 4, 2026 00:55
sarlinpe added a commit that referenced this pull request Nov 4, 2026
## Summary

Supports whitening relative-rotation residuals in
`CeresRotationAverager` for both fixed-size camera pairs and dynamic
multi-sensor rig blocks:

- Adds `DynamicCovarianceWeightedCostFunction<kNumResiduals>` in
`estimators/cost_functions/utils.h`, the runtime parameter-block
counterpart of `CovarianceWeightedCostFunction` reusing
`internal::ComputeLeftSqrtInformation`.
- Documents the right-perturbation convention of
`RelativeRotationCostFunctor` (`hat_sensor2_from_sensor1 =
sensor2_from_sensor1_prior * Exp(r)`).
- Adds an optional `cam2_from_cam1_cov` argument to
`CeresRotationAverager::AddRelativeRotationResidual` (and
`CeresRotationAverager.add_relative_rotation_residual` in pycolmap) to
whiten individual relative-rotation residuals with
`DynamicCovarianceWeightedCostFunction<3>`.

## Tests

- `DynamicCovarianceWeightedCostFunction.MatchesCovarianceWeighted` and
`CovarianceWeightedCostFunction.RejectsNonPositiveDefiniteCovariance` in
`estimators/cost_functions/utils_test.cc`.
- `CeresRotationAverager.WhitensAddedRelativeRotationResidual` in
`rotation_averaging_ceres_test.cc` (both fixed 2-block frames and
dynamic rig sensors).
- Python binding test in
`src/pycolmap/estimators/rotation_averaging_ceres_test.py`.

PR stack (3/5 splitting #4823, plus #4912):
1. #4950
2. #4951
3. **#4952** (this PR)
4. #4953
5. #4823
6. #4912
@sarlinpe sarlinpe added the full-ci Run Windows and remaining pycolmap CI label Nov 4, 2026

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few remaining comments. The missing pipeline wiring and the fallback for edges without a covariance are handled in #4823, so I left those out here.

@sarlinpe
sarlinpe force-pushed the sarlinpe/rotation-averaging-covariance-reweighting branch from 3a717c1 to b31143c Compare November 5, 2026 17:38
@sarlinpe
sarlinpe merged commit d1d92d9 into main Nov 6, 2026
16 checks passed
@sarlinpe
sarlinpe deleted the sarlinpe/rotation-averaging-covariance-reweighting branch November 6, 2026 21:49
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 7, 2026
Merge the upstream covariance-weighted rotation averaging and component pose-graph work.

## Change

- Add the upstream covariance-weighted Ceres rotation mode and global-pipeline option. Existing L1_IRLS and UNIFORM defaults remain.
- Reuse the prebuilt pipeline pose graph when constructing component subgraphs.
- Preserve the fork's option registration changes through the disjoint merge.

## Validation

- The object-level merge is conflict-free; incoming paths do not overlap the four existing dirty files.
- Native integration, affected dependent binaries and bindings remain pending. Defaulted C++ arguments do not preserve binary ABI.

## Upstream commits

- d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953)
- 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823)
- fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)

AI disclosure: Totally vibe coded
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 7, 2026
Refresh the experimental alternative with fork main while retaining its previous head as a parent. The resulting tree is exactly main plus the isolated shared-current-cost-cache deletion: one CUDA file, +1/-25.

## Change

Read the existing cost map directly during the ordered sampled-cost fold. Preserve current plane propagation, source packing, scheduling and fallback mathematics. The cache originates in custom a37c1c3, MVS: Reuse sampled current costs (GPU perf). The earlier alternative is retained through 8481f89; its original experiment was 8b1412f.

Historical Fountain11 photo+geometry at 3072 recorded 76.85 -> 78.82 seconds, one observation per arm, on the older revision. Retain this inconclusive/adverse observation with its earlier revision; current performance has not been measured. No performance experiment was reopened.

## Validation

- Exact candidate CUDA owner compiled with CUDA 13.2 for compute_86/sm_86. In a bounded retained-support context on RTX 3060, both matched targets passed all 13 selected cases: propagation, ordered source folding, checkpoint-HMM and paired-sweep preservation.
- Four complete inputs and four outputs matched across executables: 8,492,128 bytes per arm. Maps were finite with positive depth and checked ordered graph IDs. Complete witnesses use checkpoint_hmm=false; separate checkpoint tests cover both settings without full cross-executable true-mode witnesses.
- Direct owners, link maps and actual loaded providers were checked; managed jobs ended and released their bookings. Later host resize/warp changes have separate owner qualification. This is not a whole-branch build, reconstruction-quality, portability or current-performance result.

## Upstream commits carried forward

- e52aa8e Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- 8007d1a Preserve float weight ordering in maximum spanning trees (colmap#4928)
- dd84102 Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- 721e185 Add FlexGlue feature matcher (colmap#4908)
- d0a411f FlexGlue: cache the dense features on the GPU (colmap#4915)
- 105817b Hoist camera model dispatch per row in image warps (colmap#4935)
- 27ffad4 Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- e8f3522 Resolve benchmark paths before changing subprocess directories (colmap#4931)
- f664d5d Clamp visibility coordinates before unsigned conversion (colmap#4943)
- 138119b Keep thin bitmap thumbnails nonempty (colmap#4927)
- b16ed5f Count rig frame covisibility once per 3D point (colmap#4893)
- 93dbbcb Add colmap-review agent skill for code review (colmap#4934)
- 2a6f695 Fix sift_test build on MSVC without the GUI (colmap#4955)
- 78cc729 Fix default prior_position_loss_scale to sqrt(chi2_95_3dof) (colmap#4936)
- 8653c6b Estimate two-view relative rotation covariance (colmap#4950)
- e1317dd FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)
- 3db6eb0 Extract ConvertPosePriorsToENU and skip pose priors without position (colmap#4937)
- 77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
- 30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)
- e6644e7 Fix -Wunused-result warning in ThreadPool::AddTask (colmap#4958)
- 5b6da88 Clear descriptor-index results for empty queries (colmap#4960)
- d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953)
- 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823)
- fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants