Skip to content

Estimate two-view relative rotation covariance - #4950

Merged
sarlinpe merged 1 commit into
mainfrom
sarlinpe/two-view-rotation-covariance
Nov 3, 2026
Merged

sarlinpe merged 1 commit into
mainfrom
sarlinpe/two-view-rotation-covariance

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Estimates the 3×3 marginal covariance of a two-view relative rotation from its inlier correspondences:

  • For a calibrated or uncalibrated relative pose $(R, t)$ with $\|t\| = 1$, EstimateTwoViewRotationCovariance evaluates the Jacobian $J$ of the tangent Sampson error over geometry.inlier_matches w.r.t. the right perturbation $R\,\mathrm{Exp}(\delta\theta)$ and the tangent perturbation of $t \in S^2$. This yields the 5-DoF information matrix

$$ \Lambda = \frac{1}{\sigma^2} J^\top J = \begin{pmatrix} \Lambda_{RR} & \Lambda_{Rt} \ \Lambda_{tR} & \Lambda_{tt} \end{pmatrix}, \qquad \Sigma_R = \left(\Lambda_{RR} - \Lambda_{Rt} \Lambda_{tt}^{-1} \Lambda_{tR}\right)^{-1} , $$

where the translation direction is marginalized with the Schur complement and $\sigma$ is the 2D observation noise (point2D_stddev_px, 1 px by default).

  • Pure-rotation (panoramic) pairs use a rotation-only model on the 2D tangent planes orthogonal to the second camera's rays.
  • The standard deviation along poorly or unconstrained axes (e.g., geometrically degenerate configurations) is capped to max_rotation_sigma_deg (60°), with a 1e-5 relative variance floor so single-precision storage preserves positive-definiteness. Returns std::nullopt if the relative pose is missing, has too few valid inliers, or if the eigendecomposition of the information fails.
  • The covariance is expressed in camera 1's local tangent frame, matching RelativeRotationCostFunctor.
  • Exposes ExtractMatchedCamRaysWithJac in estimators/two_view_geometry.h.
  • Adds pycolmap bindings: TwoViewRotationCovarianceOptions and estimate_two_view_rotation_covariance.

Tests

  • TwoViewRotationCovarianceTests.MonteCarloCalibration: 1000-trial Monte Carlo NEES ($\delta\theta^\top \Sigma_R^{-1} \delta\theta$) checks against the rotation MLE across wide-baseline, narrow-baseline, forward-motion, planar, one-sided, and pure-rotation configurations.
  • TwoViewRotationCovarianceOptions.Check, EstimateTwoViewRotationCovariance.Nominal, UnconstrainedAxisIsCapped, and MissingPoseOrTooFewInliers.
  • ExtractMatchedCamRaysWithJac.Nominal.
  • Python binding tests in src/pycolmap/estimators/two_view_rotation_covariance_test.py.

PR stack (1/5 splitting #4823, plus #4912):

  1. Estimate two-view relative rotation covariance #4950 (this PR)
  2. Store and estimate rotation covariances on PoseGraph #4951
  3. Support covariance whitening in CeresRotationAverager #4952
  4. Add COVARIANCE reweighting for Ceres rotation averaging #4953
  5. Enable covariance-weighted rotation averaging in global pipelines #4823
  6. Reuse the pose graph of GlobalPipeline in the component mappers #4912

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM!

@ahojnnes ahojnnes added the full-ci Run Windows and remaining pycolmap CI label Nov 3, 2026
@sarlinpe
sarlinpe merged commit 8653c6b into main Nov 3, 2026
21 checks passed
@sarlinpe
sarlinpe deleted the sarlinpe/two-view-rotation-covariance branch November 3, 2026 13:54
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 3, 2026
## Change

Import relative-rotation covariance estimation and the corrected resized input
frame for FlexGlue keypoints. The merge is clean; retain the fork's default-off
optimistic scoring and both triangulation and covariance test registrations.

## Upstream commits

- `8653c6bd14d159f4f8bdf99bb220a7f54d4101b4` Estimate two-view relative rotation covariance (colmap#4950)
- `e1317ddd0e23abf70e2ad26e58139afb55bf9eac` FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)

## Validation

Private integration preview and source checks passed. Native C++ and Python
qualification remain pending; no covariance calibration or pipeline claim.

AI-assisted integration.
sarlinpe added a commit that referenced this pull request Nov 4, 2026
## Summary

Stores optional relative-rotation covariances on `PoseGraph::Edge` and
estimates them in parallel from a `DatabaseCache`:

- Adds `std::optional<Eigen::Matrix3f> cam2_from_cam1_rotation_cov` to
`PoseGraph::Edge` (single precision to keep edges lightweight),
expressed in camera 1's local tangent frame. `PoseGraph::Edge::Invert()`
conjugates it accordingly.
- Adds `EstimatePoseGraphRotationCovariances`, which populates
`cam2_from_cam1_rotation_cov` in parallel over all valid pose graph
edges that do not already have one, evaluated at each edge's relative
pose from its correspondence-graph inliers. Edges whose covariance
cannot be estimated are left unset. The database schema does not change.
- Adds pycolmap bindings: `PoseGraphEdge.cam2_from_cam1_rotation_cov`
and `estimate_pose_graph_rotation_covariances`.

## Tests

- `PoseGraph.InvertEdgeCovariance`: checks covariance conjugation and
round-trip under `Edge::Invert()`.
- `EstimatePoseGraphRotationCovariances.Nominal`, plus single-precision
inversion positive-definiteness in
`EstimateTwoViewRotationCovariance.UnconstrainedAxisIsCapped` and option
validation in `TwoViewRotationCovarianceOptions.Check`.
- Python binding tests in `pose_graph_test.py` and
`two_view_rotation_covariance_test.py`.

PR stack (2/5 splitting #4823, plus #4912):
1. #4950
2. **#4951** (this PR)
3. #4952
4. #4953
5. #4823
6. #4912
sarlinpe added a commit that referenced this pull request Nov 4, 2026
## Summary

Supports whitening relative-rotation residuals in
`CeresRotationAverager` for both fixed-size camera pairs and dynamic
multi-sensor rig blocks:

- Adds `DynamicCovarianceWeightedCostFunction<kNumResiduals>` in
`estimators/cost_functions/utils.h`, the runtime parameter-block
counterpart of `CovarianceWeightedCostFunction` reusing
`internal::ComputeLeftSqrtInformation`.
- Documents the right-perturbation convention of
`RelativeRotationCostFunctor` (`hat_sensor2_from_sensor1 =
sensor2_from_sensor1_prior * Exp(r)`).
- Adds an optional `cam2_from_cam1_cov` argument to
`CeresRotationAverager::AddRelativeRotationResidual` (and
`CeresRotationAverager.add_relative_rotation_residual` in pycolmap) to
whiten individual relative-rotation residuals with
`DynamicCovarianceWeightedCostFunction<3>`.

## Tests

- `DynamicCovarianceWeightedCostFunction.MatchesCovarianceWeighted` and
`CovarianceWeightedCostFunction.RejectsNonPositiveDefiniteCovariance` in
`estimators/cost_functions/utils_test.cc`.
- `CeresRotationAverager.WhitensAddedRelativeRotationResidual` in
`rotation_averaging_ceres_test.cc` (both fixed 2-block frames and
dynamic rig sensors).
- Python binding test in
`src/pycolmap/estimators/rotation_averaging_ceres_test.py`.

PR stack (3/5 splitting #4823, plus #4912):
1. #4950
2. #4951
3. **#4952** (this PR)
4. #4953
5. #4823
6. #4912
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 4, 2026
## Change

Import relative-rotation covariance estimation and the corrected resized input
frame for FlexGlue keypoints. The merge is clean; retain the fork's default-off
optimistic scoring and both triangulation and covariance test registrations.

## Upstream commits

- `8653c6bd14d159f4f8bdf99bb220a7f54d4101b4` Estimate two-view relative rotation covariance (colmap#4950)
- `e1317ddd0e23abf70e2ad26e58139afb55bf9eac` FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)

## Validation

Private integration preview and source checks passed. Native C++ and Python
qualification remain pending; no covariance calibration or pipeline claim.

AI-assisted integration.
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 7, 2026
Refresh the experimental alternative with fork main while retaining its previous head as a parent. The resulting tree is exactly main plus the isolated shared-current-cost-cache deletion: one CUDA file, +1/-25.

## Change

Read the existing cost map directly during the ordered sampled-cost fold. Preserve current plane propagation, source packing, scheduling and fallback mathematics. The cache originates in custom a37c1c3, MVS: Reuse sampled current costs (GPU perf). The earlier alternative is retained through 8481f89; its original experiment was 8b1412f.

Historical Fountain11 photo+geometry at 3072 recorded 76.85 -> 78.82 seconds, one observation per arm, on the older revision. Retain this inconclusive/adverse observation with its earlier revision; current performance has not been measured. No performance experiment was reopened.

## Validation

- Exact candidate CUDA owner compiled with CUDA 13.2 for compute_86/sm_86. In a bounded retained-support context on RTX 3060, both matched targets passed all 13 selected cases: propagation, ordered source folding, checkpoint-HMM and paired-sweep preservation.
- Four complete inputs and four outputs matched across executables: 8,492,128 bytes per arm. Maps were finite with positive depth and checked ordered graph IDs. Complete witnesses use checkpoint_hmm=false; separate checkpoint tests cover both settings without full cross-executable true-mode witnesses.
- Direct owners, link maps and actual loaded providers were checked; managed jobs ended and released their bookings. Later host resize/warp changes have separate owner qualification. This is not a whole-branch build, reconstruction-quality, portability or current-performance result.

## Upstream commits carried forward

- e52aa8e Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- 8007d1a Preserve float weight ordering in maximum spanning trees (colmap#4928)
- dd84102 Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- 721e185 Add FlexGlue feature matcher (colmap#4908)
- d0a411f FlexGlue: cache the dense features on the GPU (colmap#4915)
- 105817b Hoist camera model dispatch per row in image warps (colmap#4935)
- 27ffad4 Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- e8f3522 Resolve benchmark paths before changing subprocess directories (colmap#4931)
- f664d5d Clamp visibility coordinates before unsigned conversion (colmap#4943)
- 138119b Keep thin bitmap thumbnails nonempty (colmap#4927)
- b16ed5f Count rig frame covisibility once per 3D point (colmap#4893)
- 93dbbcb Add colmap-review agent skill for code review (colmap#4934)
- 2a6f695 Fix sift_test build on MSVC without the GUI (colmap#4955)
- 78cc729 Fix default prior_position_loss_scale to sqrt(chi2_95_3dof) (colmap#4936)
- 8653c6b Estimate two-view relative rotation covariance (colmap#4950)
- e1317dd FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)
- 3db6eb0 Extract ConvertPosePriorsToENU and skip pose priors without position (colmap#4937)
- 77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
- 30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)
- e6644e7 Fix -Wunused-result warning in ThreadPool::AddTask (colmap#4958)
- 5b6da88 Clear descriptor-index results for empty queries (colmap#4960)
- d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953)
- 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823)
- fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants