Repository navigation
Enable covariance-weighted rotation averaging in global pipelines - #4823
Merged
sarlinpe merged 4 commits intoNov 6, 2026
Conversation
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
October 8, 2026 08:16
fbe4090 to
5e32aa9
Compare
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
October 8, 2026 09:19
5e32aa9 to
6a43cb2
Compare
B1ueber2y
reviewed
Oct 8, 2026
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
October 8, 2026 12:22
6a43cb2 to
59936ad
Compare
ahojnnes
reviewed
Oct 9, 2026
ahojnnes
left a comment
Contributor
There was a problem hiding this comment.
Reviewed the covariance estimation math, Ceres whitening, integration, and tests. No blockers: frame conventions (right-perturb residual vs cam1-frame cov), Invert conjugation, Schur complement, degeneracy gate, and opt-in gating (default stays UNIFORM) all check out. A few minors inline.
B1ueber2y
reviewed
Oct 10, 2026
B1ueber2y
reviewed
Oct 10, 2026
B1ueber2y
reviewed
Oct 18, 2026
B1ueber2y
reviewed
Oct 25, 2026
B1ueber2y
left a comment
Contributor
There was a problem hiding this comment.
Thank you! It s unfortunate that there is no clear E2E improvement, but I think the feature itself is useful.
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
October 28, 2026 01:41
2d83ffd to
4efdb7a
Compare
sarlinpe
changed the base branch from
main
to
sarlinpe/covariance-weighted-cost-function
October 28, 2026 01:41
Base automatically changed from
sarlinpe/covariance-weighted-cost-function
to
main
October 28, 2026 20:33
sarlinpe
added a commit
that referenced
this pull request
Oct 28, 2026
## Summary `CovarianceWeightedCostFunction` leaked the wrapped cost function when computing the square root information threw, and silently produced non-finite weights for covariances that are not positive definite. - The owning `std::unique_ptr` member is now declared, and thus initialized, before the square root information, so the wrapped cost function is released if that throws. - A null cost function is rejected with `THROW_CHECK_NOTNULL`. - The covariance must be positive definite (finite inverse and successful Cholesky factorization), otherwise the constructor throws `std::invalid_argument` instead of producing NaN weights. - The square root information is computed by a shared helper, `internal::ComputeLeftSqrtInformation`, which #4823 reuses for a dynamic-size counterpart. ## Tests `CovarianceWeightedCostFunction.RejectsNonPositiveDefiniteCovariance` checks that zero and indefinite covariances, and a null cost function, throw. Under ASan, it also checks that the wrapped cost function does not leak. This is the first PR of a stack: #4823 builds on it, and #4912 builds on #4823.
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
October 28, 2026 20:33
07a5c49 to
8423e45
Compare
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
November 1, 2026 21:02
0076f4b to
c8f1dbb
Compare
ahojnnes
reviewed
Nov 2, 2026
ahojnnes
reviewed
Nov 2, 2026
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
November 3, 2026 01:19
779c4f7 to
11a62ef
Compare
sarlinpe
changed the base branch from
main
to
sarlinpe/rotation-averaging-covariance-reweighting
November 3, 2026 01:24
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
November 3, 2026 08:26
11a62ef to
e1af190
Compare
sarlinpe
added a commit
that referenced
this pull request
Nov 3, 2026
## Summary
Estimates the 3×3 marginal covariance of a two-view relative rotation
from its inlier correspondences:
- For a calibrated or uncalibrated relative pose $`(R, t)`$ with $`\|t\|
= 1`$, `EstimateTwoViewRotationCovariance` evaluates the Jacobian $`J`$
of the tangent Sampson error over `geometry.inlier_matches` w.r.t. the
right perturbation $`R\,\mathrm{Exp}(\delta\theta)`$ and the tangent
perturbation of $`t \in S^2`$. This yields the 5-DoF information matrix
$$
\Lambda = \frac{1}{\sigma^2} J^\top J = \begin{pmatrix} \Lambda_{RR} &
\Lambda_{Rt} \\ \Lambda_{tR} & \Lambda_{tt} \end{pmatrix},
\qquad
\Sigma_R = \left(\Lambda_{RR} - \Lambda_{Rt} \Lambda_{tt}^{-1}
\Lambda_{tR}\right)^{-1} ,
$$
where the translation direction is marginalized with the Schur
complement and $`\sigma`$ is the 2D observation noise
(`point2D_stddev_px`, 1 px by default).
- Pure-rotation (panoramic) pairs use a rotation-only model on the 2D
tangent planes orthogonal to the second camera's rays.
- The standard deviation along poorly or unconstrained axes (e.g.,
geometrically degenerate configurations) is capped to
`max_rotation_sigma_deg` (60°), with a `1e-5` relative variance floor so
single-precision storage preserves positive-definiteness. Returns
`std::nullopt` if the relative pose is missing, has too few valid
inliers, or if the eigendecomposition of the information fails.
- The covariance is expressed in camera 1's local tangent frame,
matching `RelativeRotationCostFunctor`.
- Exposes `ExtractMatchedCamRaysWithJac` in
`estimators/two_view_geometry.h`.
- Adds pycolmap bindings: `TwoViewRotationCovarianceOptions` and
`estimate_two_view_rotation_covariance`.
## Tests
- `TwoViewRotationCovarianceTests.MonteCarloCalibration`: 1000-trial
Monte Carlo NEES ($`\delta\theta^\top \Sigma_R^{-1} \delta\theta`$)
checks against the rotation MLE across wide-baseline, narrow-baseline,
forward-motion, planar, one-sided, and pure-rotation configurations.
- `TwoViewRotationCovarianceOptions.Check`,
`EstimateTwoViewRotationCovariance.Nominal`,
`UnconstrainedAxisIsCapped`, and `MissingPoseOrTooFewInliers`.
- `ExtractMatchedCamRaysWithJac.Nominal`.
- Python binding tests in
`src/pycolmap/estimators/two_view_rotation_covariance_test.py`.
PR stack (1/5 splitting #4823, plus #4912):
1. **#4950** (this PR)
2. #4951
3. #4952
4. #4953
5. #4823
6. #4912
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
2 times, most recently
from
November 3, 2026 14:13
4b254d2 to
e847aec
Compare
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
November 3, 2026 16:45
e847aec to
e49a10c
Compare
sarlinpe
added a commit
that referenced
this pull request
Nov 4, 2026
## Summary Stores optional relative-rotation covariances on `PoseGraph::Edge` and estimates them in parallel from a `DatabaseCache`: - Adds `std::optional<Eigen::Matrix3f> cam2_from_cam1_rotation_cov` to `PoseGraph::Edge` (single precision to keep edges lightweight), expressed in camera 1's local tangent frame. `PoseGraph::Edge::Invert()` conjugates it accordingly. - Adds `EstimatePoseGraphRotationCovariances`, which populates `cam2_from_cam1_rotation_cov` in parallel over all valid pose graph edges that do not already have one, evaluated at each edge's relative pose from its correspondence-graph inliers. Edges whose covariance cannot be estimated are left unset. The database schema does not change. - Adds pycolmap bindings: `PoseGraphEdge.cam2_from_cam1_rotation_cov` and `estimate_pose_graph_rotation_covariances`. ## Tests - `PoseGraph.InvertEdgeCovariance`: checks covariance conjugation and round-trip under `Edge::Invert()`. - `EstimatePoseGraphRotationCovariances.Nominal`, plus single-precision inversion positive-definiteness in `EstimateTwoViewRotationCovariance.UnconstrainedAxisIsCapped` and option validation in `TwoViewRotationCovarianceOptions.Check`. - Python binding tests in `pose_graph_test.py` and `two_view_rotation_covariance_test.py`. PR stack (2/5 splitting #4823, plus #4912): 1. #4950 2. **#4951** (this PR) 3. #4952 4. #4953 5. #4823 6. #4912
sarlinpe
added a commit
that referenced
this pull request
Nov 4, 2026
## Summary Supports whitening relative-rotation residuals in `CeresRotationAverager` for both fixed-size camera pairs and dynamic multi-sensor rig blocks: - Adds `DynamicCovarianceWeightedCostFunction<kNumResiduals>` in `estimators/cost_functions/utils.h`, the runtime parameter-block counterpart of `CovarianceWeightedCostFunction` reusing `internal::ComputeLeftSqrtInformation`. - Documents the right-perturbation convention of `RelativeRotationCostFunctor` (`hat_sensor2_from_sensor1 = sensor2_from_sensor1_prior * Exp(r)`). - Adds an optional `cam2_from_cam1_cov` argument to `CeresRotationAverager::AddRelativeRotationResidual` (and `CeresRotationAverager.add_relative_rotation_residual` in pycolmap) to whiten individual relative-rotation residuals with `DynamicCovarianceWeightedCostFunction<3>`. ## Tests - `DynamicCovarianceWeightedCostFunction.MatchesCovarianceWeighted` and `CovarianceWeightedCostFunction.RejectsNonPositiveDefiniteCovariance` in `estimators/cost_functions/utils_test.cc`. - `CeresRotationAverager.WhitensAddedRelativeRotationResidual` in `rotation_averaging_ceres_test.cc` (both fixed 2-block frames and dynamic rig sensors). - Python binding test in `src/pycolmap/estimators/rotation_averaging_ceres_test.py`. PR stack (3/5 splitting #4823, plus #4912): 1. #4950 2. #4951 3. **#4952** (this PR) 4. #4953 5. #4823 6. #4912
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
November 4, 2026 00:55
e49a10c to
f58255c
Compare
ahojnnes
approved these changes
Nov 5, 2026
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
2 times, most recently
from
November 6, 2026 01:17
67b7c8f to
ae325b6
Compare
…VARIANCE fallback
sarlinpe
force-pushed
the
sarlinpe/rotation-averaging-covariance
branch
from
November 6, 2026 11:34
ae325b6 to
e3bc515
Compare
fringehush-gif
added a commit
to fringehush-gif/colmap
that referenced
this pull request
Nov 7, 2026
Merge the upstream covariance-weighted rotation averaging and component pose-graph work. ## Change - Add the upstream covariance-weighted Ceres rotation mode and global-pipeline option. Existing L1_IRLS and UNIFORM defaults remain. - Reuse the prebuilt pipeline pose graph when constructing component subgraphs. - Preserve the fork's option registration changes through the disjoint merge. ## Validation - The object-level merge is conflict-free; incoming paths do not overlap the four existing dirty files. - Native integration, affected dependent binaries and bindings remain pending. Defaulted C++ arguments do not preserve binary ABI. ## Upstream commits - d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953) - 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823) - fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912) AI disclosure: Totally vibe coded
fringehush-gif
added a commit
to fringehush-gif/colmap
that referenced
this pull request
Nov 7, 2026
Refresh the experimental alternative with fork main while retaining its previous head as a parent. The resulting tree is exactly main plus the isolated shared-current-cost-cache deletion: one CUDA file, +1/-25. ## Change Read the existing cost map directly during the ordered sampled-cost fold. Preserve current plane propagation, source packing, scheduling and fallback mathematics. The cache originates in custom a37c1c3, MVS: Reuse sampled current costs (GPU perf). The earlier alternative is retained through 8481f89; its original experiment was 8b1412f. Historical Fountain11 photo+geometry at 3072 recorded 76.85 -> 78.82 seconds, one observation per arm, on the older revision. Retain this inconclusive/adverse observation with its earlier revision; current performance has not been measured. No performance experiment was reopened. ## Validation - Exact candidate CUDA owner compiled with CUDA 13.2 for compute_86/sm_86. In a bounded retained-support context on RTX 3060, both matched targets passed all 13 selected cases: propagation, ordered source folding, checkpoint-HMM and paired-sweep preservation. - Four complete inputs and four outputs matched across executables: 8,492,128 bytes per arm. Maps were finite with positive depth and checked ordered graph IDs. Complete witnesses use checkpoint_hmm=false; separate checkpoint tests cover both settings without full cross-executable true-mode witnesses. - Direct owners, link maps and actual loaded providers were checked; managed jobs ended and released their bookings. Later host resize/warp changes have separate owner qualification. This is not a whole-branch build, reconstruction-quality, portability or current-performance result. ## Upstream commits carried forward - e52aa8e Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933) - 8007d1a Preserve float weight ordering in maximum spanning trees (colmap#4928) - dd84102 Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942) - 721e185 Add FlexGlue feature matcher (colmap#4908) - d0a411f FlexGlue: cache the dense features on the GPU (colmap#4915) - 105817b Hoist camera model dispatch per row in image warps (colmap#4935) - 27ffad4 Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932) - e8f3522 Resolve benchmark paths before changing subprocess directories (colmap#4931) - f664d5d Clamp visibility coordinates before unsigned conversion (colmap#4943) - 138119b Keep thin bitmap thumbnails nonempty (colmap#4927) - b16ed5f Count rig frame covisibility once per 3D point (colmap#4893) - 93dbbcb Add colmap-review agent skill for code review (colmap#4934) - 2a6f695 Fix sift_test build on MSVC without the GUI (colmap#4955) - 78cc729 Fix default prior_position_loss_scale to sqrt(chi2_95_3dof) (colmap#4936) - 8653c6b Estimate two-view relative rotation covariance (colmap#4950) - e1317dd FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949) - 3db6eb0 Extract ConvertPosePriorsToENU and skip pose priors without position (colmap#4937) - 77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951) - 30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952) - e6644e7 Fix -Wunused-result warning in ThreadPool::AddTask (colmap#4958) - 5b6da88 Clear descriptor-index results for empty queries (colmap#4960) - d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953) - 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823) - fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912) AI disclosure: Totally vibe coded
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires
RotationAveragingReweighting::COVARIANCEinto the global SfM and rotation-averaging pipelines, and adds CLI flags:MaybeEstimatePoseGraphRotationCovariancesinestimators/rotation_averaging.h: whenreweighting == COVARIANCE, estimates the rotation covariances of valid pose graph edges that do not have one yet (usingceres.solver_options.num_threadsthreads) and invalidates edges whose covariance cannot be estimated. Throws ifbackend != CERES, warns and skips when falling back toL1_IRLSfor gravity priors, and is a no-op for other reweightings. Also bound in pycolmap asmaybe_estimate_pose_graph_rotation_covariances.MaybeEstimatePoseGraphRotationCovariancesinGlobalMapper::RotationAveraging,GlobalPipeline::ReconstructMultiComponents, andRotationAveragingPipeline::Run.--GlobalMapper.ra_ceres_covariance_sigma_floor_degand--GlobalMapper.ra_ceres_covariance_loss_scaleinOptionManager.The default reweighting remains
UNIFORM(COVARIANCEis opt-in).Tests
MaybeEstimatePoseGraphRotationCovariances.Nominalinestimators/rotation_averaging_test.ccandtest_maybe_estimate_pose_graph_rotation_covariancesinpycolmap/estimators/motion_averaging_test.py.GlobalMapper.WithNoiseAndOutliers,RotationAveragingPipeline.WithNoiseAndOutliers, andGlobalPipeline.MultiComponents(each also tested withCOVARIANCE).Evaluation
Evaluation Settings
--quality high,--use_gpu 1), exhaustive matching andCalibrateViewGraph.GlobalPipelinewith 1 thread per scene andrandom_seed = 0.0.5° / 1° / 5° / 10°, computed for two stages:GlobalPipeline.CERES: the default Ceres configuration (5 Huber warm-start iterations, then Cauchy with σ = 0.5°).COVARIANCE: the same configuration withreweighting = COVARIANCE, using the defaultscovariance_loss_scale = 3andcovariance_sigma_floor_deg = 0.02°.ceres.loss_function_scale(0.5° by default).Rotation averaging only (RA)
L1_IRLS(default)CERESCERES+COVARIANCECERES+COVARIANCE, warm-start σ = 1° (ceres.loss_function_scale = 1°)CERES+COVARIANCE, floor 0°CERES+COVARIANCE, floor 0.05°, scale 2CERES+COVARIANCE, inlier-count spanning treeCERES+COVARIANCE, warm-start σ = 1°, inlier-count spanning treeEnd-to-end (E2E, seed 0)
L1_IRLS(default)CERESCERES+COVARIANCECERES+COVARIANCE, warm-start σ = 1°E2E numbers are measured on the branch merged with the latest
main. E2E times come from runs with 12 scenes in parallel and vary with worker contention:L1_IRLSwas run separately with fewer concurrent tasks, so its IMC time is not directly comparable. In both tables, the times of theCOVARIANCErows come from runs before the latest review changes, which do not change the runtime; the latest runs were slowed down by an unrelated load on the machine.On IMC, 3 seeds (mean ± std) give the following avg AUC:
L1_IRLS(default)CERESCERES+COVARIANCECERES+COVARIANCE, warm-start σ = 1°Takeaways
COVARIANCEis +0.03 AUC aboveCERESon average. The average gap comes from a single 15-image scene,meadow, whose global positioning is bimodal across seeds. On IMC, the differences at the fine thresholds are within seed noise (2–3 AUC points), and at the coarse thresholdsCOVARIANCEis slightly aboveCERES.multiple_models, the covariances are estimated once for the component split and again in each component mapper; Reuse the pose graph of GlobalPipeline in the component mappers #4912 removes this duplicate.COVARIANCEstays opt-in. The default reweighting staysUNIFORM.Stack created with GitHub Stacks CLI • Give Feedback 💬