Skip to content

FlexGlue: match keypoints in the resized descriptor input frame - #4949

Merged
ahojnnes merged 3 commits into
mainfrom
sarlinpe/flexglue-distance-frame
Nov 3, 2026
Merged

ahojnnes merged 3 commits into
mainfrom
sarlinpe/flexglue-distance-frame

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Nov 3, 2026

Copy link
Copy Markdown
Member

Summary

FlexGlue resizes each input image to a fixed descriptor resolution (512×512 by default), and during training its mutual nearest-neighbor (MNN) search and max_distance_px threshold operate in that resized frame.

Previously, the exported matcher denormalized predicted correspondences back to each image's original (W, H) pixel coordinates and scaled max_distance_px by (max(W0, H0) / 512 + max(W1, H1) / 512) / 2 (see cvg/FlexGlue#16):

  • On non-square images, Euclidean MNN distances and max_distance_px were distorted across axes relative to the 512×512 feature grid (e.g., 50% looser along the short axis of a 3:2 image, and inconsistent on portrait–landscape pairs).
  • On image pairs with different resolutions, the higher-resolution image dominated the averaged bidirectional distance check.

This PR rescales keypoints linearly from (W, H) to (descriptor_width, descriptor_height) in KeypointsToTensorData and passes descriptor_size as image_size0 / image_size1 to the ONNX matcher:

  • Keypoint normalization inside the model (kpts / image_size * 2 - 1) is unchanged.
  • MNN search and max_distance_px now operate directly in the 512×512 descriptor input frame (pixel_scale = max(512, 512) / 512 = 1) using the already-released v0.1 ONNX weights, and give identical results with models exported both before and after Fix distance threshold frame cvg/FlexGlue#16.

Evaluation

End-to-end evaluation with DaD keypoints (2048 and 4096) and default FlexGlue options (min_covisibility = 0.5, early_stop_decay = 0.5, filter_threshold = 0.1), averaged over 3 reconstruction seeds (mean ± std), with matching speed measured on an idle A100 GPU:

ETH3D DSLR (25 scenes, 898 images, 22,569 pairs)

#kpts Version [email protected]° AUC@1° AUC@5° AUC@10° Reg. Raw matches/pair Matching (ms/pair) Verif. (s/scene) Mapper (s/scene)
2048 main 69.99 ± 0.36 82.81 ± 0.31 96.06 ± 0.09 98.00 ± 0.05 898 204.9 46.0 2.42 45.7
2048 This PR 71.02 ± 0.26 83.53 ± 0.10 96.25 ± 0.02 98.10 ± 0.01 898 199.0 45.3 1.80 (−26%) 44.5
4096 main 71.56 ± 0.50 83.70 ± 0.28 96.12 ± 0.04 97.99 ± 0.02 898 408.2 47.2 4.96 84.3
4096 This PR 71.96 ± 0.77 83.96 ± 0.36 96.15 ± 0.07 98.00 ± 0.04 898 398.8 47.4 3.93 (−21%) 87.9
  • On ETH3D, verified geometric inliers per pair slightly increase (160.9 → 161.7 at 2k, 309.0 → 312.3 at 4k) while raw matches drop, raising the inlier ratio (78.5% → 81.2% at 2k, 75.7% → 78.3% at 4k) and reducing geometric verification time by 21–26%.

IMC 2023 (7 scenes, 261 images, 10,935 pairs)

#kpts Version [email protected]° AUC@1° AUC@5° AUC@10° Reg. Raw matches/pair Matching (ms/pair) Verif. (s/scene) Mapper (s/scene)
2048 main 32.28 ± 1.16 54.03 ± 0.88 80.76 ± 0.58 85.07 ± 0.58 255 86.2 37.0 0.94 92.2
2048 This PR 31.69 ± 0.37 53.39 ± 0.32 80.65 ± 0.22 85.05 ± 0.27 255 80.6 37.0 0.72 (−23%) 85.7 (−7%)
4096 main 31.82 ± 0.47 54.26 ± 0.51 82.85 ± 0.78 87.58 ± 1.22 259 191.2 40.1 2.37 202.4
4096 This PR 27.67 ± 1.72 50.01 ± 2.09 81.34 ± 0.15 86.87 ± 0.77 260 180.9 40.1 1.91 (−19%) 186.3 (−8%)
  • Mixed portrait/landscape scenes improve (heritage/dioscuri: +2.96 [email protected]° at 2k, +1.10 [email protected]° / +1.97 AUC@5° at 4k), and geometric verification is 19–23% faster.
  • On the 7-scene IMC average, the drop in fine-threshold AUC at 4k comes primarily from heritage/wall (34 images, 6048×4032), where registering all 34/34 views across all 3 seeds (vs. 2/3 seeds on main) raises wall AUC@10° (+1.74) while 4 marginal wide-baseline views are registered with coarse poses that lower wall [email protected]°, alongside tightening the short-axis MNN window on 3:2 images from 4.5 to 3.0 resized pixels (heritage/cyprus 24 → 23 registered at 4k).

Tests

  • Extended FlexGlueTest.MatchTranslatedImages to verify that anisotropically rescaling an image's camera dimensions and keypoints (4W × 2H) leaves the matches unchanged.

sarlinpe and others added 2 commits November 3, 2026 00:38
Rescale keypoints linearly from the original image dimensions to the resized descriptor input dimensions (512x512 by default) and pass that size to the ONNX matcher, so that mutual nearest-neighbor search and max_distance_px operate in the descriptor input frame (matching cvg/FlexGlue#16, with models exported both before and after it).
@sarlinpe
sarlinpe requested a review from ahojnnes November 3, 2026 00:43
@sarlinpe sarlinpe added the full-ci Run Windows and remaining pycolmap CI label Nov 3, 2026
@ahojnnes
ahojnnes merged commit e1317dd into main Nov 3, 2026
16 checks passed
@ahojnnes
ahojnnes deleted the sarlinpe/flexglue-distance-frame branch November 3, 2026 15:06
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 3, 2026
## Change

Import relative-rotation covariance estimation and the corrected resized input
frame for FlexGlue keypoints. The merge is clean; retain the fork's default-off
optimistic scoring and both triangulation and covariance test registrations.

## Upstream commits

- `8653c6bd14d159f4f8bdf99bb220a7f54d4101b4` Estimate two-view relative rotation covariance (colmap#4950)
- `e1317ddd0e23abf70e2ad26e58139afb55bf9eac` FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)

## Validation

Private integration preview and source checks passed. Native C++ and Python
qualification remain pending; no covariance calibration or pipeline claim.

AI-assisted integration.
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 4, 2026
## Change

Import relative-rotation covariance estimation and the corrected resized input
frame for FlexGlue keypoints. The merge is clean; retain the fork's default-off
optimistic scoring and both triangulation and covariance test registrations.

## Upstream commits

- `8653c6bd14d159f4f8bdf99bb220a7f54d4101b4` Estimate two-view relative rotation covariance (colmap#4950)
- `e1317ddd0e23abf70e2ad26e58139afb55bf9eac` FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)

## Validation

Private integration preview and source checks passed. Native C++ and Python
qualification remain pending; no covariance calibration or pipeline claim.

AI-assisted integration.
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 7, 2026
Refresh the experimental alternative with fork main while retaining its previous head as a parent. The resulting tree is exactly main plus the isolated shared-current-cost-cache deletion: one CUDA file, +1/-25.

## Change

Read the existing cost map directly during the ordered sampled-cost fold. Preserve current plane propagation, source packing, scheduling and fallback mathematics. The cache originates in custom a37c1c3, MVS: Reuse sampled current costs (GPU perf). The earlier alternative is retained through 8481f89; its original experiment was 8b1412f.

Historical Fountain11 photo+geometry at 3072 recorded 76.85 -> 78.82 seconds, one observation per arm, on the older revision. Retain this inconclusive/adverse observation with its earlier revision; current performance has not been measured. No performance experiment was reopened.

## Validation

- Exact candidate CUDA owner compiled with CUDA 13.2 for compute_86/sm_86. In a bounded retained-support context on RTX 3060, both matched targets passed all 13 selected cases: propagation, ordered source folding, checkpoint-HMM and paired-sweep preservation.
- Four complete inputs and four outputs matched across executables: 8,492,128 bytes per arm. Maps were finite with positive depth and checked ordered graph IDs. Complete witnesses use checkpoint_hmm=false; separate checkpoint tests cover both settings without full cross-executable true-mode witnesses.
- Direct owners, link maps and actual loaded providers were checked; managed jobs ended and released their bookings. Later host resize/warp changes have separate owner qualification. This is not a whole-branch build, reconstruction-quality, portability or current-performance result.

## Upstream commits carried forward

- e52aa8e Move pycolmap macOS CI from macos-14 to macos-15 runner (colmap#4933)
- 8007d1a Preserve float weight ordering in maximum spanning trees (colmap#4928)
- dd84102 Bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 in /.github/workflows (colmap#4942)
- 721e185 Add FlexGlue feature matcher (colmap#4908)
- d0a411f FlexGlue: cache the dense features on the GPU (colmap#4915)
- 105817b Hoist camera model dispatch per row in image warps (colmap#4935)
- 27ffad4 Make benchmark translation-angle errors invariant to reconstruction scale (colmap#4932)
- e8f3522 Resolve benchmark paths before changing subprocess directories (colmap#4931)
- f664d5d Clamp visibility coordinates before unsigned conversion (colmap#4943)
- 138119b Keep thin bitmap thumbnails nonempty (colmap#4927)
- b16ed5f Count rig frame covisibility once per 3D point (colmap#4893)
- 93dbbcb Add colmap-review agent skill for code review (colmap#4934)
- 2a6f695 Fix sift_test build on MSVC without the GUI (colmap#4955)
- 78cc729 Fix default prior_position_loss_scale to sqrt(chi2_95_3dof) (colmap#4936)
- 8653c6b Estimate two-view relative rotation covariance (colmap#4950)
- e1317dd FlexGlue: match keypoints in the resized descriptor input frame (colmap#4949)
- 3db6eb0 Extract ConvertPosePriorsToENU and skip pose priors without position (colmap#4937)
- 77cbe48 Store and estimate rotation covariances on PoseGraph (colmap#4951)
- 30a1a81 Support covariance whitening in CeresRotationAverager (colmap#4952)
- e6644e7 Fix -Wunused-result warning in ThreadPool::AddTask (colmap#4958)
- 5b6da88 Clear descriptor-index results for empty queries (colmap#4960)
- d1d92d9 Add COVARIANCE reweighting for Ceres rotation averaging (colmap#4953)
- 761d90d Enable covariance-weighted rotation averaging in global pipelines (colmap#4823)
- fdee2e1 Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants