Skip to content

Fix distance threshold frame - #16

Merged
Phil26AT merged 5 commits into
mainfrom
resize-frame
Nov 3, 2026
Merged

Phil26AT merged 5 commits into
mainfrom
resize-frame

Conversation

@Phil26AT

Copy link
Copy Markdown
Contributor

So far the distance thresholds were rescaled to 512 / max(h,w). This is inconsistent with how we trained it for: There we apply thresholding in the actual input shape of desriptors, so (512,512). This PR applies this fix. This boosts results on some benchmarks.

fringehush-gif added a commit to fringehush-gif/FlexGlue that referenced this pull request Nov 1, 2026
## Summary
Measure cycle errors, borders and sparse endpoint distances in each image's descriptor input frame, using both axes instead of a single long-side scale. Returned point coordinates remain in original-image pixels.

Derive the frame from existing feature geometry, so cached features and direct ONNX exporter packets need no new metadata. Floating geometry retains fractional borders with integer image sizes and reduced-precision feature caches.

FountainP11, 1536 / 3072 px:
- Mean wall: eager 1.075 -> 1.076 / 1.631 -> 1.627 s; compiled 0.815 -> 0.826 / 1.402 -> 1.399 s.
- Process CPU: eager 1.408 -> 1.404 / 2.460 -> 2.446 s; compiled 1.146 -> 1.158 / 2.244 -> 2.231 s. Peak GPU allocation unchanged.
- Eager valid dense samples: 59,250 -> 59,070 / 59,230 -> 59,050. Predicted coordinates and covisibility fields match exactly; this corrects filtering rather than learned predictions.

11 images, 12 pairs, 512-square descriptor, FP32 descriptor/FP16 matcher, 12 CPU threads. Warm each arm/resolution then CTTC, two observations per arm/mode. RSS peaks: eager 2220 -> 2220 / 2294 -> 2285 MiB; compiled 2480 -> 2480 / 2608 -> 2590 MiB. Memory scope is the comparison process with both model instances resident.

Inspired by cvg#16; extends the correction to shared exporter/cache inputs and dtype contracts.

## Validation
- Analytic cases cover unequal aspect ratios and image scales, square/nonsquare grids, fractional borders, cached integer sizes and float16/32/64 feature geometry.
- A 2-pixel descriptor error is rejected at threshold 2 and accepted at 2.1; disabled Python thresholds and runtime tensor thresholds retain their distinct conventions.
- Eight fused/unfused ONNX Runtime cases pass with nonsquare exports, dynamic keypoint counts, 1536/3072 image sizes and nonempty accept/all-reject cases.
- Linux and Windows CPU suites: 84 tests run each, zero failures/errors (4/5 optional skips). Ten frame/distance tests also pass on CUDA with no skips. Full pretrained ONNX parity is checked by the existing workflow.

AI disclosure: totally vibe coded.
ahojnnes pushed a commit to colmap/colmap that referenced this pull request Nov 3, 2026
## Summary

FlexGlue resizes each input image to a fixed descriptor resolution
(`512×512` by default), and during training its mutual nearest-neighbor
(MNN) search and `max_distance_px` threshold operate in that resized
frame.

Previously, the exported matcher denormalized predicted correspondences
back to each image's original `(W, H)` pixel coordinates and scaled
`max_distance_px` by `(max(W0, H0) / 512 + max(W1, H1) / 512) / 2` (see
cvg/FlexGlue#16):
- On non-square images, Euclidean MNN distances and `max_distance_px`
were distorted across axes relative to the `512×512` feature grid (e.g.,
50% looser along the short axis of a 3:2 image, and inconsistent on
portrait–landscape pairs).
- On image pairs with different resolutions, the higher-resolution image
dominated the averaged bidirectional distance check.

This PR rescales keypoints linearly from `(W, H)` to `(descriptor_width,
descriptor_height)` in `KeypointsToTensorData` and passes
`descriptor_size` as `image_size0` / `image_size1` to the ONNX matcher:
- Keypoint normalization inside the model (`kpts / image_size * 2 - 1`)
is unchanged.
- MNN search and `max_distance_px` now operate directly in the `512×512`
descriptor input frame (`pixel_scale = max(512, 512) / 512 = 1`) using
the already-released `v0.1` ONNX weights, and give identical results
with models exported both before and after
cvg/FlexGlue#16.

## Evaluation

End-to-end evaluation with DaD keypoints (2048 and 4096) and default
FlexGlue options (`min_covisibility = 0.5`, `early_stop_decay = 0.5`,
`filter_threshold = 0.1`), averaged over 3 reconstruction seeds (`mean ±
std`), with matching speed measured on an idle A100 GPU:

### ETH3D DSLR (25 scenes, 898 images, 22,569 pairs)

| #kpts | Version | [email protected]° | AUC@1° | AUC@5° | AUC@10° | Reg. | Raw
matches/pair | Matching (ms/pair) | Verif. (s/scene) | Mapper (s/scene)
|
|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 2048 | `main` | 69.99 ± 0.36 | 82.81 ± 0.31 | 96.06 ± 0.09 | 98.00 ±
0.05 | 898 | 204.9 | 46.0 | 2.42 | 45.7 |
| 2048 | **This PR** | **71.02 ± 0.26** | **83.53 ± 0.10** | **96.25 ±
0.02** | **98.10 ± 0.01** | 898 | 199.0 | 45.3 | **1.80 (−26%)** | 44.5
|
| 4096 | `main` | 71.56 ± 0.50 | 83.70 ± 0.28 | 96.12 ± 0.04 | 97.99 ±
0.02 | 898 | 408.2 | 47.2 | 4.96 | 84.3 |
| 4096 | **This PR** | **71.96 ± 0.77** | **83.96 ± 0.36** | **96.15 ±
0.07** | **98.00 ± 0.04** | 898 | 398.8 | 47.4 | **3.93 (−21%)** | 87.9
|

- On ETH3D, verified geometric inliers per pair slightly *increase*
(`160.9 → 161.7` at 2k, `309.0 → 312.3` at 4k) while raw matches drop,
raising the inlier ratio (`78.5% → 81.2%` at 2k, `75.7% → 78.3%` at 4k)
and reducing geometric verification time by 21–26%.

### IMC 2023 (7 scenes, 261 images, 10,935 pairs)

| #kpts | Version | [email protected]° | AUC@1° | AUC@5° | AUC@10° | Reg. | Raw
matches/pair | Matching (ms/pair) | Verif. (s/scene) | Mapper (s/scene)
|
|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 2048 | `main` | 32.28 ± 1.16 | 54.03 ± 0.88 | 80.76 ± 0.58 | 85.07 ±
0.58 | 255 | 86.2 | 37.0 | 0.94 | 92.2 |
| 2048 | **This PR** | 31.69 ± 0.37 | 53.39 ± 0.32 | 80.65 ± 0.22 |
85.05 ± 0.27 | 255 | 80.6 | 37.0 | **0.72 (−23%)** | **85.7 (−7%)** |
| 4096 | `main` | 31.82 ± 0.47 | 54.26 ± 0.51 | 82.85 ± 0.78 | 87.58 ±
1.22 | 259 | 191.2 | 40.1 | 2.37 | 202.4 |
| 4096 | **This PR** | 27.67 ± 1.72 | 50.01 ± 2.09 | 81.34 ± 0.15 |
86.87 ± 0.77 | **260** | 180.9 | 40.1 | **1.91 (−19%)** | **186.3
(−8%)** |

- Mixed portrait/landscape scenes improve (`heritage/dioscuri`: `+2.96`
[email protected]° at 2k, `+1.10` [email protected]° / `+1.97` AUC@5° at 4k), and geometric
verification is 19–23% faster.
- On the 7-scene IMC average, the drop in fine-threshold AUC at 4k comes
primarily from `heritage/wall` (34 images, `6048×4032`), where
registering all 34/34 views across all 3 seeds (vs. 2/3 seeds on `main`)
raises `wall` `AUC@10°` (`+1.74`) while 4 marginal wide-baseline views
are registered with coarse poses that lower `wall` `[email protected]°`, alongside
tightening the short-axis MNN window on 3:2 images from `4.5` to `3.0`
resized pixels (`heritage/cyprus` `24 → 23` registered at 4k).

## Tests

- Extended `FlexGlueTest.MatchTranslatedImages` to verify that
anisotropically rescaling an image's camera dimensions and keypoints
(`4W × 2H`) leaves the matches unchanged.
@Phil26AT
Phil26AT merged commit c6cbc7a into main Nov 3, 2026
2 checks passed
fringehush-gif referenced this pull request in fringehush-gif/FlexGlue Nov 9, 2026
## Summary

Reuse caller-provided refinement features and update the notebook's RaCo source.
Refresh from official c6cbc7a while retaining the fork's tested defaults.

## Changes

- Adopt cfd752c, "Accept precomputed refinement features" (#10), and
  7b0b5f3, "load RaCo from main" (#17).
- Retain our descriptor-frame conversion and empty-input handling when resolving
  c6cbc7a, "Fix distance threshold frame" (#16).
- Defer 6075232, "Compile image encoding and RoMaV2 refinement" (#9).
  Earlier encoder tests had mixed end-to-end cost/output results; compiled
  refiner output lifetime needs a replay test. Existing compilation stays intact.

## Validation

- Linux Torch 2.14.1: 116 tests run, zero failures/errors; 11 CUDA tests and
  the optional ONNX test class skipped.
- Added both-side and mixed cached/missing refinement-routing regressions.
- Independent review accepted the narrowed source; no new performance claim.

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants