Skip to content

FlexGlue: share one matcher across multiple worker threads per GPU - #4957

Merged
sarlinpe merged 6 commits into
mainfrom
sarlinpe/flexglue-shared-matcher
Nov 8, 2026
Merged

sarlinpe merged 6 commits into
mainfrom
sarlinpe/flexglue-shared-matcher

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Nov 3, 2026

Copy link
Copy Markdown
Member

Summary

With a single matching worker per GPU, FlexGlue leaves the GPU underutilized while the CPU decodes high-resolution images, resizes them, prepares ONNX tensors, and processes early-stopping If checks. Spawning separate matchers per worker would duplicate both ONNX Runtime's per-session GPU arena (~2.2–2.8 GiB) and the 512 MiB GPU dense-feature cache.

Because Ort::Session::Run supports concurrent calls across threads, multiple worker threads per CUDA GPU can share a single FlexGlueFeatureMatcher instance:

  • Switch FlexGlueFeatureMatcher::dense_cache_ from LRUCache to ThreadSafeLRUCache, which also deduplicates concurrent loads when multiple workers request the same uncached image simultaneously.
  • Make DenseFeatures::checked_camera_size std::atomic<bool> so ImageSize() is lock-free and race-free across workers.
  • Add SharedFeatureMatcher in feature_matching_utils so one designated creator worker initializes the matcher (propagating initialization errors once via Setup()) while sibling workers wait for and share the same instance.
  • Add FlexGlueMatchingOptions::num_workers_per_gpu = 3 (exposed on the CLI as --FlexGlueMatching.num_workers_per_gpu, in pycolmap, and documented in doc/features.rst).

Matches are bit-exact with num_workers_per_gpu = 1; lowering num_workers_per_gpu reduces peak GPU arena memory from concurrent in-flight transformer activations.

Evaluation

Evaluated on 3 ETH3D DSLR scenes (delivery_area, terrains, office: 112 images at 24–25.7 MP, 2,132 exhaustive pairs) with DaD keypoints (2048 and 4096) and default FlexGlue options on an idle NVIDIA A100-SXM4-40GB GPU (12 vCPUs):

#kpts num_workers_per_gpu Matching (ms/pair) Speedup Active GPU Util Peak VRAM (MiB) Raw matches
2048 1 (main) 51.95 1.00× 56.9% 2,659 434,456
2048 2 37.47 1.39× 66.4% 3,793 434,456
2048 3 (default) 33.32 1.56× (−36%) 70.1% 4,937–5,905 434,456
2048 4 31.94 1.63× 73.7% 5,033 434,456
4096 1 (main) 54.93 1.00× 58.2% 3,675 868,694
4096 2 39.65 1.39× 68.3% 4,817 868,694
4096 3 (default) 35.71 1.54× (−35%) 72.3% 5,957 868,694

Tests

  • FlexGlueTest.ConcurrentMatchThreadSafety: verifies concurrent Match() calls across 3 threads hitting cold and warm ThreadSafeLRUCache entries return identical matches.
  • FeatureMatcherController.SharedFeatureMatcherWorkersMatchPairs and SetupPropagatesMatcherCreationErrors: verify shared matcher execution across 3 workers and single-rethrow error propagation when matcher creation fails.
  • FlexGlueMatchingOptions.Check: verifies num_workers_per_gpu default (3) and bounds check (> 0).

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the sharing design looks right. Please fix 1 first: a creator failure before Create() can hang sibling workers in Wait() forever. Findings 2-3 are test gaps (warm-cache coverage, no GPU concurrent-success test) and 4-5 are nits - details inline.

Comment thread src/colmap/feature/flexglue_test.cc Resolved
@sarlinpe
sarlinpe requested a review from ahojnnes November 7, 2026 14:58
@sarlinpe
sarlinpe marked this pull request as ready for review November 7, 2026 14:58
@sarlinpe sarlinpe added the full-ci Run Windows and remaining pycolmap CI label Nov 7, 2026

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the full diff at 28a748a. All five findings from the changes-requested review are addressed: SharedFeatureMatcher::Fail() unblocks waiters on creator setup failure (with the closed-database test), the concurrent FlexGlue test now covers cold and warm cache rounds behind a barrier, and the CUDA controller test asserts bit-exact matches with 3 workers. No new significant findings. Ship.

@sarlinpe
sarlinpe force-pushed the sarlinpe/flexglue-shared-matcher branch from 28a748a to f0b9255 Compare November 8, 2026 20:33
@sarlinpe
sarlinpe enabled stacked auto-merge (squash) November 8, 2026 20:34
@sarlinpe
sarlinpe merged commit df19970 into main Nov 8, 2026
16 checks passed
@sarlinpe
sarlinpe deleted the sarlinpe/flexglue-shared-matcher branch November 8, 2026 22:37
sarlinpe added a commit that referenced this pull request Nov 8, 2026
Stacked on #4957.

### Summary
Decouples CPU image decoding/preprocessing parallelism
(`FlexGlueMatchingOptions::num_workers_per_gpu`, $N$) from concurrent
GPU model execution (`FlexGlueMatchingOptions::num_inferences_per_gpu`,
$M$) in `FlexGlue`:

- **`ONNXInferenceLimiter` (`util/onnx.{h,cc}`)**: Slot-assigning
counting semaphore bounding concurrent GPU inferences across a larger
pool of worker threads, paired with `OrtCUDAProviderOptionsV2`
(`arena_extend_strategy=kSameAsRequested`,
`use_ep_level_unified_stream=1`) and lazy per-slot CUDA `Ort::Session`
replicas (`allow_session_replicas=false` on `FlexGlue`'s 344 MB dense
descriptor so its weights are never duplicated).
- **`FlexGlueMatchingOptions::num_inferences_per_gpu`
(`feature/flexglue.{h,cc}`, CLI, `pycolmap`, docs)**: Bounds concurrent
`descriptor_` + `matcher_` executions per CUDA GPU (default
`num_workers_per_gpu = 4`, `num_inferences_per_gpu = 2`), and releases
decoded full-resolution `Bitmap` buffers immediately after
`ResizeBilinearToCHW` so workers waiting on the semaphore hold only the
resized `512x512` input tensor (`~3 MB` instead of `72–150 MB`).

### Benchmark: $(N, M)$ Grid Search on `delivery_area` (44 images @ 24
MP, 946 pairs, NVIDIA A100-SXM4-40GB)

#### `FLEXGLUE` (`K=2048` keypoints)

| $N$ (`workers`) | $M$ (`inferences`) | Block ms/pair | Block speedup |
Total ms/pair | GPU util | Mem BW | Mean / Peak Power | Peak VRAM |
Matches |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| `1` | `1` | `42.94` | `1.00x` | `46.74` | `56.1%` | `9.5%` | `186 /
304 W` | `1,964 MiB` | `169,581` |
| `3` | `1` | `32.99` | `1.30x` | `36.85` | `70.0%` | `12.2%` | `226 /
310 W` | `1,960 MiB` | `169,581` |
| `6` | `1` | `30.84` | `1.39x` | `34.63` | `74.6%` | `13.0%` | `239 /
311 W` | `1,960 MiB` | `169,581` |
| `2` | `2` | `29.61` | `1.45x` | `33.36` | `63.4%` | `17.3%` | `249 /
398 W` | `2,788 MiB` | `169,581` |
| `4` | `2` | `25.46` | `1.69x` | `29.24` | `71.6%` | `19.7%` | `286 /
407 W` | `2,788 MiB` | `169,581` |
| `3` | `3` | `26.86` | `1.60x` | `30.57` | `66.4%` | `22.2%` | `272 /
421 W` | `3,600 MiB` | `169,581` |
| `6` | `3` | `24.40` | `1.76x` | `28.22` | `71.4%` | `24.2%` | `289 /
417 W` | `3,610 MiB` | `169,581` |
| `8` | `3` | **`23.67`** | **`1.81x`** | **`27.40`** | `73.6%` |
`24.7%` | `296 / 419 W` | `3,616 MiB` | `169,581` |
| `4` | `4` | `26.34` | `1.63x` | `30.13` | `67.0%` | `25.0%` | `276 /
415 W` | `4,412 MiB` | `169,581` |
| `8` | `4` | `24.17` | `1.78x` | `27.97` | `72.8%` | `26.3%` | `291 /
414 W` | `4,440 MiB` | `169,581` |
| `6` | `6` | `27.09` | `1.59x` | `30.89` | `66.9%` | `25.8%` | `272 /
416 W` | `6,038 MiB` | `169,581` |
| `8` | `8` | `27.89` | `1.54x` | `31.71` | `66.6%` | `24.5%` | `266 /
422 W` | `7,708 MiB` | `169,581` |
| `12` | `12` | `29.23` | `1.47x` | `33.04` | `65.6%` | `20.6%` | `253 /
417 W` | `10,982 MiB` | `169,581` |
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 9, 2026
## Summary

Bring in upstream's shared FlexGlue workers and bounded GPU inference slots. CPU preprocessing can overlap while inference concurrency stays independently configurable within each worker group. Preserve the fork's deferred matching-result replacement and CUDA output regression.

## Changes

- Merge the ONNX inference limiter, shared model ownership, worker/inference options and bindings.
- Keep both upstream limiter tests and the existing dynamic-shape CUDA regression when resolving onnx_test.cc.
- Give the setup-failure regression an explicit missing matcher path when model downloads are disabled, so it reaches the intended descriptor-loading error.
- Regressions cover limiter capacity/release, option bounds, worker setup errors and preservation of existing matching results on failure.

## Upstream commits

- df19970 - FlexGlue: share one matcher across multiple worker threads per GPU (colmap#4957)
- 3c1a1b9 - FlexGlue: bound concurrent GPU inferences with a semaphore (colmap#4969)

## Validation

- 42 focused native tests passed without skips: 14 ONNX, 2 FlexGlue option and 26 matching/controller cases. CUDA EP initialization occurred inside the GPU setup-error case; the dynamic-shape CUDA output regression passed.
- clang-tidy-18 completed for six affected production/test translation units with no displayed source diagnostics; generated/header warnings were suppressed. Diff and added-line formatting checks passed.
- Incremental qualification reused 47 production objects and 13 private archives; seven remaining translation units compiled and the CLI linked. Link/provider checks and five help/version smokes passed. The exported-source version reports Unknown commit/date.
- Real FlexGlue model inference, Python and Windows are not qualified by these checks.

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants