Repository navigation
FlexGlue: share one matcher across multiple worker threads per GPU - #4957
Merged
Merged
Conversation
sarlinpe
marked this pull request as draft
November 4, 2026 00:21
sarlinpe
force-pushed
the
sarlinpe/flexglue-shared-matcher
branch
from
November 5, 2026 13:05
317f3e5 to
ebd27cb
Compare
ahojnnes
requested changes
Nov 6, 2026
ahojnnes
left a comment
Contributor
There was a problem hiding this comment.
Thanks, the sharing design looks right. Please fix 1 first: a creator failure before Create() can hang sibling workers in Wait() forever. Findings 2-3 are test gaps (warm-cache coverage, no GPU concurrent-success test) and 4-5 are nits - details inline.
ahojnnes
approved these changes
Nov 8, 2026
ahojnnes
left a comment
Contributor
There was a problem hiding this comment.
Re-reviewed the full diff at 28a748a. All five findings from the changes-requested review are addressed: SharedFeatureMatcher::Fail() unblocks waiters on creator setup failure (with the closed-database test), the concurrent FlexGlue test now covers cold and warm cache rounds behind a barrier, and the CUDA controller test asserts bit-exact matches with 3 workers. No new significant findings. Ship.
sarlinpe
force-pushed
the
sarlinpe/flexglue-shared-matcher
branch
from
November 8, 2026 20:33
28a748a to
f0b9255
Compare
sarlinpe
enabled stacked auto-merge (squash)
November 8, 2026 20:34
sarlinpe
added a commit
that referenced
this pull request
Nov 8, 2026
Stacked on #4957. ### Summary Decouples CPU image decoding/preprocessing parallelism (`FlexGlueMatchingOptions::num_workers_per_gpu`, $N$) from concurrent GPU model execution (`FlexGlueMatchingOptions::num_inferences_per_gpu`, $M$) in `FlexGlue`: - **`ONNXInferenceLimiter` (`util/onnx.{h,cc}`)**: Slot-assigning counting semaphore bounding concurrent GPU inferences across a larger pool of worker threads, paired with `OrtCUDAProviderOptionsV2` (`arena_extend_strategy=kSameAsRequested`, `use_ep_level_unified_stream=1`) and lazy per-slot CUDA `Ort::Session` replicas (`allow_session_replicas=false` on `FlexGlue`'s 344 MB dense descriptor so its weights are never duplicated). - **`FlexGlueMatchingOptions::num_inferences_per_gpu` (`feature/flexglue.{h,cc}`, CLI, `pycolmap`, docs)**: Bounds concurrent `descriptor_` + `matcher_` executions per CUDA GPU (default `num_workers_per_gpu = 4`, `num_inferences_per_gpu = 2`), and releases decoded full-resolution `Bitmap` buffers immediately after `ResizeBilinearToCHW` so workers waiting on the semaphore hold only the resized `512x512` input tensor (`~3 MB` instead of `72–150 MB`). ### Benchmark: $(N, M)$ Grid Search on `delivery_area` (44 images @ 24 MP, 946 pairs, NVIDIA A100-SXM4-40GB) #### `FLEXGLUE` (`K=2048` keypoints) | $N$ (`workers`) | $M$ (`inferences`) | Block ms/pair | Block speedup | Total ms/pair | GPU util | Mem BW | Mean / Peak Power | Peak VRAM | Matches | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | `1` | `1` | `42.94` | `1.00x` | `46.74` | `56.1%` | `9.5%` | `186 / 304 W` | `1,964 MiB` | `169,581` | | `3` | `1` | `32.99` | `1.30x` | `36.85` | `70.0%` | `12.2%` | `226 / 310 W` | `1,960 MiB` | `169,581` | | `6` | `1` | `30.84` | `1.39x` | `34.63` | `74.6%` | `13.0%` | `239 / 311 W` | `1,960 MiB` | `169,581` | | `2` | `2` | `29.61` | `1.45x` | `33.36` | `63.4%` | `17.3%` | `249 / 398 W` | `2,788 MiB` | `169,581` | | `4` | `2` | `25.46` | `1.69x` | `29.24` | `71.6%` | `19.7%` | `286 / 407 W` | `2,788 MiB` | `169,581` | | `3` | `3` | `26.86` | `1.60x` | `30.57` | `66.4%` | `22.2%` | `272 / 421 W` | `3,600 MiB` | `169,581` | | `6` | `3` | `24.40` | `1.76x` | `28.22` | `71.4%` | `24.2%` | `289 / 417 W` | `3,610 MiB` | `169,581` | | `8` | `3` | **`23.67`** | **`1.81x`** | **`27.40`** | `73.6%` | `24.7%` | `296 / 419 W` | `3,616 MiB` | `169,581` | | `4` | `4` | `26.34` | `1.63x` | `30.13` | `67.0%` | `25.0%` | `276 / 415 W` | `4,412 MiB` | `169,581` | | `8` | `4` | `24.17` | `1.78x` | `27.97` | `72.8%` | `26.3%` | `291 / 414 W` | `4,440 MiB` | `169,581` | | `6` | `6` | `27.09` | `1.59x` | `30.89` | `66.9%` | `25.8%` | `272 / 416 W` | `6,038 MiB` | `169,581` | | `8` | `8` | `27.89` | `1.54x` | `31.71` | `66.6%` | `24.5%` | `266 / 422 W` | `7,708 MiB` | `169,581` | | `12` | `12` | `29.23` | `1.47x` | `33.04` | `65.6%` | `20.6%` | `253 / 417 W` | `10,982 MiB` | `169,581` |
fringehush-gif
added a commit
to fringehush-gif/colmap
that referenced
this pull request
Nov 9, 2026
## Summary Bring in upstream's shared FlexGlue workers and bounded GPU inference slots. CPU preprocessing can overlap while inference concurrency stays independently configurable within each worker group. Preserve the fork's deferred matching-result replacement and CUDA output regression. ## Changes - Merge the ONNX inference limiter, shared model ownership, worker/inference options and bindings. - Keep both upstream limiter tests and the existing dynamic-shape CUDA regression when resolving onnx_test.cc. - Give the setup-failure regression an explicit missing matcher path when model downloads are disabled, so it reaches the intended descriptor-loading error. - Regressions cover limiter capacity/release, option bounds, worker setup errors and preservation of existing matching results on failure. ## Upstream commits - df19970 - FlexGlue: share one matcher across multiple worker threads per GPU (colmap#4957) - 3c1a1b9 - FlexGlue: bound concurrent GPU inferences with a semaphore (colmap#4969) ## Validation - 42 focused native tests passed without skips: 14 ONNX, 2 FlexGlue option and 26 matching/controller cases. CUDA EP initialization occurred inside the GPU setup-error case; the dynamic-shape CUDA output regression passed. - clang-tidy-18 completed for six affected production/test translation units with no displayed source diagnostics; generated/header warnings were suppressed. Diff and added-line formatting checks passed. - Incremental qualification reused 47 production objects and 13 private archives; seven remaining translation units compiled and the CLI linked. Link/provider checks and five help/version smokes passed. The exported-source version reports Unknown commit/date. - Real FlexGlue model inference, Python and Windows are not qualified by these checks. AI disclosure: Totally vibe coded
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With a single matching worker per GPU, FlexGlue leaves the GPU underutilized while the CPU decodes high-resolution images, resizes them, prepares ONNX tensors, and processes early-stopping
Ifchecks. Spawning separate matchers per worker would duplicate both ONNX Runtime's per-session GPU arena (~2.2–2.8 GiB) and the 512 MiB GPU dense-feature cache.Because
Ort::Session::Runsupports concurrent calls across threads, multiple worker threads per CUDA GPU can share a singleFlexGlueFeatureMatcherinstance:FlexGlueFeatureMatcher::dense_cache_fromLRUCachetoThreadSafeLRUCache, which also deduplicates concurrent loads when multiple workers request the same uncached image simultaneously.DenseFeatures::checked_camera_sizestd::atomic<bool>soImageSize()is lock-free and race-free across workers.SharedFeatureMatcherinfeature_matching_utilsso one designated creator worker initializes the matcher (propagating initialization errors once viaSetup()) while sibling workers wait for and share the same instance.FlexGlueMatchingOptions::num_workers_per_gpu = 3(exposed on the CLI as--FlexGlueMatching.num_workers_per_gpu, inpycolmap, and documented indoc/features.rst).Matches are bit-exact with
num_workers_per_gpu = 1; loweringnum_workers_per_gpureduces peak GPU arena memory from concurrent in-flight transformer activations.Evaluation
Evaluated on 3 ETH3D DSLR scenes (
delivery_area,terrains,office: 112 images at 24–25.7 MP, 2,132 exhaustive pairs) with DaD keypoints (2048 and 4096) and default FlexGlue options on an idle NVIDIA A100-SXM4-40GB GPU (12 vCPUs):num_workers_per_gpu1(main)23(default)41(main)23(default)Tests
FlexGlueTest.ConcurrentMatchThreadSafety: verifies concurrentMatch()calls across 3 threads hitting cold and warmThreadSafeLRUCacheentries return identical matches.FeatureMatcherController.SharedFeatureMatcherWorkersMatchPairsandSetupPropagatesMatcherCreationErrors: verify shared matcher execution across 3 workers and single-rethrow error propagation when matcher creation fails.FlexGlueMatchingOptions.Check: verifiesnum_workers_per_gpudefault (3) and bounds check (> 0).