Skip to content

FlexGlue: bound concurrent GPU inferences with a semaphore - #4969

Merged
sarlinpe merged 1 commit into
sarlinpe/flexglue-shared-matcherfrom
sarlinpe/flexglue-inference-semaphore
Nov 8, 2026
Merged

sarlinpe merged 1 commit into
sarlinpe/flexglue-shared-matcherfrom
sarlinpe/flexglue-inference-semaphore

Conversation

@sarlinpe

@sarlinpe sarlinpe commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Stacked on #4957.

Summary

Decouples CPU image decoding/preprocessing parallelism (FlexGlueMatchingOptions::num_workers_per_gpu, $N$) from concurrent GPU model execution (FlexGlueMatchingOptions::num_inferences_per_gpu, $M$) in FlexGlue:

  • ONNXInferenceLimiter (util/onnx.{h,cc}): Slot-assigning counting semaphore bounding concurrent GPU inferences across a larger pool of worker threads, paired with OrtCUDAProviderOptionsV2 (arena_extend_strategy=kSameAsRequested, use_ep_level_unified_stream=1) and lazy per-slot CUDA Ort::Session replicas (allow_session_replicas=false on FlexGlue's 344 MB dense descriptor so its weights are never duplicated).
  • FlexGlueMatchingOptions::num_inferences_per_gpu (feature/flexglue.{h,cc}, CLI, pycolmap, docs): Bounds concurrent descriptor_ + matcher_ executions per CUDA GPU (default num_workers_per_gpu = 4, num_inferences_per_gpu = 2), and releases decoded full-resolution Bitmap buffers immediately after ResizeBilinearToCHW so workers waiting on the semaphore hold only the resized 512x512 input tensor (~3 MB instead of 72–150 MB).

Benchmark: $(N, M)$ Grid Search on delivery_area (44 images @ 24 MP, 946 pairs, NVIDIA A100-SXM4-40GB)

FLEXGLUE (K=2048 keypoints)

$N$ (workers) $M$ (inferences) Block ms/pair Block speedup Total ms/pair GPU util Mem BW Mean / Peak Power Peak VRAM Matches
1 1 42.94 1.00x 46.74 56.1% 9.5% 186 / 304 W 1,964 MiB 169,581
3 1 32.99 1.30x 36.85 70.0% 12.2% 226 / 310 W 1,960 MiB 169,581
6 1 30.84 1.39x 34.63 74.6% 13.0% 239 / 311 W 1,960 MiB 169,581
2 2 29.61 1.45x 33.36 63.4% 17.3% 249 / 398 W 2,788 MiB 169,581
4 2 25.46 1.69x 29.24 71.6% 19.7% 286 / 407 W 2,788 MiB 169,581
3 3 26.86 1.60x 30.57 66.4% 22.2% 272 / 421 W 3,600 MiB 169,581
6 3 24.40 1.76x 28.22 71.4% 24.2% 289 / 417 W 3,610 MiB 169,581
8 3 23.67 1.81x 27.40 73.6% 24.7% 296 / 419 W 3,616 MiB 169,581
4 4 26.34 1.63x 30.13 67.0% 25.0% 276 / 415 W 4,412 MiB 169,581
8 4 24.17 1.78x 27.97 72.8% 26.3% 291 / 414 W 4,440 MiB 169,581
6 6 27.09 1.59x 30.89 66.9% 25.8% 272 / 416 W 6,038 MiB 169,581
8 8 27.89 1.54x 31.71 66.6% 24.5% 266 / 422 W 7,708 MiB 169,581
12 12 29.23 1.47x 33.04 65.6% 20.6% 253 / 417 W 10,982 MiB 169,581

@sarlinpe
sarlinpe added this pull request to stack #4970 November 5, 2026 10:31
@sarlinpe
sarlinpe force-pushed the sarlinpe/flexglue-inference-semaphore branch from 1b25dcc to 5009431 Compare November 5, 2026 13:09

@ahojnnes ahojnnes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff at 5009431 (stacked on 4957 at ebd27cb) and verified it merges cleanly onto the fixed 4957 head (28a748a, no conflicts; the only overlapping file is flexglue_test.cc). The limiter's slot assignment, permit lifetime, replica-session creation, and CUDA provider options check out, and the (N, M) benchmark shows bit-identical match counts. No new significant findings; please rebase onto 4957 before merging. Ship.

@ahojnnes ahojnnes added the full-ci Run Windows and remaining pycolmap CI label Nov 8, 2026
@sarlinpe
sarlinpe force-pushed the sarlinpe/flexglue-inference-semaphore branch from 5009431 to 8e3958f Compare November 8, 2026 20:33
@sarlinpe
sarlinpe enabled stacked auto-merge (squash) November 8, 2026 20:34
@sarlinpe
sarlinpe merged commit 3c1a1b9 into main Nov 8, 2026
16 checks passed
@sarlinpe
sarlinpe deleted the sarlinpe/flexglue-inference-semaphore branch November 8, 2026 22:37
fringehush-gif added a commit to fringehush-gif/colmap that referenced this pull request Nov 9, 2026
## Summary

Bring in upstream's shared FlexGlue workers and bounded GPU inference slots. CPU preprocessing can overlap while inference concurrency stays independently configurable within each worker group. Preserve the fork's deferred matching-result replacement and CUDA output regression.

## Changes

- Merge the ONNX inference limiter, shared model ownership, worker/inference options and bindings.
- Keep both upstream limiter tests and the existing dynamic-shape CUDA regression when resolving onnx_test.cc.
- Give the setup-failure regression an explicit missing matcher path when model downloads are disabled, so it reaches the intended descriptor-loading error.
- Regressions cover limiter capacity/release, option bounds, worker setup errors and preservation of existing matching results on failure.

## Upstream commits

- df19970 - FlexGlue: share one matcher across multiple worker threads per GPU (colmap#4957)
- 3c1a1b9 - FlexGlue: bound concurrent GPU inferences with a semaphore (colmap#4969)

## Validation

- 42 focused native tests passed without skips: 14 ONNX, 2 FlexGlue option and 26 matching/controller cases. CUDA EP initialization occurred inside the GPU setup-error case; the dynamic-shape CUDA output regression passed.
- clang-tidy-18 completed for six affected production/test translation units with no displayed source diagnostics; generated/header warnings were suppressed. Diff and added-line formatting checks passed.
- Incremental qualification reused 47 production objects and 13 private archives; seven remaining translation units compiled and the CLI linked. Link/provider checks and five help/version smokes passed. The exported-source version reports Unknown commit/date.
- Real FlexGlue model inference, Python and Windows are not qualified by these checks.

AI disclosure: Totally vibe coded
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run Windows and remaining pycolmap CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants