Skip to content

Commit 3c1a1b9

Browse files
authored
FlexGlue: bound concurrent GPU inferences with a semaphore (#4969)
Stacked on #4957. ### Summary Decouples CPU image decoding/preprocessing parallelism (`FlexGlueMatchingOptions::num_workers_per_gpu`, $N$) from concurrent GPU model execution (`FlexGlueMatchingOptions::num_inferences_per_gpu`, $M$) in `FlexGlue`: - **`ONNXInferenceLimiter` (`util/onnx.{h,cc}`)**: Slot-assigning counting semaphore bounding concurrent GPU inferences across a larger pool of worker threads, paired with `OrtCUDAProviderOptionsV2` (`arena_extend_strategy=kSameAsRequested`, `use_ep_level_unified_stream=1`) and lazy per-slot CUDA `Ort::Session` replicas (`allow_session_replicas=false` on `FlexGlue`'s 344 MB dense descriptor so its weights are never duplicated). - **`FlexGlueMatchingOptions::num_inferences_per_gpu` (`feature/flexglue.{h,cc}`, CLI, `pycolmap`, docs)**: Bounds concurrent `descriptor_` + `matcher_` executions per CUDA GPU (default `num_workers_per_gpu = 4`, `num_inferences_per_gpu = 2`), and releases decoded full-resolution `Bitmap` buffers immediately after `ResizeBilinearToCHW` so workers waiting on the semaphore hold only the resized `512x512` input tensor (`~3 MB` instead of `72–150 MB`). ### Benchmark: $(N, M)$ Grid Search on `delivery_area` (44 images @ 24 MP, 946 pairs, NVIDIA A100-SXM4-40GB) #### `FLEXGLUE` (`K=2048` keypoints) | $N$ (`workers`) | $M$ (`inferences`) | Block ms/pair | Block speedup | Total ms/pair | GPU util | Mem BW | Mean / Peak Power | Peak VRAM | Matches | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | `1` | `1` | `42.94` | `1.00x` | `46.74` | `56.1%` | `9.5%` | `186 / 304 W` | `1,964 MiB` | `169,581` | | `3` | `1` | `32.99` | `1.30x` | `36.85` | `70.0%` | `12.2%` | `226 / 310 W` | `1,960 MiB` | `169,581` | | `6` | `1` | `30.84` | `1.39x` | `34.63` | `74.6%` | `13.0%` | `239 / 311 W` | `1,960 MiB` | `169,581` | | `2` | `2` | `29.61` | `1.45x` | `33.36` | `63.4%` | `17.3%` | `249 / 398 W` | `2,788 MiB` | `169,581` | | `4` | `2` | `25.46` | `1.69x` | `29.24` | `71.6%` | `19.7%` | `286 / 407 W` | `2,788 MiB` | `169,581` | | `3` | `3` | `26.86` | `1.60x` | `30.57` | `66.4%` | `22.2%` | `272 / 421 W` | `3,600 MiB` | `169,581` | | `6` | `3` | `24.40` | `1.76x` | `28.22` | `71.4%` | `24.2%` | `289 / 417 W` | `3,610 MiB` | `169,581` | | `8` | `3` | **`23.67`** | **`1.81x`** | **`27.40`** | `73.6%` | `24.7%` | `296 / 419 W` | `3,616 MiB` | `169,581` | | `4` | `4` | `26.34` | `1.63x` | `30.13` | `67.0%` | `25.0%` | `276 / 415 W` | `4,412 MiB` | `169,581` | | `8` | `4` | `24.17` | `1.78x` | `27.97` | `72.8%` | `26.3%` | `291 / 414 W` | `4,440 MiB` | `169,581` | | `6` | `6` | `27.09` | `1.59x` | `30.89` | `66.9%` | `25.8%` | `272 / 416 W` | `6,038 MiB` | `169,581` | | `8` | `8` | `27.89` | `1.54x` | `31.71` | `66.6%` | `24.5%` | `266 / 422 W` | `7,708 MiB` | `169,581` | | `12` | `12` | `29.23` | `1.47x` | `33.04` | `65.6%` | `20.6%` | `253 / 417 W` | `10,982 MiB` | `169,581` |
1 parent df19970 commit 3c1a1b9

9 files changed

Lines changed: 419 additions & 72 deletions

File tree

‎doc/features.rst‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -218,9 +218,12 @@ variant of the matcher whose transformer layers run in float16 is used by
218218
default (``--FlexGlueMatching.use_fp16``), which is faster than the float32
219219
matcher with practically identical matches; the float32 matcher is used on the CPU, on
220220
older GPUs, or if the float16 variant fails to initialize. On CUDA GPUs,
221-
``--FlexGlueMatching.num_workers_per_gpu`` (default 3) matching threads share a
221+
``--FlexGlueMatching.num_workers_per_gpu`` (default 4) matching threads share a
222222
single matcher instance and GPU feature cache per GPU so CPU image decoding
223-
overlaps with GPU inference (lower it to reduce GPU memory usage). On the CPU, a
223+
overlaps with GPU inference, while ``--FlexGlueMatching.num_inferences_per_gpu``
224+
(default 2) bounds the number of concurrent model inferences on the GPU (lower
225+
it, e.g. to 1, to reduce GPU memory usage while keeping parallel CPU image
226+
decoding). On the CPU, a
224227
single matching thread runs the models with ``--FeatureMatching.num_threads``
225228
threads, rather than one copy of the models per thread. The
226229
``automatic_reconstructor`` provides the ``--feature flexglue`` preset, which

‎src/colmap/controllers/option_manager.cc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -466,6 +466,8 @@ void OptionManager::AddFeatureMatchingOptions() {
466466
&feature_matching->flexglue->use_fp16);
467467
AddDefaultOption("FlexGlueMatching.num_workers_per_gpu",
468468
&feature_matching->flexglue->num_workers_per_gpu);
469+
AddDefaultOption("FlexGlueMatching.num_inferences_per_gpu",
470+
&feature_matching->flexglue->num_inferences_per_gpu);
469471
}
470472

471473
void OptionManager::AddTwoViewGeometryOptions() {

‎src/colmap/feature/flexglue.cc‎

Lines changed: 37 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -139,19 +139,31 @@ bool UseFlexGlueFp16(const FeatureMatchingOptions& options) {
139139
return false;
140140
}
141141

142+
std::shared_ptr<ONNXInferenceLimiter> CreateInferenceLimiter(
143+
const FeatureMatchingOptions& options) {
144+
if (SelectONNXExecutionProvider(options.use_gpu) !=
145+
ONNXExecutionProvider::CUDA) {
146+
return nullptr;
147+
}
148+
return std::make_shared<ONNXInferenceLimiter>(
149+
THROW_CHECK_NOTNULL(options.flexglue)->num_inferences_per_gpu);
150+
}
151+
142152
// Matcher with float16 or float32 transformer layers (see UseFlexGlueFp16).
143153
// The float16 matcher is loaded as a regular session, with a fallback to the
144154
// float32 matcher if its initialization fails.
145155
std::unique_ptr<ONNXModel> CreateMatcherModel(
146-
const FeatureMatchingOptions& options) {
156+
const FeatureMatchingOptions& options,
157+
const std::shared_ptr<ONNXInferenceLimiter>& inference_limiter) {
147158
const FlexGlueMatchingOptions& flexglue_options = *options.flexglue;
148159
if (UseFlexGlueFp16(options)) {
149160
try {
150161
return std::make_unique<ONNXModel>(
151162
flexglue_options.matcher_model_path_fp16,
152163
options.num_threads,
153164
options.use_gpu,
154-
options.gpu_index);
165+
options.gpu_index,
166+
inference_limiter);
155167
} catch (const Ort::Exception& e) {
156168
LOG(WARNING) << "Failed to initialize the fp16 FlexGlue matcher ("
157169
<< e.what() << "); falling back to fp32";
@@ -160,18 +172,22 @@ std::unique_ptr<ONNXModel> CreateMatcherModel(
160172
return std::make_unique<ONNXModel>(flexglue_options.matcher_model_path,
161173
options.num_threads,
162174
options.use_gpu,
163-
options.gpu_index);
175+
options.gpu_index,
176+
inference_limiter);
164177
}
165178

166179
class FlexGlueFeatureMatcher : public FeatureMatcher {
167180
public:
168181
explicit FlexGlueFeatureMatcher(const FeatureMatchingOptions& options)
169182
: options_(*THROW_CHECK_NOTNULL(options.flexglue)),
183+
inference_limiter_(CreateInferenceLimiter(options)),
170184
descriptor_(options_.descriptor_model_path,
171185
options.num_threads,
172186
options.use_gpu,
173-
options.gpu_index),
174-
matcher_(CreateMatcherModel(options)),
187+
options.gpu_index,
188+
inference_limiter_,
189+
/*allow_session_replicas=*/false),
190+
matcher_(CreateMatcherModel(options, inference_limiter_)),
175191
dense_cache_(options_.feature_cache_size,
176192
[this](const std::string& name) {
177193
return ComputeDenseFeatures(name);
@@ -413,19 +429,24 @@ class FlexGlueFeatureMatcher : public FeatureMatcher {
413429
<< "FlexGlue requires the image name to read the image pixels.";
414430
const std::filesystem::path path =
415431
std::filesystem::path(options_.image_path) / name;
416-
Bitmap bitmap;
417-
THROW_CHECK(bitmap.Read(path, /*as_rgb=*/true))
418-
<< "Failed to read image: " << path;
419-
420432
auto dense = std::make_shared<DenseFeatures>();
421-
dense->width = bitmap.Width();
422-
dense->height = bitmap.Height();
423-
424-
std::vector<float> input =
425-
ResizeBilinearToCHW(bitmap, descriptor_width_, descriptor_height_);
433+
std::vector<float> input;
434+
{
435+
// Release the full-resolution decoded bitmap before acquiring an
436+
// inference slot, so workers waiting on the GPU hold only the resized
437+
// descriptor input tensor.
438+
Bitmap bitmap;
439+
THROW_CHECK(bitmap.Read(path, /*as_rgb=*/true))
440+
<< "Failed to read image: " << path;
441+
dense->width = bitmap.Width();
442+
dense->height = bitmap.Height();
443+
input =
444+
ResizeBilinearToCHW(bitmap, descriptor_width_, descriptor_height_);
445+
}
426446
const std::vector<int64_t> input_shape = {
427447
1, 3, descriptor_height_, descriptor_width_};
428448
std::vector<Ort::Value> inputs;
449+
inputs.reserve(1);
429450
inputs.push_back(CreateONNXTensor(input, input_shape));
430451
std::vector<Ort::Value> outputs =
431452
feature_memory_info_ ? descriptor_.Run(inputs, *feature_memory_info_)
@@ -437,6 +458,7 @@ class FlexGlueFeatureMatcher : public FeatureMatcher {
437458
}
438459

439460
const FlexGlueMatchingOptions options_;
461+
std::shared_ptr<ONNXInferenceLimiter> inference_limiter_;
440462
ONNXModel descriptor_;
441463
std::unique_ptr<ONNXModel> matcher_;
442464
int descriptor_height_ = 0;
@@ -468,6 +490,7 @@ bool FlexGlueMatchingOptions::Check() const {
468490
CHECK_OPTION(!descriptor_model_path.empty());
469491
CHECK_OPTION(!matcher_model_path.empty());
470492
CHECK_OPTION_GT(num_workers_per_gpu, 0);
493+
CHECK_OPTION_GT(num_inferences_per_gpu, 0);
471494
return true;
472495
}
473496

‎src/colmap/feature/flexglue.h‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -83,10 +83,16 @@ struct FlexGlueMatchingOptions {
8383
// Number of matching worker threads per CUDA GPU that share one FlexGlue
8484
// matcher instance (including its ONNX sessions and GPU feature cache).
8585
// Multiple workers allow image decoding and preprocessing on the CPU to
86-
// overlap with model inference on the GPU without duplicating model weights
87-
// or cached features; lower this value (e.g., to 1 or 2) to reduce GPU
88-
// memory usage. Ignored when not running on a CUDA GPU.
89-
int num_workers_per_gpu = 3;
86+
// overlap with model inference on the GPU without duplicating cached
87+
// features. Ignored when not running on a CUDA GPU.
88+
int num_workers_per_gpu = 4;
89+
90+
// Maximum number of concurrent ONNX model inferences per CUDA GPU across the
91+
// shared worker threads. Setting this lower than num_workers_per_gpu (e.g.,
92+
// to 1 on GPUs with limited memory) allows multiple CPU threads to decode
93+
// and preprocess images in parallel while bounding peak GPU memory usage.
94+
// Ignored when not running on a CUDA GPU.
95+
int num_inferences_per_gpu = 2;
9096

9197
bool Check() const;
9298
};

‎src/colmap/feature/flexglue_test.cc‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -187,10 +187,15 @@ TEST(FlexGlueMatchingOptions, Check) {
187187
// Without the float16 matcher, the float32 matcher is used.
188188
options.matcher_model_path_fp16.clear();
189189
EXPECT_TRUE(options.Check());
190-
EXPECT_EQ(options.num_workers_per_gpu, 3);
190+
EXPECT_EQ(options.num_workers_per_gpu, 4);
191191
options.num_workers_per_gpu = 0;
192192
EXPECT_FALSE(options.Check());
193-
options.num_workers_per_gpu = 3;
193+
options.num_workers_per_gpu = 4;
194+
EXPECT_TRUE(options.Check());
195+
EXPECT_EQ(options.num_inferences_per_gpu, 2);
196+
options.num_inferences_per_gpu = 0;
197+
EXPECT_FALSE(options.Check());
198+
options.num_inferences_per_gpu = 1;
194199
EXPECT_TRUE(options.Check());
195200
}
196201

0 commit comments

Comments
 (0)