Skip to content

Commit fdee2e1

Browse files
authored
Reuse the pose graph of GlobalPipeline in the component mappers (colmap#4912)
## Summary With `multiple_models` (the default), `GlobalPipeline` loads the full pose graph, splits it into components with rotation averaging, and then runs a `GlobalMapper` per component. Each component mapper reloaded its pose graph from the component database cache, so all edge data computed by the pipeline was lost. With `COVARIANCE` reweighting, the rotation covariances were thus estimated twice: once for the component split, and once again in each component mapper. This PR passes each component mapper the subgraph of the pipeline's pose graph instead: - New `PoseGraph::Subgraph(image_ids)` returns the edges between the given images, copied whole (relative pose, rotation covariance, match count, validity, prior flag), so that edge data added in the future carries over too. Also exposed in pycolmap as `PoseGraph.subgraph`. - `GlobalMapper::BeginReconstruction` optionally takes a pose graph, and uses its subgraph between the images of its database cache. Without one, it loads the pose graph from the database cache as before. - `GlobalPipeline` passes its full pose graph to each component mapper. The component cache also includes the other images of the component's frames (multi-camera rigs), so their edges carry over too. The mapper then skips covariance estimation, since every edge already has one. ## Tests - `PoseGraph.Subgraph`: edge selection and whole-edge copy, including validity. - `PoseGraph.SubgraphMatchesComponentDatabaseCache`: on a two-camera rig dataset, the subgraph of a component has exactly the same edges and relative poses as the pose graph loaded from `DatabaseCache::CreateFromCache`. This guards the equivalence the pipeline relies on. - `GlobalMapper.WithGivenPoseGraph`: the mapper uses the given pose graph, and drops its edges to images outside the database cache. - `test_pose_graph_subgraph` for the binding. ## Evaluation Covariance estimation now runs once per scene instead of once plus once per component, which saves about 0.2 s per ETH3D scene single-threaded. Results are unchanged up to floating-point rounding. The subgraph's hash map iterates its edges in a different order than a pose graph loaded from the component cache, which changes the order of floating-point sums. This also affects the default `UNIFORM` reweighting, in scenes split into several components: - RA-only per-scene AUCs on ETH3D and IMC 2023 differ by less than 1e-12. - E2E averages are identical. Per scene, only IMC `dioscuri` (a scene that already amplifies tiny perturbations, −0.06 to +0.32 AUC across seeds) and ETH3D `courtyard` (+0.006 AUC) change. <sub>Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
1 parent 761d90d commit fdee2e1

10 files changed

Lines changed: 197 additions & 11 deletions

File tree

‎src/colmap/controllers/global_pipeline.cc‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -140,12 +140,13 @@ GlobalPipeline::GlobalPipeline(
140140
std::optional<std::shared_ptr<Reconstruction>>
141141
GlobalPipeline::ReconstructSingleComponent(
142142
const std::shared_ptr<const DatabaseCache>& database_cache,
143-
const GlobalMapperOptions& mapper_options) {
143+
const GlobalMapperOptions& mapper_options,
144+
const PoseGraph* pose_graph) {
144145
auto reconstruction =
145146
reconstruction_manager_->Get(reconstruction_manager_->Add());
146147

147148
GlobalMapper global_mapper(database_cache);
148-
global_mapper.BeginReconstruction(reconstruction);
149+
global_mapper.BeginReconstruction(reconstruction, pose_graph);
149150

150151
Timer run_timer;
151152
run_timer.Start();
@@ -304,8 +305,12 @@ GlobalPipeline::ReconstructionStats GlobalPipeline::ReconstructMultiComponents(
304305
const std::shared_ptr<DatabaseCache> component_cache =
305306
DatabaseCache::CreateFromCache(*database_cache_, cache_options);
306307

308+
// Reuse the edges of the full pose graph, e.g., with their estimated
309+
// rotation covariances, between the images of the component cache, which
310+
// also includes all other images of their frames.
307311
const std::optional<std::shared_ptr<Reconstruction>> reconstruction =
308-
ReconstructSingleComponent(component_cache, mapper_options);
312+
ReconstructSingleComponent(
313+
component_cache, mapper_options, &pose_graph);
309314
if (!reconstruction.has_value()) {
310315
reconstruction_manager_->Delete(reconstruction_manager_->Size() - 1);
311316
++stats.num_failed;

‎src/colmap/controllers/global_pipeline.h‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
#pragma once
44

5+
#include "colmap/scene/pose_graph.h"
56
#include "colmap/scene/reconstruction_manager.h"
67
#include "colmap/sfm/global_mapper.h"
78
#include "colmap/util/base_controller.h"
@@ -75,10 +76,12 @@ class GlobalPipeline : public BaseController {
7576
// Run the full global SfM pipeline on the given database cache and return
7677
// the resulting reconstruction, or nullopt if mapping fails. The in-progress
7778
// reconstruction is added to the manager so callbacks can render it. The
78-
// caller decides whether to keep it.
79+
// caller decides whether to keep it. If pose_graph is null, the pose graph
80+
// is loaded from the database cache.
7981
std::optional<std::shared_ptr<Reconstruction>> ReconstructSingleComponent(
8082
const std::shared_ptr<const DatabaseCache>& database_cache,
81-
const GlobalMapperOptions& mapper_options);
83+
const GlobalMapperOptions& mapper_options,
84+
const PoseGraph* pose_graph = nullptr);
8285

8386
// Partition the input view graph once using rotation averaging and
8487
// reconstruct each resulting component at most once.

‎src/colmap/scene/pose_graph.cc‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,4 +170,15 @@ void PoseGraph::InvalidatePairsOutsideActiveImageIds(
170170
}
171171
}
172172

173+
PoseGraph PoseGraph::Subgraph(const FlatHashSet<image_t>& image_ids) const {
174+
PoseGraph subgraph;
175+
for (const auto& [pair_id, edge] : edges_) {
176+
const auto [image_id1, image_id2] = PairIdToImagePair(pair_id);
177+
if (image_ids.count(image_id1) && image_ids.count(image_id2)) {
178+
subgraph.edges_.emplace(pair_id, edge);
179+
}
180+
}
181+
return subgraph;
182+
}
183+
173184
} // namespace colmap

‎src/colmap/scene/pose_graph.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,10 @@ class PoseGraph {
123123
void InvalidatePairsOutsideActiveImageIds(
124124
const FlatHashSet<image_t>& active_image_ids);
125125

126+
// Returns the subgraph of the edges between images in the given set, with
127+
// all their data and validity.
128+
PoseGraph Subgraph(const FlatHashSet<image_t>& image_ids) const;
129+
126130
private:
127131
// Map from pair ID to edge data. The pair ID is computed from the
128132
// two image IDs using ImagePairToPairId, with the smaller ID first.

‎src/colmap/scene/pose_graph_test.cc‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
#include "colmap/math/random_eigen.h"
66
#include "colmap/scene/database_cache.h"
7+
#include "colmap/scene/database_sqlite.h"
78
#include "colmap/scene/synthetic.h"
89
#include "colmap/util/hash_containers.h"
910
#include "colmap/util/testing.h"
@@ -433,6 +434,94 @@ TEST(PoseGraph, InvalidatePairsOutsideActiveImageIds) {
433434
EXPECT_FALSE(pose_graph.IsValid(ImagePairToPairId(2, 3)));
434435
}
435436

437+
TEST(PoseGraph, Subgraph) {
438+
PoseGraph pose_graph;
439+
PoseGraph::Edge edge = SynthesizeEdge(/*num_matches=*/42);
440+
edge.cam2_from_cam1 =
441+
Rigid3d(RandomEigenQuaterniond(), RandomEigenVectord<3>());
442+
edge.cam2_from_cam1_rotation_cov = 2.0f * Eigen::Matrix3f::Identity();
443+
edge.prior = true;
444+
pose_graph.AddEdge(1, 2, edge);
445+
pose_graph.AddEdge(1, 3, SynthesizeEdge());
446+
pose_graph.AddEdge(2, 3, SynthesizeEdge());
447+
pose_graph.AddEdge(2, 4, SynthesizeEdge());
448+
pose_graph.SetInvalidEdge(ImagePairToPairId(2, 4));
449+
450+
const PoseGraph subgraph = pose_graph.Subgraph({1, 2, 4});
451+
EXPECT_EQ(subgraph.NumEdges(), 2);
452+
EXPECT_FALSE(subgraph.HasEdge(1, 3));
453+
EXPECT_FALSE(subgraph.HasEdge(2, 3));
454+
455+
// Edges are copied with all their data and validity.
456+
const PoseGraph::Edge& subgraph_edge =
457+
subgraph.Edges().at(ImagePairToPairId(1, 2));
458+
EXPECT_EQ(
459+
subgraph_edge.cam2_from_cam1.params,
460+
pose_graph.Edges().at(ImagePairToPairId(1, 2)).cam2_from_cam1.params);
461+
EXPECT_EQ(subgraph_edge.num_matches, 42);
462+
EXPECT_TRUE(subgraph_edge.prior);
463+
EXPECT_TRUE(subgraph_edge.valid);
464+
ASSERT_TRUE(subgraph_edge.cam2_from_cam1_rotation_cov.has_value());
465+
EXPECT_EQ(*subgraph_edge.cam2_from_cam1_rotation_cov,
466+
*pose_graph.Edges()
467+
.at(ImagePairToPairId(1, 2))
468+
.cam2_from_cam1_rotation_cov);
469+
EXPECT_TRUE(subgraph.HasEdge(2, 4));
470+
EXPECT_FALSE(subgraph.IsValid(ImagePairToPairId(2, 4)));
471+
472+
EXPECT_TRUE(pose_graph.Subgraph({}).Empty());
473+
}
474+
475+
// The subgraph of a component equals the pose graph loaded from the database
476+
// cache of that component, which GlobalPipeline relies on.
477+
TEST(PoseGraph, SubgraphMatchesComponentDatabaseCache) {
478+
auto database = Database::Open(kInMemorySqliteDatabasePath);
479+
SyntheticDatasetOptions synthetic_options;
480+
synthetic_options.num_rigs = 2;
481+
synthetic_options.num_cameras_per_rig = 2;
482+
synthetic_options.num_frames_per_rig = 3;
483+
synthetic_options.num_points3D = 50;
484+
synthetic_options.two_view_geometry_has_relative_pose = true;
485+
Reconstruction reconstruction;
486+
SynthesizeDataset(synthetic_options, &reconstruction, database.get());
487+
488+
DatabaseCache::Options cache_options;
489+
const auto cache = DatabaseCache::Create(*database, cache_options);
490+
PoseGraph pose_graph;
491+
pose_graph.Load(*cache->CorrespondenceGraph());
492+
493+
// Select the images of the reference cameras of some frames. The component
494+
// cache also includes the other images of their frames.
495+
DatabaseCache::Options component_options;
496+
for (const auto& [image_id, image] : reconstruction.Images()) {
497+
if (image.IsRefInFrame() && image.FrameId() % 2 == 0) {
498+
component_options.image_names.insert(image.Name());
499+
}
500+
}
501+
ASSERT_FALSE(component_options.image_names.empty());
502+
const auto component_cache =
503+
DatabaseCache::CreateFromCache(*cache, component_options);
504+
PoseGraph component_pose_graph;
505+
component_pose_graph.Load(*component_cache->CorrespondenceGraph());
506+
507+
FlatHashSet<image_t> component_image_ids;
508+
for (const auto& [image_id, _] : component_cache->Images()) {
509+
component_image_ids.insert(image_id);
510+
}
511+
ASSERT_GT(component_image_ids.size(), component_options.image_names.size());
512+
const PoseGraph subgraph = pose_graph.Subgraph(component_image_ids);
513+
514+
ASSERT_GT(subgraph.NumEdges(), 0);
515+
ASSERT_EQ(subgraph.NumEdges(), component_pose_graph.NumEdges());
516+
for (const auto& [pair_id, edge] : component_pose_graph.Edges()) {
517+
ASSERT_TRUE(subgraph.Edges().count(pair_id));
518+
const PoseGraph::Edge& subgraph_edge = subgraph.Edges().at(pair_id);
519+
EXPECT_EQ(subgraph_edge.cam2_from_cam1.params, edge.cam2_from_cam1.params);
520+
EXPECT_EQ(subgraph_edge.num_matches, edge.num_matches);
521+
EXPECT_EQ(subgraph_edge.valid, edge.valid);
522+
}
523+
}
524+
436525
TEST(PoseGraph, InvertEdgeCovariance) {
437526
PoseGraph::Edge edge;
438527
const Eigen::Quaterniond q21 =

‎src/colmap/sfm/global_mapper.cc‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -113,12 +113,22 @@ GlobalMapper::GlobalMapper(std::shared_ptr<const DatabaseCache> database_cache)
113113
: database_cache_(std::move(THROW_CHECK_NOTNULL(database_cache))) {}
114114

115115
void GlobalMapper::BeginReconstruction(
116-
const std::shared_ptr<class Reconstruction>& reconstruction) {
116+
const std::shared_ptr<class Reconstruction>& reconstruction,
117+
const PoseGraph* pose_graph) {
117118
THROW_CHECK_NOTNULL(reconstruction);
118119
reconstruction_ = reconstruction;
119120
reconstruction_->Load(*database_cache_);
120-
pose_graph_ = std::make_shared<class PoseGraph>();
121-
pose_graph_->Load(*database_cache_->CorrespondenceGraph());
121+
if (pose_graph != nullptr) {
122+
FlatHashSet<image_t> image_ids;
123+
image_ids.reserve(database_cache_->NumImages());
124+
for (const auto& [image_id, _] : database_cache_->Images()) {
125+
image_ids.insert(image_id);
126+
}
127+
pose_graph_ = std::make_shared<PoseGraph>(pose_graph->Subgraph(image_ids));
128+
} else {
129+
pose_graph_ = std::make_shared<PoseGraph>();
130+
pose_graph_->Load(*database_cache_->CorrespondenceGraph());
131+
}
122132
}
123133

124134
std::shared_ptr<Reconstruction> GlobalMapper::Reconstruction() const {

‎src/colmap/sfm/global_mapper.h‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,9 +119,13 @@ class GlobalMapper {
119119
explicit GlobalMapper(std::shared_ptr<const DatabaseCache> database_cache);
120120

121121
// Prepare the mapper for a new reconstruction. This will initialize the
122-
// reconstruction and view graph from the database.
122+
// reconstruction from the database and the pose graph from the database or,
123+
// if pose_graph is given, from its subgraph between the images of the
124+
// database cache. The latter lets callers provide precomputed edge data, such
125+
// as rotation covariances.
123126
void BeginReconstruction(
124-
const std::shared_ptr<Reconstruction>& reconstruction);
127+
const std::shared_ptr<Reconstruction>& reconstruction,
128+
const PoseGraph* pose_graph = nullptr);
125129

126130
// Run the global SfM pipeline. The optional `on_progress` callback is invoked
127131
// after global positioning, after each bundle-adjustment iteration, and after

‎src/colmap/sfm/global_mapper_test.cc‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#include "colmap/sfm/global_mapper.h"
44

55
#include "colmap/scene/database_cache.h"
6+
#include "colmap/scene/database_sqlite.h"
67
#include "colmap/scene/reconstruction_matchers.h"
78
#include "colmap/scene/synthetic.h"
89
#include "colmap/util/testing.h"
@@ -158,6 +159,46 @@ TEST(GlobalMapper, WithNoiseAndOutliers) {
158159
}
159160
}
160161

162+
TEST(GlobalMapper, WithGivenPoseGraph) {
163+
auto database = Database::Open(kInMemorySqliteDatabasePath);
164+
Reconstruction gt_reconstruction;
165+
SyntheticDatasetOptions synthetic_dataset_options;
166+
synthetic_dataset_options.num_rigs = 1;
167+
synthetic_dataset_options.num_cameras_per_rig = 1;
168+
synthetic_dataset_options.num_frames_per_rig = 5;
169+
synthetic_dataset_options.num_points3D = 50;
170+
synthetic_dataset_options.two_view_geometry_has_relative_pose = true;
171+
SynthesizeDataset(
172+
synthetic_dataset_options, &gt_reconstruction, database.get());
173+
const auto database_cache = CreateDatabaseCache(*database);
174+
175+
// The mapper uses the given pose graph instead of loading it from the
176+
// database: an image without valid edges is not reconstructed.
177+
PoseGraph pose_graph;
178+
pose_graph.Load(*database_cache->CorrespondenceGraph());
179+
constexpr image_t kIsolatedImageId = 1;
180+
for (const auto& [pair_id, edge] : pose_graph.Edges()) {
181+
const auto [image_id1, image_id2] = PairIdToImagePair(pair_id);
182+
if (image_id1 == kIsolatedImageId || image_id2 == kIsolatedImageId) {
183+
pose_graph.SetInvalidEdge(pair_id);
184+
}
185+
}
186+
// Edges to images outside the database cache are dropped.
187+
constexpr image_t kUnknownImageId = 100;
188+
pose_graph.AddEdge(2, kUnknownImageId, PoseGraph::Edge(Rigid3d()));
189+
190+
auto reconstruction = std::make_shared<Reconstruction>();
191+
GlobalMapper global_mapper(database_cache);
192+
global_mapper.BeginReconstruction(reconstruction, &pose_graph);
193+
ASSERT_TRUE(global_mapper.Solve(GlobalMapperOptions()));
194+
195+
// The isolated image is not registered, and thus removed by the final
196+
// retriangulation, which tears down the reconstruction.
197+
EXPECT_FALSE(reconstruction->ExistsImage(kIsolatedImageId));
198+
EXPECT_EQ(reconstruction->NumRegImages(),
199+
gt_reconstruction.NumRegImages() - 1);
200+
}
201+
161202
TEST(GlobalMapperOptions, RefineSensorFromRigPropagatesToSubOptions) {
162203
GlobalMapperOptions options;
163204
options.refine_sensor_from_rig = false;

‎src/pycolmap/scene/pose_graph.cc‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,5 +121,10 @@ void BindPoseGraph(py::module& m) {
121121
&PoseGraph::InvalidatePairsOutsideActiveImageIds,
122122
"active_image_ids"_a,
123123
"Mark image pairs as invalid if either image is not in the active "
124-
"set.");
124+
"set.")
125+
.def("subgraph",
126+
&PoseGraph::Subgraph,
127+
"image_ids"_a,
128+
"Return the subgraph of the edges between images in the given set, "
129+
"with all their data and validity.");
125130
}

‎src/pycolmap/scene/pose_graph_test.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,3 +173,17 @@ def test_pose_graph_add_rig_edges() -> None:
173173
assert edge.prior is True
174174
assert edge.valid is False
175175
assert edge.num_matches == 0
176+
177+
178+
def test_pose_graph_subgraph() -> None:
179+
pose_graph = pycolmap.PoseGraph()
180+
edge = pycolmap.PoseGraphEdge()
181+
edge.num_matches = 42
182+
pose_graph.add_edge(1, 2, edge)
183+
pose_graph.add_edge(1, 3, pycolmap.PoseGraphEdge())
184+
pose_graph.add_edge(2, 3, pycolmap.PoseGraphEdge())
185+
subgraph = pose_graph.subgraph({1, 2})
186+
assert subgraph.num_edges == 1
187+
assert subgraph.has_edge(1, 2)
188+
assert subgraph.get_edge(1, 2).num_matches == 42
189+
assert pose_graph.num_edges == 3

0 commit comments

Comments
 (0)