Add anchor-free mode for RPF #25
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an anchor-free mode for the RPF training and inference.
We followed GARF’s data pipeline implementation, which uses an anchor-fixed setup (c.f. GARF's model and its default config). However, the anchor-fixed assumption can be unrealistic for many real-world assembly scenarios. Therefore, we also trained an anchor-free version of RPF: in this mode the model is not given the anchor part’s pose in the assembled object CoM frame during training or inference.
Comparison between Two Modes:
Non-anchor: Global Rotation + Part Centring + Part Rotation.
Non-anchor: Global Rotation + Part Centring + Part Rotation.
Evaluation in the Anchor-free Mode. To keep evaluation comparable between anchor-fixed and anchor-free models,
we perform the alignment steps at evaluation time for anchor-free predictions:
FAQs:
Q: Why does an anchor-fixed mode leak information of the GT?
A: During data augmentation, the full object is globally centered to its CoM (center of mass) frame and then normalized to unit scale. If the anchor part is not independently re-centered (i.e., Part Centering), its coordinates implicitly encode the assembled object’s CoM.
Q: Why don't we apply Part Rotation to the anchor part in anchor-free mode?
A: Anchor-free training already randomly rotates all non-anchor parts, so the anchor doesn’t provide a fixed orientation prior. Applying an additional Part Rotation to the anchor is essentially redundant (it can be absorbed into the Global Rotation), so we omit it.
Q: Why do we still return the
anchor_indicesin anchor-free mode?A: This is necessary for evaluation. At test time, we align the predicted anchor to the GT anchor using ICP and apply the same rigid transform to all predicted non-anchor parts before computing metrics. Note: we do not reset the model’s predicted anchor part to the GT in anchor-free mode.
Q: How does the anchor-free mode affect performance?
A: Training in anchor-free mode does not substantially hurt performance. In our tests, the anchor-free model shows some expected degradation due to (i) error propagation from anchor misalignment and (ii) ambiguity induced by symmetric anchor parts. Nonetheless, our model still shows strong performance under the anchor-free setting. Notably, it significantly improves anchor-free SOTA on BreakingBad-Everyday’s Part Accuracy from 76.2% (PuzzleFussion++) to 90.2%. See Appendix B of our paper for more results and details.
Code Changes:
anchor_freeparameter (defaultTrue) to therectified_point_flow/data/{dataset, datamodule}.py.align_anchorfunction torectified_point_flow/eval/metrics.pyand used it in the evaluator to align predicted anchor parts with ground truth via ICP in anchor-free mode.sample.py.model.anchor_free=trueanddata.anchor_free=truein the config. They are already enabled by default.