Repository navigation
[metrics] Raise a clear error for self-intersecting ground truth boxes - #2219
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2219 +/- ##
==========================================
+ Coverage 97.78% 97.80% +0.02%
==========================================
Files 170 170
Lines 10999 11016 +17
==========================================
+ Hits 10755 10774 +19
+ Misses 244 242 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
felixdittrich92
left a comment
There was a problem hiding this comment.
Request changes
Thanks for tracking this down. Crashing deep inside GEOS with a TopologyException is a bad experience, and the fix itself works. But I don't think we should quietly repair boxes inside polygon_iou.
A ground-truth box whose points cross each other is a broken annotation. If we reorder its points without telling anyone, the metric reports a score against a box that isn't in the dataset, and the annotation error is hidden. Rather than accept wrong ground truth, we should reject it with a clear error so the dataset gets fixed.
Requested changes:
- Remove the automatic reordering (
_polygons/order_points) frompolygon_iou. - Raise a clear
ValueErrorinstead of the GEOS exception. It should name the offending boxes, e.g."Ground truth boxes at indices [3, 17] are self-intersecting (points not in TL, TR, BR, BL order); please fix the annotations."Sincepolygon_ioucan't tell ground truth from predictions, the cleanest place for this is a small helper called ongtsin theupdate()methods that useuse_polygons(LocalizationConfusion,OCRMetric,DetectionMetric,ObjectDetectionMetric,TableCellMetric). If you think predictions should be validated the same way, I'm open to that too. - Don't reject degenerate boxes. Zero-area boxes (collinear points, repeated points) also fail
is_valid, and collinear ones even reportSelf-intersection. Main returns IoU 0 for these today and that should stay the same. Only flag boxes that are invalid and have a non-zero convex-hull area, i.e. a real box whose points are out of order. - Tests: replace
test_polygon_iou_misordered_pointswith tests that check:- a crossed ground-truth box raises
ValueErrorwith its index in the message; - a batch mixing good and bad rows reports only the bad indices;
- zero-area / collinear boxes still give IoU 0 with no error.
- a crossed ground-truth box raises
On performance: I benchmarked the is_valid check against main, and it adds about 2% end-to-end on LocalizationConfusion with polygons, so a vectorized validity check is fine to keep.
|
Thanks, that makes sense: fixing them quietly hides the broken annotation. Changed as you suggested: the reordering is gone from polygon_iou, and the update() of the five polygon metrics now raise a ValueError naming the self-intersecting ground truth indices. Zero-area / collinear boxes are left alone (IoU 0, no error). Tests cover a crossed box, a mixed batch and degenerate boxes. |
polygon_iouraisedGEOSException: TopologyException: side location conflictwhen a box's points are not in order around it, e.g. a ground-truth polygon annotated as[[0, 0], [1, 1], [1, 0], [0, 1]]: the polygon crosses itself and shapely can not intersect it. This stopsLocalizationConfusion,OCRMetricandDetectionMetricwithuse_polygons=Trueon the whole evaluation.The points of such a box are now put back in order with
order_pointsbefore computing the IoU; valid polygons are left as they are.Test added (fails on main)