Repository navigation
fix: count symmetric STS pairs once - #4958
KennethEnevoldsen merged 3 commits into
Conversation
| labels_statistics = calculate_score_statistics(score) | ||
|
|
||
| return AnySTSDescriptiveStatistics( | ||
| pair_overlap: dict[str, int] | None = None |
There was a problem hiding this comment.
We already have unique_pairs. I think would be better to just a bit update it instead
There was a problem hiding this comment.
Good point. I can make the existing STS unique_pairs count order-insensitive and drop the new pair_overlap field. That would catch swapped duplicates within a split, but not overlap across train/validation/test from #4901. Would you prefer narrowing this PR to the within-split check?
There was a problem hiding this comment.
Overall our statistics are split based. I think overlap across subsets can be computed in overall
GioiaZheng
changed the title
| column1, column2 = self.column_names | ||
| if ( | ||
| isinstance(column1, str) | ||
| and isinstance(column2, str) | ||
| and len(self.metadata.modalities) == 1 | ||
| and self.metadata.modalities[0] == "text" | ||
| ): | ||
| pair_stats["unique_pairs"] = len( | ||
| _canonical_text_pair_keys( | ||
| _load_col(column1), | ||
| _load_col(column2), | ||
| ) | ||
| ) |
There was a problem hiding this comment.
I think better to modify here
mteb/mteb/abstasks/_statistics_calculation.py
Line 517 in be0f623
There was a problem hiding this comment.
Moved this into calculate_pair_modality_statistics with an opt-in symmetric=True for STS, so directional pair-classification tasks keep their existing behavior. I also added a regression test for the default ordered case. All checks are green now.
isaac-chung
left a comment
isaac-chung
left a comment
There was a problem hiding this comment.
Seems like all of @Samoed 's comments have been addressed.
Summary
unique_pairsstatistic, so(a, b)and(b, a)count as one pair;Scope
This revision follows the review feedback to reuse
unique_pairs. It removes the proposedpair_overlapfield and the separate cross-split quality-check path. Train/validation/test overlap is not reported by this change because descriptive statistics remain split-based.Validation
pytest tests/test_abstasks/test_sts_statistics.py tests/test_abstasks/test_task_metadata.py::test_descriptive_statistics_mock_tasks -q --disable-warnings— 31 passedruff check mteb/abstasks/sts.py tests/test_abstasks/test_sts_statistics.pyruff format --check mteb/abstasks/sts.py tests/test_abstasks/test_sts_statistics.pymypy mteb/abstasks/sts.pygit diff --checkClose #4901