Skip to content

Shuffle cross-validation rows by position to support duplicate indices - #1710

Open
galafis wants to merge 1 commit into
CamDavidsonPilon:masterfrom
galafis:fix-cv-duplicate-index
Open

galafis wants to merge 1 commit into
CamDavidsonPilon:masterfrom
galafis:fix-cv-duplicate-index

Conversation

@galafis

@galafis galafis commented Oct 24, 2026

Copy link
Copy Markdown

k_fold_cross_validation shuffles index labels and reindexes the dataset. A DataFrame formed by concatenating cohorts without resetting the index raises ValueError: cannot reindex on an axis with duplicate labels. If every label is identical, the shuffle silently leaves rows in their original order.

Shuffle row positions with iloc instead. This preserves labels while assigning the same folds for the same rows and seed, regardless of their index. Regression tests compare fold scores against the original RangeIndex for repeated labels, a constant string index and a non-unique MultiIndex, and check that the input is unchanged.

Validation: all three new parameterized cases fail on the original implementation; the complete utility module passes after the fix (92 passed, two approximation warnings). Python 3.12.14, NumPy 2.5.4, pandas 2.3.3, lifelines 0.30.3. Repository-pinned Black 22.8.0 reports formatting differences in unchanged baseline sections of both files; its diff does not touch the new regression or changed shuffle line. git diff --check passes. The full model suite was not run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant