fix: multi-column groupby crashing on non-string observations by barlowa124 · Pull Request #4385 · scverse/scanpy

barlowa124 · 2026-09-28T17:29:49Z

For multi-column groupby, _prepare_dataframe joins the key values
with "_".join over raw values and takes ordering levels from
.cat.categories. A non-string key column (integer categorical, plain
numeric) or a NaN row then fails with TypeError: sequence item 1: expected str instance. Single-key groupby handles the same columns
correctly. The breakage is specific to the multi-key join. The shared
helper is used by dotplot, matrixplot, stacked_violin,
tracksplot, heatmap and the dendrogram path, so they all gain the
fix.

Joined elements pass through str() individually.
Ordering levels come from .cat.categories on categorical columns
and np.unique otherwise.
Rows missing any key value join to no category, matching the
single-column path where NaN never forms one.

Tests

test_plotting.py gains parametrized test_multi_groupby_nonstring
and test_multi_groupby_partial_nan. Integer-categorical and numeric
second keys plus partially-NaN keys are pushed through sc.pl.dotplot
and sc.pl.matrixplot, with joined labels asserted on the returned
plot object. The pre-existing test_violin image-compare failures in
this file are seaborn drift, identical on main.

…#4384) _prepare_dataframe joined multi-column groupby values with "_".join directly and took ordering levels from .cat.categories, so integer categoricals and plain numeric columns raised TypeError. Convert values with astype(str), fall back to np.unique for non-categorical columns, and order unmatched joined labels (NaN combinations) last.

codecov · 2026-09-28T17:47:01Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.11%. Comparing base (50bb2b0) to head (93ac9f0).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4385   +/-   ##
=======================================
  Coverage   85.11%   85.11%           
=======================================
  Files         139      139           
  Lines       12318    12318           
=======================================
  Hits        10484    10484           
  Misses       1834     1834

Flag	Coverage Δ
hatch-test.low-vers	`78.68% <100.00%> (ø)`
hatch-test.pre	`84.89% <100.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/scanpy/plotting/legacy/_anndata.py	`85.72% <100.00%> (ø)`

…e#4384) _prepare_dataframe joined multi-column groupby values with "_".join directly and took ordering levels from .cat.categories, so integer categoricals and plain numeric columns raised TypeError. Join values through per-element str(), take levels from np.unique for non-categorical columns, and give rows missing a groupby value no joined category, matching the single-column path.

barlowa124 · 2026-09-30T18:52:01Z

The only failing check is the title/milestone/labels metadata check, which needs maintainer-side labels.

Zethson · 2026-10-02T12:21:26Z

You're committing hundreds of times in weeks. Have you human reviewed your own PR? Otherwise, I'll close this as maintainers can guide an agent to exactly the needed solution much better.

barlowa124 · 2026-10-02T17:41:12Z

Yes, this diff was human-reviewed before posting. Multi-column groupby joined obs values with "_".join, which raised TypeError on integer or float columns. The fix maps values through str, keeps the single-column NaN rule (a row missing any key gets no category), and falls back to np.unique for ordering when a column is not categorical. The added tests run under the hatch-test CI matrices (3.12 low-vers, 3.14 stable/pre/few-extras), which pass.

On commit volume, I can squash the branch if that is easier to review.

sylvesterkaczmarek · 2026-11-08T17:36:18Z

+                    *(
+                        obs_tidy[g].cat.categories
+                        if isinstance(obs_tidy[g].dtype, CategoricalDtype)
+                        else np.unique(obs_tidy[g])


np.unique sorts object values, so this still raises for an ordinary non-categorical grouping column containing mixed strings and integers, or strings and missing values. For example, np.unique(pd.Series([1, 'control'], dtype=object)) raises TypeError: '<' not supported between instances of 'str' and 'int'; ['control', None] fails similarly. This PR is intended to support non-string observations. Could we use pd.unique/a categorical-safe unique operation (excluding missing values) and add a mixed-type and missing-value regression for multi-column grouping?

Thanks for the correction, @barlowa124. I checked head b927638: the mixed int/string and string/missing cases now fall back to pd.unique with missing values dropped, while ordinary numeric values retain the sorted path. I independently exercised those three cases, and the new hosted test matrices pass. That resolves the correctness issue I raised. One small CI fix remains: the ruff check job reports D205 at tests/plotting/legacy/test_plotting.py:444 because the new multi-line test docstring needs a blank line between its summary and description. Appreciate the follow-through.

np.unique raises TypeError on object columns mixing ints and strings or containing missing values. Fall back to pd.unique in appearance order with missing values dropped, and cover the case with a mixed-type/missing-value regression test. Addresses review on scverse#4385.

barlowa124 · 2026-11-11T05:35:34Z

Fixed in d31f6df. np.unique now sits behind a helper that falls back to pd.unique in appearance order with missing values dropped when sorting raises TypeError, covering mixed int/string and string/None object columns. test_multi_groupby_mixed_type exercises a mixed-type column with missing entries for dotplot and matrixplot. pytest tests/plotting/legacy/test_plotting.py -k multi_groupby passes locally, 8 tests.

read-the-docs-business · 2026-11-12T15:12:06Z

Documentation build overview

📚 scanpy | 🛠️ Build #4720987 | 📁 Comparing 93ac9f0 against latest (7db89c6)

🔍 Preview build

318 files changed · + 44 added · ± 269 modified · - 5 deleted

+ Added

± Modified

- Deleted

…g on non-string observations) (#4517) Co-authored-by: Angus Barlow <[email protected]>

sylvesterkaczmarek · 2026-11-13T09:30:39Z

Thanks for following up on the mixed-type case, @barlowa124, and for taking it through to merge. Glad the fallback now covers mixed string/integer and missing values, with the grouping regressions in place. Appreciate the careful correction.

barlowa124 added 2 commits September 28, 2026 13:29

Rename release note to PR number

b478865

barlowa124 changed the title ~~Fix multi-column groupby crashing on non-string observations~~ fix: multi-column groupby crashing on non-string observations Sep 28, 2026

barlowa124 force-pushed the fix/multi-groupby-nonstr branch from ebffb76 to 682a0c3 Compare September 28, 2026 18:13

Zethson added this to the 1.12.5 milestone Oct 2, 2026

flying-sheep modified the milestones: 1.12.5, 1.12.6, 1.12.7 Oct 29, 2026

sylvesterkaczmarek reviewed Nov 8, 2026

View reviewed changes

autofix-ci Bot and others added 3 commits November 11, 2026 05:35

[autofix.ci] apply automated fixes

b927638

simpler

9f12524

Merge branch 'main' into pr/barlowa124/4385

5cd0d25

simplify tests

93ac9f0

flying-sheep merged commit e099d9b into scverse:main Nov 12, 2026
14 checks passed

meeseeksmachine mentioned this pull request Nov 12, 2026

Backport PR #4385 on branch 1.12.x (fix: multi-column groupby crashing on non-string observations) #4517

Merged

flying-sheep pushed a commit that referenced this pull request Nov 13, 2026

Backport PR #4385 on branch 1.12.x (fix: multi-column groupby crashin…

4064e4f

…g on non-string observations) (#4517) Co-authored-by: Angus Barlow <[email protected]>

flying-sheep modified the milestones: 1.12.7, 1.12.6 Nov 13, 2026

Navigation Menu

Repository navigation

fix: multi-column groupby crashing on non-string observations - #4385

fix: multi-column groupby crashing on non-string observations#4385
flying-sheep merged 8 commits into
scverse:mainfrom
barlowa124:fix/multi-groupby-nonstr

barlowa124 commented Sep 28, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

barlowa124 commented Sep 30, 2026 •

edited

Loading

Uh oh!

Zethson commented Oct 2, 2026

Uh oh!

barlowa124 commented Oct 2, 2026 •

edited

Loading

Uh oh!

sylvesterkaczmarek Nov 8, 2026

Uh oh!

sylvesterkaczmarek Nov 11, 2026

Uh oh!

barlowa124 commented Nov 11, 2026

Uh oh!

read-the-docs-business Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

sylvesterkaczmarek commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

barlowa124 commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Uh oh!

codecov Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

barlowa124 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zethson commented Oct 2, 2026

Uh oh!

barlowa124 commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sylvesterkaczmarek Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

sylvesterkaczmarek Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

barlowa124 commented Nov 11, 2026

Uh oh!

read-the-docs-business Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Uh oh!

sylvesterkaczmarek commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

barlowa124 commented Sep 28, 2026 •

edited

Loading

codecov Bot commented Sep 28, 2026 •

edited

Loading

barlowa124 commented Sep 30, 2026 •

edited

Loading

barlowa124 commented Oct 2, 2026 •

edited

Loading

read-the-docs-business Bot commented Nov 12, 2026 •

edited

Loading