Adding shape validation for add_matrix_to_collection by ktsitsi · Pull Request #4258 · single-cell-data/TileDB-SOMA

ktsitsi · 2025-10-01T17:28:02Z

Issue and/or context:
The SOMA specification defines strict requirements for the shape of layers within pre-defined collections:

X: (O, V)
obsm: (O, M)
varm: (V, M)
obsp: (O, O)
varp: (V, V)

Where O is the domain of obs.soma_joinid, V is the domain of var.soma_joinid in the containing SOMAMeasurement, and M is user-defined.

However, the soma.io.add_matrix_to_collection() and soma.io.add_X_layer() functions do not validate the shape of incoming matrices.
Changes:

Introducing an internal validation function for testing the shape of the incoming Layer/Matrix to be added in a SOMA measurement.
Tests for add_matrix_to_collection and add_X_layer

Notes for Reviewer:

**DEFAULT BEHAVIOR: schema_validation = True **

codecov · 2025-10-01T18:09:21Z

Codecov Report

❌ Patch coverage is 96.96970% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 89.45%. Comparing base (e21bff2) to head (b2f8ad5).
⚠️ Report is 36 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #4258       +/-   ##
===========================================
+ Coverage   76.12%   89.45%   +13.32%     
===========================================
  Files         233       61      -172     
  Lines       32559     7176    -25383     
  Branches     1240        0     -1240     
===========================================
- Hits        24786     6419    -18367     
+ Misses       7349      757     -6592     
+ Partials      424        0      -424

Flag	Coverage Δ
libtiledbsoma	`?`
python	`89.45% <96.96%> (+<0.01%)`	⬆️
r	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	`89.45% <96.96%> (+<0.01%)`	⬆️
libtiledbsoma	`∅ <ø> (∅)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bkmartinjr

target shape determination is based upon the axis dataframe, not X['data']

bkmartinjr

Two items that definitely need fix:

obsp/varp checks are incorrect - see inline note
need to test pre-1.15 (old-style shape)
There are other suggested fixes also noted inline

bkmartinjr

Couple of requests:

Need a test case that covers pre-1.15 experiment, schema validation on, and a "good" layer being added. Ie. a case that should succeed
The handling of pre-1.15 data appears to get the wrong result for its shape, so schema validation on an old Experiment will always fail. See inline comments on this bug.

bkmartinjr

Two tiny nits in the tests. Would be great if you can fix as they are super easy.

Otherwise, it LGTM.

github-actions · 2025-12-05T18:03:28Z

The backport to release-2.2 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-release-2.2 release-2.2
# Navigate to the new working tree
cd .worktrees/backport-release-2.2
# Create a new branch
git switch --create backport-4258-to-release-2.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 2d91da66948febfcd128c6d27e90aa4227e27dc7
# Push it to GitHub
git push --set-upstream origin backport-4258-to-release-2.2
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-release-2.2

Then, create a pull request where the base branch is release-2.2 and the compare/head branch is backport-4258-to-release-2.2.

* Adding shape validation for add_matrix_to_collection * Adding test for add_X_layer * Add context to the validation function * Schema validation optional default False * PR fixes * PR fixes vol 2 * Adding all versions in tests * Adding full docstrings * Add entry in HISTORY.md * Minor fixes for PR * Opening internal var dataframe in read mode when writing & text fix * PR fixes - timestamp, closed property * Add unit tests for previous versions * PR comments * PR comments * Add warning instead of log msg * Rebase with main * Final misc * Match argument to all tests (cherry picked from commit 2d91da6)

* Adding shape validation for add_matrix_to_collection * Adding test for add_X_layer * Add context to the validation function * Schema validation optional default False * PR fixes * PR fixes vol 2 * Adding all versions in tests * Adding full docstrings * Add entry in HISTORY.md * Minor fixes for PR * Opening internal var dataframe in read mode when writing & text fix * PR fixes - timestamp, closed property * Add unit tests for previous versions * PR comments * PR comments * Add warning instead of log msg * Rebase with main * Final misc * Match argument to all tests (cherry picked from commit 2d91da6) Co-authored-by: Konstantinos Tsitsimpikos <[email protected]>

ktsitsi requested review from bkmartinjr and jp-dark October 1, 2025 17:28

ktsitsi force-pushed the kt/SOMA-387/add-shape-validation-to-soma-io-add-functions branch from 71f98e2 to b37b395 Compare October 1, 2025 19:33

ktsitsi requested a review from aaronwolen October 1, 2025 20:13

bkmartinjr reviewed Oct 2, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr requested changes Oct 2, 2025

View reviewed changes

ktsitsi force-pushed the kt/SOMA-387/add-shape-validation-to-soma-io-add-functions branch from 727dddb to 260630c Compare October 7, 2025 17:05

ktsitsi requested a review from bkmartinjr October 7, 2025 17:30

bkmartinjr reviewed Oct 7, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr reviewed Oct 7, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr reviewed Oct 7, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr reviewed Oct 7, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py

bkmartinjr reviewed Oct 7, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr reviewed Oct 7, 2025

View reviewed changes

Comment thread apis/python/tests/test_basic_anndata_io.py

bkmartinjr requested changes Oct 7, 2025

View reviewed changes

ktsitsi requested a review from bkmartinjr October 7, 2025 20:11

jp-dark reviewed Oct 8, 2025

View reviewed changes

ktsitsi force-pushed the kt/SOMA-387/add-shape-validation-to-soma-io-add-functions branch 2 times, most recently from 3d54161 to 6d6cb26 Compare October 10, 2025 19:07

ktsitsi requested a review from jp-dark October 13, 2025 14:24

ktsitsi force-pushed the kt/SOMA-387/add-shape-validation-to-soma-io-add-functions branch from 6d6cb26 to 08cb444 Compare October 13, 2025 20:01

bkmartinjr reviewed Oct 14, 2025

View reviewed changes

Comment thread apis/python/HISTORY.md Outdated

bkmartinjr reviewed Oct 14, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr reviewed Oct 14, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

bkmartinjr reviewed Oct 14, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py Outdated

ktsitsi requested a review from bkmartinjr October 14, 2025 15:12

bkmartinjr reviewed Oct 14, 2025

View reviewed changes

Comment thread apis/python/src/tiledbsoma/io/ingest.py

bkmartinjr reviewed Oct 14, 2025

View reviewed changes

Comment thread apis/python/tests/test_basic_anndata_io.py Outdated

bkmartinjr requested changes Oct 14, 2025

View reviewed changes

ktsitsi added 17 commits October 20, 2025 14:33

Adding shape validation for add_matrix_to_collection

3689c15

Adding test for add_X_layer

cdb8dce

Add context to the validation function

6e9b24b

Schema validation optional default False

a2879d0

PR fixes

7b9fa9f

PR fixes vol 2

771825b

Adding all versions in tests

b3f5e2b

Adding full docstrings

4f76592

Add entry in HISTORY.md

7e1eed5

Minor fixes for PR

34191e7

Opening internal var dataframe in read mode when writing & text fix

9e9f208

PR fixes - timestamp, closed property

95836fd

Add unit tests for previous versions

a84c36b

PR comments

e06930d

PR comments

63e3f05

Add warning instead of log msg

b1d3d60

Rebase with main

6e4f91e

ktsitsi force-pushed the kt/SOMA-387/add-shape-validation-to-soma-io-add-functions branch from d4917cd to 6e4f91e Compare October 20, 2025 18:34

ktsitsi requested a review from bkmartinjr October 20, 2025 18:51

bkmartinjr reviewed Oct 21, 2025

View reviewed changes

Comment thread apis/python/tests/test_basic_anndata_io.py Outdated

Final misc

ccad8f3

bkmartinjr approved these changes Oct 21, 2025

View reviewed changes

Match argument to all tests

b2f8ad5

ktsitsi merged commit 2d91da6 into main Oct 21, 2025
12 checks passed

ktsitsi deleted the kt/SOMA-387/add-shape-validation-to-soma-io-add-functions branch October 21, 2025 20:23

jp-dark added the backport release-2.2 label Dec 5, 2025

jp-dark mentioned this pull request Dec 5, 2025

Adding shape validation for add_matrix_to_collection (#4258) #4329

Merged

Conversation

ktsitsi commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

bkmartinjr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkmartinjr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bkmartinjr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bkmartinjr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ktsitsi commented Oct 1, 2025 •

edited

Loading

codecov Bot commented Oct 1, 2025 •

edited

Loading

bkmartinjr left a comment •

edited

Loading