[python] add axis delete to Experiment by bkmartinjr · Pull Request #4215 · single-cell-data/TileDB-SOMA

bkmartinjr · 2025-08-29T14:34:36Z

Add Experiment.var_axis_delete and Experiment.obs_axis_delete and their unit tests.

codecov · 2025-08-29T17:38:54Z

Codecov Report

❌ Patch coverage is 92.70833% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.71%. Comparing base (8ecd7e3) to head (f856eb5).
⚠️ Report is 72 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #4215       +/-   ##
===========================================
+ Coverage   69.46%   89.71%   +20.25%     
===========================================
  Files         169       61      -108     
  Lines       18972     7284    -11688     
  Branches     1233        0     -1233     
===========================================
- Hits        13179     6535     -6644     
+ Misses       5359      749     -4610     
+ Partials      434        0      -434

Flag	Coverage Δ
libtiledbsoma	`?`
python	`89.71% <92.70%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	`89.71% <92.70%> (+0.01%)`	⬆️
libtiledbsoma	`∅ <ø> (∅)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jp-dark · 2025-09-02T15:59:37Z

I keep forgetting that we need to explicitly add new Python functions to the Sphinx docs (see doc/source dir). We can add that to a separate PR since I need to do it for all the new delete methods as well.

jp-dark · 2025-09-02T16:49:59Z

+    if isinstance(obj, (DataFrame, SparseNDArray, GeometryDataFrame, PointCloudDataFrame)) and all(
+        obj.schema.get_field_index(col) >= 0 for col in join_on
+    ):


The support for deletes in GeometryDataFrame isn't implemented yet.

True, but I didn't want to duplicate the error checking already implemented in delete_cells.

That said, we can remove this easily and ignore the spatial arrays, at the cost of potential user confusion. My sense is we need to raise a proactive error if the Experiment contains any unsupported sub-array.

What do you recommend?

I think we raise a NotImplementedError here if there is a GeometryDataFrame. I think it's unlikely anyone is using the GeometryDataFrame in the wild right now, but I don't want to corrupt an Experiment if they are.

jp-dark

I have a couple small comments, but overall this looks good to me.

jp-dark · 2025-09-02T16:55:48Z

+# var_id, n_cells
+# len  1837
+


Stray comments?

jp-dark · 2025-09-02T18:03:12Z

+
+
+@pytest.mark.spatialdata
+def test_experiment_obs_axis_delete_spatial(soma_spatial_experiment, soma_tiledb_context) -> None:  # noqa: F811


Missing a similar test for var_axis_delete

aaronwolen

Looks great and the docstrings are all very clear/informative. Most of my suggestions are formatting-related.

aaronwolen · 2025-09-02T17:40:01Z

+        This operation affects all Measurements within an Experiment. Deleted observations are specified:
+        - by `obs` DataFrame soma_joinid: a sequence of soma_joinid or a slice
+        - or, by `obs` DataFrame value filter / query condition


Suggested change

This operation affects all Measurements within an Experiment. Deleted observations are specified:

- by `obs` DataFrame soma_joinid: a sequence of soma_joinid or a slice

- or, by `obs` DataFrame value filter / query condition

This operation affects all Measurements within an Experiment. Deleted observations are specified:

- by ``obs`` DataFrame soma_joinid: a sequence of soma_joinid or a slice

- or, by ``obs`` DataFrame value filter / query condition

Inserted space so the list will render properly and fixed inline code formatting.

aaronwolen · 2025-09-02T17:41:45Z

+        Sub-objects affected by this operation:
+        - exp.obs - delete matching DataFrame elements by soma_joinid


Suggested change

Sub-objects affected by this operation:

- exp.obs - delete matching DataFrame elements by soma_joinid

Sub-objects affected by this operation:

- exp.obs - delete matching DataFrame elements by soma_joinid

aaronwolen · 2025-09-02T17:42:50Z

+        The following sub-objects are unaffected by this operation:exp.ms[*].var , exp.ms[*].varm[*], exp.ms[*].varp[*]
+        and any user-defined arrays within the collection.


Suggested change

The following sub-objects are unaffected by this operation:exp.ms[*].var , exp.ms[*].varm[*], exp.ms[*].varp[*]

and any user-defined arrays within the collection.

The following sub-objects are unaffected by this operation:

- exp.ms[*].var , exp.ms[*].varm[*], exp.ms[*].varp[*]

- any user-defined arrays within the collection

aaronwolen · 2025-09-02T17:43:58Z

+        If any affected sub-objects are of type DenseNDArray (dense TileDB array), the entire operation results in an error.
+
+        Either ``coords`` or ``value_filter`` must be provided. When both ``coords`` and ``value_filter`` are provided,
+        the cells that match both constraints will be removed.


Suggested change

the cells that match both constraints will be removed.

the observations that match both constraints will be removed.

aaronwolen · 2025-09-02T18:24:32Z

+        Deleted features are specified:
+        - by `var` DataFrame joinid: a sequence of joinid or a slice
+        - by `var` DataFrame value filter / query condition


Suggested change

Deleted features are specified:

- by `var` DataFrame joinid: a sequence of joinid or a slice

- by `var` DataFrame value filter / query condition

Deleted features are specified:

- by ``var`` DataFrame joinid: a sequence of joinid or a slice

- by ``var`` DataFrame value filter / query condition

aaronwolen · 2025-09-02T18:27:11Z

+        Sub-objects affected by this operation, given a named Measurement (ms_name):
+        - exp.ms[*ms_name*].var - delete matching DataFrame elements


Suggested change

Sub-objects affected by this operation, given a named Measurement (ms_name):

- exp.ms[*ms_name*].var - delete matching DataFrame elements

Sub-objects affected by this operation, given a named Measurement (ms_name):

- exp.ms[*ms_name*].var - delete matching DataFrame elements

aaronwolen · 2025-09-02T18:27:59Z

+        - exp.ms[*ms_name*].varp[*] - delete matching SparseNDMatrix values, joining on the soma_dim_0 and soma_dim_1 dimensions
+        - exp.spatial[*].varl[*ms_name*][*] - delete matching PointCloudDataFrame/GeometryDataFrame elements, joining on the soma_joinid dimension
+
+        The following Measurement sub-objects are unaffected by this operation: exp.ms[*ms_name*].obsm[*] , exp.ms[*ms_name*].obsp[*] ,


Suggested change

The following Measurement sub-objects are unaffected by this operation: exp.ms[*ms_name*].obsm[*] , exp.ms[*ms_name*].obsp[*] ,

The following Measurement sub-objects are unaffected by this operation:

- exp.ms[*ms_name*].obsm[*] and exp.ms[*ms_name*].obsp[*]

aaronwolen · 2025-09-02T18:30:39Z

+        If any affected sub-objects are a DenseNDArray (dense TileDB array), the entire operation results in an error.
+
+        Either ``coords`` or ``value_filter`` must be provided. When both ``coords`` and ``value_filter`` are provided,
+        the cells that match both constraints will be removed.


Suggested change

the cells that match both constraints will be removed.

the variables that match both constraints will be removed.

aaronwolen · 2025-09-02T18:30:56Z

+            >>> with tiledbsoma.Experiment.open(experiment_uri, mode="d") as exp:
+            ...     exp.var_axis_delete(value_filter="feature_biotype == 'gene'")
+
+        Note: Deleting cells does not change the size of the current domain or possible enumeration values.


Suggested change

Note: Deleting cells does not change the size of the current domain or possible enumeration values.

Note: Deleting variables does not change the size of the current domain or possible enumeration values.

aaronwolen · 2025-09-02T18:32:02Z

+    """Append the candidate array to the final selection list if:
+    - not a dense array


Suggested change

"""Append the candidate array to the final selection list if:

- not a dense array

"""Append the candidate array to the final selection list if:

- not a dense array

bkmartinjr · 2025-09-03T02:01:17Z

I believe I have addressed all of the issues:

formatting & copy editing in docstrings
added missing unit test
added silent filtering of any joinids outside the domain/shape of the target array

aaronwolen

Awesome, thank you!

bkmartinjr added 8 commits August 29, 2025 07:29

first pass at axis delete

fddb8e3

Merge branch 'main' into bkm/soma-187

357d3cb

change log entry

5679f70

add var axis test; fix bugs found in TDD

cb2469d

cleanup

8ad5f07

further reduction in test memory use

850b76e

refine error handling and unit tests for same

de70edf

lint

73307de

improve handling of intermittent errors

dbe7267

bkmartinjr marked this pull request as ready for review August 29, 2025 18:06

bkmartinjr requested review from aaronwolen and jp-dark August 29, 2025 18:06

fix typo in pytest mark

56d271a

jp-dark reviewed Sep 2, 2025

View reviewed changes

jp-dark approved these changes Sep 2, 2025

View reviewed changes

aaronwolen requested changes Sep 2, 2025

View reviewed changes

bkmartinjr added 3 commits September 2, 2025 12:31

Merge branch 'main' into bkm/soma-187

df25e3a

PR f/b

b57f334

silently filter joinids out of domain/shape

f856eb5

bkmartinjr requested review from aaronwolen and jp-dark September 3, 2025 02:01

jp-dark approved these changes Sep 3, 2025

View reviewed changes

aaronwolen approved these changes Sep 3, 2025

View reviewed changes

bkmartinjr merged commit 5550c6f into main Sep 3, 2025
17 checks passed

bkmartinjr deleted the bkm/soma-187 branch September 3, 2025 13:52



		@pytest.mark.spatialdata
		def test_experiment_obs_axis_delete_spatial(soma_spatial_experiment, soma_tiledb_context) -> None: # noqa: F811

		Sub-objects affected by this operation:
		- exp.obs - delete matching DataFrame elements by soma_joinid

		The following sub-objects are unaffected by this operation:exp.ms[].var , exp.ms[].varm[], exp.ms[].varp[*]
		and any user-defined arrays within the collection.

	the cells that match both constraints will be removed.
	the observations that match both constraints will be removed.

		Sub-objects affected by this operation, given a named Measurement (ms_name):
		- exp.ms[ms_name].var - delete matching DataFrame elements

	the cells that match both constraints will be removed.
	the variables that match both constraints will be removed.

	Note: Deleting cells does not change the size of the current domain or possible enumeration values.
	Note: Deleting variables does not change the size of the current domain or possible enumeration values.

		"""Append the candidate array to the final selection list if:
		- not a dense array

Conversation

bkmartinjr commented Aug 29, 2025

Uh oh!

codecov Bot commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jp-dark commented Sep 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jp-dark left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaronwolen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkmartinjr commented Sep 3, 2025

Uh oh!

aaronwolen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Aug 29, 2025 •

edited

Loading