Skip to content

[python] deprecate tiledbsoma.io resume mode#4083

Merged
bkmartinjr merged 3 commits intobkm/soma-148from
bkm/soma-152
May 23, 2025
Merged

[python] deprecate tiledbsoma.io resume mode#4083
bkmartinjr merged 3 commits intobkm/soma-148from
bkm/soma-152

Conversation

@bkmartinjr
Copy link
Copy Markdown
Member

@bkmartinjr bkmartinjr commented May 22, 2025

Issue and/or context:

"resume" mode for tiledbsoma.io ingestion methods is deprecated

Fixes SOMA-152

Changes:

  • added function to check mode and conditionally generate the deprecation warning
  • added checks for the warning in unit tests
  • a very slight refactoring of from_anndata to allow consistent stacklevel warning

Example deprecation message:

test.py:276: DeprecationWarning: The 'resume' ingest_mode is deprecated and will be removed in a future version. The current implementation has a known issue that can can cause multi-dataset appends to not resume correctly.
  
  The recommended and safest approach for recovering from a failed ingestion is to delete the partially written SOMA Experiment and restart the ingestion process from the original input files or a known-good backup.
    tiledbsoma.io.from_h5ad(output_path2, resume_mode_h5ad_file.as_posix(), "RNA", ingest_mode="resume")

@bkmartinjr bkmartinjr marked this pull request as ready for review May 22, 2025 23:01
@bkmartinjr bkmartinjr requested review from aaronwolen and jp-dark May 22, 2025 23:02
@bkmartinjr bkmartinjr requested a review from ktsitsi May 23, 2025 13:26
@bkmartinjr bkmartinjr merged commit 851b634 into bkm/soma-148 May 23, 2025
4 checks passed
@bkmartinjr bkmartinjr deleted the bkm/soma-152 branch May 23, 2025 14:10
bkmartinjr pushed a commit that referenced this pull request May 23, 2025
[python] deprecate tiledbsoma.io.create_from_matrix (#4082)
92780ce 
* deprecate tiledbsoma.io.create_from_matrix

* add change log entry

* [python] deprecate tiledbsoma.io resume mode (#4083)

* add deprecation for ingest resume mode

* add change log entry

* add docstring
bkmartinjr pushed a commit that referenced this pull request May 23, 2025
[python] deprecate tiledbsoma.io.append_ functions (#4081)
038a87f 
* deprecate io.append_ functions

* changelog entry

* mark previously deprecated functions with decorator

* clean up whitespace

* [python] deprecate tiledbsoma.io.create_from_matrix (#4082)

* deprecate tiledbsoma.io.create_from_matrix

* add change log entry

* [python] deprecate tiledbsoma.io resume mode (#4083)

* add deprecation for ingest resume mode

* add change log entry

* add docstring
@johnkerl johnkerl mentioned this pull request May 27, 2025
Merged
@maarten-devries
Copy link
Copy Markdown

maarten-devries commented Jun 3, 2025 •
edited
Loading

I'm a bit sad to see this go. We use this option quite a lot - never for multi-dataset appends.

When ingestions error out, "just re-ingest from scratch" (if I'm reading the above correctly) is not a very satisfying alternative to this resume option.

In addition, this function has nice properties that we use for other use cases. From a conversation with James.

We use it as following:
I have a query SOMA already successfully on TileDB. Now I have the exact same query SOMA plus a few arrays in uns , obsm , obsp .
The code will successfully only add those arrays that were added compared to the SOMA already in TileDB.

In short, given SOMA A1 and SOMA A2, add the "extra" arrays that are in A2 to A1.

The user guarantees that SOMA A2 is just SOMA A1 plus some added arrays.

@jggatter
Copy link
Copy Markdown

jggatter commented Jun 3, 2025

I still don't fully understood the various ingestion mode behaviors, so I suppose I'm not sad to see it go personally.

@maarten-devries We should probably instead use the API to check whether we have these slots in the SOMA. If we don't then we can likely use tiledbsoma.io.add_matrix_to_collection. I'm less certain about uns, I would have to double check how we're doing this in our wrapper.

@aaronwolen
Copy link
Copy Markdown
Member

aaronwolen commented Jun 3, 2025

@maarten-devries and @jggatter, thanks to you both for the feedback. Mind if I follow-up on Slack to hear more about the workflows described here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aaronwolen aaronwolen aaronwolen approved these changes

@ktsitsi ktsitsi ktsitsi approved these changes

@jp-dark jp-dark Awaiting requested review from jp-dark

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bkmartinjr @maarten-devries @jggatter @aaronwolen @ktsitsi