Repository navigation
Labels
Area - Out of core 💾Working with on disk dataWorking with on disk dataArea – Performance 🐌MetaA meta-request to be implemented in multiple stepsA meta-request to be implemented in multiple steps
Description
What kind of feature would you like to request?
Additional function parameters / changed functionality / changed defaults?
Please describe your wishes
Meta issue tracking scanpy functions without dask support
Related #921
First we should test things more generically:
- add global
array_typefixture to use in all tests for features below
@ivirshup came up with places where we should prioritize dask support:
-
normalize_total- seems to work: Fix normalize_total for dask #2466
- has tests: Allow dask arrays to propagate through _normalize_data() #1663
-
log1p,normalize_per_cell,filter_cells/*_genes:- accept Dask arrays: Support running on distributed compute engines like Dask, and Spark via Zap #283
- Test that they are kept as Dask arrays throughout: Test full Dask support for
log1p,normalize_per_cell,filter_cells/filter_genes#2814
-
sc.pp.pca: Dask PCA support #2563 Implement sparsecovariance_eighPCA using Dask #3263 -
sc.pp.calculate_qc_metrics(feat):calculate_qc_metricswithdask#3307 sc.pp.highly_variable_genes:-
sc.tl.rank_genes_groups- depends on Handle Dask arrays in some utilities #2621
-
sc.pp.subsample/sc.pp.sample: so simple it has always been supported - Aggregate: https://flox.readthedocs.io/en/latest/
sc.get.aggregatewith dask #3659
Later:
- scale updates sparse scale #2942
- nearest-neighbors: Nearest Neighbors dask/dask-ml#982 may need to implement ourselves...
- umap: implement ourselves?
- metrics: depednent on squidpy decisions but should be dask-ifiable
- scrublet?
Metadata
Metadata
Assignees
Labels
Area - Out of core 💾Working with on disk dataWorking with on disk dataArea – Performance 🐌MetaA meta-request to be implemented in multiple stepsA meta-request to be implemented in multiple steps
Type
Projects
- StatusShow more project fieldsIn Progress
- StatusShow more project fieldsMeta
Activity
mumichae commentedon Feb 14, 2024
For the highly_variable_genes dask support, are you also considering cases where the dask chunks are scipy sparse matrices?
flying-sheep commentedon Feb 15, 2024
Not currently, but since the scope of that PR got reduced, it shouldn’t be too much work.
jan-engelmann commentedon Mar 12, 2025
would be great to be able to use
sc.get.aggregatewith dask arrays!sc.get.aggregatewith dask #3659daskmap_blocks#3723[-]Expand dask support[/-][+]Dask support[/+]