Skip to content

User documentation on how to setup jobs or run large analysis #768

Open

Description

@berombau

Is your feature request related to a problem? Please describe.
Current documentation shows small examples. Working on real large datasets varies in some ways and has specific needs:

  • limiting Dask workers to limit memory usage
  • job-based instead of interactive
  • sometimes needs to use a specific HPC setup
  • requesting resources via SLURM or using workflow managers like Nextflow, Snakemake...
  • working with a distributed Dask cluster

Describe the solution you'd like
A documentation page should explain this and link to existing resources. It would also be interesting to gather existing documentation of executing large jobs with SpatialData.

Some resources:

Activity

  1. berombau commented on Nov 13, 2024

    @berombau
    ContributorAuthor

    Some new notebook ideas:

  2. moved this to 🏗️ In Progress in Basel Hackathon Nov 2024on Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions