Phase 3 Funding: Dask compatibility for nddata and quantity - #283

afaik nddata is already fine with dask? Or are there specific features that you know break dask support?

Sure, this seems like a fun task to do, @eteq. I reviewed the associated issue, astropy/astropy#8227. I don't see a clear definition of work order. Would you like to huddle on this and see if we can hash out some clear goal posts?

For example, something we can automate completly against. Example: loading files larger than memory and having the quantities correctly calculated using ASDF or FITS.

Harder example: Does cloud support have to be tested too? How does that translate into the Dask Scheduler distributing data?

How does this request differ from: #275?

@mrocklin, any chance for a Coiled account with some free credits for deving on this using Coiled.io?

@mrocklin, Also, can the Dask data distribution be modified in the scheduler? Or do we still have to role our own data distribution algorithm when working with AWS s3 or another static site service?

Because the estimated total of all the proposals is a bit higher than the amount available for allocation, we suggest that you (@eteq) provide a budget range - i.e., a "minimum for this project to be viable" and "maximum you could use effectively for this project".

I think this proposal needs a goal, like "implement X". If the community votes for it in this stage, we have no idea what we get for the money since dask-support is a pretty fluid thing that could touch a lot of astropy data structures. The proposal would be helped by describing one or two use cases that are currently not possible, but will be implemented in this proposal.

Happy to talk about things from the dask side. Note that fsspec support is coming to FITS in astropy astropy/astropy#13238 ; but kerchunk also has an interest in FITS and would provide dasky parallel out-of-core access via zarr.

It seems like the quantity part of this proposal will need to wait for #269 to be completed, so I wonder if it would make more sense to propose this in the next cycle once #269 has been done if it is approved?

Yeah I'm leaning in the direction @astrofrog proposes. I'm inclined to leave this up to gauge general interest, but given all this work happening kind of in parallel, I think it might be safer to give it a better try next cycle if it doesn't get a lot of interest in the current hslf-finished state. (Note, though, a proposal from @jbcurtin or @martindurant in the next cycle would be very welcome even if this doesn't go through now!)

I am not likely to propose anything in astropy, but more than happy to provide advice and perhaps integration work.

It seems to me that there are some conflicting ideas going on. Perhaps I can help differentiate them.

• the storage layer. astropy supports primarily FITS. Other IO is possible right now, such as ASDF, and yet more would be easy to implement, such as zarr. The latter is the one I am familiar with, and the easiest way to hook into kerchunk and dask.
• the in-memory representation, astropy numpy-augmented structures, versus ndcube, versus xarray or others, and where these are stored (RAM/GPU). The structure must be able to hold units, uncertainties, WCS ...
• the compute engine. Numpy, xarray as storage containers also come with compute APIs, and can both be daskified. JAX has its own compute layer over several storage formats.

Specifically about dask, these are the reasons an astro data handler might want it:

• laziness, you can specify big datasets, and though selections and further operations, don't compute anything until the expression is ready, and so only load what you need.
• out-of-core, work an algorithm by processing chunk-by-chunk, so you can work on datasets much bigger than memory. This, combined with the previous point, means you can load data directly from remote and not need either disk or memory size that can hold all of it.
• parallel, working on many tasks at once, potentially across many machines, pooling resources such as GPUs, memory, network bandwidth. This is only worthwhile when the job is big, since any parallel execution comes with overhead costs.

You might find it interesting, that creating a new "collection" for dask is not that hard. We have been working on this recently in dask-awkward, if anyone is interested. In the case here, there is a lot more pre-existing work tying in quantities, so presumably less new code needed. However, you cannot expect existing algorithms to just work!

Please react to this comment to vote on this proposal (👍, 👎, or no reaction for +0)

Really needs focus before committing funding.

Unfortunately, this FR does not have support from the community, so it will be closed without merging. Please feel free to try again in future Astropy funding cycles.