Repository navigation
Description
I'm loading a big bunch of tomographic data, crop out small regions, rechunk those with chunks='auto' and want to save them to disk with .to_zarr.
dask.__version__='2024.11.2' complains about doing so, while dask.__version__='2024.9.1' does so without issue.
Minimal Complete Verifiable Example:
# Load reconstructions
Reconstructions = [dask_image.imread.imread(os.path.join(f, '*rec*.png')) for f in Data.Folder]
# Save to zarr
Data['OutputNameZarr'] = [f + '.zarr' for f in Data.Folder]
for c, row in Data.iterrows():
Reconstructions[c].rechunk(chunks='auto').to_zarr(row['OutputNameZarr'])
# Load zarr files back in
Reconstructions = [dask.array.from_zarr(file) for file in Data['OutputNameZarr']]
# Select one reconstruction and crop a small region
whichsample = 12
center=(2000,2000,1250)
sidelength=333
Crop = Reconstructions[whichsample][center[2] - (sidelength // 2):center[2] + ((1+sidelength) // 2), center[1] - (sidelength // 2):center[1] + ((1+sidelength) // 2), center[0] - (sidelength // 2):center[0] + ((1+sidelength) // 2)]
Crop.rechunk(chunks='auto').to_zarr(OutputNameCrop)The code above works nicely with dask=2024.9.1 (on one machine), but with dask=2024.11.2 (on another machine), I get
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[43], line 2
1 whichsample = 2
----> 2 Cutout = cutout(whichsample,
3 center=(2000,2000,1250),
4 sidelength=333,
5 verbose=True)
Cell In[42], line 39, in cutout(whichsample, center, sidelength, verbose)
35 Crop = Reconstructions[whichsample][center[2] - (sidelength // 2):center[2] + ((1+sidelength) // 2),
36 center[1] - (sidelength // 2):center[1] + ((1+sidelength) // 2),
37 center[0] - (sidelength // 2):center[0] + ((1+sidelength) // 2)]
38 # Save to disk
---> 39 Crop.rechunk().to_zarr(OutputNameCrop)
40 if verbose:
41 for d, direction in enumerate(directions):
File [~\AppData\Local\anaconda3\envs\laminitis\Lib\site-packages\dask\array\core.py:3015](http://localhost:8888/~/AppData/Local/anaconda3/envs/laminitis/Lib/site-packages/dask/array/core.py#line=3014), in Array.to_zarr(self, *args, **kwargs)
3004 def to_zarr(self, *args, **kwargs):
3005 """Save array to the zarr storage format
3006
3007 See https://zarr.readthedocs.io for details about the format.
(...)
3013 dask.array.to_zarr : equivalent function
3014 """
-> 3015 return to_zarr(self, *args, **kwargs)
File ~\AppData\Local\anaconda3\envs\laminitis\Lib\site-packages\dask\array\core.py:3875, in to_zarr(arr, url, component, storage_options, overwrite, region, compute, return_stored, **kwargs)
3872 raise ValueError("Cannot use `region` keyword when url is not a `zarr.Array`.")
3874 if not _check_regular_chunks(arr.chunks):
-> 3875 raise ValueError(
3876 "Attempt to save array to zarr with irregular "
3877 "chunking, please call `arr.rechunk(...)` first."
3878 )
3880 storage_options = storage_options or {}
3882 if storage_options:
ValueError: Attempt to save array to zarr with irregular chunking, please call `arr.rechunk(...)` first.If I change the last line to Crop.rechunk((100, 100, 100)).to_zarr(OutputNameCrop), my code works, but I thus "loose" the auto-chunk-size selection, which worked quite nicely for my use case...
The cropping/rechunking code is part of a function, you can CTRL + f for 'cropper' on https://nbviewer.org/github/habi/Laminitis/blob/main/DistanceTransformation.ipynb to find it.
Activity
habi commentedon Nov 20, 2024
dask.array.to_zarr()does not parse forchunkskwarg #10522, but I haven't seen the issue pop up before.daskwithcondaand this environment file: https://github.com/habi/Laminitis/blob/main/environment.ymlphofl commentedon Nov 20, 2024
Thanks for your report. We changed how auto stacks up chunks, one result of this is that auto is not smoothing chunks over anymore. Let me look into this a bit
phofl commentedon Nov 21, 2024
Could you post
basically the chunking of the array before you call to_zarr. That will help me narrow this down
habi commentedon Nov 21, 2024
When I call my function with
print(Crop.rechunk(chunks='auto').chunks)results in "((50, 56, 56, 56, 56, 59), (333,), (333,))" and subsequently fails.When I call my function with
print(Crop.rechunk(chunks='auto').chunks)results in "((100,), (100,), (100,))" and my code works.sidelength=128returns "((116, 12), (128,), (128,))" and workssidelength=150returns "((99, 51), (150,), (150,))" and workssidelength=170returns "((95, 75), (170,), (170,))" and workssidelength=175returns "((83, 92), (175,), (175,))" and failsHope that helps.
phofl commentedon Nov 21, 2024
Yep that helps!
Could you try my pr? That should fix things
habi commentedon Nov 29, 2024
I honestly have no idea how I can apply your PR to my
conda-installeddask.I'll try for sure once the release is out and report back.
habi commentedon Dec 6, 2024
As said, I wasn't able to apply the PR directly.
But I simply reinstalled the environment and running the same code with
dask.__version__='2024.12.0'works fine and as expected.Thanks for the quick fix!