Skip to content

.rechunk()-Issue with dask 2024.11.2 #11544

Closed

Description

@habi

I'm loading a big bunch of tomographic data, crop out small regions, rechunk those with chunks='auto' and want to save them to disk with .to_zarr.

dask.__version__='2024.11.2' complains about doing so, while dask.__version__='2024.9.1' does so without issue.

Minimal Complete Verifiable Example:

# Load reconstructions
Reconstructions = [dask_image.imread.imread(os.path.join(f, '*rec*.png')) for f in Data.Folder]
# Save to zarr
Data['OutputNameZarr'] = [f + '.zarr' for f in Data.Folder]
for c, row in Data.iterrows():
        Reconstructions[c].rechunk(chunks='auto').to_zarr(row['OutputNameZarr'])
# Load zarr files back in
Reconstructions = [dask.array.from_zarr(file) for file in Data['OutputNameZarr']]
# Select one reconstruction and crop a small region
whichsample = 12
center=(2000,2000,1250)
sidelength=333
Crop = Reconstructions[whichsample][center[2] - (sidelength // 2):center[2] + ((1+sidelength) // 2), center[1] - (sidelength // 2):center[1] + ((1+sidelength) // 2), center[0] - (sidelength // 2):center[0] + ((1+sidelength) // 2)]
Crop.rechunk(chunks='auto').to_zarr(OutputNameCrop)

The code above works nicely with dask=2024.9.1 (on one machine), but with dask=2024.11.2 (on another machine), I get

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[43], line 2
      1 whichsample = 2
----> 2 Cutout = cutout(whichsample,
      3                 center=(2000,2000,1250),
      4                 sidelength=333,
      5                 verbose=True)

Cell In[42], line 39, in cutout(whichsample, center, sidelength, verbose)
     35     Crop = Reconstructions[whichsample][center[2] - (sidelength // 2):center[2] + ((1+sidelength) // 2),
     36                            center[1] - (sidelength // 2):center[1] + ((1+sidelength) // 2),
     37                            center[0] - (sidelength // 2):center[0] + ((1+sidelength) // 2)]
     38     # Save to disk
---> 39     Crop.rechunk().to_zarr(OutputNameCrop)
     40 if verbose:
     41     for d, direction in enumerate(directions):

File [~\AppData\Local\anaconda3\envs\laminitis\Lib\site-packages\dask\array\core.py:3015](http://localhost:8888/~/AppData/Local/anaconda3/envs/laminitis/Lib/site-packages/dask/array/core.py#line=3014), in Array.to_zarr(self, *args, **kwargs)
   3004 def to_zarr(self, *args, **kwargs):
   3005     """Save array to the zarr storage format
   3006 
   3007     See https://zarr.readthedocs.io for details about the format.
   (...)
   3013     dask.array.to_zarr : equivalent function
   3014     """
-> 3015     return to_zarr(self, *args, **kwargs)

File ~\AppData\Local\anaconda3\envs\laminitis\Lib\site-packages\dask\array\core.py:3875, in to_zarr(arr, url, component, storage_options, overwrite, region, compute, return_stored, **kwargs)
   3872     raise ValueError("Cannot use `region` keyword when url is not a `zarr.Array`.")
   3874 if not _check_regular_chunks(arr.chunks):
-> 3875     raise ValueError(
   3876         "Attempt to save array to zarr with irregular "
   3877         "chunking, please call `arr.rechunk(...)` first."
   3878     )
   3880 storage_options = storage_options or {}
   3882 if storage_options:

ValueError: Attempt to save array to zarr with irregular chunking, please call `arr.rechunk(...)` first.

If I change the last line to Crop.rechunk((100, 100, 100)).to_zarr(OutputNameCrop), my code works, but I thus "loose" the auto-chunk-size selection, which worked quite nicely for my use case...

The cropping/rechunking code is part of a function, you can CTRL + f for 'cropper' on https://nbviewer.org/github/habi/Laminitis/blob/main/DistanceTransformation.ipynb to find it.

Activity

  1. habi commented on Nov 20, 2024

    @habi
    Author
  2. phofl commented on Nov 20, 2024

    @phofl
    Collaborator

    Thanks for your report. We changed how auto stacks up chunks, one result of this is that auto is not smoothing chunks over anymore. Let me look into this a bit

  3. phofl commented on Nov 21, 2024

    @phofl
    Collaborator

    Could you post

    Crop.rechunk(chunks='auto').chunks
    

    basically the chunking of the array before you call to_zarr. That will help me narrow this down

  4. added and removed
    needs triageNeeds a response from a contributor
    on Nov 21, 2024
  5. habi commented on Nov 21, 2024

    @habi
    Author

    When I call my function with

    Cutout = cutout(whichsample,
                    center=(2000,2000,1250),
                    sidelength=333,
                    verbose=True)
    

    print(Crop.rechunk(chunks='auto').chunks) results in "((50, 56, 56, 56, 56, 59), (333,), (333,))" and subsequently fails.

    When I call my function with

    Cutout = cutout(whichsample,
                    center=(2000,2000,1250),
                    sidelength=100,
                    verbose=True)
    

    print(Crop.rechunk(chunks='auto').chunks) results in "((100,), (100,), (100,))" and my code works.

    • sidelength=128 returns "((116, 12), (128,), (128,))" and works
    • sidelength=150 returns "((99, 51), (150,), (150,))" and works
    • sidelength=170 returns "((95, 75), (170,), (170,))" and works
    • sidelength=175 returns "((83, 92), (175,), (175,))" and fails

    Hope that helps.

  6. phofl commented on Nov 21, 2024

    @phofl
    Collaborator

    Yep that helps!

    Could you try my pr? That should fix things

  7. habi commented on Nov 29, 2024

    @habi
    Author

    Could you try my pr? That should fix things

    I honestly have no idea how I can apply your PR to my conda-installed dask.
    I'll try for sure once the release is out and report back.

  8. habi commented on Dec 6, 2024

    @habi
    Author

    As said, I wasn't able to apply the PR directly.
    But I simply reinstalled the environment and running the same code with dask.__version__='2024.12.0' works fine and as expected.
    Thanks for the quick fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions