Skip to content

Reading/writing F-order data w/o compression is broken #1279

Closed

Description

@normanrz

Zarr version

v2.13.3

Numcodecs version

v0.10.2

Python Version

3.10.8

Operating System

tested on Linux, Mac

Installation

poetry

Description

I was trying to upgrade zarr-python in our webKnossos libs and noticed failing tests. The tests essentially write some data, reopen the array, read the data back, and compare before/after.
We use F-order to write our data. I tried to replace F with C order and the tests succeeded. Interestingly, the issue does not occur when writing with compression (tested with Blosc). I didn't check whether the writing or reading part is the cause for this problem.

Steps to reproduce

from pathlib import Path
from shutil import rmtree

import numpy as np
import zarr


def get_multichanneled_data(dtype: type) -> np.ndarray:
    data: np.ndarray = np.zeros((3, 250, 200, 10), dtype=dtype)
    for h in range(10):
        for i in range(250):
            for j in range(200):
                data[0, i, j, h] = i * 256
                data[1, i, j, h] = j * 256
                data[2, i, j, h] = 100 * 256
    return data


def main():
    ds_path = Path("testoutput") / "test_zarr"
    rmtree(ds_path, ignore_errors=True)

    data = get_multichanneled_data(np.uint16)
    ds = zarr.create(
        shape=data.shape,
        chunks=(3, 32, 32, 32),
        dtype=data.dtype,
        compressor=(None),
        store=zarr.storage.FSStore(url=str(ds_path), mode="a"),
        order="F",
    )

    ds[:, :, :, :] = data

    ds_reopened = zarr.open_array(
        store=zarr.storage.FSStore(url=str(ds_path), mode="r")
    )

    written_data = ds_reopened[:, :, :, :]

    assert data.shape == written_data.shape
    assert np.array_equal(data, written_data)


if __name__ == "__main__":
    main()

Additional output

No response

Activity

  1. added
    bugPotential issues with the zarr-python library
    on Dec 1, 2022
  2. normanrz commented on Dec 1, 2022

    @normanrz
    MemberAuthor

    I just checked: the write path seems to be the culprit.

  3. joshmoore commented on Dec 6, 2022

    @joshmoore
    Member

    Can confirm that using FSStore fails while DirectoryStore passes:

        ds = zarr.create(
            shape=data.shape,
            chunks=(3, 32, 32, 32),
            dtype=data.dtype,
            compressor=(None),
            store=zarr.storage.DirectoryStore(str(ds_path)),
            order="F",
        )
    
        ds[:, :, :, :] = data
    
        ds_reopened = zarr.open_array(
            store=zarr.storage.DirectoryStore(str(ds_path)),
        )
    

    cc: @martindurant

  4. normanrz commented on Dec 6, 2022

    @normanrz
    MemberAuthor

    I debugged a bit further. Previously, bytes were passed to the FSStore and now the arrays are passed directly. That causes issues with the order while serializing in fsspec. Not sure whether to fix on fsspec or zarr side (leaning towards zarr).

    The change is that ensure_bytes is not called anymore for FSStore:
    https://github.com/zarr-developers/zarr-python/blob/main/zarr/core.py#L2232-L2233
    Change: 43266ee#diff-f640a0c570765203632323d71a9d1145ec6121c90db73f56e1271ab0efd5b8d8R2232

  5. normanrz commented on Dec 6, 2022

    @normanrz
    MemberAuthor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugPotential issues with the zarr-python library

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions