Repository navigation
Extension proposal: chunk statistics #319
Description
This is a draft idea for a possible Zarr extension, looking for feedback before putting more work into it.
The idea in short
For each chunk, save a separate json file containing statistics about values in that chunk (including e.g. minimum, maximum, average, number of non-nan values).
Readers can read this very small file instead of the chunk itself to perform some kinds of computation more efficiently.
Advantages
- Efficient where queries on variables (e.g. to build a mask of all points with variable x larger than 5 you can look at all chunk stats and only have to load the chunks where
min(x)<5andmax(x)>5) - Efficient computation of mean/sum over multiple chunks
- "quick preview" of large portions of data by loading the mean values for each chunk rather than the data itself
Costs
- Storing an extra file for each chunk (but each file will be less than 100B, so this is a low cost)
- Overhead of querying more objects: even when the total number of bytes read gets decreased by this extension it is likely that the number of files read will increase (all stata files plus some chunk files)
- Added complexity to zarr readers to support the extension (too much to be worth considering?)
Is this something that could work? Something that already exists? Something that would never be useful?
I haved worked with Zarr for a while but I haven't worked on Zarr yet, so any feedback is appreciated.
Activity
d-v-b commentedon Nov 14, 2024
This is a cool and useful idea, and people have proposed it before. The challenge for the exact thing you suggest here will be managing
O(num_chunks)statistics files. For large Zarr arrays, we will be basically doubling the number of objects in storage. But maybe that's worth it?Alternatively, if we view "chunk statistics" in the abstract as a tabular data structure, with 1 row per chunk, then there are lots of ways to store that data structure, some of which might not generate so many files.
If you have the time, maybe it would be worth hacking together a prototype? If that prototype proves useful to you, that will be sure proof that the idea might be useful to someone else, and we can worry about scaling details later.
d-v-b commentedon Nov 14, 2024
I think this idea dovetails with concepts used in tools like kerchunk and virtualizarr, where an explicit list of chunks (a "chunk manifest") is created and used for data retrieval, instead of relying on the discovered state of the storage backend. That same chunk manifest could perhaps be augmented with columns for chunk statistics?
cc @martindurant @TomNicholas
martindurant commentedon Nov 14, 2024
Agreed, manifests are an obvious place to put this kind of thing - just one more value per chunk. This will increase the size of the manifests, but they ought to end up small regardless. The idea sort of suggests that manifests ought to be the only way to store extra information per chunk, which remain encoded data only. (Noting that checksum codec implementations currently do store data in the chunk along with the data)
How to make use of the metadata is another issue. We are talking about adding a "query plan" layer to what zarr currently does.
The related suggestion was for codecs where the parameters vary between chunks, e.g., a different scale/offset for each. That also requires updating the read pipelin, but in a different way.
TomNicholas commentedon Nov 14, 2024
I agree - which is why I raised #305 to discuss the general problem of storing metadata that scales with
O(num_chunks).EDIT: Since that issue was raised Icechunk was released, which currently uses
msgpackto store the manifests.jhamman commentedon Nov 14, 2024
Thanks @barbuz for opening this issue. Its a a topic we'e already been thinking about with Icechunk and we've intentionally left room in the Chunk Manifest spec for such a feature. As @TomNicholas points out, we're currently using MessagePack for the manifests but this is extensible. We could pick a different container for very large manifests in the future.
I also agree with @martindurant - we'll need to think carefully about how Zarr implementations would take advantage of such statistics.
barbuz commentedon Nov 21, 2024
Thank you all for your feedback on this. I appreciate the points you made, which I can summarise in two main items:
How to manage O(num_chunks) metadata
The approach I envisioned is based on a separate file for each chunk because this would make it easy to do parallel writes and reads across several chunks, in line with the main philosophy of Zarr. I know that this is not the most compact way of storing this information, but maybe it won't be a big issue in practice? Some tests could be needed to be sure.
The approaches based on manifests used by some tools are great, but maybe they should be kept as options to be decided by each tool, while the base Zarr specs should focus on defining simple and interoperable ways to store data.
How to use the statistics in queries
This, I believe, is the hard part. My naive idea would be for the Zarr libraries to expose access to these statistics, and for the data analysis libraries built on top of Zarr to use them to make their queries more efficient. I will admit though that I have been mostly managing Zarr data through
xarrayand I am not sure where the line is between whatxarraydoes and whatzarr-pythondoes. I will need to look into this, but if anyone has a clear picture of how these interactions are structured and an idea about how these chunk statistics could fit in I would love to hear it!bendichter commentedon Nov 3, 2026
We've prototyped this in shadowzarr, with a draft spec in docs/block-statistics.md. Our motivating case was finding a time window in a long irregular
timestampsarray without reading the whole array. Here is how it handles the points raised above.O(num_chunks) metadata. Each statistic is one ordinary Zarr array laid out on the block grid, not one file per chunk.
minmaxhas shape(*block_grid, 2), is chunked and compressed like any other array, and is read lazily. It works unchanged in plain Zarr stores, kerchunk references and Icechunk.Location. The statistics sit in a hidden
.statsgroup next to the array they describe, at.stats/<array name>/<type>/<block shape>. Copying a group keeps them. The array lists them in ablock_statisticsattribute, and readers use those paths rather than parsing names.Blocks, not chunks.
block_shapeis given in elements and is independent of the chunk grid. Along each dimension it must divide the chunk size, be a multiple of it, or cover the whole dimension. A size of 1 keeps a dimension, so[30000, 1]on(time, channel)data gives per-channel statistics every 30,000 samples. One statistic can have several block shapes, which gives levels.Query layer.
searchsorted(array, values, side)matchesnp.searchsortedon a sorted array. It reads only theminmaxarray plus the blocks the values fall in. On an NWB LFP timestamps array on DANDI (12.1M float64 values, 97 MB in 1,024 chunks), finding a 2 s window took 0.13 s, 3 requests and 0.14 MB. Reading the whole array took 7.4 s, 1,024 requests and 63 MB. The statistics were 16 KB.To see it in action, the README section has a short example on that DANDI file, and scripts/block_statistics_benchmark.py reproduces the numbers on any remote NWB file.
So far only
minmaxis defined. Next arenan_countandmoments:sum/sum_sqfor integers andmean/M2for floats. After that come rollups that build coarser levels from finer ones.Two questions:
block_statisticswork?