You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
agreed, but I think you can get the same functionality today using a modified store class: see this example, which uses a primary store (where the data is coming from) and a cache store (where you want to store the data as a cache). This might be better than the LRUStoreCache because in this case you can easily re-use the cache later, but I haven't tested this approach extensively
I mainly use zarr is for Machine Learning, for efficient data loading during model training.
ML however relies on mostly random sampling (without replacement, i.e. for each epoch one datapoint is only loaded once) from a huge dataset.
That is, any caching similar to LRUCache that relies on caching the more frequently accessed data would be essentially useless in that scenario. A more 'stupid' caching (i.e. just cache everything accessed until limit is reached, without any concept of 'staleness') might be really helpful for such applications.
Of course implementing ones own is always an option, but I'd guess this is not such a niche case and would benefit from a default support.
Activity
d-v-b commentedon Aug 7, 2025
agreed, but I think you can get the same functionality today using a modified store class: see this example, which uses a primary store (where the data is coming from) and a cache store (where you want to store the data as a cache). This might be better than the LRUStoreCache because in this case you can easily re-use the cache later, but I haven't tested this approach extensively
ruaridhg commentedon Aug 7, 2025
Happy to work on this approach using the example and see how far I get.
d-v-b commentedon Aug 7, 2025
great! if this is useful, then we could definitely include it in the library
[-]Add LRUStoreCache to Zarr 3.0[/-][+]Add general StoreCache to Zarr 3.0[/+][-]Add general StoreCache to Zarr 3.0[/-][+]Add general CacheStore to Zarr 3.0[/+]FabricioArendTorres commentedon Sep 10, 2025
A general comment regarding use of caches:
I mainly use zarr is for Machine Learning, for efficient data loading during model training.
ML however relies on mostly random sampling (without replacement, i.e. for each epoch one datapoint is only loaded once) from a huge dataset.
That is, any caching similar to LRUCache that relies on caching the more frequently accessed data would be essentially useless in that scenario. A more 'stupid' caching (i.e. just cache everything accessed until limit is reached, without any concept of 'staleness') might be really helpful for such applications.
Of course implementing ones own is always an option, but I'd guess this is not such a niche case and would benefit from a default support.