All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
This release includes many breaking changes, particularly for extension developers. The most impactful breaking changes are summarised below and detailed in the sections that follow.
-
Breaking: Array operations are defined by the
ArrayOps,Array{Read,Write,Update,Mut}Opstraits and their async variants, implemented byArrayand the newArrayCached- Explicit-options variants (
_opt) are removed; derive an array with different options instead
- let chunk: Vec<f32> = array.retrieve_chunk_opt(&[0, 0], &codec_options)?; + let chunk: Vec<f32> = array.with_codec_options(codec_options).retrieve_chunk(&[0, 0], &Resources::default())?;
retrieve_chunk_subsetandstore_chunk_subsetare renamed and accept any&dyn Indexer
- let data: Vec<f32> = array.retrieve_chunk_subset(&[0, 0], &[0..2, 0..2])?; + let data: Vec<f32> = array.retrieve_partial_chunk(&[0, 0], &[0..2, 0..2], &Resources::default())?;
- Explicit-options variants (
-
Breaking: Chunk caches are used through
ArrayCached, which pairs an array with a cache and supports async stores- let cache = ChunkCacheDecodedLruChunkLimit::new(array.clone(), 32); - let chunk: Vec<f32> = cache.retrieve_chunk(&[0, 0], &CodecOptions::default())?; + let array = ArrayCached::new(array, ChunkCacheDecodedLruChunkLimit::new(32)); + let chunk: Vec<f32> = array.retrieve_chunk(&[0, 0], &Resources::default())?;
-
Breaking: Pass
Resourcesto each array operation to control compute concurrency, andComputeResourcesto in-memory codec operations- Compute-target control moves out of
CodecOptionsandConfig; arrays captureConfig::chunk_concurrent_minimumat construction instead of storing it inCodecOptions
- Compute-target control moves out of
-
Breaking: Codecs are bound to the data type and fill value of an array at array creation, so invalid codec chains are rejected when an array is built or opened rather than on first use
- Codec implementations are split into
UnboundArrayTo{Array,Bytes}CodecTraits(created from metadata) andArrayTo{Array,Bytes}CodecTraits(bound withwith_context); seezarrs_codec0.3.0
- Codec implementations are split into
-
Breaking: Encoded and raw bytes are
CowBytesthroughout, which removes copies on many store, codec, andTensorpaths, andTensormay borrow its bytes -
Breaking: Arrays have an immutable dimensionality, and chunk grids no longer support out-of-bounds chunks or unlimited (zero-sized) dimensions; resize an array before extending it
-
New extensions:
- Codecs:
cast_value,numcodecs.packbits(V2:packbits)
- Codecs:
- Breaking: Pass
Resourcesper array and chunk-cache operation instead of setting the compute target inCodecOptionsorConfigchunk_concurrent_minimumis private array state initialized fromConfig;concurrency_chunks_and_codectakes the minimum directly instead ofCodecOptions
- Breaking: Add operation traits decoupling array methods from the
Arraytype:ArrayOps,ArrayReadOps,ArrayWriteOps,ArrayUpdateOps,ArrayMutOps, and async variants- These are implemented as inherent traits on
ArrayandArrayCached - Promote previously private methods to public:
[async_]retrieve_chunk_intoand[async_]retrieve_partial_chunk_into
- These are implemented as inherent traits on
- Breaking: Remove explicit-options variants and parameters from synchronous and asynchronous
Arrayoperations; configure a derived array with the correspondingwith_*method, then call the operation- Add
ArrayOps::{with_codec_options,with_metadata_options,with_metadata_erase_version}()andArrayMutOps::set_metadata_erase_version() ArrayOps::metadata_opt()no longer takes an options argument and applies the array's stored metadata options
- Add
- Breaking: Rename
retrieve_chunk_subsettoretrieve_partial_chunkandstore_chunk_subsettostore_partial_chunk, including async and_intovariants, which accept&dyn Indexer - Breaking:
[Async]ArrayReadOps::[async_]retrieve_{chunk,partial_chunk,array_subset}_intotake&mut ArrayBytesDecodeIntoTarget - Breaking:
[Async]ArrayWriteOps::erase_chunksand[Async]ArrayReadOps::retrieve_encoded_chunksaccept&dyn Indexer, validate the chunks against the chunk grid, and return anArrayErrorinstead of aStorageError, as doerase_chunkandretrieve_encoded_chunk- Behavioural change: out-of-bounds chunks or an incompatible dimensionality are an error rather than a silent no-op
[async_]erase_chunkslimit the number of concurrent erases to the concurrent target of the array, rather than being unbounded
- Breaking:
[Async]ArrayReadOps::retrieve_encoded_chunk[s]returnBytesinstead ofVec<u8>, and[Async]ArrayWriteOps::store_encoded_chunktakesCowBytes - Breaking:
ArrayOps::chunk_keyreturnsResult<StoreKey, ArrayError>, and chunk keys from a chunk key encoding are validated rather than trusted- Add
ArrayError::{InvalidStoreKey,ChunkKeyEncodingError}, and returnArrayError::InvalidChunkGridIndicesErrorif the chunk indices do not match the dimensionality of the array [async_]store_encoded_chunk,[async_]erase_chunk, and[async_]retrieve_encoded_chunkno longer store, erase, or retrieve a key that does not belong to the chunk grid
- Add
- Breaking: Make array dimensionality immutable; dimensionality-changing shape updates return
ArrayCreateError::ChangedDimensionality - Breaking: Make
ArrayMutOps::set_dimension_names()fallible, validate and persist names, and retain them when converting Zarr V2 arrays to V3 - Breaking:
ArrayCreateError::CodecsCreateErrorcontains aCodecCreateErrorrather than aPluginCreateError, andArrayCreateError::ChunkGridCreateErrorcontains aChunkGridCreateError- Remove
ArrayCreateError::InvalidSubchunkShape, superseded byChunkGridCreateError
- Remove
- Breaking:
ArrayError::{InvalidArraySubset,InvalidChunkSubset}box theirArraySubset, so array results stay small now that subsets are stored inline - Breaking: Remove the deprecated
_elements/_ndarraymethod variants ofArray, array extension traits, andChunkCache; use the genericstore_*andretrieve_*methods withVec<T>orndarray::Array<T, D>instead - Chunk caching
- Breaking: Add
ArrayCached<TStorage, C>, which pairs anArraywith a chunk cache, and refactorChunkCacheinto a pure key/chunk value container- Remove the
ChunkCacheretrieve_*methods, which are provided byArrayCached, and chunk cache constructors no longer take an array (e.g.ChunkCacheDecodedLruChunkLimit::new(capacity)) ChunkCacheTypeDecodedis anOption, andChunkCacheTypeEncodedisOption<Bytes>instead ofOption<Arc<CowBytes<'static>>>[Async]ChunkCache::try_get_or_insert_withtakes the chunk indices asArrayIndicesrather thanVec<u64>, so a cache access does not allocate a key, and is no longer#[doc(hidden)]ChunkCachehas aValueassociated type and is no longer dyn compatible, andChunkCacheTypeis sealed- Add
invalidatemethods
- Remove the
- Add support for async caching
ArrayCachedimplementsAsyncArrayReadOps,AsyncArrayWriteOps, andAsyncArrayUpdateOps, so that chunk caches can be used with asynchronous stores- Add the
AsyncChunkCachetrait andAsyncChunkCacheLru{ChunkLimit,SizeLimit}implementations withAsyncChunkCache{Encoded,Decoded,PartialDecoder}Lru{ChunkLimit,SizeLimit}aliases - Add
ChunkCacheTypeAsyncPartialDecoderfor cached asynchronous partial decoding - Add the
SyncChunkCacheTypeandAsyncChunkCacheTypesubtraits ofChunkCacheType
- Retrieve complete cached chunks without extracting a subset
- Fix chunk subset retrieval from a chunk cache to validate the chunk subset and chunk indices if a chunk is absent, rather than returning fill values
- Breaking: Add
- Subchunks
- Breaking: Remove
[Async]ArrayShardedReadableExt[Cache], whose methods are now part of[Async]ArrayReadOps[async_]retrieve_encoded_subchunkand[async_]subchunk_byte_rangeare removed; use[Async]ShardingPartialDecoder::{retrieve_subchunk_encoded,subchunk_byte_range}for low-level encoded subchunk access
- Breaking: Remove
ArrayShardedExt, whosesubchunk_gridandsubchunk_shapeare now part ofArrayOpsis_shardedandis_exclusively_shardedare removed; queryArrayOps::subchunk_grid()instead
- Breaking:
ArrayOps::subchunk_grid()returnsChunkGridDecodedRef<'_>, which distinguishes an absent subchunk grid from one that is only resolvable per chunk- Use
ChunkGridDecodedRef::as_chunk_grid()to get the subchunk grid only if it is resolvable for the whole array - Re-export
ChunkGridDecodedandChunkGridDecodedReffromzarrs::array
- Use
- Breaking: Remove
ArrayOps::subchunk_grid_shape(); query thesubchunk_grid()directly - Breaking: Remove
effective_subchunk_shape(), superseded byArrayOps::subchunk_shape()- Behavioural change:
ArrayOps::subchunk_shape()returns the subchunk shape of the subchunk grid (if regular), rather than thesharding_indexedcodecchunk_shape
- Behavioural change:
- Add
ArrayReadOps::{retrieve_subchunk,retrieve_subchunks}and_at_levelvariants, andArrayError::MissingSubchunkGridfor requests on arrays without a subchunk grid - Add
ArrayOps::{subchunk_grids,subchunk_grid_at_level,subchunk_shape_at_level}for querying nested subchunk grid hierarchies, ordered outermost to innermost - Add
ArrayOps::{subchunk_codec,subchunk_codecs,subchunk_codec_at_level}for querying the codec of each subchunk level- Codec subchunking is opt-in through
ArrayToBytesCodecTraits::as_subchunking, returningArrayToBytesCodecSubchunkingTraits; codecs without subchunks require no additional trait implementation
- Codec subchunking is opt-in through
- Add
ArrayReadOps::local_subchunk_grid[_at_level]for chunk-local subchunk grids
- Breaking: Remove
- Tensors and bytes
- Breaking: Use
cowbytes::CowBytesinstead ofCow<'a, [u8]>orbytes::Bytesfor encoded and raw bytes throughout the library, removing copies on some arrayretrieve_/store_paths with select stores and codecs - Breaking:
Tensorhas a lifetime parameter and may borrow its bytes, andTensor::into_partsreturnsCowBytes - Breaking: Replace the
TensorLikeimplementation forTensorwithTensor::into_dlpack(), which exports aTensor<'static>as a read-only, versioned DLPack managed tensor (requiresdlpack)- Support DLPack 1.4, including
complex_bfloat16export
- Support DLPack 1.4, including
- Add
Tensor::into_static()for converting aTensorinto aTensor<'static>, copying only if its bytes are borrowed - Add
Tensor::num_elements()and implementCloneandDebugforTensor - Implement
IntoArrayBytesfor&bytes::Bytes; storing a chunk from abytes::Bytesis zero-copy where the codec chain passes its input through unchanged - Fix DLPack export to validate that tensor bytes cover the shape and data type, adding
TensorError::{InsufficientBytes,UnsupportedShape}
- Breaking: Use
- Implement
CloneforArrayBuilder - Reserve capacity for fallible array, chunk-grid, codec, fill-value, and async result collections
- Clarify in the
ArrayParallel Writing documentation that a chunk must not be retrieved while it is being written, not just that it must not be written concurrently, and fix a missing negation in thepartial_encoderrule - Fix
ArrayBuilderto validate aFillValueagainst the data type before binding the codecs, so an invalid fill value returnsArrayCreateError::InvalidFillValuerather than a codec error - Fix multi-chunk operations to treat a concurrent target of zero as unconstrained, and avoid stalled async encoded-chunk retrieval and suffix stripping
- Fix
[Async]ArrayReadOps::[async_]retrieve_array_subset_intofailing across multiple chunks if the output target has the same number of elements as the array subset, but a different shape or dimensionality - Fix variable-length fill-value detection to respect element boundaries, preventing non-fill chunks from being omitted or erased when
store_empty_chunksis disabled - Fix storing an owned
ndarrayarray sliced in place, which used every element of its backing vector rather than only its own elements
- Breaking:
Group::open_opttakes aGroupOpenOptionsparameter rather thanMetadataRetrieveVersion, andHierarchy::open_opttakes aHierarchyOpenOptions - Breaking: Remove explicit-options variants and parameters from synchronous and asynchronous
Groupoperations; configure a derived group with the correspondingwith_*method instead- Add
Group::{metadata_options,with_metadata_options,metadata_erase_version,with_metadata_erase_version}() Group::metadata_opt()no longer takes an options argument and applies the group's stored metadata options
- Add
- Breaking:
node::data_keytakes the chunk key as a&strand returnsResult<StoreKey, StoreKeyError> - Add
GroupOpenOptionsandGroup::new_with_metadata_opt - Implement
DefaultforMetadataRetrieveVersionandCopyforGroupMetadataOptions NodePathusescamino::Utf8PathBufinternally instead ofstd::path::PathBuf- Add
NodePath::as_utf8_path()for direct access tocamino::Utf8Path - Implement
FromStrforNodePath, preservingTryFrom<&str>support
- Add
- Retrieve child-node metadata concurrently in asynchronous hierarchy discovery
zarrs_codec 0.2.1 → 0.3.0 (Breaking): per-operation resources and deferred decoding; codecs bind to a data type and fill value at array creation; owned, subdividable decode targets; CowBytes bytes; subchunk-grid traits replace partial decode granularity; u32/u64 ArrayBytesOffsets; ArrayBytesMask guarantees optional masks are 0 or 1
- Breaking:
update_array_bytestakes&DataTypeinstead ofDataTypeSize, validating each optional layer, andCodecError::CodecCreateErrorholds aBox<CodecCreateError>to keep array results small - Return errors when fill requests or expected optional mask lengths exceed platform limits
- Breaking: Bind array codec chains eagerly during array construction and use the bound chain for runtime and representation queries
- Add
CodecChainBoundandArrayOps::codecs_boundfor data type and fill value context-bound codec runtime operations CodecChain,ShardingCodecBuilder, anddefault_array_to_bytes_codechold or return unbound codecs (UnboundArrayTo{Array,Bytes}CodecTraits), and codec constructors returnCodecCreateErrorrather thanPluginCreateError
- Add
- Breaking: Apply codec-specific options when binding a codec chain rather than by reconfiguring its codecs
CodecChain::with_contexttakesCodecSpecificOptions, and anOption<FillValue>that isNoneif the fill value is undefined- Remove
CodecChain::with_codec_specific_options; passCodecSpecificOptionstoCodecChain::with_contextinstead Array::{with,set}_codec_specific_optionsare fallible, replace any previously set codec-specific options, and no longer changeArrayOps::codecs()- Add
ArrayOps::codec_specific_options() ArrayBuilder::from_arrayretains the codec-specific options of the array- Fix codec-specific options to propagate to all nested codecs, such as those in
sharding_indexed,optional, andvlencodecs, andCodecChains used as an array-to-bytes codec - Fix
ArrayBuilderto apply codec-specific options before validating the codec chain - Fix
Array::with_codec_specific_options()to refresh decoded subchunk grids consistently withArrayMutOps::set_codec_specific_options()
- Cache data type support when binding
bytes,packbits,bitround, andcast_value, and re-export their{Codec}DataTypeTraitsArcaliases - Copy small contiguous regions and elements directly when extracting array subsets and writing or filling output views
- Map ascending element positions in output views by stepping through their contiguous regions, rather than unravelling each position
- Add pure element, subset, and chunk-local subchunk grid mappings for built-in array-to-array codecs
- Re-export
ArrayBytesDecodeIntoTargetOptional,ArrayBytesMask[CreateError],IncompatibleOptionalMaskError, andWithOptionalMaskErrorfromzarrs::array - Decode through codec chains directly into output views, forwarding deferred decoding through transparent nested chains
- Partial encoding
- The
bytescodec and default array-to-array/array-to-bytes partial encoders replace the chunk without checking for or reading the existing chunk if an update covers the entire chunk- The chunk is erased if the update is entirely the fill value and
store_empty_chunksis disabled, as with other partial encoders - Following bytes-to-bytes codecs (e.g.
zstd) no longer decode the existing chunk, so an undecodable chunk can be replaced
- The chunk is erased if the update is entirely the fill value and
- Fix
Array::[async_]partial_encoderleaving trailing bytes of the previous chunk if a bytes-to-bytes codec (e.g.zstd) re-encoded the chunk shorter, which could make it undecodable - Fix partial chunk writes with
experimental_partial_encodingstoring a missing chunk if only the fill value is written, unlessstore_empty_chunksis enabled- Behavioural change: an empty partial chunk write no longer stores a missing chunk, even if
store_empty_chunksis enabled
- Behavioural change: an empty partial chunk write no longer stores a missing chunk, even if
- Fix partial codecs to validate indexers against their own decoded shape
- The
- Fix the partial decode granularity potentially being incorrect with multiple array-to-array codecs
- Fix the partial decoder cache in a codec chain to receive the encoded shape and data type
- Support subchunk shapes that do not evenly divide the shard shape (#233)
- Subchunks straddling the shard boundary are clipped to the shard shape, as proposed in zarr-specs #370
- Existing arrays with such subchunk shapes can be opened, but
ArrayBuilder::buildrejects them until they are part of the specification
- Add
SubchunkWriteOrder::{Lexicographic,Colexicographic,Morton}for writing subchunks in row-major, column-major, or Morton (Z-order) order- Deprecated:
SubchunkWriteOrder::Cis soft deprecated in favour ofSubchunkWriteOrder::Lexicographicand will be removed in a future release - Breaking: Remove
SubchunkWriteOrder::Random; useSubchunkWriteOrder::Unorderedfor the same behaviour
- Deprecated:
- Expose
ShardingCodecBoundand[Async]ShardingPartialDecoderAPIs for low-level encoded subchunk access (seeshardingmodule docs)- Breaking:
ShardingCodec::{compact,decode_into}move toShardingCodecBound, andShardingCodec::decode_indexis removed
- Breaking:
- Support efficient asynchronous partial encoding, and partial encoding with generic indexers
- Limit partial-encode subchunk concurrency and split the concurrent target between subchunks and their inner codecs, including async partial encoding
- Avoid copying each encoded inner chunk
- Fix decoding optional data types, which could be encoded but not decoded
- Fix a panic or incorrect data when partially decoding empty subchunks with generic indexers
- Fix async
partial_decode_intoto use the same subchunk-aware path as sync decoding - Fix
decode_intoandpartial_decode_intofailing if the output target has the same number of elements as the decoded shard or subset, but a different shape or dimensionality - Fix the partial encoder
- Rewrite the shard compactly when its output does not support partial encoding (e.g. when followed by bytes-to-bytes codecs), rather than accumulating superseded subchunks
- Replace a rewritten shard rather than erasing it first, so a failed write leaves the shard intact
- Reject overflowing shard-index offsets and append spans in sync and async partial encoders before changing storage or the cached index
- Reset the cached shard index when it erases its shard, so a subsequent partial encode does not reference subchunks of the erased shard
- The sync partial encoder reports
size_heldin bytes and includes its input handle, matching the async partial encoder - Treat a concurrent target of zero as unconstrained
- Breaking: Correct missing
packbitsfirst_byte/last_bytepadding bytes for whole-byte data. Chunks written byzarrs0.20-0.23 without them are no longer readable, and those releases cannot read the corrected chunks - Breaking:
packbitsrequires components of at most 64 bits and matching data type/component sizes - Add
NumcodecsPackBitsCodecfor Numcodecs-compatible boolean packing (V2packbits, V3numcodecs.packbits) - Bind full-width whole-byte layouts without padding to the little-endian
bytescodec for direct output placement - Copy only the packed bytes that are unpacked when unpacking part of a chunk
- Pack and unpack one-bit components of one byte (e.g.
bool) eight at a time, about 20 and 14 times faster - Fix
packbitssign extension, which was missing when partially decodingint2/int4and stopped at the byte containing the sign bit for multi-byte signed values - Fix
packbitspartial decoding with non-zerofirst_bit, big-endian encoding, and input validation
- Breaking: Codecs whose encoded representation depends on the fill value (e.g.
sharding_indexed) are rejected within thedata_codecsof anoptionalcodec with anullfill value (unless the inner data type isoptional), since the fill value of the data codecs is undefined- Such arrays created by
zarrs0.23 fail to be opened
- Such arrays created by
- Behavioural change: Bind
data_codecswith the unwrapped fill value if it is non-null (e.g.[42]->42), rather than a zero fill value- This changes the encoded representation if
data_codecscontain a codec dependent on the fill value (e.g.sharding_indexed). Subchunks of zeros omitted by prior versions are decoded as the unwrapped fill value
- This changes the encoded representation if
- Fix storing and decoding optional masks of the wrong length or with values other than 0/1, which could erase chunks or store values as null; these are now rejected
- Return an error if the dense data or offset buffer exceeds platform limits when decoding an all-null chunk
- Decoding can reuse shared data and aligned index bytes with pass-through codec chains, and encoding avoids offset-width conversion when the stored width matches
index_data_type - Fix index endianness handling to use the actual index data type rather than
uint64 - Return an error rather than panicking if the encoded index length exceeds the chunk length
- Decode directly into contiguous output views, converting endianness while copying and placing deferred output directly into the view
- Convert endianness while copying borrowed or shared bytes into strided output views with regions of at least 256 bytes, rather than into a temporary buffer
- Return an error rather than panicking if a decoded shape's element count or byte size overflows
- Reject short decoded byte ranges in synchronous and asynchronous partial decoding
- Decode directly into preallocated output on native and WebAssembly backends
- Fix incorrect partial decoding for byte ranges not aligned to the blosc
typesize(e.g. optional data types) - Return an error instead of panicking or returning short output when decompression yields fewer bytes than expected, and from
blosc_decompress_bytes_partialif the requested item range exceedsc_int::MAX
- Reuse per-thread compression and bulk decompression contexts
- Compression reuse defaults to inputs up to 1 MiB and is configurable with
ZstdCodecOptions::{with_,set_,}compression_context_reuse_threshold, viaZstdCodec::with_optionsorCodecSpecificOptions
- Compression reuse defaults to inputs up to 1 MiB and is configurable with
- Use bulk decompression bounds on WebAssembly when available
- Cap the decompression buffer by the decoded size rather than the content size claimed by a frame header
- Decode directly into preallocated output, validating frame lengths against the decoded size
- Decode directly into preallocated output (requires
gdeflate) - Decode chunks of more than one page, and return an error rather than panicking if a page is corrupt
- Check decoded representation bounds and validate unbounded input page by page before allocating the full decoded length
vlen-bytes/vlen-utf8: Breaking:ArrayBuilderreturns an error for thevlen-bytescodec with a data type other thanbytes, and thevlen-utf8codec with a data type other thanstring, as the registered codecs are only compatible with those data types (such arrays are still read)cast_value: Add thecast_valuearray-to-array codecpcodec: Decode directly into aligned contiguous output, retaining fallback for other views and rejecting incomplete decompression (requirespcodec)bz2: Decode streams directly into preallocated, uninitialised output (requiresbz2)zlib: Decode streams directly into preallocated, uninitialised output (requireszlib)gzip: Decode streams directly into preallocated output with bounded oversized-stream detectionzfp/zfpy: Replacezfp-syswith the pure-Rustzfp-rstranspose: Support optional data types, permuting values and validity masks togetherreshape: Remove warnings now that the codec is stablereshape: Fix empty partial reads and writes, which are now mapped to an empty subset of the encoded dimensionality instead of an incompatible empty list of indicesshuffle: Encode and decode 4–28× faster for 2, 4, 8 and 16-byte elements by transposing eight elements at a time without zero-initialising the output, decode directly into preallocated output, and pass one-byte elements through without copyingshuffle: Reject anelementsizeof zero, which panicked when encoding or decoding an empty chunkcrc32c/adler32/fletcher32: Reuse the input allocation (if possible) when appending/stripping the checksumcrc32c/adler32/fletcher32: Fix partial decoders returning incorrect bytes for aByteRange::Suffixrequest with the checksum at the endnumcodecs.fletcher32: Fix checksums of data with an odd length, which omitted the last byte, and read those written byzarrs0.19 to 0.23 for backwards compatibilitynumcodecs.fixedscaleoffset: Decode twice as fast, and encode faster, by transforming elements into a pre-sized outputnumcodecs.fixedscaleoffset: Fix encoding to matchnumcodecs- Compute in
f32forfloat32data and otherwisef64, rounding encoded values with ties to even and decoded integers to the nearest integer - Values are no longer saturated to the range of the data type (or
astype) before the transform when their ranges differ - Represent
offsetandscaleasf64rather thanf32 - Read single byte
dtype/astypedata types with a|byteorder (e.g.|i1, as written bynumcodecs) or without a byteorder (exceptu1)
- Compute in
numcodecs.zlib: Read configurations without alevel(as written byzarr-pythonby default) or with the zlib default level of -1 (#503)
zarrs_data_type 0.9.0 → 0.10.0 (Breaking): a layout enum and specialized traits; cached codec support; Any/Debug trait bounds; in-place byte decoding; CowBytes encode/decode; cast_value codec traits; PackBitsDataTypeTraits requires BytesDataTypeTraits
- Breaking:
DataTypehasFixed,Variable, andOptionalvariants backed by specialized traits extending the commonDataTypeTraits, which requiresinto_data_type(self: Arc<Self>)- Query byte sizes through
FixedDataTypeTraits::sizeafter matching; remove the commonsize,DataType::fixed_size, and optional sizing helpers, and traverse optional layers explicitly when needed is_fixedandis_variableclassify the variant, andas_fixed,as_variable, andas_optionalreturn their trait objects, allowing external optional implementations
- Query byte sizes through
FixedLengthUTF32DataType::newreturns an error iflength_bytesexceedsusize::MAX- Fix
optionaldata type metadata without an inner data typeconfigurationbeing rejected, as it is optional in the specification - Fix malformed fill values of the
optionaldata type being accepted, which bounddata_codecswith a fill value of the wrong size - Fix
r*(raw bits) fill values whose length differs from the data type size being accepted - Compare
r*(raw bits) data types by width, including when wrapped in optional data types
zarrs_chunk_grid 0.5.1 → 0.6.0 (Breaking): inline ArrayIndices/ArraySubset (≤4 dimensions); ChunkGridCreateError; Indexer::validate; zero-sized dimensions are no longer unlimited
- Breaking:
ArrayIndicesholds up to four indices inline, so chunk grid lookups, chunk cache keys and the indices of planned chunks do not allocateunravel_indexis re-exported fromzarrs_chunk_grid, and returnsNonerather than panicking for a shape with a zero-sized dimension
- Behavioural change: Chunk grids no longer support out-of-bounds operations or unlimited dimensions; resize before extending arrays
- Reading/writing completely out-of-bounds chunks is an error
- Querying completely out-of-bounds chunks always returns
None - Zero sized array dimensions are no longer functionally unlimited with certain chunk grids (e.g.
regular)
- Avoid an intermediate allocation in
RegularBoundedChunkGrid::subset
zarrs_chunk_key_encoding 0.2.0 → 0.3.0 (Breaking): encode returns Result<String, ChunkKeyEncodingError>, validated by the caller; drops the zarrs_storage dependency
- Encode chunk store keys directly with
ChunkKeyEncodingTraits::encode_intoin one allocation, rather than allocating per index and growing the key
zarrs_storage 0.4.5 → 0.5.0 (Breaking): set/set_partial take CowBytes; zarrs_filesystem 0.3.14 → 0.4.0 (Breaking): no per-key serialisation, byte-range fixes
- Fix partial encoding to check whether the storage transformed by the storage transformers of an array supports partial sets, rather than the underlying store
- Fix
[async_]erase_metadatato erase array metadata through the storage transformers of the array, consistent with[async_]store_metadata
zarrs_metadata 0.7.5 → 0.7.6: always serialise a configuration if present; FromStr.
zarrs_metadata_ext 0.4.4 → 0.4.5: legacy pcodec/zfp/zfpy and f64 fixedscaleoffset configurations.
zarrs_plugin 0.4.1 → 0.4.2: defaulted error/match generics on [Runtime]Plugin[2]
- Improve compatibility with other Zarr implementations:
- Store codec and storage transformer metadata as objects with a
configuration(empty if absent), including short-hand names, as required byzarr-pythonandtensorstore - Write
bytesdata type fill values as base64-encoded strings rather than arrays of bytes (both conformant), aszarr-pythononly reads the former
- Store codec and storage transformer metadata as objects with a
- Read codec metadata with names and configurations written by past
zarrsversions:numcodecs.bz2:bz2(unregistered, 0.11-0.12) andhttps://codec.zarrs.dev/bytes_to_bytes/bz2(0.13-0.19)numcodecs.pcodec:pcodec(unregistered, 0.11-0.12) and legacy configurations (0.11-0.15)zarrs.gdeflate:gdeflate(unregistered, 0.16-0.19) andhttps://codec.zarrs.dev/bytes_to_bytes/gdeflate(documented)numcodecs.fletcher32:fletcher32(unregistered, 0.19)zarrs.vlen_v2:vlen_v2(unregistered, 0.19)
- Breaking: Bump MSRV to 1.92 (11 December, 2025)
- Breaking: Bump
float8to 0.7.0,dlparkto 0.11.1, andjiffto 1.0.0 - Add the
cowbytesandcaminodependencies, whose types are part of the public API - Deprecated: Soft deprecate the
shardingfeature flag, as the sharding codec and associated utilities are always available - Avoid unused asynchronous dependencies and unnecessary native-library builds
- Bump internal dependencies:
rayon_iter_concurrent_limit0.3.0,base640.23.1,getrandom0.4.3,itertools0.15.0,lru0.19.0,quick_cache0.7.0,blusc0.0.7,serial_test3.5.0,unsafe_cell_slice0.2.4,pco1.1.0,flate21.1.9,bzip20.6.1,zstd0.14.4
- Add array-operation, sharding encode/decode, and sync/async I/O latency benchmarks, and allocation regression checks for general array operations
0.23.14 - 2026-08-15
- Fixed
GzipCodec::encoded_representation()under-estimating the worst-case encoded size, which could cause an out-of-bounds panic or aCodecErrorwhengzipwas an inner codec ofsharding_indexed(#444)- The bound now matches zlib's
deflateBound()
- The bound now matches zlib's
- The
sharding_indexedcodec now returns aCodecErrorinstead of panicking if an inner codec exceeds its reported bounded encoded size - Zarr V2 arrays with a
zstdcompressor configuration that includes achecksumfield (as written by recentnumcodecs/zarr-pythonreleases) failed to open with anunknown field checksumerror async_get_child_nodes_optnow ignores unrecognised listed prefixes consistently with sync discovery- Fixed a panic when retrieving an array subset spanning multiple chunks through a chunk cache with a nested optional data type (e.g.
Option<Option<u8>>)- Only the outermost mask was allocated, so inner masks were also dropped
0.23.13 - 2026-05-24
- Implement
IntoArrayBytesfor&ndarray::ArrayBase
- Support partial encoding and decoding in the
reshapecodec- This codec remains unregistered and aliased as
zarrs.reshape
- This codec remains unregistered and aliased as
0.23.12 - 2026-05-16
- Add the
fixed_length_utf32data type- Supports
[char; N],Vec<char>, and&[char]asElement[Owned]
- Supports
- Bump
zarrs_metadata_extto 0.4.4
0.23.11 - 2026-05-13
- Add a
microfloatfeature for expanded subfloat and complex subfloat data type / element support- The
float8feature is considered deprecated as the associated types are not IEEE 754-compliant
- The
- Add
NodePath::is_root,NodePath::parent, andNodePath::joinfor safer hierarchy path manipulation - Use Zarr V3 consolidated metadata (when present on a root group) to populate child nodes — avoiding
list_dirand per-array reads — forHierarchy::open,Node::open,Group::children,Group::traverse,Group::child_arrays,Group::child_groups,Group::child_paths,Group::child_array_paths,Group::child_group_paths, and their async variants- Add
UseConsolidatedMetadata(Auto/Must/Never) and a newConfig::use_consolidated_metadataglobal setting (defaultAuto) - When
Mustis set but consolidated metadata is absent, returnNodeCreateError::MissingMetadatawith a descriptive message (a more specific variant may be added in a future breaking release)
- Add
- Bump
zarrs_metadata_extto 0.4.3
NodePath::validatenow rejects components starting with__(the reserved prefix) and components composed only of period characters (./../...), matchingNodeNamerules
0.23.10 - 2026-04-09
- Add
SubchunkWriteOrder::Unorderedwith the same behaviour asSubchunkWriteOrder::Random- The
Randomvariant is a misnomer and it will be removed in a future release
- The
- Add
ArrayBuilder::codec_specific_options - Add
ShardingCodec::with_optionsandShardingCodec::with_subchunk_write_orderfor directly configuring aShardingCodecbefore passing it toArrayBuilder
ArrayBuilder::buildnow preserves codec objects (and their embeddedCodecSpecificOptions) passed directly viaarray_to_bytes_codecetc., rather than losing them through a metadata serialisation/deserialisation round-trip
0.23.9 - 2026-04-02
- Fix
zstdandblosccodecs on WASM (#369 by @keller-mark)- On WASM, the
blosccodec usesblusc(a pure Rust reimplementation ofc-blosc2) rather thanc-blosc
- On WASM, the
0.23.8 - 2026-03-26
- Add
rectilinearchunk grid
- Bump
zarrs_metadata_extto 0.4.2 - Replace
rectangular_array_write_readexample withrectilinear_chunk_grid
0.23.7 - 2026-03-21
- Add
ShardingCodecOptionsfor runtime configuration of the sharding codec- Subchunks can now be written in C-major order via
ShardingCodecOptions.{set_,with_,}subchunk_write_order(by @ilan-gold)
- Subchunks can now be written in C-major order via
- Add
Array::with_codec_specific_optionsandArray::set_codec_specific_optionsto reconfigure codec chains withCodecSpecificOptions - Add
CodecChain::with_codec_specific_options
- Bump
zarrs_codecto 0.2.1
0.23.6 - 2026-03-08
- Avoid a potential copy in
ShardingPartialDecoder::partial_decode_into - Bump
zarrs_chunk_gridto 0.5.1
0.23.5 - 2026-03-03
- Avoid a potential copy in
ShardingCodec::decode - Bump
zarrs_storageto 0.4.3
- Fixed missing
ndarrayfeature gate forasync_store_chunk_subset_ndarray_opt
0.23.4 - 2026-02-28
- Fix enabling
array_async_sharded_readable_extmodule without theshardingfeature
0.23.3 - 2026-02-21
- Fix
CodecChaininternal validation of decoded bytes with an array-to-array codec that changes the data type
0.23.2 - 2026-02-17
- Restore
i64Element[Owned]support fornumpy.{datetime64,timedelta64}(broken in 0.23.0)
0.23.1 - 2026-02-08
- Add
Array::[async_]retrieve_array_subset_into[_opt]for decoding array subsets directly into preallocatedArrayBytesDecodeIntoTarget
- Bump
pcodecto 1.0.0
- Support Zarr V2
|VXdata with anullfill value
0.23.0 - 2026-02-02
This release includes many breaking changes, particularly for extension developers. The most impactful breaking changes are summarised below.
-
Improved performance!
-
Breaking:
DataTypeis now a newtype holding (Arc<dyn DataTypeExtension>) rather than an enum- Use factory functions in
zarrs::array::data_typeto create data types (e.g.data_type::int8(),data_type::float32(), etc.)
- let data_type = DataType::Float32; + let data_type: DataType = data_type::float32();
- Use factory functions in
-
Breaking: Generic
Arraystore_*andretrieve_*methods- let data = array.retrieve_array_subset(&subset)?; - let data: Vec<f32> = array.retrieve_array_subset_elements(&subset)?; - let data: ndarray::ArrayD<f32> = array.retrieve_array_subset_ndarray(&subset)?; + let data: ArrayBytes = array.retrieve_array_subset(&subset)?; + let data: Vec<f32> = array.retrieve_array_subset(&subset)?; + let data: ndarray::ArrayD<f32> = array.retrieve_array_subset(&subset)?;
- array.store_array_subset_elements(&subset, &data)?; - array.store_array_subset_ndarray(&subset, &data)?; + array.store_array_subset(&subset, &data)?;
-
Breaking: split
zarrsinto more crates, addzarrs_codec,zarrs_chunk_grid, andzarrs_chunk_key_encoding- Many types that were previously accessible in these modules are not individually re-exported, but are accessible via
apisubmodules - Various types are now only re-exported higher in
zarrs::array, e.g.zarrs::array::data_type::DataType->zarrs::array::DataType
- Many types that were previously accessible in these modules are not individually re-exported, but are accessible via
-
Breaking: Rename sharding "inner chunk" terminology to "subchunk" (
inner_chunk_shape()→subchunk_shape(),retrieve_inner_chunk_opt()→retrieve_subchunk_opt(), etc.) -
Many parameters are now
&dyn ArraySubsetTraitsinstead ofArraySubsetfor ergonomic array subset indexing- array.retrieve_array_subset(&ArraySubset::new_with_ranges(&[0..3, 10..20]))?; + array.retrieve_array_subset(&[0..3, 10..20])?;
-
Add
ArrayBuilder::subchunk_shape(). Recommended overShardingCodecBuilderfor most use cases- array_builder.array_to_bytes_codec( - ShardingCodecBuilder::new( - vec![4, 4].try_into()?, // subchunk shape - ) - .build_arc(), -) +.subchunk_shape(vec![4, 4])
-
Add the
zarrs.optionaldata type and codec (spec proposal zarr-developers/zarr-extensions#33) -
Much improved API for Zarr extension point registration and aliasing, with runtime extension registration support
- See https://book.zarrs.dev/extensions.html for a tutorial on custom data types and codecs
- Add
ElementErrortype forElementandElementOwnedtrait methods - Add
ZfpyCodecfornumcodecs.zfpycompatibility - Add
zarrs::convertmodule (moved fromzarrs_metadata_ext::v2_to_v3) - Add data type marker types (
BoolDataType,Int8DataType,Float32DataType, etc.) inzarrs::array::data_type(implementingExtensionAliasesfor per-data-type alias management andDataTypeTraits::compatible_element_types()forElementtype compatibility) - Add
ArrayBytesVariableLength,ArrayBytesDecodeIntoTarget, andArrayBytesOptionaltypes - Add support for the
optionaldata type and codec (OptionalCodec,OptionalDataType,Element[Owned]forOption<T>, examples) - Add chunk compaction API (
Array::[async_]compact_chunk(),ArrayToBytesCodecTraits::compact()) - Add
zarrs::array::codec::default_array_to_bytes_codec() - Add
ArrayBuilder::subchunk_shape()andInvalidSubchunkShapevariant toArrayCreateError - Add
Tensortype for tensor library interop (implementsdlpack::traits::TensorLike) andTensorError/ArrayError::TensorError - Add
Config::{codec_options,codec_metadata_options,array_metadata_options,group_metadata_options}()methods - Add
Defaultimplementation forMetadataConvertVersionandMetadataEraseVersion - Add runtime extension registration for codecs, data types, chunk grids, chunk key encodings, and storage transformers
- Add
ArrayError::ArraySubsetErrorvariant andCodecOptions::[{set_,with_}]chunk_concurrent_minimum() - Add
register_data_type_extension_codec!macro andDataTypeExttrait - Add
From<ExpectedFixedLengthBytesError>,From<ExpectedVariableLengthBytesError>, andFrom<ExpectedOptionalBytesError>forArrayError - Add
ArrayError::ElementErrorvariant
- Breaking:
ElementandElementOwnedtrait methods now returnElementErrorinstead ofArrayError - Breaking: Replace
ArrayError::IncompatibleElementTypeandArrayError::InvalidElementValuewithArrayError::ElementError - Breaking: Bump MSRV to 1.91 and use Rust 2024 edition
- Breaking: Bump public dependencies:
zarrs_metadata0.7.2,zarrs_data_type0.9.0,zarrs_metadata_ext0.4.1,zarrs_plugin0.4.1,zarrs_filesystem0.3.9,zarrs_storage0.4.2,float80.5.0,dlpark0.6.0,ndarray0.17.1 - Bump internal dependencies:
zfp-sys0.4.2,pco0.4.9,criterion(dev) 0.8.1 - Breaking: Replace
DataTypeenum withArc<dyn DataTypeExtension> - Breaking: Revise extension alias handling for codecs, chunk grids, and chunk key encodings
- Extensions now implement
ExtensionAliases<V>trait for per-extension alias management - Remove
ExtensionAliasesCodecV3parameter fromCodec::from_metadata(),CodecChain::from_metadata(),default_array_to_bytes_codec(), and codec constructor methods
- Extensions now implement
- Breaking: Change
StorageTransformerPluginto the newPluginsystem fromzarrs_plugin - Breaking: Add node to
NodeCreateError::MissingMetadatamessage (#280 by @mannreis) - Breaking: Refactor
ArrayBytesand related types:ArrayBytes::new_fill_value()now takes adata_typeandnum_elementsand is fallibleArrayBytes::Variablevariant now holdsArrayBytesVariableLengthrather than bytes and offsetsArrayBytes::validate()now takes aDataTypeinstead of aDataTypeSizeArrayBytes::into_variable()now returnsArrayBytesVariableLengthinstead of a bytes and offsets tuple- Rename
RawBytestoArrayBytesRaw,RawBytesOffsetstoArrayBytesOffsets
- Breaking: Change various methods to take
ArrayBytesDecodeIntoTargetinstead ofArrayBytesFixedDisjointView([Async]ArrayPartialDecoderTraits::partial_decode_into(),ArrayToBytesCodecTraits::decode_into(),[Async]Array::retrieve_chunk[_subset]_into(),copy_fill_value_into()) - Breaking: Return borrowed references instead of owned collections:
{Array,Chunk}Representation::shape_u64()returns&[u64]},Array::chunk_grid_shape()returns&[u64] - Breaking: Move modules into
zarrs_chunk_gridand re-export;ArraySubsetmoves fromarray_subsettoarraymodule - Breaking: Add
DataTypeparameter toShardingCodecBuilder::new()so that it can choose an appropriate default array-to-bytes codec - Breaking:
Element::into_array_bytes()now takes an ownedVec<T>instead of a slice&[T](addedElement::to_array_bytes()matching the old signature) - Breaking: Refactor array store and retrieve methods to be generic over input and output types
{Array,ArrayShardedReadableExt,ChunkCache}::retrieve_*methods are now generic over the return type (any type implementingFromArrayBytes)Array::store_*methods are now generic over the input type (any type implementingIntoArrayBytes)- Deprecate
Array::retrieve_*_{ndarray,elements}andArray::{store,retrieve}*_{ndarray,elements}in favour of the generic methods - Remove
Array::store_*_dlparkandArray::retrieve_*_dlparkmethods (useTensortype instead) - Added
ChunkCache::retrieve_*_bytesmethods that returnChunkCacheTypeDecodeddirectly
- Breaking: Change various methods to take
ChunkShape,DataType, andFillValueinstead ofChunkRepresentation(CodecPartialDefault::new(), various codec trait methods) - Breaking: Replace
ChunkShapenewtype withVec<NonZeroU64>, useChunkShapeTraitsinstead - Breaking: Refactor low-level codec API to reduce global config retrieval
CodecOptionsandCodecMetadataOptionsnow implementCopywith hardcoded defaults; useConfig::codec_options()/Config::codec_metadata_options()to get options from global config- Remove
CodecOptionsBuilder,CodecOptions::builder(),CodecOptions::into_builder() - Rename
ArrayMetadataOptions::codec_options[_mut]()tocodec_metadata_options[_mut](),CodecTraits::configuration_opt()toconfiguration(),CodecChain::create_metadatas_opt()tocreate_metadatas() - Add
ZarrVersionparameter toCodecTraits::configuration() - Remove
CodecTraits::default_name()
- Breaking: Refactor
DataTypeAPI: replace{DataType,DataTypeExtension}::name()withExtensionNametrait, replaceDataType::metadata()withconfiguration() - Performance improvements: avoid unnecessary copies/allocations in
Arraymethods and codecs, improve index iterator performance Array,ChunkCache, andArrayShardedReadableExtmethods take&dyn ArraySubsetTraitsinstead of&ArraySubset- Breaking:
CodecTraitschanges: requireExtensionNameimplementation, addis_any()method, removenameparameter fromconfiguration - Breaking: Change
Codec::from_metadataparameter to&CodecMetadatainstead of&MetadataV3 - Breaking: Change the representation of the
ArrayError::{InvalidFillValue,InvalidFillValueMetadata}variants - Breaking: Rename
ArrayBytesFixedDisjointViewCreateError::IncompatibleIndexerErrortoIndexerError - Breaking: Do not re-export
array::FillValueMetadata{V2,V3}, useFillValueMetadatainstead - Breaking: Rename
StorageTransformerExtensiontoStorageTransformerTraitsfor alignment with other extension traits- add
createandconfigurationmethods, removemetadatamethod
- add
- Breaking: Rename sharding "inner chunk" terminology to "subchunk" throughout:
ArrayShardedExt:inner_chunk_shape()→subchunk_shape(),effective_inner_chunk_shape()→effective_subchunk_shape(),inner_chunk_grid()→subchunk_grid(),inner_chunk_grid_shape()→subchunk_grid_shape()ArrayShardedReadableExt/AsyncArrayShardedReadableExt:inner_chunk_byte_range()→subchunk_byte_range(),retrieve_encoded_inner_chunk()→retrieve_encoded_subchunk(),retrieve_inner_chunk_opt()→retrieve_subchunk_opt(),retrieve_inner_chunks_opt()→retrieve_subchunks_opt(), and related_elements/_ndarraymethods
- Refactor
Element/ElementOwnedimplementations into separate submodules and useDataTypeTraits::compatible_element_types()for type validation (enables custom data types to declare compatibility with existingElementtypes)
- Breaking: Remove
zarrs_registrydependency - Breaking: Remove
Config::codec_aliases_{v2,v3}()andConfig::data_type_aliases_{v2,v3}()methods - Breaking: Remove
ZfpCodec::new_with_configuration_zfpy()(useZfpyCodecinstead) - Breaking: Remove
ArraySizeand{Array,Chunk}Representation::size()(usenum_elements()andelement_size()instead) - Breaking: Remove DLPack legacy API:
RawBytesDlPack,ArrayDlPackExtError,AsyncArrayDlPackExt,ArrayError::DlPackError - Breaking: Remove
array::ArrayRepresentationBaseandArray::chunk_array_representation()(usechunk_shape(),data_type(),fill_value()) - Breaking: Remove
CodecError::DataTypeExtension,ArrayError::IncompatibleStartEndIndicesError, andIncompatibleStartEndIndicesError - Breaking: Remove
indexerandarray_subsetmodules (types re-exported inarraymodule) - Breaking: Remove
Named*types and methods:Named[{ArrayToArray,ArrayToBytes,BytesToBytes}]Codec,NamedDataType,ShardingCodecBuilder::*_named(),DataType::into_named() - Breaking: Rename
codec_v2_to_v3_name()tocodec_v2_to_v3()and change parameter to metadata from name - Breaking: Remove
ArrayBuilderChunkGridtype - Breaking: Remove
array::ArrayBuilder{ChunkGridMetadata,DataType,FillValue}re-exports - Breaking: Remove unused
ElementFixedLengthmarker trait
- Fix
transposecodec decoding with variable-size data types - Various fixes to aliased data type handling
- Fixes handling of Zarr V2 arrays with bool fill values
- Disallow
zstdZarr V2 metadata in V3 arrays
0.23.0-beta.6 - 2026-01-14
0.23.0-beta.5 - 2026-01-13
0.23.0-beta.4 - 2026-01-09
0.23.0-beta.3 - 2025-12-31
0.23.0-beta.2 - 2025-12-31
0.23.0-beta.1 - 2025-12-29
0.23.0-beta.0 - 2025-12-29
0.22.10 - 2025-11-29
- Fix
bz2codec bounded size - Fix
zlibcodec bounded size
0.22.9 - 2025-11-27
0.22.8 - 2025-11-14
Config: implementClone,Serialize, andDeserializeMetadata{Convert,Erase}Version: ImplementSerializeandDeserialize
- Bump
zarrs_filesystemto 0.3.5 - Bump
zarrs_registryto 0.1.8
0.22.7 - 2025-11-05
- Bump
zarrs_registryto 0.1.7
- Avoid needlessly decoding entire chunks with
ChunkCache::retrieve_array_subseton first read with a partial decoder cache - Upper bound
ndarrayto<0.17instead of<17
0.22.6 - 2025-10-31
- Bump
zarrs_data_typeto 0.4.2 - Bump
zarrs_filesystemto 0.3.4
0.22.5 - 2025-10-26
- Add
Array::set_shape_and_chunk_grid - Add
node::[async_]get_child_nodes_opt - Add
Group::storage() - Add
DataType::is_fixed()andis_variable()
- Bump
zarrs_data_typeto 0.4.1 - Bump
zarrs_metadatato 0.6.2 - Bump
zarrs_metadata_extto 0.2.2
- Respect
versionargument for children inNode::[async_]open_opt
0.22.4 - 2025-10-13
- Log warnings when using experimental or deprecated extensions
- Add
NamedDataType
- Bump
zarrs_metadata_extto 0.2.1 - Document that the
binarydata type alias is deprecated
- Preserve aliased data type names in array metadata
- Interpret string fill values as base64 encoded for the
variable_length_bytesdata type
0.22.3 - 2025-10-11
- Add test with v3
byteswritten byzarr-python3.1 (#282 by @kylebarron)
- Bump
zarrs_registryto 0.1.6
- Replace use of removed
doc_auto_cfgfeature withdoc_cfg(fixesdocs.rsbuild) - Properly support Zarr V3 data type aliases, which were only being applied to data type plugins
0.22.2 - 2025-10-07
- Bump
zarrs_filesystemto 0.3.2- Add direct I/O read support (#249 by @ilan-gold)
- Open and warn zarr stores with non-zarr nodes (#279 by @mannreis)
0.22.1 - 2025-09-19
- Bump
zarrs_data_typeto 0.4.0- 0.3.3 was yanked as it missed a breaking change
0.22.0 - 2025-09-18
This release was yanked.
- This release includes several Major Breaking changes that will certainly require changes to user code.
- Many foundational traits have been refactored for consistency, simplicity, and to support new capabilities.
- Initial generic indexing and WASM support
- More partial encoding support. It remains experimental and lightly documented.
- New experimental extensions (codecs, chunk grids, chunk key encodings).
- Add
index_locationsupport tovlencodec- Add
VlenCodec::with_index_location()
- Add
- Add
numcodecs.adler32codec - Add
ChunkCacheTypePartialDecoder,ChunkCachePartialDecoderLru{Chunk,Size}Limit[ThreadLocal] - Add
Array::storage()andArray::with_storage() - Add
Array<T>::[async_]readable()whereT: [Async]ReadableWritableStorageTraits - Implement
CloneforErrorstructs - Add
{Indices,Chunks,LinerisedIndices,ContiguousIndices,ContiguousLinearisedIndices}IntoIteratorandPar{Indices,Chunks}IntoIterator- Implement
Into[Parallel]IteratorforIndicesandIntoParallelRefIteratorfor&Indices - Implement
Into[Parallel]IteratorforChunksandIntoParallelRefIteratorfor&Chunks - Implement
IntoIteratorfor{Linearised,Contiguous,ContiguousLinearised}Indices
- Implement
- Impl
From<ChunkKeySeparator>forchar - Impl
From<RegularChunkGridCreateError>forIncompatibleDimensionalityError - Add
ChunkGridTraits::[par_]iter_chunk_indices() - Add
ArraySubset::chunk_shape() - Impl
IntoIteratorforChunkShape - Add
RegularBoundedChunkGrid(zarrs.regular_bounded) - Add
DefaultSuffixChunkKeyEncoding(zarrs.default_suffix) - Add initial generic indexing support to partial decoders
- Add
indexermodule withIndexertrait andIncompatibleIndexerError - Implement
IndexerforArraySubset,&ArraySubset,&[ArrayIndices],&[T]whereT: Indexer, and more - Breaking: Partial decoders and encoders use
&dyn Indexerinstead of&ArraySubset
- Add
- Add
[StorageTransformerChain,StorageTransformerExtension]::create[_async]_readable_writable_transformer - Add
CodecPartialDefault - Add partial encoding support for the
bytescodec - Add experimental
reshapecodec - Add
Array::async_partial_encoderand partial encoding support toArray::async_store_chunk_subset_opt
-
Major Breaking: Refactor
ArrayBuilder- All fields are now private
- Add
ArrayBuilder::{new_with_chunk_grid,chunk_grid_metadata,build_metadata,attributes_mut}() - Add
ArrayBuilder{ChunkGrid,DataType,FillValue} - Change
ArrayBuilder::new()to take a broader range of types for each parameter, and swap order ofchunk_grid/data_type. See below
ArrayBuilder::new( // array shape vec![8, 8], // or [8, 8], &[8, 8], etc. - DataType::Float32, - vec![4, 4].try_into()?, // no longer valid - f32::NAN.into(), // no longer valid + // regular chunk shape or chunk grid metadata + vec![4, 4], // or [4, 4], &[4, 4], "{"name":"regular",...}", MetadataV3::new_with_configuration(...) + // data type or data type metadata + DataType::Float32, // or "float32", "{"name":"float32"}", MetadataV3::new("float32"). + // fill value or fill value metadata + f32::NAN, // or "NaN", FillValue, FillValueMetadataV3 ) .build() -
Major Breaking: change the
{Array,Chunk}Representation::new[_unchecked]fill_valueparameter to takeimpl Into<FillValue>instead ofFillValue- ChunkRepresentation::new(chunk_shape(), DataType::Float32, 0.0f32.into())?, + ChunkRepresentation::new(chunk_shape(), DataType::Float32, 0.0f32)?,
-
Major Breaking:
[Async]ArrayPartialDecoderTraitstrait changes:partial_decode[_into](): parameterarray_subsets: &[ArraySubset]changed toindexer: &dyn Indexerpartial_decode[_into](): returnsArrayBytes<'_>instead ofVec<ArrayBytes<'_>>partial_encode(): parametersubsets_and_bytes: &[(&ArraySubset, ArrayBytes<'_>)]changed toindexer: &dyn Indexerandbytes: &ArrayBytes<'_>- Rename
size()tosize_held()and add async version - Add
supports_partial_decode - Add
exists()
-
Breaking:
[Async]BytesPartialDecoderTraitstrait changes:- Rename
partial_decodetopartial_decode_manyand change parameterdecoded_regions: &[ByteRange]toByteRangeIterator - Add
partial_decodefor decoding a single byte range - Remove
partial_decode_concat - Rename
size()tosize_held()and add async version - Add
supports_partial_decode - Add
exists()
- Rename
-
Breaking:
Array::set_shape()now returns aResult- Previously it was possible to resize an array to a shape incompatible with a
rectangularchunk grid
- Previously it was possible to resize an array to a shape incompatible with a
-
Breaking: Refactor
ChunkGridTraitsandChunkGridPlugin, chunk grids are initialised with the array shapeChunkGridTraitsis now anunsafetrait with invariantsChunkGrid::from_metadata()and{Regular,Rectangular}ChunkGrid::new()now have anArrayShapeparameter- Add
ChunkGridTraits::array_shape() - Remove
_uncheckedmethods and default implementations of associated checked methods- Implementations must implement the checked methods instead
-
Breaking:
VlenCodec::new()gains anindex_locationparameter -
Breaking:
ArrayShardedExt::inner_chunk_grid_shape()no longer returns anOption -
Breaking: change
array::codecs()to return anArcd instead of borrowedCodecChain -
Breaking: Add
size()method to{Array,Bytes}PartialDecoderTraits -
Breaking: Refactor the
ChunkCachetrait- The previous API supported misuse (e.g. using a chunk cache with different arrays)
- Breaking: Add
retrieve_chunk_subset()andarray()methods (required) - Add
retrieve_{array_subset,chunks}()methods with_elements()and_ndarray()variants (provided) - Remove
arrayfrom method parameters, theChunkCachemust own anArc<Array>instead. See below
- let cache = ChunkCacheEncodedLruChunkLimit::new(50); - array.retrieve_chunk_opt_cached(&cache, &[0, 1], &CodecOptions::default()), + let cache = ChunkCacheEncodedLruChunkLimit::new(array, 50); + cache.retrieve_chunk(&[0, 1], &CodecOptions::default()),
-
Breaking: Change
byte_range::extract_byte_ranges_read_seekto takeTinstead of&mut T -
Breaking: Auto implement
[Async]BytesPartialDecoderTraitsforT: AsRef<[u8]> + ... -
Breaking:
ArctheChunkCachetypes -
Breaking: Change
Contiguous[Linearised]Indicesiterators to include the number of contiguous indices in theirItem -
Breaking:
ravel_indicesandunravel_indexnow return anOption, out-of-bounds access returnsNone -
Breaking:
CodecErrorenum revisions- Rename
InvalidArraySubsetErrortoIncompatibleIndexer - Remove
InvalidArraySubsetDimensionalityError, included inIncompatibleIndexer
- Rename
-
Breaking: Refactor partial encoding and the
[Async]{Array,Bytes}PartialEncoderTraitstraits:- The partial decoder traits are now a supertrait of the partial encoder traits
- Input/output handle parameters are fused in relevant methods
- input_handle: Arc<dyn ArrayPartialDecoderTraits>, - output_handle: Arc<dyn ArrayPartialEncoderTraits>, + input_output_handle: Arc<dyn ArrayPartialEncoderTraits>,
- Add
[Async]{Array,Bytes}PartialEncoderTraits::into_dyn_decoder - Impl
{Array,Bytes}PartialEncoderTraitforMutex<Option<Vec<u8>>> - Add
supports_partial_encode [Async]BytesPartialEncoderTraitsspecific trait changes:- Rename
partial_encodetopartial_encode_manyand change parameteroffsets_and_bytes: &[(ByteOffset, RawBytes<'_>)]tooffset_values: OffsetBytesIterator<crate::array::RawBytes<'_>> - Add
partial_encodefor decoding a single byte range
- Rename
-
Breaking: Remove
ArraySubset::byte_ranges- Replaced by
Indexer::iter_contiguous_byte_ranges - Add
CodecTraits::partial_decoder_capabilityandPartialDecoderCapabilitypartial_decoder_capabilityreplacespartial_decoder_should_cache_inputandpartial_decoder_decodes_all
- Add
CodecTraits::partial_encoder_capabilityandPartialEncoderCapability
- Replaced by
-
Breaking: remove
experimental_prefix fromcodec_store_metadata_if_encode_onlyand switch default totrue -
Breaking: Make
StoragePartialEncodergeneric over the underlying storage and add async support -
Optimised chunk key encoders
-
Conditional use of
Send/Sync/async_trait(?Send)based ontarget_archfor WASM compatibility (#245 by @keller-mark) -
Use WASM compatible
rayon_iter_concurrent_limitinternally -
Bump
zarrs_metadata_extto 0.2.0 -
Bump
zarrs_storageto 0.4.0 -
Bump
blosc-srcto 0.3.6 -
Bump
criterion(dev) to 0.7.0 -
Bump
float8to 0.4.1 -
Bump
lruto 0.16 -
Bump minimum
ndarrayto 0.15.4 -
Move
zarrs_opendalto a new repository:zarrs/zarrs_opendal
- Breaking: Remove
ArrayChunkCacheExt. Use theChunkCachemethods instead - Breaking: Remove
Par{Chunks,Indices}IteratorProducer, which were unneeded - Breaking: Remove
[Async]BytesPartialDecoderTraitsimplementations forstd::io::Cursorvariants - Breaking: Remove
ArraySubset::chunks()andarray_subset::iterators::Chunks - Breaking: Remove
storage::byte_rangere-export - Breaking: Remove
array_subset::IncompatibleArraySubsetAndShapeError, replaced byindexer::IncompatibleIndexerError - Breaking: Remove
[Maybe]AsyncBytes, which are identical to[Maybe]Bytes - Breaking: Remove
[Async]{ArrayToArray,ArrayToBytes,BytesToBytes}Partial{Encoder,Decoder}Default- Replaced by generic
CodecPartialDefault
- Replaced by generic
- Permit data types with empty configurations that do not require one
- Erase chunks before writing the updated chunk in
ArrayTo{Array,Bytes}PartialEncoderDefault - Fix
squeezeandtransposecodec partial encoding - Fix
bitroundpartial decoder needlessly rounding on decode
0.22.0-beta.3 - 2025-09-17
0.22.0-beta.2 - 2025-09-13
0.22.0-beta.1 - 2025-09-07
0.22.0-beta.0 - 2025-09-06
0.21.2 - 2025-06-19
- Add complex variants of
float4_e2m1fn,float6_{e2m3fn,e3m2fn},float8_{e3m4,e4m3,e4m3b11fnuz,e4m3fnuz,e5m2,e5m2fnuz,e8m0fnu}data types - Add
float8feature enabling the use offloat8::F8E4M3andfloat8::F8E5M2inArray::*_elementsmethods
- Cleanup
zarrsroot docs and README - Bump
bzip2to 0.6.0 - Bump
criterion(dev) to 0.6.0 - Bump
zip(dev) to 4.0.0 - Bump
zarrs_registryto 0.1.4 - Bump
zarrs_data_typeto 0.3.2 - Bump
zarrs_zipto 0.2.3
0.21.1 - 2025-06-16
- Add
numpy.datetime64andnumpy.timedelta64data type supportchronoandjiffnative element types are supported
- Add
OthertoArrayErrorenum - Bump
zarrs_registryto 0.1.3 - Bump
zarrs_metadata_extto 0.1.1
0.21.0 - 2025-06-08
- Implement
Elementfor&[u8; N]forr*data type - Add
complex_{bfloat16,float16,float32,float64}data types - Add
float4_e2m1fn,float6_{e2m3fn,e3m2fn},float8_{e3m4,e4m3,e4m3b11fnuz,e4m3fnuz,e5m2,e5m2fnuz,e8m0fnu}data types- These have no associated
Element/ElementOwnedtype and cannot be used withArray::*_elements()methods - Only hex string fill values are supported
- These have no associated
- Add
[u]int{2,4}data types
- Sign extend to the nearest byte when decoding in the
packbitscodec
- Breaking behaviour: The
bytesdata type now uses thevlen-bytescodec rather thanvlenby default- This was intended for 0.20.0 but incorrectly implemented (the changelog has been amended)
- Breaking: Resolve bugs in
Group::child_*methods and remove no-oprecursiveboolean (#200 by @jder) - Fix missing support for
bitround[u]int8partial decoding - Fix missing support for
pcodec{int,uint,float}16partial decoding - Fixed
packbitscodec with a non-zerofirst_bit
0.20.1 - 2025-06-01
- Improve
zstdcodec performance
0.20.0 - 2025-05-17
- Added support for ZEP0009 — Zarr Extension Naming
- Added support for ZEP0010 — Generic Extensions (Draft as at 2025/05/17)
- array/group
extensionsmetadata, broadermust_understandsupport
- array/group
- Added support for data type extensions
- Add various new codecs
- Add codecs:
numcodecs.zlib,numcodecs.shuffle,numcodecs.fixedscaleoffset,packbits,squeeze - Add support for data type extensions
- The data type extension API is defined in the
zarrs_data_typecrate - Breaking:
DataType::metadata_fill_value()is now fallible - Breaking:
DataType::from_metadata()now returns aPluginCreateErroron error instead ofUnsupportedDataTypeError - Breaking:
DataType::from_metadata()has an additionalExtensionAliasesDataTypeV3parameter - Breaking:
DataType::[fixed_]size()are no longerconst - Breaking: Remove
TryFrom<DataTypeMetadataV3>forDataType - Breaking: Remove
DataType::identifier() - Breaking: move the
zarrs::array::{data_type,fill_value}modules into thezarrs_data_typecrate - Breaking: Rename
IncompatibleFillValueErrortoDataTypeFillValueError - Breaking: Rename
IncompatibleFillValueMetadataErrortoDataTypeFillValueMetadataError - Add
Extensionvariant toDataType - Add
custom_data_type_{fixed_size,variable_size,uint4,uint12,float8_e3m4}examples
- The data type extension API is defined in the
- Add
zarrs_registrycrate that tracks extensionname/idaliases- Add
config::{{codec,data_type}_aliases_{v3,v2}}[_mut]
- Add
- Add
array:codec::{InvalidBytesLengthError,InvalidArrayShapeError,InvalidNumberOfElementsError,SubsetOutOfBoundsError} - Add
ArraySubset::inbounds_shape()(matches the oldArraySubset::inboundsbehaviour) - Add
ArrayBytesFixedDisjointView[CreateError] - Add
[Async]ArrayDlPackExttraits that add methods toArrayforDLPacktensor interop (needsdlpackfeature) - Add missing
Group::async_child_*methods - Add
[Async]{Array,Bytes}PartialDecoderDefault - Add
Config::{codec,data_type}_aliases_{v2,v3}[_mut] - Add
Async{Array,Bytes}PartialEncoderTraitsand*CodecTraits::async_partial_encoder() - Add
array_subset::ArraySubsetError#156 by @ilan-gold - Add
array_subset::IncompatibleOffsetError - Add
AsyncArrayShardedReadableExtandAsyncArrayShardedReadableExtCache - Add
ArrayBytesFixedDisjointViewCreateError::IncompatibleArraySubsetAndShapeError#156 by @ilan-gold - Add
CodecError::IncompatibleDimensionalityErrorvariant #156 by @ilan-gold - Add
CodecError::{DataTypeExtension,IncompatibleFillValueError,InvalidArrayShape,InvalidNumberOfElements,SubsetOutOfBounds,RawBytesOffsetsCreate,RawBytesOffsetsOutOfBounds,InvalidIndexerError}variants - Add
ArrayError::{ArrayBytesFixedDisjointViewCreateError,IncompatibleStartEndIndicesError,IncompatibleOffset,DlPackError}variants - Add
CodecMetadataOptionsandArrayMetadataOptions::codec_options[_mut] - Implement
From<T: IntoIterator<Item = Range<u64>>>forArraySubset
- Breaking Behaviour: Use the
vlen-utf8codec by default forstringdata typeszarrspreviously usedvlen, an experimental codec not supported by other implementations
- Breaking Behaviour: Refactor
codecname handling andCodecTraitsin alignment with ZEP0009 and thezarr-extensionsrepository- All "experimental" codecs now use the
zarrs.prefix (ornumcodecs.if fully compatible) - Add support for aliased codec names
- Enables pass-through of codecs from Zarr V2 to V3 without converting to a V3 equivalent (if supported)
- All "experimental" codecs now use the
- Breaking: Split the
zarrs_metadatacrate intozarrs_metadata(core) andzarrs_metadata_ext(extensions)zarrs_metadata_extis re-exported aszarrs::metadata_ext
- Breaking: Split the
pluginmodule to thezarrs_plugincratezarrs_pluginis re-exported aszarrs::plugin- Breaking:
Pluginis now generic over the creation arguments - Breaking:
StorageTransformerPluginnow uses aPlugin
- Breaking: change
ArraySubset::inboundsto take another subset rather than a shape - Breaking: Change
CodecError::UnexpectedChunkDecodedSizeto anInvalidBytesLengthError - Breaking: Make the following safe and change output args to
ArrayBytesFixedDisjointView:Array::[async_]retrieve_chunk[_subset]_into[Async]ArrayPartialDecoderTraits::partial_decode_intoArrayToBytesCodecTraits::decode_intozarrs::array::copy_fill_value_intozarrs::array::update_array_bytes
- Breaking: change
RawBytesOffsetsinto a validated newtype - Breaking:
ArrayBytes::new_vlen()not returns aResultand validates bytes/offsets compatibility - Breaking:
ArrayErrorandCodecErorrare now marked as non-exhaustive - Breaking: Add
CodecTraits::{identifier,default_name,configuration[_opt]}() - Breaking: Change the error type of
node::[async_]get_child_nodes()andGroup::{children,child_*}()toNodeCreateErrorinstead ofStorageError - Breaking: Refactor
ArrayToArrayCodecTraits:- Rename
compute_encoded_size()toencoded_representation()and add a default implementation - Rename
compute_decoded_shape()todecoded_shape() - Add
encoded_shape()andencoded_fill_value()
- Rename
- Breaking: Rename
{ArrayToArray,ArrayToBytes,BytesToBytes}CodecTraits::compute_encoded_size()toencoded_representation() - Breaking: Rename
{ArrayToArray,ArrayToBytes,BytesToBytes}CodecTraits::dynamic()tointo_dyn() - Breaking: Rename
[Async]ArrayPartial{Encoder,Decoder}Defaultto[Async]ArrayToBytesPartial{Encoder,Decoder}Default - Breaking: Rename
[Async]BytesPartial{Encoder,Decoder}Defaultto[Async]BytesToBytesPartial{Encoder,Decoder}Default - Breaking:
ArraySubset::bounderror type changed toArraySubsetError#156 by @ilan-gold - Breaking:
ArraySubset::relative_toerror type changed toArraySubsetError - Breaking:
Group::consolidated_metadatanow returns an ownedConsolidatedMetadatainstead of a reference - Breaking: Move
zarrs_metadata::v3::UnsupportedAdditionalFieldErrortozarrs::array::AdditionalFieldUnsupportedError - Breaking: Move
ArrayMetadataOptions::*experimental_codec_store_metadata_if_encode_onlyintoCodecMetadataOptions ArrayCreateError::DataTypeCreateErrornow uses aPluginCreateErrorinternally- Add default implementations for
{ArrayToArray,ArrayToBytes,BytesToBytes}CodecTraits::[async_]partial_{encoder,decoder} - Breaking: Bump
zarrs_metadatato 0.5.0 - Bump
thiserrorto 2.0.2 - Bump
lruto 0.14.0 - Bump
halfto 2.3.1 - Bump
derive_moreto 2.0.0
- Breaking: Remove
ArraySubsetunchecked methods #156 by @ilan-gold - Breaking: Remove
{Array,Group}::additional_fields[_mut] - Breaking: Remove
CodecTraits::create_metadata[_opt]() - Breaking: Remove
Config::experimental_codec_names[_mut] - Breaking: Remove
CodecError::InvalidArraySubsetError
- Fixed reserving one more element than necessary when retrieving
stringorbytesarray elements - Check offset is valid in
ArraySubset::relative_to - Reenable broken compatibility tests since fixed in
zarr-python/numcodecs - Reject arrays and groups with unsupported
"must_understand": trueextensions - Allow
{Array,Group}::new_with_metadata()and{Array,Group}Builderto create arrays with"must_understand": trueadditional fields{Array,Group}::[async_]open[_opt]continue to fail with additional fields with"must_understand": true
0.20.0-beta.2 - 2025-05-16
0.20.0-beta.1 - 2025-05-12
0.20.0-beta.0 - 2025-05-03
0.19.2 - 2025-02-13
- Bump
zarrs_metadatato 0.3.4 which includes a number of Zarr metadata fixes- See the
zarrs_metadataCHANGELOG.md
- See the
0.19.1 - 2025-01-19
- Document that elements in
ArrayBytesmust be in C-contiguous order
- Use new language/library features added between Rust 1.78-1.82 (internal)
- Cleanup root docs and README removing ZEPs table and ecosystem table
- New clippy lints
- Mark
StringandBytesdata types as experimental in their docs - Mark
rectangularchunk grid as experimental since it is based on a draft ZEP - Add missing invariant to
[partial_]decode_intosafety docs
0.19.0 - 2025-01-10
zarr-python3.0.0 has been released today!- The focus of this release has been in maximising compatibility with unstandardised extensions in
zarr-python(e.g. experimental codecs, consolidated metadata, etc.)
- Add
ArrayShardedReadableExt::retrieve_encoded_inner_chunk - Add
ArrayShardedReadableExt::inner_chunk_byte_range - Add
ArrayShardedExt::is_exclusively_sharded - Add
ArrayShardedReadableExtCache::array_is_exclusively_sharded - Add
Vlen{Array,Bytes,Utf8}Codec, replacingVlenV2Codec - Add
ZstdCodecConfigurationNumCodecs- Adds support for Zarr V2
zstdencoded data created withnumcodecs< 0.13
- Adds support for Zarr V2
- Add support for pcodec
Auto,None, andTryLookbackdelta specs - Add
Group::[set_]consolidated_metadata - Add
Node::consolidate_metadata- Consolidated metadata is not currently used to optimise node hierarchy requests
- Add experimental
fletcher32checksum codec based on the numcodecs implementation- Adds
fletcher32feature flag
- Adds
- Add ecosystem compatibility notes in the experimental codecs docs
- Breaking: Seal
Arrayextension traits:ArraySharded[Readable]ExtandArrayChunkCacheExt - Breaking: Make
{Array,Bytes}PartialDecoderCacheprivate - Breaking: Make
Anya supertrait of partial encoder/decoder traits - Breaking: Add
ArrayError::UnsupportedMethod - Breaking: Rename
DataType::BinarytoBytesfor compatibility withzarr-python - Breaking: Make
array::codec::array_to_bytes::bytes::reverse_endiannessprivate - Breaking: Make
VlenV2Codecprivate - Bump
itertoolsto 0.14 - Indicate that
zfp/zfpycodecs have different metadata in codec table - Use
zarr-python3.0.0 in compatibility tests - Breaking: Bump MSRV to 1.82 (17 October, 2024)
- Remove support for pcodec
Try{FloatMult,FloatQuant,IntMult}mode specs- These may be reimplemented when supported by
zarr-python/numcodecs
- These may be reimplemented when supported by
- Cleanup unnecessary lifetime constraints in partial decoders
- Fix
clippy::useless_conversionlint
0.18.3 - 2024-12-30
- impl
From<Node>forNodePath(#112 by @niklasmueboe) - Add
Group::child[_{group,array}]_paths(#112 by @niklasmueboe)
0.18.2 - 2024-12-25
- functions to get children of Group (#104 by @niklasmueboe)
- adds
Group::[async_]children,Group::[async_]child_groups,Group::[async_]child_arrays
- adds
- Impl
From<Node>forNodeMetadata
- Reduce metadata code duplication in the
Nodemodule
0.18.1 - 2024-12-17
- Bump
zfp-systo 0.3.0 - Bump
bzip2to 0.5.0 - Minor readme/ecosystem updates
- Fix
unsafe_op_in_unsafe_fnlint - Clarify that the
zstdcodec is draft in docs - Clarify that the
gdeflatecodec is experimental in docs
0.18.0 - 2024-11-23
zarrs-pythonwas recently released- It implements a high-performance codec pipeline for the reference
zarr-pythonimplementation - It is supported by downstream libraries like
dask - See zarr_benchmarks for benchmarks
- It implements a high-performance codec pipeline for the reference
- The
zarrsBook has been created and is under construction
- Experimental partial encoding support (enabling writing shards incrementally)
- Improve Zarr V2 interoperability and conversion
- Add a
makefileand simplifyBUILD.md - Add chunk-by-chunk update example in
Arraydocs - Add
array::copy_fill_value_into() - Add experimental partial encoding support (sync only):
- Enables writing shards incrementally
- With
{CodecOptions,Config}::experimental_partial_encodingenabled,Array::store_{array,chunk}_subsetwill use partial encoding - This is an experimental feature for now until it has more comprehensively tested and support is added in the async API
- Adds
ArrayPartialEncoderTraits,BytesPartialEncoderTraits,StoragePartialEncoder,ArrayPartialEncoderDefault,BytesPartialEncoderDefault - Breaking: Add
{ArrayToArray,ArrayToBytes,BytesToBytes}CodecTraits::partial_encoder
- Add
with_methods to{Array,Group}MetadataOptions - Add
zarr_v2_to_v3example - Add
{Array,Group}::to_v3() - Add
ShardingCodecBuilder::build_arc() - Add
zarrs::version::version_{str,pre} - Add "The zarrs Book" and
zarrs-pythonto docs
- Bump
unsafe_cell_sliceto 0.2.0 - Breaking: Change
outputparameter ofdecode_intocodec trait methods to&UnsafeCellSlice<u8> - Breaking: Add
dynamic()to allCodecTraits - Breaking: Add
optionsparameter to[Async]ArrayPartialDecoderTraits::partial_decodeand removepartial_decode_opt - Make
array::update_array_bytes()public - Breaking: Bump MSRV to 1.77 (21 March, 2024)
- Bump
zfp-systo 0.2.0 - Display
ArraySubsetas a list of ranges - Relax
output_subsetrequirements onArrayToBytesCodecTraits::decode_intoandArrayPartialDecoderTraits::partial_decode_into- The subset shape and dimensionality no longer has to match, only the number of elements
- Bump
pco(pcodec) to 0.4 - Breaking: Change
experimental_codec_namesconfig hashmap toHashMap<String, String>fromHashMap<&'static str, String> - Breaking: Add
nameparameter tovlen_v2codec constructors - Register
vlen-array,vlen-bytes, andvlen-utf8codecs - Bump
zarrs_metadatato 0.2.0 - Bump
zarrs_storageto 0.3.0 - Bump
zarrs_filesystemto 0.2.0 - Make
zarrs::version::version_{,major,minor,patch}const
- Remove
async-recursiondependency - Breaking: Remove
Defaultimplementation forVlenV2Codec
- Fix panics that could occur with with empty byte ranges / empty array subsets in
Array,ByteRangeand codec methods
0.18.0-beta.0 - 2024-11-15
0.17.1 - 2024-10-18
- Add
zarrs_icechunkto ecosystem docs
- Fix
data_keyencoding on windows (it contained '//') - Fix
clippy::needless_lifetimeslint
0.17.0 - 2024-10-02
zarrshas been split into 3 core crates:zarrs,zarrs_metadata, andzarrs_storagezarrs_storageandzarrs_metadataare re-exported as thestorageandmetadatamodules
- Store implementations have been moved into separate crates:
zarrs_{filesystem,http,object_store,opendal,zip}zarrs_filesystemis re-exported as thefilesystemmodule with thefilesystemfeature (enabled by default)
- Direct IO support for Linux in
FilesystemStore - Implicit groups are no longer supported
- It is the responsibility of the
zarrsconsumer to explicitly write group metadata when creating a hierarchy
- It is the responsibility of the
- Codecs must now be
Arc'd instead ofBox'd - Fixes a performance regression introduced in 0.16 when reading sharded arrays
- Check the full release notes for all changes. This release has many breaking changes due to items being removed, moved, renamed, and deprecated.
- Add
ChunkGridTraits::chunks_in_array_subset() - Add chunk cache support
- Add
ArrayChunkCacheExtextension trait forArray - Add traits:
ChunkCache,ChunkCacheType(implemented byChunkCacheType{Encoded,Decoded}) - Add chunk cache implementations:
ChunkCache{En,De}codedLru{Size,Chunk}Limit[ThreadLocal]
- Add
- #58 Add direct I/O support in
FilesystemStoreby @sk1p- Adds
FilesystemStoreOptionsandFilesystemStore::new_with_options
- Adds
- #64 Add
Array::[async_]store_encoded_chunkfor writing already-encoded chunks by @sk1p - Add
key(),start(),value()toStoreKeyStartValue - Add
StorageError::MissingMetadatanow that implicit groups are not supported - Add
ChunkShape::to_array_shape() - Add
DataTypeMetadataV3 - Add
ArrayShardedExt::effective_inner_chunk_shape- This is the effective inner chunk shape (i.e. read granularity) of a sharded array
- It is equal to the inner chunk shape unless the
transposecodec precedessharding_indexed
- Breaking: Add
ArrayToArrayCodecTraits::compute_decoded_shape()- Needed for
ArrayShardedExt::effective_inner_chunk_shape
- Needed for
- Add
ArrayToBytesCodecTraits::decode_intoand[Async]ArrayPartialDecoderTraits::partial_decode_into- This revisits the array view API that was removed in 0.16 #39, but simpler and less public
- Resolves a performance regression introduced in 0.16 when decoding sharded arrays with
Array::[async_]retrieve_array_subset_opt.
- Add
Array::subset_all() - Add
ArraySubset::to_ranges()
- Breaking:
Arcinstead ofBoxpartial decoders - Expand
set_partial_valuestests - Specialise
set_partial_valuesforMemoryStore - Bump maximum supported
ndarrayversion from 0.15 to 0.16 - Breaking: Make
create_chunk_grid_{regular,rectangular}pub(crate)in alignment with other internal create from metadata methods - Breaking: Bump MSRV to 1.76 (8 February, 2024)
- #59 Add
ReadableWritableStorageTraitsautomatically for all implementations by @dustinlagoy - Remove implicit group support
- This is a post-acceptance change of Zarr V3: zarr-developers/zarr-specs#292
- #63 Make
StoreKeysPrefixesconstructible by @sk1p - Breaking: Use values of
Metadata{Convert,Erase}Versioninstead of references in parameters/return values - Restructure
metadatamodule- Move conversion to
metadata::v2_to_v3 - Breaking: Remove many re-exports to the root of
metadata - Move various items to the
metadatamodule, keeping re-exports in thearraymodule{Array,Chunk}ShapeEndiannessZARR_NAN_{F16, BF16, F32, F64}ChunkKeySeparatorDimensionName
- Split
DataTypeintoarray::DataTypeandmetadata::_::DataTypeMetadataV3
- Move conversion to
- Breaking:
data_keyinzarrs::storagenow take achunk_keyinstead of achunk_key_encodingandchunk_indices - Breaking: Move
metadata::{v2,v3}::{codec,chunk_grid,chunk_key_encoding}tometadata::{v2,v3}::array:: - Breaking: Rename
ArrayMetadataV2DataTypetoDataTypeMetadataV2 - Breaking: Rename
FillValueMetadatatoFillValueMetadataV3 - Move
crate::byte_rangeintocrate::storage::byte_rangemodule, add re-export - Add
impl TryInto<StorePrefix> for &NodePath- Removes
TryFrom<&NodePath> for StorePrefix
- Removes
- Breaking: Move
extract_byte_ranges_read[_seek]fromarray::codectostorage::byte_range - Breaking: Move
storage::[async_]{get_child_nodes,node_exists,node_exists,listable}tonodemodule - Breaking: Move
storage::{meta_key*,data_key}intonodemodule - Split the crate into multiple crates:
zarrszarrs_storage(re-exported asstoragemodule)zarrs_metadata(re-exported asmetadatamodule)zarrs_filesystem(re-exported asfilesystemmodule withfilesystemfeature, enabled by default)zarrs_httpzarrs_object_storezarrs_opendalzarrs_zip
- Breaking: Codecs no longer need to implement
Clonebut must be inArcinstead ofBox - Breaking: Change
Contiguous[Linearised]Indicesiterators to return indices only - Breaking: Remove lifetime constraints in partial decoder API by utilising
Arc'd codecs - Breaking:
ShardingCodec::newCodecChainparameters now must be in anArc - Breaking: Change
UsageLogStorageTransformertoUsageLogStorageAdapterand move tozarrs_storage - Breaking: Change
PerformanceMetricsStorageTransformertoPerformanceMetricsStorageAdapterand move tozarrs_storage - Breaking: Storage transformer refactor:
- Add
StorageTransformerPluginfor storage transformer registration instead of genericPlugin - Remove several
StorageTransformerExtensiontrait methods no longer needed - Change
StorageTransformerExtension::create_metadatato returnMetadataV3instead of anOption StorageTransformerExtension::[async_]create_*_transformermethods are now fallibleStorageTransformerExtension::async_create_*_transformermethods are now asyncStorageTransformerChain::from_metadataandtry_create_storage_transformernow has apath: &NodePathparameter
- Add
- Breaking: Remove
array::NonZeroError, usestd::num::TryFromIntErrorinstead - Breaking: Move storage transformers from the
storageto thearraymodule - Breaking: Remove many functions in the storage module:
[async_]create_{array, group}[async_]erase_{chunk,metadata},[async_]{retrieve,store}_chunk[async_]retrieve_partial_values[async_]discover_nodes
- Breaking: Remove
Defaultimplementation forMetadata{Convert,Erase}Version- Explicitly use
global_config()instead
- Explicitly use
- Breaking: Remove
array::UnsafeCellSlice- Replaced by
UnsafeCellSlicein theunsafe_cell_slicecrate
- Replaced by
- Breaking: Remove
NATIVE_ENDIAN, useEndianness::native() - Breaking: Remove unused
DataTypeExtension - Remove remnants of old synchronisation API
[async_]store_set_partial_valuesno longer truncates- this could corrupt values depending on the order of
set_partial_valuescalls
- this could corrupt values depending on the order of
- Fix
FilesystemStore::fspath_to_keyon windows - Make
ArrayRepresentationBasepub so that{Array,Chunk}Representationare not opaque - Fix
ArrayShardedExt::inner_chunk_gridwhen applied on a sharded array with thetransposecodec precedingsharding_indexed - Fix
ZipStorageAdapteron windows - Fixed an unnecessary copy in
Array::[async_]retrieve_chunk_if_exists_opt - Fixed
CodecOptionsnot being forwarded inArray::retrieve_chunk_subset_opton the fast path - Fixed missing fast path in
Array::[async_]retrieve_chunk_subset_opt - Fixed
blosccodec partial decoding withnoshuffle
0.17.0-beta.3 - 2024-09-26
0.17.0-beta.2 - 2024-09-23
0.17.0-beta.1 - 2024-09-16
0.17.0-beta.0 - 2024-09-06
0.16.4 - 2024-08-22
- Reduce maximum supported
opendalversion from 0.49 to 0.48- This reverts a change in 0.16.3 that was not semver compatible
0.16.3 - 2024-08-14
This release was yanked.
- Bump
derive_moreto 1.0.0 - Bump maximum supported
opendalversion from 0.48 to 0.49 - Box the metadata in
PluginMetadataInvalidError
- Fix
cargo testwith--no-default-features - Fix new clippy warnings in nightly
0.16.2 - 2024-08-05
- Add the experimental
gdeflatebytes-to-bytes codec - Add
Array::chunk_key() - Add
ArrayShardedExt::inner_chunk_grid_shape()
0.16.1 - 2024-07-30
- Bump maximum supported
opendalversion from 0.47 to 0.48
- Fixed handling of empty shards with a variable length data type with sharding partial decoder
0.16.0 - 2024-07-28
- Add experimental support for the
stringandbinarydata types and thevlenandvlen_v2codecs - Cleanup the
ArrayAPI for retrieving elements (Vec<T>) andndarrays - Support more Zarr V2 array configurations and make various experimental codecs
numcodecscompatible
- Add
ArrayBytes,RawBytes,RawBytesOffsets, andArrayBytesError- These can represent array data with fixed and variable length data types
- Add
array::Element[Owned]traits representing array elements- Supports conversion to and from
ArrayBytes
- Supports conversion to and from
- Add
array::ElementFixedLengthmarker trait - Add experimental
vlenandvlen_v2codec for variable length data typesvlen_v2is for legacy support of Zarr V2vlen-utf8/vlen-bytes/vlen-arraycodecs
- Add
DataType::{String,Binary}data types- These are likely to become standardised in the future and are not feature gated
- Add
ArraySubset::contains() - Add
FillValueMetadata::{String,Unsupported}ArrayMetadatacan be serialised and deserialised with an unsupportedfill_value, butArraycreation will fail.
- Implement
From<{[u8; N],&[u8; N],String,&str}>forFillValue - Add
ArraySizeandDataTypeSize - Add
DataType::fixed_size()that returnsOption<usize>. ReturnsNonefor variable length data types. - Add
ArrayError::IncompatibleElementType(replacesArrayError::IncompatibleElementSize) - Add
ArrayError::InvalidElementValue - Add
ChunkShape::num_elements_u64 - Add global
Configoption for manipulating experimental codec names - Add
metadata::v2::codec::ZfpyCodecConfigurationNumcodecsand associated structures
- Use
[async_]retrieve_array_subset_optinternally inArray::[async_]retrieve_chunks_opt - Breaking: Replace
[Async]ArrayPartialDecoderTraits::element_size()withdata_type() - Array
_storemethods now useimpl Into<ArrayBytes<'a>>instead of&[u8]for the input bytes - Breaking: Array
_store_{elements,ndarray}methods now useT: Elementinstead ofT: bytemuck::Pod - Breaking: Array
_retrieve_{elements,ndarray}methods now useT: ElementOwnedinstead ofT: bytemuck::Pod - Breaking: Simplify array store
_ndarraymethods to 2 generic type parameters - Optimised
Array::[async_]store_array_subset_optwhen the subset is a subset of a single chunk - Make
transmute_to_bytespublic - Relax
ndarray_into_vecfromT: bytemuck:PodtoT: Clone - Breaking:
DataType::size()now returns aDataTypeSizeinstead ofusize - Breaking:
ArrayCodecTraits::{encode/decode}have been specialised intoArrayTo{Array,Bytes}CodecTraits::{encode/decode} - Various changes to the experimental
zfpcodec- Breaking: Remove
Zfp{Expert,FixedAccuracy,FixedPrecision,FixedRate}Configurationand just embed these structures directly inZfpMode - Breaking:
ZfpCodec::new_expertnow takesminbits,maxbits,maxprec, andminexpinstead ofZfpExpertConfiguration - Breaking: All
ZfpCodec::new_*methods now take awrite_header: boolparameter
- Breaking: Remove
- Breaking: Add
ArrayMetadataV2ToV3ConversionError::Other - Make all v2 metadata available even without experimental codec features
- Breaking: Change pcodec
max_page_nconfiguration toequal_pages_up_toto match numcodecs - Improve the
Arraydocs
- Breaking: Remove
into_array_viewarray and codec API- This was not fully utilised, not applicable to variable sized data types, and quite unsafe for a public API
- Breaking: Remove internal
ChunksPerShardErrorand just useCodecError::Other - Breaking: Remove
array_subset::{ArrayExtractBytesError,ArrayStoreBytesError} - Breaking: Remove
ArraySubset::{extract,store}_bytes[_unchecked], they are replaced by methods inArrayBytes - Breaking: Remove
array::validate_element_sizeandArrayError::IncompatibleElementSize- The internal validation in array
_elementmethods is now more strict than just matching the element size - Example:
u16must matchuint16data type and will not matchint16orfloat16
- The internal validation in array
- Fix an unnecessary copy in
async_store_set_partial_values - Fix error when
bytesmetadata is encoded without a configuration, even if empty - Fix an error in
ChunkGriddocs - Fixed
[async_]store_set_partial_valuesandMemoryStore::setto correctly truncate the bytes of store value if they shrink
0.15.1 - 2024-07-11
- Add
CITATION.cff
- Implement
From<&String>forDimensionName - Cleanup macro usage in array
- Fix unnecessary allocations in
_elementsvariants of array store methods
0.15.0 - 2024-07-07
- Zarr V2 support (a Zarr V3 compatible subset)
- Codec and array optimisations
- Array store methods previously taking
Vec<u8>now take&[u8] - Codec methods previously taking
Vec<u8>now takeCow<'_, [u8]>
- Array store methods previously taking
AsyncToSyncStorageAdapter: use an async store (e.g. HTTP, S3, etc.) in a sync context- Snappy codec support for the
blosccodec
- Add support for a V3 compatible subset of Zarr V2
- Compatible subset: Zarr V2 data that is Zarr V3 compatible with only a metadata change
- Zarr V2 metadata (
.zarray/.zgroup/.zattrs) can be transformed to V3 (zarr.json)
- Add
ArrayBuilder::build_arcmethod - Add
Array::[async_]retrieve_encoded_chunk[s]method - Add
Group::metadata_optmethod - Add
{Array,Group}::{store,erase}_metadata_optmethods - Add
metadata::Metadata{Retrieve,Convert,Erase}Versionenums - Add
Config::[set_]metadata_{convert,erase}_versionmethods - Add
{Array,Group}MetadataOptions::[set_]metadata_convert_versionmethods - Add
{Config,ArrayMetadataOptions}::[set_]include_zarrs_metadatamethods - Add
Array::set_dimension_names - Add
storage::[Maybe]AsyncBytes - Add
array::{convert_from_bytes_slice,convert_to_bytes_vec} - Add
AdditionalField - Add
AsyncToSyncStorageAdapterandAsyncToSyncBlockOn - Add internal
fill_array_view_with_fill_valuefunction
- Breaking: Deprecate
{Array,Group,Node}::[async_]newfor[async_]open, and addopen_opt - Breaking:
Arraystore methods now take slices instead ofVecs - Breaking: Change various store methods to take
&Arc<TStorage>instead of&TStorage - Breaking: Sync and async stores now consume and return
bytes::Bytesinstead ofVec<u8> - Breaking:
{Array,Group}::metadata()now return references instead of values - Breaking:
AdditionalFieldsis now an alias forBTreeMap<String, AdditionalField>instead of an opaque struct - Breaking: Move
[Async]ReadableStorageTraits::{size[_prefix]}to[Async]ListableStorageTraitsand add default implementation forsize - Use
monostateforzarr_format,node_type, andmust_understandin unknown fields in array and group metadata- These fields must be be valid on deserialisation rather than array/group initialisation
- Breaking: Remove associated field validation functions
- Support
object_store0.9-0.10 - Breaking: Support
opendal0.46-0.47, drop support for 0.45 - Bump
rayonto 1.10.0 - Bump
itertoolsto 0.13 - Bump
reqwestto 0.12 - Bump
zipto 2.1 - Bump
blosc-srcto 0.3.4- Adds
snappycodec support
- Adds
- Bump minimum supported
flate2to 1.0.30 andthiserrorto 1.0.61 - Add default implementations for
[Async]ReadableStorageTraits::{get,get_partial_values} - Use
futures::TryStreamExt::try_for_each_concurrentinstead ofFuturesUnorderedwhere appropriate - Move all metadata/configuration structures into the metadata module (non breaking with re-exports)
- Rename
MetadatatoMetadataV3, an alias is retained - Improve various docs
- Breaking: Add
MissingMetadatatoGroupCreateErrorenum - Change internal structure of various iterators to use
std::ops::Rangeand remove redundantlength - Breaking:
Indices::new_with_start_endnow takes arangerather than astartandend RecommendedConcurrency::newtakesimpl std::ops::RangeBounds<usize>instead ofstd::ops::Range- Breaking: Move
array::MaybeBytestostorage::MaybeBytes - Breaking: Move
storage::storage_adapter::ZipStorageAdapter[CreateError]tostorage::storage_adapter::zip:: - The
{async,sync}_http_array_readexamples now demonstrate usage ofopendalandobject_storestorage backends - Bump
pcodecto 0.3- Adds support for
uint16,int16andfloat16data types to the experimentalpcodeccodec - Breaking: The schema for
PcodecCodecConfigurationV1has changed
- Adds support for
- Exclude integration tests and data from published package
- Breaking: Remove re-exports of public dependencies
- Breaking: Remove
Array::set_include_zarrs_metadata. Use{Config,ArrayMetadataOptions}::set_include_zarrs_metadata - Breaking: Remove
ArrayMetadataV2ToV3ConversionError::InvalidZarrFormat - Breaking: Remove
{Array,Group}CreateError::{InvalidZarrFormat,InvalidNodeType}
- Breaking: Change
ZfpExpertParamstoZfpExpertConfigurationreplacing existingZfpExpertConfiguration- The previous
ZfpExpertConfigurationwas incorrect and was identical toZfpFixedRateConfiguration
- The previous
- Fix
FilesystemStore::list_prefixwith an empty prefix - Fix
storage::[async_]discover_nodesand add tests
0.14.0 - 2024-05-16
- Breaking: Remove
store_locksmodule,[Async]ReadableWritableStorageTraits::mutex(), andnew_with_locksconstructors from storesDefaultStoreLockscould result in a deadlock- It is now the responsibility of zarrs consumers to ensure that:
Array::store_chunk_subsetis not called concurrently on the same chunkArray::store_array_subsetis not called concurrently on regions sharing chunks
- Chunk locking may be revisited in a future release
0.13.3 - 2024-05-16
This release was yanked and changes reverted.
0.13.2 - 2024-05-08
- Make the
bz2andpcodeccodecs public - The
"name"of experimental codecs in array metadata now points to a URI to avoid potential future incompatibilities with other implementations - Improve docs of several experimental codecs
0.13.1 - 2024-05-06
- Added the
array_sharded_ext::{ArrayShardedExt,ArrayShardedReadableExt}extension traits forArrayto simplify working with sharded arrays- Abstracts the chunk grid to an "inner chunk grid" to simplify inner chunk retrieval.
- Shard indexes are cached in a
ArrayShardedReadableExtCache - Retrieval and chunk grid methods have fallbacks for unsharded arrays. For example, an inner chunk in an unsharded array is just a chunk
- Sync API only,
AsyncArrayShardedReadableExtandAsyncArrayShardedReadableExtCacheare planned for a future release
- Added
ChunkGridTraits::chunks_subset()with default implementation
- Allow float fill values to be created from int fill value metadata
- Make
chunk_grid::{regular,rectangular}public - Support 8 and 16 bit integer data types with zfp codec by promoting to 32 bit
- Fix
compute_encoded_size()forBitroundCodecincorrectly indicating various data types were unsupported - Fix a link in chunk grid docs
- Fix incorrect minimum dependency versions and add CI check
- Fix clippy
unexpected_cfgswarning with recent nightly
0.13.0 - 2024-04-20
- Add "experimental codec store metadata if encode only" option to global config
- Add "store empty chunks" option to global config and
CodecOptions - Add
ArrayMetadataOptions - Add small example to
README.mdand crate root documentation - Add
uint8/int8support to the bitround codec - Add
len()andis_empty()methods to array subset iterator producers - Add
Array::chunk_origin()
- Breaking: Bump MSRV to 1.75 (28 December, 2023)
- Bump
pco(pcodec) to 0.2.1 - Breaking: Add
CodecTraits::create_metadata_opt() - Breaking: Rename
data_type::IncompatibleFillValueErrorMetadataErrortoIncompatibleFillValueMetadataError - Add a useful warning if the shard index references out-of-bounds bytes in a chunk encoded with the sharding codec
- Breaking: Require
ndarray::Arrayparameters in various async array methods to implementSend
- Fix implementation of
from_chunkgrid_regular_configurationmacro leading to recursion - Fix arithmetic overflow possible with the bitround codec with integer data types
- Address
todo!()s inretrieve_chunks_into_array_view_optmethods
0.12.5 - 2024-03-17
- Implement
TryFrom<&str>for{Array,Group}Metadata
- Make
array::codec::array_to_bytes::bytes::{Endianness::is_native(),NATIVE_ENDIAN,reverse_endianness}public
0.12.4 - 2024-03-09
- Remove unnecessary copy in
OpendalStore::set - Fixed
zfpcodec encode not truncating compressed data zfpcodeccompute_encoded_size()now correctly outputs a bounded size instead of an unbounded size
0.12.3 - 2024-03-07
- Implement
DeserializeforDataType- A convenience for
zarrsconsumers.ArrayMetadatacontinues to useMetadatato parse unknown data types.
- A convenience for
- Add
{Array,Group}::{async_}erase_metadata()andstorage::{async_}erase_metadata()
- Fixed various errors in storage docs
- Blosc codec config allow deserialization with missing
typesize/shuffle - Blosc codec encoding with
typesizeof 0 with shuffling
0.12.2 - 2024-02-26
- Added "Getting Started" in root documentation
- Disabled
blosccodec partial decoding pending a faster implementation
- Remove an unnecessary allocation in
IndicesIterator
0.12.1 - 2024-02-24
- Add
byte_range::extract_byte_ranges_concat() - Add
{Async}BytesPartialDecoderTraits::partial_decode_concat()with default implementation - Add
ArrayCodecTraits::partial_decode_granularity()with default implementation
- Cleanup "Implementation Status" and "Crate Features" in root documentation
- Minor changes to docs
0.12.0 - 2024-02-22
- This release targeted:
- Improving performance and reducing memory usage
- Increasing test coverage (82% from 66% in v0.11.6)
- There are a number of breaking changes, the most impactful user facing changes are
Arraypar_variants have been removed and the default variants are now parallel instead of serialArray_optvariants use newCodecOptionsinstead ofparallel: boolArraySubsetiter_methods have had theiter_prefix removed. Returned iterators now implementinto_iter()and some also implementinto_par_iter()Array::{set_}parallel_codecsandArrayBuilder::parallel_codecshave been removed. New methods supportingCodecOptionsoffer far more concurrency controlDimensionNames can now be created less verbosely. E.g.array_builder.dimension_names(["y", "x"].into()). Type inference will fail on old syntax like.dimension_names(vec!["y".into(), "x".into()].into())Arraystore_ndarrayvariants now take any type implementingInto<ndarray::Array<T, D>>instead of&ndarray::ArrayViewD<T>.
- Add
array::concurrency::RecommendedConcurrency - Add
array::ArrayView - Add array
into_array_viewmethodsArray::{async_}retrieve_chunk{_subset}_into_array_view{_opt}Array::{async_}retrieve_chunks_into_array_view{_opt}Array::{async_}retrieve_array_subset_into_array_view{_opt}
- Add
Array::{async_}retrieve_chunk_if_exists{_elements,_ndarray}_{opt} - Add
_optvariants to various array store/retrieve methods - Add
Array::dimensionality() - Add
Array::chunk_shape_usize()
- Add
codec::CodecOptions{Builder} - Add
ArrayCodecTraits::decode_into_array_viewwith default implementation - Add
{Async}ArrayPartialDecoderTraits::partial_decode_into_array_view{_opt}with default implementation - Add
TestUnboundedcodec for internal testing
- Add
Contiguous{Linearised}IndicesIterator::contiguous_elements{_usize}() - Implement
DoubleEndedIteratorfor{Indices,LinearisedIndices,ContiguousIndices,ContiguousLinearisedIndicesIterator}Iterator - Add
ParIndicesIteratorandParChunksIterator
- Add
Default Codec Concurrent TargetandDefault Chunk Concurrency Minimumglobal configuration options toConfig - Add
{Async}ReadableWritableListableStorageTraitsand{Async}ReadableWritableListableStorage - Add
{Chunk,Array}Representation::shape_u64 - Implement
AsyncBytesPartialDecoderTraitsforstd::io::Cursor<{&[u8],Vec<u8>}> - Implement
From<ChunkShape>forVec<NonZeroU64> - Add
ChunkShape::num_elements() - Implement
From<String>forDimensionName - Add
array::unsafe_cell_slice::UnsafeCellSlice::len() - Add
{Array,Chunk}Representation::dimensionality() - Add
ArraySubset::new_empty()andArraySubset::is_empty() - Add missing
IncompatibleArraySubsetAndShapeError::new() - Add
--usage-logargument to examples to useUsageLogstorage transformer - Add more tests for
Array, codecs, store locks, and more - Add
array_write_read_ndarrayexample - Add
array::bytes_to_ndarray()and makearray::elements_to_ndarray()public - #13 Add
node::Node::path(),metadata(), andchildren()by @lorenzocerrone - #13 Derive
Clonefornode::Nodeandnode::NodeMetadataby @lorenzocerrone - Add
bytemuckfeature to thehalfdependency (public)
- Breaking:
Array_optmethods now use acodec::CodecOptionsparameter instead ofparallel: bool - Behaviour change: default variants without
_optare no longer serial but parallel by default - Breaking:
Arraystore_ndarrayvariants now take any type implementingInto<ndarray::Array<T, D>>instead of&ndarray::ArrayViewD<T>- This is to reflect that these methods consume the underlying
Vecin the ndarray - It also removes the constraint that arrays have a dynamic dimension
- This is to reflect that these methods consume the underlying
- Breaking: remove
par_variants and many_optvariants in favor of a single method with acodec::CodecOptionsparameterpartial_decodeandpartial_decode_optremain- Behaviour change:
partial_decodeis no longer serial but parallel by default
- Breaking: add
{ArrayCodecTraits,BytesToBytesCodecTraits}::recommended_concurrency() - Breaking: add
ArrayPartialDecoderTraits::element_size()
- Breaking:
ArraySubset::iter_methods no longer have aniter_prefix and return structures implementingIntoIteratorincludingIndices,LinearisedIndices,ContiguousIndices,ContiguousLinearisedIndices,ChunksIndicesandChunksalso implementIntoParallelIter
- Array subset iterators are moved into public
array_subset::iteratorsand no longer in thearray_subsetnamespace
- Breaking: Storage transformers must be
Arcwrapped asStorageTransformerExtensiontrait methods now takeself: Arc<Self> - Breaking:
GroupandArraymethods generic on storage now require the storage have a'staticlifetime - Removed lifetimes from
{Async}{Readable,Writable,ReadableWritable,Listable,ReadableListable}Storage
- Breaking:
ArrayBuilder::dimension_names()generalised to acceptOption<I>whereI: IntoIterator<Item = D>andD: Into<DimensionName>- Can now write
builder.dimension_names(["y", "x"].into())instead ofbuilder.dimension_names(vec!["y".into(), "x".into()].into())
- Can now write
- Breaking: Remove
Array::{set_}parallel_codecsandArrayBuilder::parallel_codecs - Breaking: Add
ChunkGridTraits::chunk_shape_u64{_unchecked}toChunkGridTraits - Breaking: Add
create{_async}_readable_writable_listable_transformertoStorageTransformerExtensiontrait - Breaking: Rename
IncompatibleArrayShapeErrortoIncompatibleArraySubsetAndShapeError - Breaking: Use
IncompatibleArraySubsetAndShapeErrorinArrayStoreBytesError::InvalidArrayShape - Breaking: Add
ArrayError::InvalidDataShape - Add a fast path to
Array::retrieve_chunk_subset{_opt}if the entire chunk is requested DimensionName::new()generalised to accept a name implementingInto<String>- Cleanup uninitialised
Vechandling - Dependency bumps
- Make
UnsafeCellSlicepublic
- Breaking: Remove
InvalidArraySubsetErrorandArrayExtractElementsError - Breaking: Remove non-default store lock constructors
- Breaking: Remove unused
storage::store::{Readable,Writable,ReadableWritable,Listable}Store
- Breaking:
ArraySubset::end_incnow returns anOption, which isNonefor an empty array subset Array::retrieve_array_subsetand variants now correctly return the fill value if the array subset references out-of-bounds elements- Add missing input validation to some
partial_decodemethods - Validate
ndarrayarray shape in{async_}store_{chunk,chunks}_ndarray{_opt} - Fixed transpose partial decoder and its test, elements were not being correctly transposed
- Minor docs fixes
0.11.6 - 2024-02-06
- Add a global configuration
config::Configaccessible viaconfig::{get_config,get_config_mut}- Currently it exposes a single configuration option:
validate_checksums(default:true)
- Currently it exposes a single configuration option:
- Document correctness issues with past versions and how to correct errant arrays in crate root
0.11.5 - 2024-02-05
- Major bug Fixed the
crc32ccodec so it usesCRC32Crather thanCRC32- All arrays written prior to this release that use the
crc32ccodec are not correct
- All arrays written prior to this release that use the
- Fixed the
crc32ccodec reserving more memory than necessary
0.11.4 - 2024-02-05
- Add
codecovsupport to CI
- Fixed a regression introduced in v0.11.2 (89fc63f) where codecs with an empty configuration would serialise as a string rather than a struct with a
namefield, which goes against the zarr spec- Fixes the
codec_bytes_configuration_nonetest and addscodec_crc32c_configuration_nonetest
- Fixes the
0.11.3 - 2024-01-31
- Added support for miri testing and accompanying notes in
BUILD.md
- Make
IDENTIFIERpublic for codecs, chunk key encodings, and chunk grids
- Fix formatting of
pcodecfeature inlib.rsdocs - Remove
println!inPcodecCodec - Fixed
FillValue::equals_allwith unaligned inputs
0.11.2 - 2024-01-30
- Added experimental
bz2(bzip2) codec behindbz2feature - Added experimental
pcodeccodec behindpcodecfeature
- docs: clarify that
bitroundandzfpcodec configurations are draft
- Do not serialise
configurationinMetadataif is empty - Do not serialise
endianinBytesCodecConfigurationV1if it is none
0.11.1 - 2024-01-29
- Fixed build with
bitroundorzfpfeatures withoutasyncfeature
0.11.0 - 2024-01-26
- This release targeted
- Improving documentation
- Increasing coverage and correctness (line coverage increased from 70.66% to 78.46% since
0.10.0) - Consolidating and improving errors and their messages
- Major breaking changes
Arrayretrieve_methods now returnVec<u8>/Vec<T>instead ofBox<[u8]>/Box<[T]>- Added
ChunkShape(which wrapsVec<NonZeroU64>) and addedChunkRepresentation- Chunks can no longer have any zero dimensions
- Creating an array now requires specifying a chunk shape like
vec![1, 2, 3].try_into()?instead ofvec![1, 2, 3].into()
- Tests for
ByteRange,BytesRepresentation,StorePrefix,StoreKey,ArrayBuilder,ArraySubset,GroupBuilder,Group,NodeName,NodePath,Node,AdditionalFields,Metadata,FillValue,Group,Metadata array_subset::IncompatibleStartEndIndicesError- Add
array::transmute_from_bytes_vec - Re-export public dependencies at the crate root:
bytes,bytemuck,dyn_clone,serde_json,ndarray,object_store, andopendal - Implement
DisplayforByteRange,StoreKeyRange,NodeName - Add
HexString::new - Add
PluginMetadataInvalidError
- Breaking:
Arrayretrieve_methods now returnVec<u8>/Vec<T>instead ofBox<[u8]>/Box<[T]>- This avoids potential internal reallocations
- Breaking:
StoreKey::parentnow returnsStorePrefixinstead ofOption<StorePrefix> - Breaking:
ZipStorageAdapter::{new,new_with_path}now take aStoreKey - Breaking:
ArraySubset::new_with_start_end_{inc,exc}now returnIncompatibleStartEndIndicesErrorinstead ofIncompatibleDimensionalityError- It is now an error if any element of
endis less thanstart
- It is now an error if any element of
- Remove
#[must_use]fromGroupBuilder::{attributes,additional_fields} - Breaking: Rename
Node::new_with_storetoNode::new, andNode::newtoNode::new_with_metadatafor consistency withArray/Group - Use
serde_jsonfloat_roundtripfeature - Breaking: Use
bytemuckinstead ofsafe_transmute- Array methods now have
<T: bytemuck::Pod + ..>instead of<T: safe_transmute::TriviallyTransmutable + ..>
- Array methods now have
- Breaking: Rename
array::safe_transmute_to_bytes_vectoarray::transmute_to_bytes_vec - Breaking: Make
zfpa private dependency by changingZfp{Bitstream,Field,Stream}frompubtopub(super) - Breaking: Make
zipa private dependency by not exposingZipErrorinZipStorageAdapterCreateError - Refine
UsageLogStorageTransformeroutputs and add docs - Improve
PerformanceMetricsStorageTransformerdocs - Breaking:
InvalidByteRangeErrornow holds aByteRangeand bytes length and returns a more informative error message - Breaking: Remove
StorageError::InvalidJSONand addStorageError::InvalidMetadataInvalidMetadataadditionally holds aStoreKeyfor more informative error messages
- More informative
Metadatadeserialisation error message with an invalid configuration - Breaking:
PluginCreateError::Otherchanged to unit struct and addedPluginCreateError::from<{String,&str}> PluginCreateError::Unsupportednow includes aplugin_typefield for more informative error messages- Add
array::ChunkShapewrappingVec<NonZeroU64>andarray::ChunkRepresentationwhich is essentiallyArrayRepresentationwith aNonZeroU64shape- Breaking: Relevant codec and partial decoder methods now use
ChunkRepresentationinstead ofArrayRepresentation - Breaking: Relevant chunk grid methods now use
ChunkShapeinstead ofArrayShape - Breaking: Relevant array methods now use
ChunkShapeinstead ofArrayShape
- Breaking: Relevant codec and partial decoder methods now use
- Breaking: Remove
StorePrefixError::new, deprecated sincev0.7.3 - Breaking: Remove
ArraySubset::{in_subset,in_subset_unchecked}, deprecated sincev0.7.2 - Breaking: Remove
impl From<&StorePrefix> for NodeName, unused and not useful - Breaking: Remove
NodeCreateError::MetadataNodeCreateError::StorageErrorwithStorageError::InvalidMetadatais used instead
- Breaking: Remove
{ArrayCreateError,GroupCreateError}::MetadataDeserializationError{ArrayCreateError,GroupCreateError}::StorageErrorwithStorageError::InvalidMetadatais used instead
- Breaking: Remove
GroupCreateError::Metadataas it was unused - Breaking: Remove
PluginCreateError::ConfigurationInvalidError
- Disallow an empty string for a
StoreKey ArrayBuildernow validates additional fieldsFillValue::equals_allincorrect behaviour with aFillValuewith size not equal to 1, 2, 4, 8, or 16 bytes.- Fix
NodePathdisplay output - Fix handling of non-standard
NaNvalues forf16andbf16 - Fix potential missed error in
Metadata::to_configuration
0.10.0 - 2024-01-17
- Bump
opendalto 0.44 - Bump
object_storeto 0.9 - Breaking
async_store_chunkandAsyncWritableStorageTraits::setnow takebytes::Bytesbytes::Bytesare used by both supported async stores (object_storeandopendal), and this avoids a copy
- Various clippy warnings
0.9.0 - 2024-01-03
- New
Arraymethods to store/retrieve/erase multiple chunks - Many
Arrayinternal revisions and removal of some unnecessary methods
- Reexport
safe_transmute::TriviallyTransmutableasarray::TriviallyTransmutable - Add
Array::chunks_subset{_bounded} - Add
store_chunks,retrieve_chunks,erase_chunksand variants toArray
- Use macros to reduce common code patterns in
Array - Separate
Arraymethods into separate files for each storage trait - Breaking: Remove
_optandpar_variants ofasync_retrieve_array_subsetandasync_store_array_subset(including_elementsand_ndarrayvariants) - Revise
array_write_readandasync_array_write_readexamples - Breaking: Storage
erase/erase_values/erase_prefixmethods andArray::erase_chunknow return()instead ofbooland succeed irrespective of the whether the key/prefix exists
0.8.0 - 2023-12-26
- Feature changes:
- Added:
object_storeandopendalwith generalised support for stores from these crates - Removed:
s3,gcp, andazure(useobject_storeoropendalinstead) - Changed:
httpandzipare no longer default features
- Added:
ReadableStorageTraitsis no longer a supertrait ofWritableStorageTraits- Moved chunk locking from
Arrayinto stores - Improved documentation and code coverage
- Added
ReadableWritableStorageandReadableWritableStoreand async variants - Added
{async_}store_set_partial_values - Breaking Added
create_readable_writable_transformertoStorageTransformerExtensiontrait - Added
storage::store_lockmodule- Adds generic
StoreLocks,StoreKeyMutex, andStoreKeyMutexGuardwith associated traits and async variants - Includes
DefaultStoreLocksandDisabledStoreLocksimplementations
- Adds generic
- Readable and writable stores include a
new_with_locksmethod to choose the store lock implementation - Added
ArraySubset::new_with_ranges - Added
ByteRange::offset - Added
object_storefeature withAsyncObjectStorestore (wrapsobject_store::ObjectStore)- Breaking Removes the explicit
object_store-based stores (e.g.AsyncAmazonS3Store,AsyncHTTPStore) - Breaking Removes the
object_store_implmacro - Breaking Removes the
s3,gcp, andazurecrate features
- Breaking Removes the explicit
- Added
opendalfeatureOpendalStorestore (wrapsopendal::BlockingOperator)AsyncOpendalStorestore (wrapsopendal::Operator)
- Breaking
ReadableStorageTraitsis no longer a supertrait ofWritableStorageTraitsWritableStorageis no longer implicitly readable. UseReadableWritableStorage
- Breaking:
{Async}WritableStorageTraits::set_partial_values()no longer include default implementations- Use new
{async_}store_set_partial_valuesutility functions instead
- Use new
- Add
#[must_use]toArray::builder,Array::chunk_grid_shape, andArrayBuilder::from_array - Breaking Remove
httpandzipfrom default features - Locking functionality for arrays is moved into stores
- Improved
Arraydocumentation - Add store testing utility functions for unified store testing
- Make various
Arraymethods withparallelparameterpub - Remove
#[doc(hidden)]from various functions which areunsafeand primarily intended for internal use - Breaking Bump minimum supported rust version (MSRV) to
1.71(13 July, 2023)
- Fixed
MemoryStore::get_partial_values_keyif given an invalid byte range, now returnsInvalidByteRangeErrorinstead of panicking
0.7.3 - 2023-12-22
- Add
From<ChunkKeyEncodingTraits>forChunkKeyEncoding - Add chunk key encoding tests
- Revise code coverage section in
BUILD.mdto usecargo-llvm-cov - Increased code coverage in some modules
- Add
--all-featuresto clippy usage inBUILD.mdandci.yml
- Fixed chunk key encoding for 0 dimensional arrays with
defaultandv2encoding - Fixed various clippy warnings
0.7.2 - 2023-12-17
ArraySubset::{extract_elements/extract_elements_unchecked}andArrayExtractElementsError
- Add
ArraySubset::{overlap,overlap_unchecked}andArraySubset::{relative_to,relative_to_unchecked}- These replace
ArraySubset::{in_subset,in_subset_unchecked}, which are now deprecated
- These replace
- Add
From<String>forStorePrefixErrorand deprecateStorePrefixError::new
- Fix
cargo testwithasynccrate feature disabled
0.7.1 - 2023-12-11
- Fix use of
impl_trait_projectionsin{Array/Bytes}PartialDecoderCache, which was only stabilised in Rust 1.74
0.7.0 - 2023-12-05
- Experimental
asyncfeature: Support async stores and array/group operations- See
async_array_write_readandasync_http_array_readexamples
- See
- Experimental
s3/gcp/azurefeatures for experimental async Amazon S3/Google Cloud Storage/Microsoft Azure Blob Storage stores
- Add
storage_asyncmodule with asynchronous storage traits:Async{Readable,Writable,Listable,ReadableListable}StorageTraits - Implemented for
StorageHandle - Add async filesystem/http/memory stores in
storage::store::asyncmodule and a wrapper for any store provided by theobject_storecrate - Add async support to zarr
GroupandArray - Add
Async{Readable,Writable,Listable,ReadableListable}Storage - Add
StorageTransformerExtension::create_async_{readable,writable,listable,readable_listable}_transformer - Add
Node::async_new_with_storeand implementstorage::async_get_child_nodes - Add
async_array_write_readandasync_http_array_readexamples - Add experimental
asyncfeature (disabled by default) - Add experimental async
Amazon S3,Google Cloud,Microsoft Azurestores (untested)
- Bump itertools to
0.12and zstd to0.13 - Set minimum supported rust version (MSRV) to
1.70(1 June, 2023)- Required by
halfsince2.3.1(26 June, 2023)
- Required by
- Make
StoreKeyRangeandStoreKeyStartValueclonable - Breaking: Remove
ReadableWritableStorageTraits,ReadableWritableStorage,ReadableWritableStore, andStorageTransformerExtension::create_readable_writable_transformer- These were redundant because
WritableStorageTraitsrequiresReadableStorageTraitssince 6e69a4d
- These were redundant because
- Move sync stores to
storage::store::sync - Move sync storage traits to
storage_sync.rs - Move array sync storage trait impls into
array_sync.rs - Use
required-featuresfor examples
0.6.0 - 2023-11-16
- Revisions for recent updates to the Zarr V3 specification (e.g. sharding
index_locationand removal of"order": "C"/"F"from transpose codec) - API changes to improve usability
- Performance improvements and a few bug fixes
- Experimental
zfpandbitroundcodecs
- Breaking: Added
UnsupportedDataTypeError - Breaking: Added
CodecError::UnsupportedDataType - Added
array_subset::IncompatibleArrayShapeError - Added
array_subset::iter_linearised_indices_unchecked - Added parallel encoding/decoding to tests
- Added
array_subset::ArrayStoreBytesError,store_bytes, andstore_bytes_unchecked - Added experimental
zfpcodec implementation behindzfpfeature flag (disabled by default) - Added experimental
bitroundcodec implementation behindbitroundfeature flag (disabled by default)- This is similar to numcodecs BitRound, but it supports rounding integers from the most significant set bit
- Added
ShardingCodecBuilder - Added
ReadableListableStorage,ReadableListableStorageTraits,StorageTransformerExtension::create_readable_listable_transformer - Added
ByteRange::to_range()andto_range_usize() - Added
StoreKeyStartValue::end() - Added default implementation for
WritableStorageTraits::set_partial_valuesWritableStorageTraitsnow requiresReadableStorageTraits
- Added
From<&[u8]>forFillValue - Added
FillValue::all_equaland fill value benchmark- Implements a much faster fill value test
- Added
Array::chunk_grid_shape/chunk_subset - Added
par_variants for thestore_array_subset/retrieve_array_subsetvariants inArray, which can encode/decode multiple chunks in parallel - Added
ArraySubset::boundandArray::chunk_subset_bounded - Added methods to
CodecChainto retrieve underlying codecs - Added
Array::builder()andArrayBuilder::from_array() - Added more
ArrayBuildermodifiers and made internals public - Added a fast path to
Array::retrieve_array_subsetmethods if the array subset matches a chunk - Added
array::{ZARR_NAN_F64,ZARR_NAN_F32,ZARR_NAN_F16,ZARR_NAN_BF16}which match the zarr nan bit representation on all implementations - Added
size_usize()toArrayRepresentation - Add generic implementations of storage supertraits (
ReadableWritableStorageTraitsandReadableListableStorageTraits) - Add
size_prefix()to stores
- Breaking:
array::data_type::DataTypeis now marked#[non_exhaustive] - Breaking: Promote the
r*(raw bits),float16andbfloat16data types to standard data types inarray::data_type::DataType, rather than extension data types- Breaking: Remove the crate features:
raw_bits,float16,bfloat16 - Breaking: Removes
array::data_type::RawBitsDataType/Bfloat16DataType/Float16DataType - Breaking:
halfis now a required dependency
- Breaking: Remove the crate features:
- Breaking: Rename
TransposeCodec::newtonew_with_configuration - Breaking: Array subset methods dependent on an
array_shapenow use theIncompatibleArrayShapeErrorerror type instead ofIncompatibleDimensionalityError - Breaking: Various array subset iterators now have validated
newand unvalidatednew_uncheckedconstructors - Blosc codec: disable parallel encoding/decoding for small inputs/outputs
- Bytes codec: optimise implementation
- Breaking:
ArrayToArrayCodecTraits::compute_encoded_sizeandArrayToBytesCodecTraits::compute_encoded_sizecan now return aCodecError ArrayBuilder::build()andGroupBuilder::buildnow accept unsized storage- Breaking:
StoragePartialDecodernow takes aReadableStorageinput sharded_array_write_readexample now prints storage operations and demonstrates retrieving inner chunks directly from a partial decoder- the zarrs version and a link to the source code is now written to the
_zarrsattribute in array metadata, this can be disabled withset_include_zarrs_metadata(false) - Breaking: Rename
FillValueMetadata::UinttoUIntfor consistency with theDataType::UIntvariants. StorePrefix,StoreKey,NodeNamenewmethods are now: impl Into<String>- Breaking:
ArrayBuilder::dimension_namesnow takes anOptioninput, matching the output ofArray::dimension_names - Breaking:
ArrayError::InvalidChunkGridIndicesErrornow holds array indices directly, not achunk_grid::InvalidArrayIndicesError - Breaking:
Array::chunk_array_representationnow returnsArrayErrorinstead ofInvalidChunkGridIndicesErrorand thearray_shapeparameter is removed - Breaking:
Array::chunks_in_array_subsetnow returnsIncompatibleDimensionalityErrorinstead ofArrayError - Breaking:
ArraySubset::new_with_start_end_inc/new_with_start_end_excnow takeend: ArrayIndicesinstead of `end: &[u64] - Breaking: Move
array_subset::validate_array_subsettoArraySubset::inbounds - Allow out-of-bounds
Array::store_array_subsetandArray::retrieve_array_subset- retrieved out-of-bounds elements are populated with the fill value
- Derive
CloneforStorageTransformerChain - Breaking:
ArrayBuilder::storage_transformersuseStorageTransformerChain - Breaking: change
ArrayError::InvalidFillValuetoInvalidFillValueMetadataand add a newInvalidFillValue - Breaking: The transpose codec order configuration parameter no longer supports the constants "C" or "F" and must instead always be specified as an explicit permutation zarr-developers/zarr-specs #264
- Removes
TransposeOrderImpl TransposeOrderis now validated on creation/deserialisation andTransposeCodec::newno longer returns aResult
- Removes
- Breaking: Change
HexString::as_bytes()toas_be_bytes() - Support
index_locationfor sharding codec - Optimise
unravel_index - Breaking: Array subset iterator changes
- Simplify the implementations
- Remove
nextinputs - Make constructors consistent, remove
innerin constructors - Add
size_hintto all iterators, implementExactSizeIterator/FusedIterator
- Major breaking: Output boxed slices
Box<[..]>from array retrieve methods instead ofVec<..> - Major breaking: Input
Vec<..>instead of&[..]to array store methods- Supports some perf improvements
- Breaking: Change
BytesRepresentationenum fromKnownSize(u64)/VariableSizetoFixedSize(u64)/BoundedSize(u64)/UnboundedSize - Preallocate sharding codec encoded output when the encoded representation has a fixed or bounded size
- Add
par_encodeforzstdcodec - Breaking: Codecs now must implement
encode_opt,decode_opt, andpartial_decoder_opt - Breaking: Partial decoders now must implement
partial_decode_optand thedecoded_representationis now supplied on creation, rather than when callingpartial_decode/partial_decode_par/partial_decode_opt - Sharding partial decoder now decodes inner chunks in full rather than partially decoding them, this is much faster with some codecs (e.g. blosc)
- In future, this will probably become configurable
- Moved
storage/store/{key.rs,prefix.rs}tostorage/{store_key.rs,store_prefix.rs}
- Bytes codec handling of complex and raw bits data types
- Additional debug assertions to validate input in array subset
_uncheckedfunctions - Breaking:
array_subset::iter_linearised_indicesnow returns aResult<_, IncompatibleArrayShapeError>, previously it could not fail even if thearray_shapedid not enclose the array subset - The
array_subset_iter_contiguous_indices3test was incorrect as the array shape was invalid for the array subset ArraySubset::extract_bytesnow reserves the appropriate amount of memory- Sharding codec performance optimisations
FilesystemStore::erase_prefixnow correctly removes non-empty directories- Breaking:
ArrayBuilder::storage_transformersremove#[must_use] - Validate data type and fill value compatibility in
ArrayBuilder - Handling of
"NaN"fill values, they are now guaranteed to match the byte representation specified in the zarr v3 spec - Add a fast path to array retrieve methods which avoids a copy
- Optimise sharding codec decode by removing initial population by fill value
- Include checksum with
zstdcodec if enabled, previously this did nothing
- Breaking: Disabled data type extensions
array::data_type::DataType::Extension. - Breaking: Remove
StoreExtensiontraits - Breaking: Remove
chunk_grid::InvalidArrayIndicesError/ChunkGridShapeError - Breaking: Remove
ArrayError::InvalidArrayIndicesError
0.5.1 - 2023-10-10
- Tests for
DataType::fill_value_from_metadata - A paragraph on concurrency in the
Arraydocs
- Fix some docs typos
FillValueMetadata::try_as_uint/intcan both now handleFillValueMetadata::Uint/IntArray::store_chunknow erases empty chunks- Fixed a race in
Array::store_chunk_subsetand add a fast path if the subset spans the whole chunk - Fix complex number handling in
DataType::fill_value_from_metadata - Fix byte arrays being interpreted as complex number fill value metadata
0.5.0 - 2023-10-08
MaybeBytesan alias forOption<Vec<u8>>- When a value is read from the store but the key is not found, the returned value is now
Noneinstead of an error indicating a missing key
- When a value is read from the store but the key is not found, the returned value is now
- Add
storage::erase_chunkandArray::erase_chunk- The
array_write_readexample is updated to demonstrateerase_chunk
- The
- Add
TryFrom::<&str>forMetadataandFillValueMetadata - Add
chunk_key_encodingandchunk_key_encoding_default_separatortoArrayBuilder - Add
TryFrom<char>forChunkKeySeparator - Add
ArraySubset::new_with_start_end_inc/new_with_start_end_exc - Add
codec::extract_byte_ranges_readutility function to read byte ranges from aReadsource which does not supportSeek
- Breaking: Remove
StorageError::KeyNotFoundin favour of returningMaybeBytes - Breaking: Add
StorageError::UnknownKeySizeif a method requires a known key size, but the key size cannot be determined - Breaking: Add
ArrayCreateError::MissingMetadataif attempting to create an array in a store without specifying metadata, and the metadata does not exist - Breaking:
UsageLogStorageTransformernow takes a prefix function rather than a string- The
http_array_readexample now logs store requests to stdout usingUsageLogStorageTransformer
- The
- Breaking:
WritableStorageTraits::erase/erase_values/erase_prefixreturn a boolean indicating if they actually deleted something - Breaking: Add
ReadableStorageTraits::get_partial_values_keywhich reads byte ranges for a store key - Breaking: Changed
data_type::try_create_data_typetoDataType::from_metadata - Breaking: Changed
try_create_codectoCodec::from_metadata - Make
ArrayBuilderandGroupBuildernon-consuming - Add a fast-path to
Array::store_array_subsetif the array subset matches a chunk subset - Breaking: Make
ChunkKeyEncodinga newtype.- Breaking: Changed
try_create_chunk_key_encodingtoChunkKeyEncoding::from_metadata.
- Breaking: Changed
- Breaking: Make
ChunkGrida newtype.- Breaking: Changed
try_create_chunk_gridtoChunkGrid::from_metadata.
- Breaking: Changed
- Breaking: Rename
StorageTransformerChain::new_with_metadatastofrom_metadata - Breaking: Rename
CodecChain::new_with_metadatastofrom_metadata - Breaking: Rename
DataTypeExtension::try_create_fill_valuetofill_value_from_metadata - Breaking: Rename
codec::extract_byte_ranges_rstoextract_byte_ranges_read_seek
BytesCodecnow defaults to native endian encoding as opposed to no encoding (only valid for 8 bit data types).storage::get_child_nodesandNode::new_with_storenow correctly propagate storage errors instead of treating all errors as missing metadataGroup::newnow handles an implicit group (with a missingzarr.json)ZipStorehandle missing filesZipStoreno longer reads an internal file multiple times when extracting multiple byte rangesHTTPStoreimprove error handling, check status codes
0.4.2 - 2023-10-06
- Add
From<&str>andFrom<String>forOthervariants ofCodecError,StorageError,ChunkGridShapeError,StorePluginCreateError
- Support parallel encoding, parallel decoding, and partial decoding with the
blosccodec- Previously the blosc codec would read the whole chunk when partial decoding
HTTPStorerange requests are now batched by default
- Fixed
retrieve_chunk_subsetreturning fill values on anyStorageErrorinstead of justStorageError::KeyNotFound
0.4.1 - 2023-10-04
- Add
StorageHandle, a clonable handle to borrowed unsized storage
- Support unsized storage (
TStorage: ?Sizedeverywhere)
0.4.0 - 2023-10-04
- Readable and listable
ZipStorageAdapterbehindzipfeature - Readable
HTTPStorebehindhttpfeature - Add
StorageValueIO, astd::io::Readinterface to a storage value. - Add
zip_array_write_readandhttp_array_readexamples
- Relax some dependency minimum versions
- Add
size_hint()to some array subset iterators - Breaking: Fix
LinearisedIndicesIteratorto use array shape instead of subset shapeLinearisedIndicesIterator::newandArraySubset::iter_linearised_indicesnow requirearray_shapeparameter
- Breaking: Array subset shape no longer needs to be evenly divisible by chunk shape when creating a chunk iterator
- Removes
ChunksIteratorError
- Removes
- Add array subset iterator tests
- Breaking: Remove unvalidated
from(String)for store key/prefix - Validate that store prefixes and keys do not start with
/ - Breaking: Add
StorageError::OtherandStorageError::Unsupportedvariants - Breaking
FilesystemStorenow accepts a file and does not create directory on creation- adds
FilesystemStoreCreateError:InvalidBasePathand removesFilesystemStoreCreateError:InvalidBaseDirectory/ExistingFile
- adds
- Breaking:
ReadableStorageTraits::size()tou64fromusize - Breaking: Add
ReadableStorageTraits::size_key() - Storage and store traits no longer require
Debug - Add a default implementation for
WritableStorageTraits::erase_values - Make array/group explicitly store
Arc<TStorage> StorageTransformerChainnow only acceptsArcstorage.- Breaking: Various group methods are now
#[must_use] - Change
NodePathinternal representation toPathBuf - Remove unneeded '/' prefix strip in
StorePrefix - Breaking: Remove storage trait implementations for
Arc<TStorage>, now must explicitly deref - Implement
EqandPartialEqforDataType - Breaking: Use
u64instead ofusizefor byte ranges/array index/shape etc. This makes it possible to index into larger arrays on 32-bit systems.
- Fix store prefix to node path conversion and vice versa
- Fix
StorePrefix::parent()so it outputs a validStorePrefix - Fix
StoreKey::parent()for top level keys, e.g"a"now has parent prefix"/"instead ofNone - Print root node with
Node::hierarchy_tree
0.3.0 - 2023-09-27
- Add
CHANGELOG.md
- Require the
ndarrayfeature for examples - Remove the
ndarraydev dependency - Remove the
ndarraydependency for theshardingfeature - Replace deprecated
tempdirwithtempfilefor tests - Breaking: Change
ByteRangeenum to haveFromStartandFromEndvariants - Substitute
bloscdependency forblosc-srcwhich builds blosc from source - Breaking: Rename
compressionfield tocnameinBloscCodecConfigurationV1for consistency with the zarr spec
0.2.0 - 2023-09-25
- Initial public release