Skip to content

Instantly share code, notes, and snippets.

@d-v-b
Last active September 29, 2026 20:46
Show Gist options
  • Select an option

  • Save d-v-b/406b5d314cdc458021d8b42c58b13e30 to your computer and use it in GitHub Desktop.

Select an option

Save d-v-b/406b5d314cdc458021d8b42c58b13e30 to your computer and use it in GitHub Desktop.
zarr-python store API: the design gaps behind #4450, and archives versus stores

🤖 AI text below 🤖

zarr-python store API: the design gaps behind #4450, and archives versus stores

Written by an agent (Claude Code) at Davis Bennett's request, 2026-09-29, after seven adversarial review rounds of zarr-developers/zarr-python#4450. The archive/store distinction in the second half is Davis's.

The bugs fixed in #4450 come from five gaps in the Store base class, not from ZipStore alone. The 4.0 plan in zarr-python-planning already closes four of them for stateless stores, and keeps ZipStore as a writable, resource-holding exception; every ZipStore item still in the plan reproduces a bug the PR fixed. Distinguishing key-value archives, written whole, from key-value stores, which mutate individual keys, removes the exception: ZipStore becomes an archive with a pack() step, and the plan's open lifecycle question closes.

Bugs by root cause

The review found 12 bugs in ZipStore on main. The PR fixes ten, narrows one and leaves one open, and each traces to one of five design gaps.

Trigger What happens on main Gap
get, get_partial_values, set_if_not_exists or clear on a store never opened AttributeError: _lock (#3588) Lifecycle
Two threads use a new "w" store at once Both open the file; the second truncates the first's writes (14 of 16 keys kept) Lifecycle
A thread uses the store while another calls close() Reopens before the central directory is written; entries lost Lifecycle
A thread uses the store during move() Reopens the old path; the move fails with store is already open Lifecycle
clear() fails partway Store marked open on a closed handle; every call fails Lifecycle
Any call after close() on a "w" store, even a read Reopens with "w" and erases the archive Creation policy
move() or unpickling a "w" or "x" store "w" truncates; "x" refuses its own file Creation policy
close() fails while writing the central directory Next use starts a new archive; earlier entries silently lost Durability
A write-only file object reused after close() A second archive is appended; earlier entries lost Durability
da.to_zarr into a ZipStore Unpickled copies reopen the file and corrupt it (#3516) Copies
Two stores write one archive Silent corruption; the PR detects only a copy made during the first writing session Copies
Rewriting a key after a reopen, e.g. zarr.json on resize Duplicate zip entries (#3580); not fixed Overwrite

After the first fix commit the PR took 11 more fix commits across nine review passes, and several of those fixes introduced bugs that a later pass caught. That churn is the clearest sign the problem is structural: each fix adds a flag or a lock to one store, and the next edge case falls between them.

Five design weaknesses

Each gap is in the Store contract, so every store pays for it; ZipStore shows it first because a zip file punishes mistakes that a directory tolerates.

Weakness Where it shows What it costs
1. Lifecycle is a boolean the subclasses police Store._is_open is one flag for never opened, open, closed and failed. Store._open raises if already open, and Store._ensure_open checks without a lock. LocalStore._ensure_open carries a comment explaining it bypasses _open for that reason. MemoryStore and FsspecStore copy the open check into six methods. ZipStore added _was_opened and _close_failed on top. Missed checks (AttributeError), races between first uses, stores stuck after a failed step
2. Creation policy is stored as the open mode ZipStore(mode="w") means "truncate when created", but main replayed it on every open; the PR now rewrites _zmode to "a" after the first open. StorePath.open has its own mode vocabulary (r, r+, a, w, w-) where w means "delete keys under this path". Data loss on reopen; one word with two meanings
3. Durability is implicit A zip is readable only after close() writes its central directory. No API says when data is committed, close() is synchronous and does I/O, there is no async close, and zarr never closes a store itself. ZipStore alone lacks with_read_only (#2448). A forgotten close() leaves an unreadable file; a failed one silently loses entries on reuse
4. Copies have no contract Each store picks its own pickling behaviour; main's ZipStore reopened the file on unpickle. Nothing states whether a store may be used from several threads or processes. dask corrupts a ZipStore (#3516); concurrent writers go undetected
5. Overwrite is assumed set overwrites in the API, but a zip is append-only. supports_deletes exists; nothing says whether overwrite is real. zarr rewrites zarr.json on every resize or attribute change. Duplicate entries and growing files (#3580, #828); readers disagree on which copy wins

Weaknesses 1 and 2 are general: LocalStore and MemoryStore show the same structure; their open step is cheap and idempotent, so no bug from it has been seen there yet. Weaknesses 3 to 5 bite stores that buffer or pack data. The next section checks each against the 4.0 plan.

What the 4.0 plan already covers

The store proposals in zarr-python-planning resolve four of the five weaknesses for stateless stores. ZipStore is carved out of each resolution as "the resource-holding exception".

Weakness The plan's decision ZipStore in the plan
1. Lifecycle Stores are stateless; _is_open, _open, _ensure_open and Store.open are removed, and LocalStore moves its existence check to __init__ (stores.md § Lifecycle). Keeps a file handle and a lock with context managers; whether use outside a context manager raises, warns or lazy-opens is an open question with five options (stores-api.md § Open questions, deferred-items).
2. Creation policy Explicit constructors replace mode= at the zarr.open level; LocalStore gains mkdir= (missing-apis.md § 3). The stub still takes mode: "r" | "a" | "w" | "x" (stores-api.md § ZipStore).
3. Durability Per-key atomic Put, rename-into-place for LocalStore, and Transactional[S] on TensorStore's model (stores-transactional.md). Put not atomic, no generation, and the torn-write conformance test is skipped for it.
4. Copies Serializable produces a StoreDeclaration; pickling goes through it, and ThreadSafe/AsyncSafe markers advertise sharing (stores.md § Equality, performance.md § Multiprocessing). "The file handle and lock are recreated on the unpickling side, which lets ZipStore survive distributed scheduling." That is the mechanism behind #3516.
5. Overwrite Not addressed: Put is assumed to overwrite. Advertises Put when opened for writing; flush() and reopen() stubs support "write to a ZipStore, then read from the same object" (missing-apis.md § 5, backends.py). That reopen is the path that truncated archives on main.

The five lifecycle options in the plan all keep a writable, resource-holding ZipStore, and zarr-python-planning#6 on coordinated writes already concludes that safe uncoordinated writes need a storage format with a manifest, not a zip file. The sections below drop the exception instead of choosing among the five.

Prior art: TensorStore

TensorStore has none of the first four weaknesses, by construction rather than by guards, and sidesteps the fifth by refusing to write zip files.

Weakness What TensorStore does Source
1. Lifecycle No open/closed state and no close(). tensorstore.open() returns a Future for a handle that is already open. Shared resources such as caches and concurrency limits live in a Context and are released when the last handle is dropped. open, Context
2. Creation policy open, create and delete_existing are arguments to open(), applied once; the handle does not keep them. A pickled handle is restored with assume_cached_metadata only, so a copy can never replay create or delete_existing. driver.cc
3. Durability Every write() returns a copy future and a commit future. commit resolves when the data is durable: after fsync for local files, or when Google Cloud Storage acknowledges the write. write
4. Copies A handle pickles as its spec plus its read/write mode, and unpickling reopens the store. A handle inside an open transaction refuses to pickle. Concurrent writers are handled by conditional writes: every kvstore write can carry an if_equal storage generation, so a conflicting write fails instead of overwriting. serialization.h, operations.h
5. Overwrite The zip driver is read-only: "Writing is not supported." Packed single-location storage is OCDBT: immutable, append-only data files plus a versioned manifest, where each commit is a new B+tree version that becomes visible when the manifest is updated. zip, OCDBT

The plan has already taken rows 3 and 4: Transactional[S] is TensorStore's transaction model, and StoreDeclaration is its identity-as-data. The lesson it has not taken is row 5. TensorStore's zip driver is read-only, and packed storage that needs mutation is a different format with a manifest. A format that only becomes readable when it is closed does not get a writable driver.

Archives and stores

flowchart LR
    R["Both satisfy the read capabilities: Get, GetRange, List, Head"]
    S["Key-value store<br/>put and delete change one key at a time<br/>Local, Memory, Fsspec, Obstore stores"]
    A["Key-value archive<br/>written whole by pack(); read-only after<br/>ZipArchive, formerly ZipStore"]
    R --- S
    R --- A
    S -->|"pack(store, target)"| A
    A -->|"copy(archive, store)"| S
Loading

A key-value archive is written whole and read by key. A key-value store supports mutation of individual keys. Reading is one capability set that both satisfy; the distinction is in how they are written. In the plan's protocol terms, an archive advertises the read capabilities and Serializable, never Put or Delete, and adds one constructor:

class Packable(Protocol):
    """Capability: build this backend whole from any readable source."""

    @classmethod
    def pack(cls, source: Readable, target: Path | IO[bytes]) -> Self: ...


class ZipArchive:
    """Get, GetRange, List, ListWithDelimiter, Head, Serializable, Packable, ThreadSafe."""

    def __init__(self, path: Path | str | IO[bytes]) -> None: ...   # read-only; no mode

    @classmethod
    def pack(cls, source: Readable, target: Path | IO[bytes]) -> Self:
        """Write every key of `source` once. A path target is replaced
        atomically with os.replace of a temporary file; a file object is
        written in one pass."""

ZipArchive holds a read handle it can drop and reacquire at will, so it needs no lifecycle contract, and it pickles as its declaration like every other store. Adding keys to an archive is producing a new one, pack(source=[archive, store], target). Zip allows appending entries and a new central directory, so pack can do that without rewriting when the target is the archive's own path and no key repeats; that is an optimization inside pack, not a Put.

Item in the plan or the tracker Under the split
ZipStore lifecycle contract, five options Closed. Nothing is written at close, so there is no contract to choose.
Roundtrip through ZipStore (#3194) via flush() and reopen() Write to a MemoryStore or LocalStore, pack, read the archive. The stubs are not needed.
ZipStore pickling re-acquires the handle on each worker Workers reopen for reading only; #3516 cannot recur.
ZipStore rows in the transactional and conformance tables Out of those tables by type. Archive conformance is: pack(source) reads equal to source, and a failed pack leaves the previous target intact.
Duplicate entries on rewrite (#3580, #828) Each key is written once per pack.
Read-only ZipStore copy (#2448) Every archive is read-only; ReadOnly[S] is a no-op on it.
File-like ZipStore (#1018), an M0 item Reads need read and seek; pack writes to any file object in one pass.

zarr already ships a key-value archive: a shard is an index plus entries, written whole and read by key, nested inside a store value. The same reader shape could back both. That is worth a note in the plan, not a commitment.

What it costs

Packing makes a second copy of the data, and incremental writes into an existing zip are not offered. That is the trade TensorStore made, and the writable ZipStore has never been safe for more than one writer. Users who write directly into a ZipStore today get a shim for one deprecation window: the write modes route to an internal MemoryStore that is packed on close() or __exit__. The #4450 fixes stay in 3.x as the safety net for that shim.

Migration

In the plan's three streams:

  1. Stream 1, additive 3.x minors. Merge #4450 as the 3.x safety net. Add Packable to the capability protocols, ZipStore.pack(), and the archive conformance spec. Mark the ZipStore read path ThreadSafe.
  2. Stream 2, deprecations across 3.x. Deprecate mode="w", "a" and "x" on ZipStore; the shim above keeps them working. Deprecate the class name if it is renamed.
  3. Stream 3, the late major. ZipStore is an archive. The write modes, flush() and reopen() never ship, the lifecycle open question is closed, and StoreDeclaration covers it like every other store.

The rest of the plan's store work is unchanged; nothing here blocks on the Prefixed[S] or sync-by-default migrations.

Open decisions

  • Is "archive" a nominal base class, or a Packable capability protocol plus the absence of Put, matching the plan's protocol style?
  • Name: ZipArchive with ZipStore as a deprecated alias, or keep ZipStore for a class that no longer takes writes?
  • Where does pack live: a classmethod on each archive type, or zarr.pack(source, target, format="zip")?
  • Accept that incremental writes into a zip go away, with the MemoryStore-plus-pack shim across Stream 2?
  • #4450: merge as the 3.x safety net, or shrink it to the AttributeError and reopen-truncation fixes?
  • Should the plan note the shard as an archive?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment