Skip to content

Commit cffa16c

Browse files
d-v-b-agentclaude
andcommitted
fix(config): address branch-review findings
Fixes issues found reviewing the typed-config branch: - B1: `ZARR_CODECS__VLEN_UTF8`/`VLEN_BYTES` env overrides were silently dead (env vars can't contain hyphens, so they landed under a dead `vlen_utf8` key). `build_config` now canonicalizes an underscore codec name onto its hyphenated built-in default when one exists. - M1: a dotted key that walked past a scalar leaf resolved stray Python attributes (`array.order.upper` -> `str.upper`) instead of raising. `get_path`, `_replace_recursive`, and `_resolve_for_suggestion` now descend only into config nodes and match declared fields. - M2: the `codecs` subtree is now an immutable `MappingProxyType`, so a snapshot can't be mutated in place or leaked across `replace`-shared snapshots. - M3: `set` validates keys but not values; the docstring/changelog previously claimed value validation. Corrected the wording (value validation is deferred to zarr-developersgh-3285). - M4: subtree nodes gained `__contains__`/`__iter__`/`keys`/`__len__`, so `"order" in config.get("array")`, iteration, `keys()`, and `dict(...)` work instead of raising a cryptic error. - M5: the manager gained `__getitem__`/`__contains__` so `config["array.order"]` and `"array.order" in config` mirror `get`. - m1: concurrent permanent `set`s to distinct keys could lose updates (lock-free read-modify-write of the global snapshot). Added a lock around `_base` transitions. - m4: restored `Config` as an importable alias of `ZarrConfigManager`. Adds regression tests for each. The `with config.set(...)` vs concurrent permanent-set race remains a documented caveat (a proper fix needs a design change to the scope/base model). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VnqmZHVj4yCecb8oSDjCB3
1 parent ae7d5b3 commit cffa16c

3 files changed

Lines changed: 253 additions & 47 deletions

File tree

‎changes/4101.changed.md‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,23 @@
11
Gave `zarr.config` a statically-typed, dataclass-backed representation.
22
`zarr.config` now provides precise static types for attribute access
33
(`zarr.config.array.order`) and for the dotted-string API
4-
(`zarr.config.get("array.order")`), and validates keys and values at runtime
5-
with suggestions for typos. `donfig` is retained as the reader for environment
4+
(`zarr.config.get("array.order")`), and validates configuration *keys* at
5+
runtime with suggestions for typos. (Value validation is not yet performed; an
6+
out-of-range value such as `config.set({"array.order": "Q"})` is accepted and
7+
surfaces at its use site.) `donfig` is retained as the reader for environment
68
variables (`ZARR_FOO__BAR`) and YAML config files, so their locations and
79
precedence are unchanged. The string API, `config.set` (permanent and as a
810
context manager, including cross-thread visibility of permanent overrides),
911
`config.reset`, `config.enable_gpu`, and the `deprecations` mechanism are all
10-
preserved.
12+
preserved. The former `Config` class name remains importable from
13+
`zarr.core.config` as an alias of the new manager.
1114

1215
Note: `zarr.config.defaults` now returns a nested `dict` directly; donfig
1316
previously returned a one-element `list[dict]`, so callers that used
1417
`config.defaults[0]` must be updated to use `config.defaults`. Subtree reads
1518
such as `config.get("array")` now return a typed settings object rather than a
16-
plain `dict`; attribute access (`config.get("array").order`) and item access
17-
(`config.get("array")["order"]`) both work, but `dict`-only operations like
18-
`.keys()` or `dict(...)` no longer apply to a subtree.
19+
plain `dict`; it supports attribute access (`config.get("array").order`), item
20+
access (`config.get("array")["order"]`), membership (`"order" in ...`),
21+
iteration, `.keys()`, and `dict(...)`, but is immutable. The `codecs` mapping is
22+
likewise read-only (a `MappingProxyType`); change codec selections via
23+
`config.set` rather than by mutating the mapping in place.

‎src/zarr/core/config.py‎

Lines changed: 144 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,12 @@
3535
from __future__ import annotations
3636

3737
import difflib
38+
import threading
3839
import warnings
39-
from collections.abc import Mapping
40+
from collections.abc import Iterator, Mapping
4041
from contextvars import ContextVar, Token
4142
from dataclasses import dataclass, field, fields, is_dataclass, replace
43+
from types import MappingProxyType
4244
from typing import Any, Literal, Self, cast, overload
4345

4446
from zarr.errors import ZarrDeprecationWarning, ZarrUserWarning
@@ -102,6 +104,31 @@ def __getitem__(self, key: str) -> object:
102104
# such nodes structurally, so the cast is sound.
103105
return get_path(cast("ZarrConfig", self), key)
104106

107+
def __contains__(self, key: object) -> bool:
108+
# Mirror donfig's `"array" in config.get(...)` support. Only string keys
109+
# can resolve; a non-string (or unknown key) is simply not contained.
110+
if not isinstance(key, str):
111+
return False
112+
try:
113+
get_path(cast("ZarrConfig", self), key)
114+
except KeyError:
115+
return False
116+
return True
117+
118+
def __iter__(self) -> Iterator[str]:
119+
# Yield immediate child key names (serialized), so iteration, `keys()`,
120+
# and `dict(node)` behave like the plain dicts donfig returned. Note this
121+
# deliberately does NOT make the node a `collections.abc.Mapping`: the
122+
# internal `isinstance(obj, Mapping)` checks must stay false for config
123+
# nodes so they are distinguished from the open `codecs` mapping.
124+
return iter(_children(self))
125+
126+
def keys(self) -> list[str]:
127+
return _children(self)
128+
129+
def __len__(self) -> int:
130+
return len(_children(self))
131+
105132

106133
@dataclass(frozen=True, slots=True)
107134
class ArraySettings(_ConfigNode):
@@ -139,7 +166,9 @@ class ZarrConfig(_ConfigNode):
139166
threading: ThreadingSettings = field(default_factory=ThreadingSettings)
140167
json_indent: int = 2
141168
codec_pipeline: CodecPipelineSettings = field(default_factory=CodecPipelineSettings)
142-
codecs: Mapping[str, str] = field(default_factory=lambda: dict(DEFAULT_CODECS))
169+
codecs: Mapping[str, str] = field(
170+
default_factory=lambda: MappingProxyType(dict(DEFAULT_CODECS))
171+
)
143172
buffer: str = "zarr.buffer.cpu.Buffer"
144173
ndbuffer: str = "zarr.buffer.cpu.NDBuffer"
145174

@@ -172,8 +201,15 @@ def get_path(cfg: ZarrConfig, key: str) -> object:
172201
return obj[remainder]
173202
except KeyError:
174203
raise KeyError(key) from None
204+
# A prior segment resolved to a scalar leaf, but the key has more
205+
# segments — descend no further. Without this guard, `hasattr` would
206+
# match ordinary Python attributes/methods (e.g. `array.order.upper`
207+
# returning `str.upper`, or `default_zarr_format.numerator` returning
208+
# the int's numerator) instead of raising for the invalid key.
209+
if not is_dataclass(obj):
210+
raise KeyError(key)
175211
field_name = _resolve_field(obj, segment)
176-
if not hasattr(obj, field_name):
212+
if field_name not in {f.name for f in fields(obj)}:
177213
raise KeyError(key)
178214
obj = getattr(obj, field_name)
179215
return obj
@@ -192,15 +228,22 @@ def _replace_recursive(obj: Any, segments: list[str], value: object, key: str) -
192228
segment = segments[0]
193229
if isinstance(obj, Mapping):
194230
remainder = ".".join(segments)
195-
return {**obj, remainder: value}
231+
# Keep the open mapping immutable, like every other snapshot value.
232+
return MappingProxyType({**obj, remainder: value})
233+
if not is_dataclass(obj):
234+
# `key` tries to descend past a scalar leaf (e.g. `array.order.upper`).
235+
raise KeyError(key)
196236
field_name = _resolve_field(obj, segment)
197-
if not hasattr(obj, field_name):
237+
if field_name not in {f.name for f in fields(obj)}:
198238
raise KeyError(key)
239+
# `is_dataclass` narrows `obj` to `... | type[...]`, which `replace` rejects;
240+
# at runtime `obj` is always a dataclass *instance* here, so re-widen to Any.
241+
node: Any = obj
199242
if len(segments) == 1:
200-
return replace(obj, **{field_name: value})
201-
child = getattr(obj, field_name)
243+
return replace(node, **{field_name: value})
244+
child = getattr(node, field_name)
202245
new_child = _replace_recursive(child, segments[1:], value, key)
203-
return replace(obj, **{field_name: new_child})
246+
return replace(node, **{field_name: new_child})
204247

205248

206249
_ROSTER_LIMIT = 10
@@ -231,8 +274,11 @@ def _resolve_for_suggestion(cfg: ZarrConfig, key: str) -> tuple[str, list[str],
231274
if isinstance(obj, Mapping):
232275
# the remainder indexes into an open mapping as a single key
233276
return prefix, _children(obj), ".".join(segments[i:])
277+
if not is_dataclass(obj):
278+
# a prior segment resolved to a scalar leaf; nothing deeper is valid
279+
return prefix, _children(obj), segment
234280
field_name = _resolve_field(obj, segment)
235-
if not hasattr(obj, field_name):
281+
if field_name not in {f.name for f in fields(obj)}:
236282
return prefix, _children(obj), segment
237283
obj = getattr(obj, field_name)
238284
prefix = f"{prefix}.{segment}" if prefix else segment
@@ -321,6 +367,28 @@ def apply_overrides(cfg: ZarrConfig, overrides: Mapping[str, object]) -> ZarrCon
321367
_DONFIG_META_KEYS: frozenset[str] = frozenset({"config", "root_config"})
322368

323369

370+
def _canonicalize_override_keys(overrides: Mapping[str, object]) -> dict[str, object]:
371+
"""Map underscore codec names onto their hyphenated built-in defaults.
372+
373+
Environment variables cannot contain hyphens, so ``ZARR_CODECS__VLEN_UTF8``
374+
flattens to the key ``codecs.vlen_utf8``. The built-in codec is registered
375+
under the hyphenated name ``vlen-utf8`` (likewise ``vlen-bytes``), so without
376+
this remapping the override would land under a dead ``vlen_utf8`` key and be
377+
silently ignored while the registry keeps reading the untouched default.
378+
When a ``codecs.<name>`` key does not match a default but its hyphenated
379+
variant does, rewrite it to the hyphenated form. New (non-default) codec
380+
names and underscore-named defaults (e.g. ``sharding_indexed``) are untouched.
381+
"""
382+
out: dict[str, object] = {}
383+
for key, value in overrides.items():
384+
if key.startswith("codecs."):
385+
name = key[len("codecs.") :]
386+
if name not in DEFAULT_CODECS and name.replace("_", "-") in DEFAULT_CODECS:
387+
key = f"codecs.{name.replace('_', '-')}"
388+
out[key] = value
389+
return out
390+
391+
324392
def build_config() -> ZarrConfig:
325393
"""Build the base snapshot: typed defaults overlaid with donfig's ingest.
326394
@@ -341,6 +409,7 @@ def build_config() -> ZarrConfig:
341409
for key, value in overrides.items()
342410
if key.split(".", 1)[0] not in _DONFIG_META_KEYS
343411
}
412+
overrides = _canonicalize_override_keys(overrides)
344413
return apply_overrides(make_default_config(), overrides)
345414

346415

@@ -396,6 +465,10 @@ class ZarrConfigManager:
396465
def __init__(self) -> None:
397466
self._base: ZarrConfig = build_config()
398467
self._scope: ContextVar[ZarrConfig] = ContextVar("zarr_config_scope")
468+
# Serializes read-modify-write of the process-global `_base` so
469+
# concurrent permanent `set`s to different keys don't lose updates
470+
# (each `set` rebuilds a whole immutable snapshot from `_base`).
471+
self._lock = threading.Lock()
399472

400473
# --- state resolution -------------------------------------------------
401474
def _current(self) -> ZarrConfig:
@@ -408,7 +481,8 @@ def _enter_scope(self, prev_base: ZarrConfig, new: ZarrConfig) -> Token[ZarrConf
408481
# Promote a `with config.set(...)` to a context-local override: undo the
409482
# immediate global apply performed by `set`, then pin `new` to this
410483
# context only. Concurrent threads/tasks keep seeing the global base.
411-
self._base = prev_base
484+
with self._lock:
485+
self._base = prev_base
412486
return self._scope.set(new)
413487

414488
def _exit_scope(self, token: Token[ZarrConfig]) -> None:
@@ -537,57 +611,81 @@ def set(self, updates: Mapping[str, object] | None = None, **kwargs: object) ->
537611
config.set({"array.order": "F"})
538612
config.set(default_zarr_format=2)
539613
540-
Unlike `get`, `set` does not statically type-check values: an invalid
541-
value such as `config.set({"array.order": "Q"})` is reported at runtime,
542-
not by the type checker. See the implementation comment above for the
543-
rationale (the open `codecs.*` namespace prevents a precise TypedDict
544-
under current mypy).
614+
`set` validates *keys* — an unknown key raises with a suggestion — but
615+
does **not** validate *values*: `config.set({"array.order": "Q"})` is
616+
accepted, and the invalid value surfaces later at its use site rather
617+
than here. Static value typing is prevented by the open `codecs.*`
618+
namespace (see the implementation comment above); runtime value
619+
validation is planned via the unified `parse_json` checker (gh-3285).
545620
"""
546621
all_updates: dict[str, object] = {}
547622
if updates:
548623
all_updates.update(updates)
549624
all_updates.update(kwargs)
550-
prev_base = self._base
551-
# Two snapshots so an override applies to the right layer:
552-
# - `scoped` layers on the current view (any active `with` overlay), and
553-
# is what a `with config.set(...)` pins as its context-local scope;
554-
# - `permanent` layers on the *global* base, and is what a bare `set`
555-
# writes. Basing the permanent write on `prev_base` rather than the
556-
# current view keeps a bare `set` nested inside a `with` block from
557-
# leaking that block's overlay into the global base.
558-
# Outside any `with` block the two are identical (`_current()` is `_base`).
559-
scoped = self._current()
560-
permanent = prev_base
561-
for key, value in all_updates.items():
562-
resolved = self._apply_deprecation(key, raise_on_removed=True)
563-
try:
564-
scoped = replace_path(scoped, resolved, value)
565-
permanent = replace_path(permanent, resolved, value)
566-
except KeyError:
567-
raise _unknown_key_error(key, permanent) from None
568-
# Apply immediately to the global base. A bare `set` stays here (permanent,
569-
# cross-thread); a `with config.set(...)` is promoted to a context-local
570-
# overlay by `_ConfigSet.__enter__`, which undoes this global apply and
571-
# pins `scoped` instead.
572-
self._base = permanent
625+
# Hold the lock across the whole read-modify-write of `_base` so two
626+
# concurrent permanent `set`s to different keys can't clobber each other
627+
# (each rebuilds a full snapshot from the base it read).
628+
with self._lock:
629+
prev_base = self._base
630+
# Two snapshots so an override applies to the right layer:
631+
# - `scoped` layers on the current view (any active `with` overlay),
632+
# and is what a `with config.set(...)` pins as its context-local
633+
# scope;
634+
# - `permanent` layers on the *global* base, and is what a bare `set`
635+
# writes. Basing the permanent write on `prev_base` rather than the
636+
# current view keeps a bare `set` nested inside a `with` block from
637+
# leaking that block's overlay into the global base.
638+
# Outside any `with` block the two are identical (`_current()` is
639+
# `_base`).
640+
scoped = self._current()
641+
permanent = prev_base
642+
for key, value in all_updates.items():
643+
resolved = self._apply_deprecation(key, raise_on_removed=True)
644+
try:
645+
scoped = replace_path(scoped, resolved, value)
646+
permanent = replace_path(permanent, resolved, value)
647+
except KeyError:
648+
raise _unknown_key_error(key, permanent) from None
649+
# Apply immediately to the global base. A bare `set` stays here
650+
# (permanent, cross-thread); a `with config.set(...)` is promoted to a
651+
# context-local overlay by `_ConfigSet.__enter__`, which undoes this
652+
# global apply and pins `scoped` instead.
653+
self._base = permanent
573654
return _ConfigSet(self, prev_base, scoped)
574655

575656
# --- lifecycle --------------------------------------------------------
576657
def reset(self) -> None:
577658
# Rebuild the global base. A bare `set` no longer pins a context-local
578659
# scope, so the rebuilt base is visible in every context that is not
579-
# inside an active `with config.set(...)` block.
580-
self._base = build_config()
660+
# inside an active `with config.set(...)` block. Build outside the lock
661+
# (it reads env/YAML) and swap atomically under it.
662+
new_base = build_config()
663+
with self._lock:
664+
self._base = new_base
581665

582666
def refresh(self) -> None:
583-
self._base = build_config()
667+
self.reset()
584668

585669
def enable_gpu(self) -> _ConfigSet:
586670
return self.set(
587671
{"buffer": "zarr.buffer.gpu.Buffer", "ndbuffer": "zarr.buffer.gpu.NDBuffer"}
588672
)
589673

590674
# --- compat / introspection ------------------------------------------
675+
def __getitem__(self, key: str) -> Any:
676+
# donfig-style item access: `config["array.order"]` mirrors `get`.
677+
return self.get(key)
678+
679+
def __contains__(self, key: object) -> bool:
680+
# donfig-style membership: `"array.order" in config`.
681+
if not isinstance(key, str):
682+
return False
683+
try:
684+
self.get(key)
685+
except KeyError:
686+
return False
687+
return True
688+
591689
@property
592690
def defaults(self) -> dict[str, Any]:
593691
return to_nested_dict(make_default_config())
@@ -667,6 +765,11 @@ class BadConfigError(ValueError):
667765
"array.v3_default_compressors": None,
668766
}
669767

768+
# Backwards-compatible alias: before the typed rewrite, this module exposed the
769+
# donfig subclass as `Config`. Keep the name so `from zarr.core.config import
770+
# Config` and `isinstance(x, Config)` continue to work for downstream code.
771+
Config = ZarrConfigManager
772+
670773
config = ZarrConfigManager()
671774

672775

0 commit comments

Comments
 (0)