Support global configuration? #153

Activity

1. kylebarron commented on Aug 7, 2026

   

   kylebarron

   on Aug 7, 2026

   Last edited by kylebarron

   MemberAuthor

   More actions

   Written by Claude:

   ---

   I looked into this. Short version: a rasterio.Env-style scoped context manager is possible, and zarrista can implement it without touching the zarrs global config at all.

   Why not build on zarrs' global config

   zarrs::config::Config is a single process-wide LazyLock<RwLock<Config>> behind global_config() / global_config_mut(). There is no scoped-set API and no thread-local variant.

   A context manager on top of that would be save-mutate-restore of a process global. That is worse than the plain global config this issue is already wary of: a with block on one thread silently changes behavior for every other thread, and rayon workers inside zarrs read the same global. Nesting plus an exception on a second thread can leave it wrong.

   We don't need it

   Two facts make this easy:

   1. CodecOptions::default() in zarrs_codec is a hardcoded default. It does not consult Config.
   2. zarrista already resolves options at the pyo3 boundary and calls only the *_opt methods.

   So we are already insulated. The one thing missing is what fills in unwrap_or_default():

   let codec_options = codec_options.map(|o| o.into_inner()).unwrap_or_default();
   self.inner.retrieve_chunk_opt(&chunk_indices, &codec_options)?

   Change that single expression to consult a scoped default and we get rasterio.Env semantics with zero global mutable state. Options are resolved once on the calling thread and passed down into zarrs by value, so tokio and rayon workers never consult anything.

   Sync and async share one hook point

   Every async method already resolves codec_options in the synchronous prologue, before the future is built:

   let inner = self.inner.clone();
   let codec_options = codec_options.map(|o| o.into_inner()).unwrap_or_default();
   
   future_into_py(py, async move { ... })

   The sync path does the same before crate::py::detach. So both APIs read on the caller's thread while holding the GIL, and there is exactly one change shape for both.

   Where the scope lives: contextvars

   Three candidates:

   1. A Python contextvars.ContextVar, read from Rust. Correct for OS threads and asyncio tasks, since each Task runs in a copied Context. Cost is one Python call per operation, which is nothing next to chunk IO and decode.
   2. A Rust thread_local! stack. What rasterio does. Wrong for asyncio: two concurrent tasks on one event-loop thread share the thread-local, so an inner scope leaks across await into a sibling task.
   3. A pure-Python wrapper layer. Invasive. __init__.py re-exports everything from _zarrista, so we would have to shadow every method.

   Option 1. Worth noting that rasterio itself is option 2 (a threading.local stack in env.py, plus CPLSetThreadLocalConfigOption off the main thread). It predates contextvars and is not asyncio-safe. We can do better.

   Semantics

   • Precedence: explicit codec_options= kwarg > innermost scope > hardcoded default. An explicit argument wins outright and is never merged with the scope, so it stays a complete, self-describing value.
   • Nesting: an inner scope inherits the outer one and overrides only the keys it names.
   • Resolution timing: at call time, not at Array open time. That makes with config(...): arr.retrieve_chunk(...) do the obvious thing on an array opened outside the block.
   • No process-global setter. The point of the scoped form is that there is no ambient state to miss. Anyone who wants process-wide defaults can wrap main() in the context manager.

   I modeled this exact shape in Python and ran 14 cases (sync default / scoped / nested / restore-on-exception, explicit-kwarg-wins, thread behavior, asyncio siblings, task inheritance, deferred await). All 14 behaved as designed.

   Two asymmetries fall out. Both are inherent to contextvars, and I think we document them rather than fix them:

   1. A bare threading.Thread does not inherit the enclosing scope. A new thread starts with a fresh empty Context. rasterio behaves the same way, so this is not a regression against the model we are borrowing. But it does surprise anyone who is thinking "global config". asyncio.create_task does inherit, because it copies the context at creation.

   2. For async, the scope binds at coroutine-creation time, not await time.

   with zarrista.config(validate_checksums=False):
       coro = arr.retrieve_chunk(idx)   # bound here
   await coro                            # still validate_checksums=False

   The mirror case (created outside, awaited inside) does not pick up the scope. This is a direct consequence of resolving in the prologue, which is also what makes sync and async identical. The alternative, resolving inside the future, would make the answer depend on which tokio worker runs it. That is strictly worse. Since with config(...) normally wraps the await too, this only shows up if you deliberately split creation from await.

   A plain @contextmanager works inside async def, so we do not need __aenter__.

   What goes in scope

   The zarrs Config fields split three ways:

   Group	Fields	Status
   Codec / chunk	validate_checksums, store_empty_chunks, concurrent_target, chunk_concurrent_minimum, experimental_partial_encoding	Already plumbed. Exactly the five keys PyCodecOptions accepts
   Metadata	include_zarrs_metadata, codec_store_metadata_if_encode_only, convert_aliased_extension_names, metadata_erase_version	Needs *_opt threading at store_metadata / erase_metadata
   Node open	use_consolidated_metadata	Needs *_opt threading at the open sites

   I would ship the first row alone as v1. It is a one-line change per call site and covers the settings people actually want to tune. metadata_convert_version stays out: zarrista deliberately pins V3.

   Sketch

   Python side keeps the merge and token logic (store.py is precedent for a Python-side module):

   # python/zarrista/_config.py
   _CONFIG: ContextVar[Mapping[str, Any]] = ContextVar("zarrista_config", default={})
   
   @contextmanager
   def config(**overrides: Any) -> Iterator[None]:
       token = _CONFIG.set({**_CONFIG.get(), **overrides})
       try:
           yield
       finally:
           _CONFIG.reset(token)

   Rust side only reads it:

   fn scoped_codec_options(py: Python<'_>) -> PyResult<CodecOptions> {
       static VAR: GILOnceCell<Py<PyAny>> = GILOnceCell::new();
       let var = VAR.get_or_try_init(py, || {
           py.import("zarrista._config")?.getattr("_CONFIG").map(Into::into)
       })?;
       let mapping = var.bind(py).call_method0("get")?;
       PyCodecOptions::extract(mapping.as_borrowed()).map(Into::into)
   }

   and each call site becomes:

   let codec_options = match codec_options {
       Some(opts) => opts.into_inner(),
       None => scoped_codec_options(py)?,
   };

   Note that this reuses PyCodecOptions' existing FromPyObject, which already extracts from any mapping. So the key-name parsing and its error message live in one place, and zarrista.config(validate_checksm=False) raises the same TypeError as the keyword form. The only constraint is that the read must happen before py::detach (sync) or future_into_py (async), which is where the resolution already sits.
2. kylebarron commented on Aug 11, 2026

   

   kylebarron

   on Aug 11, 2026

   MemberAuthor

   More actions

   From @d-v-b in zarr-developers/zarr-python#4149 (comment)

   that local config could be queried.

   this is a pattern I don't really like. if an object like a function or class instance depends on some other data, then that should be formalized as part of the function signature or class structure. zarr-python today has a lot of places where code reaches for a free-floating config object, when IMO the simpler solution is to add parameters to functions / classes.
3. d-v-b commented on Aug 11, 2026

   

   d-v-b

   on Aug 11, 2026

   Contributor

   More actions

   yeah i'm not seeing the need for global configuration. It should definitely be modelled as part of zarrs' API, but at least in zarr-python every time global configuration made controlling some knob easy, it also made doing the same thing hard, when someone wants to turn a knob without modifying the global config.

   👍React with 👍1Reacted by Kyle Barron
4. kylebarron commented on Aug 11, 2026

   

   kylebarron

   on Aug 11, 2026

   MemberAuthor

   More actions

   I agree. I think for now we should just close this. Even though zarrs exposes a global configuration, for now I think that's something we shouldn't expose through zarrista.
5. kylebarron

   closed this as completedon Aug 11, 2026
6. kylebarron commented on Aug 11, 2026

   

   kylebarron

   on Aug 11, 2026

   MemberAuthor

   More actions

   @d-v-b this seems to conflict a bit with the upcoming changes in zarrs/zarrs#443. In particular, calls that previously had per-method configuration, like retrieve_array_subset_opt no longer exist. Instead the CodecOptions are placed onto the Array directly.

   I'm not sure I fully agree with that approach. I could agree that whether to store empty chunks is truly a property of the array, but I don't think the concurrency settings should be there.
7. d-v-b commented on Aug 11, 2026

   

   d-v-b

   on Aug 11, 2026

   Contributor

   More actions

   i might be misreading but it looks like the new pattern is reconfigured_array = array.with_config(...); reconfigured_array.do_config_sensitive_thing(), which seems like a pattern that's fine to bubble up to python?
8. kylebarron commented on Aug 11, 2026

   

   kylebarron

   on Aug 11, 2026

   MemberAuthor

   More actions

   I think the question is to what extent config options should be a property of the array vs a read-time/write-time property.

   I think things like concurrency settings aren't a property of the array, but rather of how you read/write the array, and for that reason shouldn't be part of with_config
9. d-v-b commented on Aug 12, 2026

   

   d-v-b

   on Aug 12, 2026

   Last edited by d-v-b

   Contributor

   More actions

   I think things like concurrency settings aren't a property of the array, but rather of how you read/write the array, and for that reason shouldn't be part of with_config

   I tend to agree in principle but for python consumers __setitem__ and __getitem__ don't take config-like parameters, and that means there's no way to configure the underlying IO routines except via something bound to self. Ideally I think we would have an API not so different from do_x and do_x_opt, where __setitem__ and __getitem__ take their configuration implicitly from self, and methods like set_values / get_values (name tbd) could take all the config they need explicitly.