You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
obstore is currently a dependency, but I don't think VirtualiZarr requires accessing object stores, since it should be agnostic to the storage?
Could it be an extra-dependency?
OK I think I was a bit confused. It seems that obstore is the generic library to access any object store. I am particularly interested in using an HTTPStore from a WASM environment, so I guess obstore is not going to work (I tried adding it to emscripten-forge but I have little hope).
What I'd like is an httpx2-backed object store, since this one works in WASM (both in Pyodide and in emscripten-forge). But currently I cannot use VirtualiZarr in WASM since it depends on obstore. I'm not sure what the solution is, maybe have an obstore package on emscripten-forge that only supports an HTTPStore (for now) usgin httpx2? Or should the solution be in obstore itself, by having extra-dependencies to support only the needed backends?
I don't think VirtualiZarr requires accessing object stores
VirtualiZarr requires reading data from arbitrary storage (both metadata and sometimes actual chunks), and obstore gives us a nice abstraction that works for reading data from object storage, file systems, and HTTP. FYI Obstore is not named that because its specific to object storage, IIUC its named that because it provides an object-storage-like interface to various storage types.
Given that obstore does have a HTTP store already, this feature request seems like something to bring up with the obstore maintainers, and if that isn't possible for some reason we can discuss more here.
Separately I'm very curious what your use case for WASM VirtualiZarr is @davidbrochart ?
Thanks for the feedback @TomNicholas.
My use case was to use VirtualiZarr to fetch non-chunked data on the web from https://notebook.link, a JupyterLite-based hosted service. I've found another solution for now, but I think it could still be interesting for VirtualiZarr to work in the browser. Since obstore is mainly Rust with a thin Python layer, I don't think it will be easy for it to use httpx2 internally, but I'll post an issue there anyway.
My use case was to use VirtualiZarr to fetch non-chunked data
I'm confused - the whole point of VirtualiZarr is to organise references to chunks of numerical data. Are you merely wanting the storage indirection, and none of the Zarr features?
I was not clear, there is data on the web that I can access using HTTP range requests, and I want a Zarr API on top of it that would allow me to get only the data I need. Isn't VirtualiZarr a good fit for that?
Yes, I was just confused because you said "non-chunked" but all Zarr data is chunked! (Even if there is only one chunk.)
there is data on the web that I can access using HTTP range requests, and I want a Zarr API on top of it
FYI you can use VirtualiZarr to organise and write virtual references into an Icechunk store, and Icechunk supports WASM. When reading data back from the Icechunk store, VirtualiZarr (and obstore) is no longer involved, so that might potentially allow you to read Zarr chunks from HTTP via WASM.
Yes, I was just confused because you said "non-chunked" but all Zarr data is chunked! (Even if there is only one chunk.)
Yes I was referring to the "non-Zarr" data that I want to fetch on the web, which is not chunked.
FYI you can use VirtualiZarr to organise and write virtual references into an Icechunk store, and Icechunk supports WASM. When reading data back from the Icechunk store, VirtualiZarr (and obstore) is no longer involved, so that might potentially allow you to read Zarr chunks from HTTP via WASM.
Thanks, that's good to know. I've been away from Zarr/Xarray for some years, I need to catch up with all the improvements. It looks like the ecosystem is moving fast, which is great :)
There was some discussion here where @kylebarron mentioned VirtualiZarr could ideally depend only on obspec protocols, not on the specific obstore implementation. I think that would let me achieve what I'm trying to do, that I'll try to explain.
Each square represents a 10x10 degree tile, and each tile is a COG whose name has coordinates in it, like https://data.hydrosheds.org/file/hydrosheds-v2/ACC/1s/s10w060_ACC_1s_v2r0.tif (bottom left corner is at -10°, -60° lat/lon). This is a 1.8 GB file that I don't want to download in the browser, so being a COG is convenient because I can only access part of it. But there is no global representation of the whole dataset, i.e. I don't have a single API to access data that can be in any of these tiles, so I think VirtualiZarr could still help here. That would let me use xarray to slice through these tiles seamlessly, accessing the corresponding COGs under the hood.
I just started https://github.com/davidbrochart/ipygeotiff (a Python wrapper to geotiff.js), that lets me access remote COGs in the browser, and I'd like to use it in VirtualiZarr to do what I described.
Do you think it's a valid approach? Is there a better way? Would you be open to only depend on obspec in VirtualiZarr?
Each square represents a 10x10 degree tile, and each tile is a COG whose name has coordinates in it, like https://data.hydrosheds.org/file/hydrosheds-v2/ACC/1s/s10w060_ACC_1s_v2r0.tif (bottom left corner is at -10°, -60° lat/lon). This is a 1.8 GB file that I don't want to download in the browser, so being a COG is convenient because I can only access part of it. But there is no global representation of the whole dataset, i.e. I don't have a single API to access data that can be in any of these tiles, so I think VirtualiZarr could still help here. That would let me use xarray to slice through these tiles seamlessly, accessing the corresponding COGs under the hood. I just started https://github.com/davidbrochart/ipygeotiff (a Python wrapper to geotiff.js), that lets me access remote COGs in the browser, and I'd like to use it in VirtualiZarr to do what I described.
Hey there @davidbrochart, not sure if it would fit your use case, but have you tried out virtual-tiff? You can use it as a Parser for Virtualizarr to read COGs, then use Xarray to concat/merge and then write the Virtual Zarr to Icechunk. We've used this to visualize COGs in a web-map. That way, you can keep the COG's on the server and only request the chunks what you need.
Thanks @norlandrhagen! That looks like exactly what I am looking for, but it also depends on obstore, which means it's not going to work in the browser?
@davidbrochart To second what @norlandrhagen was mentioning, Virtualizarr is the library (along with its format specific parsers) that we use to build a "map" of the chunks in an archival file and represent that archival file as Zarr array whose chunks are "virtual" (still living in their archival file container). As @norlandrhagen described, once we've built this representation of one (or many files) we can use Xarray to combine them along a dimension to form a datacube and then serialize that to Icechunk. From this point on Virtualizarr is no longer in the picture and users/applications are now interacting directly with the Icechunk via a Zarr client to retrieve data.
Thanks for the detailed explanation @sharkinsspatial, so it seems that Icechunk has become an essential piece of the Zarr ecosystem (I'm currently learning about it).
IIUC I would need to create a Jupyter widget for Icechunk WASM (similar to ipygeotiff), since what I want is Python bindings in the end. I was already planning to do ipyzarrita but I'll do a widget that includes zarrita.js and Icechunk WASM, host the Icechunk store in the Jupyter filesystem and read it from Python.
I'll report back and hopefully have something working on Notebook.link :)
Seconding what the others said - I would just use VirtualiZarr to assemble the tiles in one (sparse) global datacube and write that to Icechunk, then build your application to read from Icechunk.
There are a few different threads here which make this confusing.
Making obstore optional
Regardless of the usage discussion of virtualizarr/virtual-tiff/etc above, I think making obstore optional is the right design.
None of my projects that usually use obstore: async-tiff nor async-geotiff nor async-pmtiles, depend on obstore. Each of them only depend on obspec, which is only the interface for obstore. This is the practice I recommend because it doesn't force a dependency of obstore onto users, which is useful whether the user might be constrained in the browser or constrained in AWS Lambda or just want to use their own backend with minimal dependencies.
I have a PR open to zarr-python to move to obspec, which would allow removing obstore as a required dependency (it's not required, but auto-enabled from the remote extra). I haven't been focused on that PR but it would be great to push it over the line for circumstances like these.
Python in the browser in Emscripten
Python in emscripten has some limitations. For one, the only way to make requests in the browser is via the JS fetch APIs. This means that some packages that manage IO themselves won't work out of the box.
Also, it's currently impossible (or at least very, very hard) to compile any Rust-Python async library to emscripten because of a mess around tokio. So obstore, async-tiff, zarrista, etc will not realistically be distributed for emscripten anytime soon (zarrista is compiled to emscripten but only with its synchronous APIs).
Icechunk Python is also async Rust based, so that also won't compile to emscripten FYI.
Because of all of these issues around emscripten, I'm not personally working on client-side Python because I don't think the core ecosystem is ready yet.
If you want to interact with this Icechunk store via JS/TS
@davidbrochart FYI if you'd prefer an alternative API to GeoTIFFs than geotiff.js that has a cleaner Typescript API (I'm biased) you can evaluate @developmentseed/geotiff. Its API is much more type secure.
Python in emscripten has some limitations. For one, the only way to make requests in the browser is via the JS fetch APIs. This means that some packages that manage IO themselves won't work out of the box.
I agree, this is why I'm currently leaning towards wraping existing JS libraries in Jupyter widgets, instead of e.g. the httpx2 way of detecting if we're running in the browser and using a different API for IO (doing that for every Python package that makes IO will take a lot of time).
Also, it's currently impossible (or at least very, very hard) to compile any Rust-Python async library to emscripten because of a mess around tokio. So obstore, async-tiff, zarrista, etc will not realistically be distributed for emscripten anytime soon (zarrista is compiled to emscripten but only with its synchronous APIs).
I think we should adopt the Sans-IO pattern when making an async library. For instance, there could be obstore-sans-io that obstore would depend on and integrate with tokio. Then we would make Python bindings to obstore-sans-io and integrate it with asyncio or AnyIO. But I agree it has to be designed this way from the start, it's much harder to refactor when the code is already very integrated with the event loop.
Icechunk Python is also async Rust based, so that also won't compile to emscripten FYI.
Yes but I'll use icechunk-js, right?
Because of all of these issues around emscripten, I'm not personally working on client-side Python because I don't think the core ecosystem is ready yet.
Well we're making good progress in emscripten-forge (conda-forge for emscripten), and the promise of Notebook.link is that this is becoming a viable solution.
If you want to interact with this Icechunk store via JS/TS
Yes but Python will talk to icechunk-js through a widget, which is essentially Jupyter's way of making Python bindings to code running in the browser, so that will work. Ipygeotiff does just that.
@davidbrochart FYI if you'd prefer an alternative API to GeoTIFFs than geotiff.js that has a cleaner Typescript API (I'm biased) you can evaluate @developmentseed/geotiff. Its API is much more type secure.
Thanks, I'll look at it. It seems you did quite a lot of work around making things run in the browser too. My needs are a little bit different because I still want to use Python in Notebook.link, which unlocks so many possibilities with regards to data processing. For instance, we recently managed to support Numba in emscripten-forge, so Python in the browser could also have good performances.
FYI I made some progress in ipygis (it requires earth-mover/icechunk#2363).
I can use xarray to slice through the HydroSHEDS dataset with a Zarr store that uses icechunk-js. All of that can run in the browser (Python included), for instance in JupyterLite, but unfortunately not in Notebook.link because we don't serve COOP and COEP headers (yet), which is required by the icechunk-js WASM build that uses SharedArrayBuffer.
Also, it's currently impossible (or at least very, very hard) to compile any Rust-Python async library to emscripten because of a mess around tokio. So obstore, async-tiff, zarrista, etc will not realistically be distributed for emscripten anytime soon (zarrista is compiled to emscripten but only with its synchronous APIs).
I think we should adopt the Sans-IO pattern when making an async library.
Making obstore sans-io wouldn't change anything because it's not the IO that requires tokio, it's the async runtime management. As long as we use pyo3-async-runtimes, which is currently the only stable way to make an async pyo3-based package, we'll need tokio.
Native pyo3 support for async has been worked on for a while and seems like it's still a long way off.
In general all my libraries other than obstore are sans-io, because they require a store parameter that implements the IO. Obspec exists to facilitate sans-io libraries with a stable, predictable protocol.
Making obstore sans-io wouldn't change anything because it's not the IO that requires tokio, it's the async runtime management.
OK but we could imagine that at a low level "obstore-base" only consists of sync functions, that obstore would use in an async context if its runtime needs it. I don't think obstore should need asynchronous programming for a MemoryStore, for instance. It would be similar to what zarr-developers/zarr-python#4217 is trying to do. But maybe I'm missing something and the async runtime is something deeply embedded that you can't change.
Obstore exists to be a binding to the Rust object_store library. Because object_store is exclusively async, so also Obstore must be async. I don't think it's realistic to consider a synchronous obstore.
This is why obspec exists, however, so that downstream libraries can use anything that looks like obstore, without needing to depend directly on obstore.
But maybe I'm missing something and the async runtime is something deeply embedded that you can't change.
So it's object_store that drags in async.
I don't think it's realistic to consider a synchronous obstore.
I think a MemoryStore is a good candidate for a synchronous store. But I understand that splitting obstore into a pure sync API and an async API could be difficult.
Yes, object_store is async. Even the MemoryStore implementation in object_store is fully async, because the point of object_store is for all backends to share exactly one API. That primary API is exclusively async.
You could create a synchronous MemoryStore in Python! As long as it implements the obspec interface, the goal is that it will work with any existing library that supports obspec.
fwiw, Zarr's needs are not exactly the same as the design goals of obstore. Zarr does not need a single API that only supports the overlap of many object storage backends; if that was Zarr's goal, we wouldn't be able to support partial writes, which are potentially very useful on file systems and in-memory stores. Instead, Zarr needs an API that's compatible with object storage APIs, but not bounded by object storage APIs. So an obstore restriction (no synchronous storage API) is not a Zarr restriction.
Activity
davidbrochart commentedon Aug 28, 2026
OK I think I was a bit confused. It seems that
obstoreis the generic library to access any object store. I am particularly interested in using anHTTPStorefrom a WASM environment, so I guessobstoreis not going to work (I tried adding it to emscripten-forge but I have little hope).What I'd like is an httpx2-backed object store, since this one works in WASM (both in Pyodide and in emscripten-forge). But currently I cannot use VirtualiZarr in WASM since it depends on
obstore. I'm not sure what the solution is, maybe have anobstorepackage on emscripten-forge that only supports anHTTPStore(for now) usginhttpx2? Or should the solution be inobstoreitself, by having extra-dependencies to support only the needed backends?TomNicholas commentedon Aug 31, 2026
VirtualiZarr requires reading data from arbitrary storage (both metadata and sometimes actual chunks), and obstore gives us a nice abstraction that works for reading data from object storage, file systems, and HTTP. FYI Obstore is not named that because its specific to object storage, IIUC its named that because it provides an object-storage-like interface to various storage types.
Given that obstore does have a HTTP store already, this feature request seems like something to bring up with the obstore maintainers, and if that isn't possible for some reason we can discuss more here.
Separately I'm very curious what your use case for WASM VirtualiZarr is @davidbrochart ?
davidbrochart commentedon Aug 31, 2026
Thanks for the feedback @TomNicholas.
My use case was to use VirtualiZarr to fetch non-chunked data on the web from https://notebook.link, a JupyterLite-based hosted service. I've found another solution for now, but I think it could still be interesting for VirtualiZarr to work in the browser. Since
obstoreis mainly Rust with a thin Python layer, I don't think it will be easy for it to usehttpx2internally, but I'll post an issue there anyway.TomNicholas commentedon Aug 31, 2026
I'm confused - the whole point of VirtualiZarr is to organise references to chunks of numerical data. Are you merely wanting the storage indirection, and none of the Zarr features?
davidbrochart commentedon Aug 31, 2026
I was not clear, there is data on the web that I can access using HTTP range requests, and I want a Zarr API on top of it that would allow me to get only the data I need. Isn't VirtualiZarr a good fit for that?
TomNicholas commentedon Aug 31, 2026
Yes, I was just confused because you said "non-chunked" but all Zarr data is chunked! (Even if there is only one chunk.)
FYI you can use VirtualiZarr to organise and write virtual references into an Icechunk store, and Icechunk supports WASM. When reading data back from the Icechunk store, VirtualiZarr (and obstore) is no longer involved, so that might potentially allow you to read Zarr chunks from HTTP via WASM.
davidbrochart commentedon Aug 31, 2026
Yes I was referring to the "non-Zarr" data that I want to fetch on the web, which is not chunked.
Thanks, that's good to know. I've been away from Zarr/Xarray for some years, I need to catch up with all the improvements. It looks like the ecosystem is moving fast, which is great :)
davidbrochart commentedon Sep 3, 2026
There was some discussion here where @kylebarron mentioned VirtualiZarr could ideally depend only on obspec protocols, not on the specific
obstoreimplementation. I think that would let me achieve what I'm trying to do, that I'll try to explain.I want to lazily fetch data from https://www.hydrosheds.org/downloads-core-data-v2, which is organized like this:
Each square represents a 10x10 degree tile, and each tile is a COG whose name has coordinates in it, like https://data.hydrosheds.org/file/hydrosheds-v2/ACC/1s/s10w060_ACC_1s_v2r0.tif (bottom left corner is at -10°, -60° lat/lon). This is a 1.8 GB file that I don't want to download in the browser, so being a COG is convenient because I can only access part of it. But there is no global representation of the whole dataset, i.e. I don't have a single API to access data that can be in any of these tiles, so I think VirtualiZarr could still help here. That would let me use xarray to slice through these tiles seamlessly, accessing the corresponding COGs under the hood.
I just started https://github.com/davidbrochart/ipygeotiff (a Python wrapper to geotiff.js), that lets me access remote COGs in the browser, and I'd like to use it in VirtualiZarr to do what I described.
Do you think it's a valid approach? Is there a better way? Would you be open to only depend on obspec in VirtualiZarr?
norlandrhagen commentedon Sep 3, 2026
Hey there @davidbrochart, not sure if it would fit your use case, but have you tried out virtual-tiff? You can use it as a Parser for Virtualizarr to read COGs, then use Xarray to concat/merge and then write the Virtual Zarr to Icechunk. We've used this to visualize COGs in a web-map. That way, you can keep the COG's on the server and only request the chunks what you need.
davidbrochart commentedon Sep 3, 2026
Thanks @norlandrhagen! That looks like exactly what I am looking for, but it also depends on
obstore, which means it's not going to work in the browser?sharkinsspatial commentedon Sep 3, 2026
@davidbrochart To second what @norlandrhagen was mentioning, Virtualizarr is the library (along with its format specific parsers) that we use to build a "map" of the chunks in an archival file and represent that archival file as Zarr array whose chunks are "virtual" (still living in their archival file container). As @norlandrhagen described, once we've built this representation of one (or many files) we can use Xarray to combine them along a dimension to form a datacube and then serialize that to Icechunk. From this point on Virtualizarr is no longer in the picture and users/applications are now interacting directly with the Icechunk via a Zarr client to retrieve data.
If you want to interact with this Icechunk store via JS/TS you can leverage Zarrita.js with Earthmover's Icechunk WASM bindings https://icechunk.io/en/latest/reference/icechunk-js/. Icechunk uses it's own custom storage backends for different protocols (s3, https, etc) rather than
obstoresoobstoreis not ever involved in the chunk retrieval pipeline (in your case since yourtiffiles are referenced via anhttpsendpoint it will be using this storage backend). We have slightly outdated example of a full Icechunk/Virtual array GL rendering application here https://github.com/developmentseed/deck.gl-raster/pull/577/changes.So in your case you would use
davidbrochart commentedon Sep 3, 2026
Thanks for the detailed explanation @sharkinsspatial, so it seems that Icechunk has become an essential piece of the Zarr ecosystem (I'm currently learning about it).
IIUC I would need to create a Jupyter widget for Icechunk WASM (similar to ipygeotiff), since what I want is Python bindings in the end. I was already planning to do ipyzarrita but I'll do a widget that includes zarrita.js and Icechunk WASM, host the Icechunk store in the Jupyter filesystem and read it from Python.
I'll report back and hopefully have something working on Notebook.link :)
TomNicholas commentedon Sep 3, 2026
Seconding what the others said - I would just use VirtualiZarr to assemble the tiles in one (sparse) global datacube and write that to Icechunk, then build your application to read from Icechunk.
kylebarron commentedon Sep 4, 2026
There are a few different threads here which make this confusing.
Making obstore optional
Regardless of the usage discussion of virtualizarr/virtual-tiff/etc above, I think making obstore optional is the right design.
None of my projects that usually use obstore: async-tiff nor async-geotiff nor async-pmtiles, depend on obstore. Each of them only depend on obspec, which is only the interface for obstore. This is the practice I recommend because it doesn't force a dependency of obstore onto users, which is useful whether the user might be constrained in the browser or constrained in AWS Lambda or just want to use their own backend with minimal dependencies.
I have a PR open to zarr-python to move to obspec, which would allow removing obstore as a required dependency (it's not required, but auto-enabled from the
remoteextra). I haven't been focused on that PR but it would be great to push it over the line for circumstances like these.Python in the browser in Emscripten
Python in emscripten has some limitations. For one, the only way to make requests in the browser is via the JS fetch APIs. This means that some packages that manage IO themselves won't work out of the box.
Also, it's currently impossible (or at least very, very hard) to compile any Rust-Python async library to emscripten because of a mess around tokio. So obstore, async-tiff, zarrista, etc will not realistically be distributed for emscripten anytime soon (zarrista is compiled to emscripten but only with its synchronous APIs).
Icechunk Python is also async Rust based, so that also won't compile to emscripten FYI.
Because of all of these issues around emscripten, I'm not personally working on client-side Python because I don't think the core ecosystem is ready yet.
@sharkinsspatial; @davidbrochart seems to want to use Python in the browser, not JS/TS.
ipygeotiff
@davidbrochart FYI if you'd prefer an alternative API to GeoTIFFs than geotiff.js that has a cleaner Typescript API (I'm biased) you can evaluate
@developmentseed/geotiff. Its API is much more type secure.davidbrochart commentedon Sep 4, 2026
I agree, this is why I'm currently leaning towards wraping existing JS libraries in Jupyter widgets, instead of e.g. the
httpx2way of detecting if we're running in the browser and using a different API for IO (doing that for every Python package that makes IO will take a lot of time).I think we should adopt the Sans-IO pattern when making an async library. For instance, there could be
obstore-sans-iothatobstorewould depend on and integrate with tokio. Then we would make Python bindings toobstore-sans-ioand integrate it with asyncio or AnyIO. But I agree it has to be designed this way from the start, it's much harder to refactor when the code is already very integrated with the event loop.Yes but I'll use icechunk-js, right?
Well we're making good progress in emscripten-forge (conda-forge for emscripten), and the promise of Notebook.link is that this is becoming a viable solution.
Yes but Python will talk to icechunk-js through a widget, which is essentially Jupyter's way of making Python bindings to code running in the browser, so that will work. Ipygeotiff does just that.
Thanks, I'll look at it. It seems you did quite a lot of work around making things run in the browser too. My needs are a little bit different because I still want to use Python in Notebook.link, which unlocks so many possibilities with regards to data processing. For instance, we recently managed to support Numba in emscripten-forge, so Python in the browser could also have good performances.
davidbrochart commentedon Sep 8, 2026
FYI I made some progress in ipygis (it requires earth-mover/icechunk#2363).
I can use xarray to slice through the HydroSHEDS dataset with a Zarr store that uses icechunk-js. All of that can run in the browser (Python included), for instance in JupyterLite, but unfortunately not in Notebook.link because we don't serve COOP and COEP headers (yet), which is required by the icechunk-js WASM build that uses
SharedArrayBuffer.kylebarron commentedon Sep 11, 2026
Making obstore sans-io wouldn't change anything because it's not the IO that requires tokio, it's the async runtime management. As long as we use pyo3-async-runtimes, which is currently the only stable way to make an async pyo3-based package, we'll need tokio.
Native pyo3 support for async has been worked on for a while and seems like it's still a long way off.
In general all my libraries other than obstore are sans-io, because they require a
storeparameter that implements the IO. Obspec exists to facilitate sans-io libraries with a stable, predictable protocol.davidbrochart commentedon Sep 11, 2026
OK but we could imagine that at a low level "obstore-base" only consists of sync functions, that obstore would use in an async context if its runtime needs it. I don't think obstore should need asynchronous programming for a
MemoryStore, for instance. It would be similar to what zarr-developers/zarr-python#4217 is trying to do. But maybe I'm missing something and the async runtime is something deeply embedded that you can't change.kylebarron commentedon Sep 11, 2026
Obstore exists to be a binding to the Rust
object_storelibrary. Becauseobject_storeis exclusively async, so also Obstore must be async. I don't think it's realistic to consider a synchronous obstore.This is why obspec exists, however, so that downstream libraries can use anything that looks like obstore, without needing to depend directly on obstore.
davidbrochart commentedon Sep 11, 2026
So it's
object_storethat drags in async.I think a
MemoryStoreis a good candidate for a synchronous store. But I understand that splitting obstore into a pure sync API and an async API could be difficult.kylebarron commentedon Sep 11, 2026
Yes,
object_storeis async. Even theMemoryStoreimplementation inobject_storeis fully async, because the point ofobject_storeis for all backends to share exactly one API. That primary API is exclusively async.You could create a synchronous
MemoryStorein Python! As long as it implements the obspec interface, the goal is that it will work with any existing library that supports obspec.d-v-b commentedon Sep 11, 2026
fwiw, Zarr's needs are not exactly the same as the design goals of obstore. Zarr does not need a single API that only supports the overlap of many object storage backends; if that was Zarr's goal, we wouldn't be able to support partial writes, which are potentially very useful on file systems and in-memory stores. Instead, Zarr needs an API that's compatible with object storage APIs, but not bounded by object storage APIs. So an obstore restriction (no synchronous storage API) is not a Zarr restriction.