Repository navigation
DOC: create_array(..., data=,...) #2809
Description
Describe the issue linked to the documentation
I am very confused about the argument data in create_array. A common use case is to simply serialize an in memory array, in which case I tend to pass it as the data=in_memory_array argument. However, I cannot find the data argument in the documentation.
Using IPyhon, on the other hand, zarr.create_array clearly has the argument, while zarr.Group.create_array doesn't seem to expose the interface. I am quite confused about the discrepancy. If this is intentional, please document it.
LLM also suggest that
zarr.create_array("store.zarr", data=in_memory_data)is more efficient than
arr = zar.create_arra("store.zarr", shape=in_memory_data.shape, dtype=in_memory_data.dtype)
arr[...] = in_memory_dataI have no idea whether this is true or not. zarr.create_array(..., data=in_memory_data) might be indeed more efficient as it seems to be written asynchronously. But the documentation seems to by quite lacking, what the best practice is.
This might be a bit out of scope for this issue, this issue, so please tell me if it's out of scope. But from the documentation, I don't really see how to leverage the asynchronous nature of the zarr implementation. A common pattern I encounter is, that data is generated in parallel using multiprocessing (as it is CPU bound) and persisted using zarr (probably disc bound). Is there a preferred pattern, to use zarr as an asynchronous sink for the generated data? If so, it would be great to include it in the docs.
Suggested fix for documentation
No response
Activity
d-v-b commentedon Feb 9, 2025
thanks for this issue @DerWeh. the
datakeyword argument forcreate_arraywas added relatively recently and it looks like I forgot to add it to theGroup.create_arraymethod. This should be a simple fix.I agree that the documentation should say more about this. Basically all of the non-async functions like (like
Array.__setitem__are designed to take advantage of concurrency. But this concurrency is only useful for performance if your underlying storage layer is actually async. I don't think python's interface to the local file system is asynchronous so there's nothing for you to leverage. But if you were writing to cloud storage like s3, then you would gain a lot of performance from the async layer, even without accessing it.DerWeh commentedon Feb 9, 2025
Thanks for the clarification. Adding the
datakeyword seems simple enough. You're also right with it to be recent, on version 3.0.2 it is in fact documented on rtfd, while on 3.0.1 it's not available yet. Sorry for not making sure that I am reading the latest documentation.As far as I know, Python's standard library uses synchronous operations for files. There are, however, libraries like aiofiles (which I haven't tried so far). If I understand you correctly, using such a library as storage backend, we could expect performance improvements?
OPMTerra commentedon Mar 4, 2025
Hi @d-v-b 👋,
I'd like to help complete the documentation for
create_array(data=...).Proposed Changes:
dataparameter toGroup.create_arraydocstringIs this approach okay?
d-v-b commentedon Mar 4, 2025
that sounds good! note that the docstring changes might already by handled by this pr : #2819, but please weigh in on that PR if you would like to see anything changed there
OPMTerra commentedon Mar 4, 2025
@d-v-b 👋,
Thanks for pointing me to #2819 ! I've reviewed the changes and see that it addresses parameter consistency for
create_array.Observations:
create_arraysignatures consistent #2819 addsdatatoGroup.create_arraysignatures ✅dataparameter in code ✅Remaining Gaps:
datausagearr[...] = dataCan I focus on updating
docs/user-guide/arrays.rstwith:data=...data=vs assignmentdata=...)Would this be helpful?
d-v-b commentedon Mar 4, 2025
that would be great, thank you!
DOC: Add data= examples and async note (fix zarr-developers#2809)
OPMTerra commentedon Mar 5, 2025
Hi @d-v-b 👋,
I've opened a PR #2890 to address this issue!
Changes included:
data=example toarrays.rstLet me know if any adjustments are needed!
Thank you for your guidance 🙏
d-v-b commentedon Oct 27, 2026
docs for the
dataparameter was added in #2819