Repository navigation
Use the Zarr's new getitems() API - #131
Merged
Merged
Conversation
madsbk
added
improvement
This was referenced Sep 13, 2022
madsbk
force-pushed
the
zarr_getitems
branch
2 times, most recently
from
April 20, 2023 12:44
aca8ed9 to
dd82a5e
Compare
Merged
madsbk
marked this pull request as ready for review
June 19, 2023 12:55
wence-
requested changes
Jun 20, 2023
Comment on lines
+96
to
+98
| finally: | ||
| for f in files: | ||
| f.close() |
Contributor
There was a problem hiding this comment.
The idiomatic way to do this is...
import contextlib
...
with contextlib.ExitStack() as stack:
for key in keys:
f = stack.enter_context(kvikio.CuFile(filepath, "r"))
ret[key] = ..
io_results.append((f.pread(ret[key]), nbytes)
for fut, nbytes in io_results:
nbytes_read = fut.get()
...
Contributor
There was a problem hiding this comment.
I'll note that both the new code (and my proposed approach) leave dangling futures if the block produces an exception anywhere. I guess that is fine because that propagates and the program is hosed at that point anyway?
Co-authored-by: Lawrence Mitchell <[email protected]>
Member
Author
|
Thanks @wence-, I think I have addressed all of your issues. |
Member
Author
|
/merge |
wence-
approved these changes
Jun 21, 2023
Contributor
|
I think we need to wait for the other codeowners to approve too. |
jjacobelli
approved these changes
Jun 21, 2023
Member
|
Thanks all! 🙏 Great work 🥳 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
By using the new API in zarr-developers/zarr-python#1131, we do not have to guess whether to read into host or device memory. That is, no more filtering of specify keys like:
Notice, this PR is on hold until Zarr v2.15 is released
Closes #119
UPDATE: Zarr v2.15 has been released