[data] Add iterator batch_format=None support, which will yield batches in the current batch format with zero copies #33562
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
Signed-off-by: Eric Liang <[email protected]>
ericl
requested review from
c21,
clarkzinzow,
jianoaix,
jjyao and
scv119
as code owners
amogkam
approved these changes
Mar 21, 2023
| batch_size: Optional[Union[int, Literal["default"]]] = "default", | ||
| compute: Optional[Union[str, ComputeStrategy]] = None, | ||
| batch_format: Literal["default", "pandas", "pyarrow", "numpy"] = "default", | ||
| batch_format: Optional[str] = "default", |
Contributor
There was a problem hiding this comment.
keep it as Optional[Literal] for full explicitness of supported batch formats?
Contributor
Author
There was a problem hiding this comment.
I feel like that is hard to maintain (per the inconsistencies in the code already), so opted to go unify on the shorter signature.
Contributor
8 tasks
Signed-off-by: Eric Liang <[email protected]>
Contributor
Author
Done |
clarkzinzow
approved these changes
Mar 22, 2023
| ``"numpy"`` to select ``numpy.ndarray`` for tensor datasets and | ||
| ``Dict[str, numpy.ndarray]`` for tabular datasets. Default is "default". | ||
| ``Dict[str, numpy.ndarray]`` for tabular datasets, or None to return | ||
| the underlying block exactly as is with no additional formatting. |
Contributor
There was a problem hiding this comment.
Nice, I like batch_size=None a good bit more than adding another literal string!
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: elliottower <[email protected]>
elliottower
pushed a commit
to elliottower/ray
that referenced
this pull request
Apr 22, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
…es in the current batch format with zero copies (ray-project#33562) This PR is a cleanup of ray-project#33536 It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format. Signed-off-by: Jack He <[email protected]>
ProjectsByJackHe
pushed a commit
to ProjectsByJackHe/ray
that referenced
this pull request
May 4, 2023
…project#324… (ray-project#33601) The failure in rllib should have been fixed by ray-project#33562 Verified with `python -m pytest rllib/core/learner/torch/tests/test_torch_learner.py::TestLearner::test_end_to_end_update`. Signed-off-by: Jack He <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR is a cleanup of #33536
It uses "None" instead of "zero-copy" as a batch format, since None has a similar meaning for batch_size, where it means a system-chosen batch size. Here "None" also means the system chosen optimal batch format.