You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
DataFrame.collect() should return async stream rather than a Vec<RecordBatch> #47
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently, any DataFrame implementation must load the entire result set into RAM in the collect() method because it has to return a Vec<RecordBatch>.
Describe the solution you'd like
Change the signature to return an async stream of batches.
Activity
kination commentedon May 16, 2021
@andygrove hope to touch on this if it's not on work yet 🙏
kination commentedon May 23, 2021
@andygrove currently
collectis collectingSendableRecordBatchStreamto Vector inside function before returning.Are you intend to make this not collecting before return?
feat: Support Emit::First for SumDecimalGroupsAccumulator (apache#47)
feat: add with_column_renamed funcation for dataframe (apache#47)
Unprotect 2 (apache#47)