feat: document the new streaming dataset for data loading#294
feat: document the new streaming dataset for data loading#294westonpace wants to merge 6 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
Thanks for the PR @westonpace! As I understand it, this is in main, but not yet in the official release, right? If so, we should wait till we release it before merging. |
It's not even in main yet 😅 so yes, we should hold off for now. |
prrao87
added
the
needs_new_release
prrao87
added
the
wip
prrao87
removed
needs_new_release
prrao87
left a comment
prrao87
left a comment
There was a problem hiding this comment.
Updated to the latest state of the code and improved conciseness/clarity. LGTM!
| stored in memory but they can be persisted to storage as well. This is useful when you want to share a permutation | ||
| table across processes or nodes. | ||
| `StreamingDataset` relies on the permutation API and requires direct access to an embedded LanceDB table. The table | ||
| data can live on local or cloud object storage, but a remote table accessed through the LanceDB Cloud API is not |
There was a problem hiding this comment.
Is it still true that the remote table API isn't supported?
There was a problem hiding this comment.
No, the remote table API should be supported.
There was a problem hiding this comment.
I'll clean that up.
2 participants
No description provided.