Skip to content

Comments

feat(core): Expose Reader::into_stream and Writer::into_sink#5698

Merged
Xuanwo merged 2 commits intomainfrom
expose-buffer-stream-and-sink
Mar 6, 2025
Merged

feat(core): Expose Reader::into_stream and Writer::into_sink#5698
Xuanwo merged 2 commits intomainfrom
expose-buffer-stream-and-sink

Conversation

@Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Mar 6, 2025

Which issue does this PR close?

Closes #4592

Rationale for this change

This PR will expose into_stream and into_sink API for users allowing them to fetch or sink fixed size buffer inside.

What changes are included in this PR?

Expose new APIs and add docs.

Are there any user-facing changes?

Expose new APIs.

@Xuanwo Xuanwo requested review from dqhl76 and tisonkun March 6, 2025 10:31
@github-actions github-actions bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Mar 6, 2025
state: State,
}

/// # Notes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment would now go to State's? Looks weird.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment would now go to State's? Looks weird.

Yep, I don't want users (from public API) to read this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it's not for State? Maybe // on BufferStream.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved to inner of BufferStream, PTAL.

Signed-off-by: Xuanwo <[email protected]>
@Xuanwo Xuanwo merged commit 8586a9e into main Mar 6, 2025
175 of 177 checks passed
@Xuanwo Xuanwo deleted the expose-buffer-stream-and-sink branch March 6, 2025 11:02
@chitralverma
Copy link
Contributor

@Xuanwo so now with this API in place if i want to copy a file from local fs to s3 in near zero copy manner, how will a sample code look?

because using tokio::io::copy or futures::io::copy will require AsyncRead and AsyncWrite inplementations.

@Xuanwo
Copy link
Member Author

Xuanwo commented Mar 6, 2025

@Xuanwo so now with this API in place if i want to copy a file from local fs to s3 in near zero copy manner, how will a sample code look?

Hi, we can use SinkExt::send_all to send stream into the sink in this way:

let mut osink = op.writer(path).await?.into_sink();
let mut ostream = op.reader(path).await?.into_stream(..).await?;
osink.send_all(&mut ostream).await?;
osink.close().await?;

@chitralverma
Copy link
Contributor

@Xuanwo BufferStream is still not completely public, probably needs to be added to exports.

pub(crate) use buffer_stream::BufferStream;

@Xuanwo
Copy link
Member Author

Xuanwo commented Mar 10, 2025

@Xuanwo BufferStream is still not completely public, probably needs to be added to exports.

pub(crate) use buffer_stream::BufferStream;

Fixed in #5730

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat"

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose BufferStream and BufferSink as public API

3 participants