Repository navigation
[API Proposal]: System.Buffers : BuffersExtensions.AsStream #100434
Description
Background and motivation
This is tangentially related to #100290 and the .NET 9 distributed caching epic; the proposed hybid-cache API will use ReadOnlySequence<byte> and IBufferWriter<byte> as the primary serializer APIs, but: not all serializers support these APIs (many do, note) - with Stream being the most common fallback.
API Proposal
// assembly: System.Memory
namespace System.Buffers;
public static class BuffersExtensions // pre-existing
{
+ public static Stream AsStream(this ReadOnlySequence<byte> value) => new ReadOnlySequenceStream(value);
+ public static Stream AsStream(this IBufferWriter<byte> value) => new BufferWriterStream(value);
}
+ internal sealed class ReadOnlySequenceStream : Stream {...}
+ internal sealed class BufferWriterStream : Stream {...}API Usage
ReadOnlySequence<byte> payload = ...
Customer obj = SomeRandomSerializer.Deserialize<Customer>(payload.AsStream());and
Customer obj = ...
IBufferWriter<byte> target = ...
SomeRandomSerializer.Serialize<Customer>(target.AsStream(), obj);The implementations would be internal, but:
ReadOnlySequenceStreamhas aCanRead: true,CanWrite: falseimplementation that supports seek etc; no additional buffer copies, just a few "where are we" counters, usingSequencePositioniteration and aReadOnlyMemory<byte>snapshot of the current segmentBufferWriterStreamhas aCanRead: false,CanWrite: trueimplementation that does not support seek; position and length are read-only and report the bytes written so far; no double-buffering - it is assumed (as a fundamental part ofIBufferWriter<byte>) that the underlyingIBufferWriter<byte>already does some internal work there when responding toGetSpan/GetMemory(such that they are affordable), so this would be duplicated effort and an additional mem-copy- in both cases, all APIs are synchronous, with the async APIs implemented as async-over-sync as efficiently as possible
- no need for fancy recycling, since we don't have any local buffers etc
Alternative Designs
The alternative is to use MemoryStream, which involves multiple additional copy operations and additional byte[] buffers.
Risks
None seen
Additional
I am happy to contribute the implementation effort.
Reacted by Paulus Pärssinen, kronic, Jason Nelson, Cole Tobin, Giorgi Chakhidze, Frank Gehann, TOM_RUS, Alexander Köpke, Linus Schwartz Hamlin, Jody Donetti and 1 more
Activity
dotnet-policy-service commentedon Mar 29, 2024
Tagging subscribers to this area: @dotnet/area-system-buffers
See info in area-owners.md if you want to be subscribed.
[-][API Proposal]: System.Buffers : BufferExtensions.AsStream[/-][+][API Proposal]: System.Buffers : BuffersExtensions.AsStream[/+]stephentoub commentedon Mar 29, 2024
The
ReadOnlySequence<T>side of this is #27156. @jozkee, where did we land on a design for these extra streams, e.g. AsStream vs dedicated publicly named streams vs etc. Implementation is straightforward, so if we have an agreed upon design, we can make quick forward progress here.jozkee commentedon Apr 1, 2024
#82801 (comment)
For
[ReadOnly]Memory<byte>andReadOnlySequence<byte>, we want to have static methods that return the Stream wrapping them. If we want to have them as extension methods inStreamwe would need the static extensions language feature to haveAsStream(this ReadOnlySequence<byte>), or we move ReadOnlySequence down to S.P.CoreLib.I discarded having
Memory<byte>support directly in MemoryStream based on #84103.mgravell commentedon Apr 1, 2024
Re corelib: that's why I proposed the existing BuffersExtensions as the root (or something in the same package/namespace) - it should have all the right refs.
PaulusParssinen commentedon Apr 2, 2024
Noting that we now have
System.IO.Pipelinesavailable in .NET 9 shared runtime meaning that there are now two types implementingIBufferWriter<byte>in the BCL;PipeWriterandArrayBufferWriter<byte>.When I use
S.IO.PipelinesI often find myself having to pullCommunityToolkit.HighPerformancein for itsAsStream(this IBufferWriter<byte> writer)fallback.jozkee commentedon Apr 2, 2024
Considering that static extensions is in the backlog dotnet/csharplang#192, I think we should consider this for 9 while keeping #82801 as future.
PaulusParssinen commentedon Jan 31, 2025
Is the plan for this and all other static extension helpers proposals to wait until dotnet/csharplang#192 is done before any further triage?
jeffhandley commentedon Feb 9, 2025
At this point, we are not prioritizing this proposal. But I am taking note that when dotnet/csharplang#192 lands, we should consider this proposal as a use case to implement.
aradalvand commentedon Feb 1, 2026
This is still just as needed as two years ago. And dotnet/csharplang#192 has now made it to C# 14. @jeffhandley Did you guys ever re-consider this issue since then?
jeffhandley commentedon Feb 1, 2026
Hi @aradalvand. Indeed, @ViveliDuCh has been working in this space and on related issues. I didn't have this one assigned to her yet, but I know it's part of what she's been looking into.
Add Stream wrappers for memory and text-based types (#126669)
Add Stream wrappers for memory and text-based types (#126669)