Skip to content

Commit b9c742d

Browse files
authored
.Net Kernel and ISKFunction Streaming Option #1 (Phase 1) (#3496)
## Context and Problem Statement Resolves #1649 Resolves #1298 It is quite common in co-pilot implementations to have a streamlined output of messages from the LLM (large language models) and currently that is not possible while using ISKFunctions.InvokeAsync or Kernel.RunAsync methods, which enforces users to work around the Kernel and Functions to use `ITextCompletion` and `IChatCompletion` services directly as the only interfaces that currently support streaming. Currently streaming is a capability that not all providers do support and this as part of our design we try to ensure the services will have the proper abstractions to support streaming not only of text but be open to other types of data like images, audio, video, etc. Needs to be clear for the sk developer when he is attempting to get streaming data. ## Decision Drivers 1. The sk developer should be able to get streaming data from the Kernel and Functions using Kernel.RunAsync or ISKFunctions.InvokeAsync methods 2. The sk developer should be able to get the data in a generic way, so the Kernel and Functions can be able to stream data of any type, not limited to text. 3. The sk developer when using streaming from a model that does not support streaming should still be able to use it with only one streaming update representing the whole data. ## User Experience Goal ```csharp //(providing the type at as generic parameter) // Getting a Raw Streaming data from Kernel await foreach(string update in kernel.StreamingRunAsync<byte[]>(function, variables)) // Getting a String as Streaming data from Kernel await foreach(string update in kernel.StreamingRunAsync<string>(function, variables)) // Getting a StreamingResultUpdate as Streaming data from Kernel await foreach(StreamingResultUpdate update in kernel.StreamingRunAsync<StreamingResultUpdate>(variables, function)) // OR await foreach(StreamingResultUpdate update in kernel.StreamingRunAsync(function, variables)) // defaults to Generic above) { Console.WriteLine(update); } ``` ## Out of Scope - Streaming with plans will not be supported in this phase. Attempting to do so will throw an exception. - Kernel streaming will not support multiple functions (pipeline). ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
1 parent bc2d83c commit b9c742d

32 files changed

Lines changed: 1506 additions & 70 deletions
Lines changed: 350 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,350 @@
1+
---
2+
# These are optional elements. Feel free to remove any of them.
3+
status: proposed
4+
date: 2023-11-13
5+
deciders: rogerbarreto,markwallace-microsoft,SergeyMenshykh,dmytrostruk
6+
consulted:
7+
informed:
8+
---
9+
10+
# Streaming Capability for Kernel and Functions usage - Phase 1
11+
12+
## Context and Problem Statement
13+
14+
It is quite common in co-pilot implementations to have a streamlined output of messages from the LLM (large language models)M and currently that is not possible while using ISKFunctions.InvokeAsync or Kernel.RunAsync methods, which enforces users to work around the Kernel and Functions to use `ITextCompletion` and `IChatCompletion` services directly as the only interfaces that currently support streaming.
15+
16+
Currently streaming is a capability that not all providers do support and this as part of our design we try to ensure the services will have the proper abstractions to support streaming not only of text but be open to other types of data like images, audio, video, etc.
17+
18+
Needs to be clear for the sk developer when he is attempting to get streaming data.
19+
20+
## Decision Drivers
21+
22+
1. The sk developer should be able to get streaming data from the Kernel and Functions using Kernel.RunAsync or ISKFunctions.InvokeAsync methods
23+
24+
2. The sk developer should be able to get the data in a generic way, so the Kernel and Functions can be able to stream data of any type, not limited to text.
25+
26+
3. The sk developer when using streaming from a model that does not support streaming should still be able to use it with only one streaming update representing the whole data.
27+
28+
## Out of Scope
29+
30+
- Streaming with plans will not be supported in this phase. Attempting to do so will throw an exception.
31+
- Kernel streaming will not support multiple functions (pipeline).
32+
- Input streaming will not be supported in this phase.
33+
- Post Hook Skipping, Repeat and Cancelling of streaming functions are not supported.
34+
35+
## Considered Options
36+
37+
### Option 1 - Dedicated Streaming Interfaces
38+
39+
Using dedicated streaming interfaces that allow the sk developer to get the streaming data in a generic way, including string, byte array directly from the connector as well as allowing the Kernel and Functions implementations to be able to stream data of any type, not limited to text.
40+
41+
This approach also exposes dedicated interfaces in the kernel and functions to use streaming making it clear to the sk developer what is the type of data being returned in IAsyncEnumerable format.
42+
43+
`ITextCompletion` and `IChatCompletion` will have new APIs to get `byte[]` and `string` streaming data directly as well as the specialized `StreamingContent` return.
44+
45+
The sk developer will be able to specify a generic type to the `Kernel.RunStreamingAsync<T>()` and `ISKFunction.InvokeStreamingAsync<T>` to get the streaming data. If the type is not specified, the Kernel and Functions will return the data as StreamingContent.
46+
47+
If the type is not specified or if the string representation cannot be cast, an exception will be thrown.
48+
49+
If the type specified is `StreamingContent` or another any type supported by the connector no error will be thrown.
50+
51+
## User Experience Goal
52+
53+
```csharp
54+
//(providing the type at as generic parameter)
55+
56+
// Getting a Raw Streaming data from Kernel
57+
await foreach(string update in kernel.RunStreamingAsync<byte[]>(function, variables))
58+
59+
// Getting a String as Streaming data from Kernel
60+
await foreach(string update in kernel.RunStreamingAsync<string>(function, variables))
61+
62+
// Getting a StreamingContent as Streaming data from Kernel
63+
await foreach(StreamingContent update in kernel.RunStreamingAsync<StreamingContent>(variables, function))
64+
// OR
65+
await foreach(StreamingContent update in kernel.RunStreamingAsync(function, variables)) // defaults to Generic above)
66+
{
67+
Console.WriteLine(update);
68+
}
69+
```
70+
71+
Abstraction class for any stream content, connectors will be responsible to provide the specialized type of `StreamingContent` which will contain the data as well as any metadata related to the streaming result.
72+
73+
```csharp
74+
75+
public abstract class StreamingContent
76+
{
77+
public abstract int ChoiceIndex { get; }
78+
79+
/// Returns a string representation of the chunk content
80+
public abstract override string ToString();
81+
82+
/// Abstract byte[] representation of the chunk content in a way it could be composed/appended with previous chunk contents.
83+
/// Depending on the nature of the underlying type, this method may be more efficient than <see cref="ToString"/>.
84+
public abstract byte[] ToByteArray();
85+
86+
/// Internal chunk content object reference. (Breaking glass).
87+
/// Each connector will have its own internal object representing the content chunk content.
88+
/// The usage of this property is considered "unsafe". Use it only if strictly necessary.
89+
public object? InnerContent { get; }
90+
91+
/// The metadata associated with the content.
92+
public Dictionary<string, object>? Metadata { get; set; }
93+
94+
/// The current context associated the function call.
95+
internal SKContext? Context { get; set; }
96+
97+
/// <param name="innerContent">Inner content object reference</param>
98+
protected StreamingContent(object? innerContent)
99+
{
100+
this.InnerContent = innerContent;
101+
}
102+
}
103+
```
104+
105+
Specialization example of a StreamingChatContent
106+
107+
```csharp
108+
//
109+
public class StreamingChatContent : StreamingContent
110+
{
111+
public override int ChoiceIndex { get; }
112+
public FunctionCall? FunctionCall { get; }
113+
public string? Content { get; }
114+
public AuthorRole? Role { get; }
115+
public string? Name { get; }
116+
117+
public StreamingChatContent(AzureOpenAIChatMessage chatMessage, int resultIndex) : base(chatMessage)
118+
{
119+
this.ChoiceIndex = resultIndex;
120+
this.FunctionCall = chatMessage.InnerChatMessage?.FunctionCall;
121+
this.Content = chatMessage.Content;
122+
this.Role = new AuthorRole(chatMessage.Role.ToString());
123+
this.Name = chatMessage.InnerChatMessage?.Name;
124+
}
125+
126+
public override byte[] ToByteArray() => Encoding.UTF8.GetBytes(this.ToString());
127+
public override string ToString() => this.Content ?? string.Empty;
128+
}
129+
```
130+
131+
`IChatCompletion` and `ITextCompletion` interfaces will have new APIs to get a generic streaming content data.
132+
133+
```csharp
134+
interface ITextCompletion + IChatCompletion
135+
{
136+
IAsyncEnumerable<T> GetStreamingContentAsync<T>(...);
137+
138+
// Throw exception if T is not supported
139+
}
140+
141+
interface IKernel
142+
{
143+
// Get streaming function content of T
144+
IAsyncEnumerable<T> RunStreamingAsync<T>(ContextVariables variables, ISKFunction function);
145+
}
146+
147+
interface ISKFunction
148+
{
149+
// Get streaming function content of T
150+
IAsyncEnumerable<T> InvokeStreamingAsync<T>(SKContext context);
151+
}
152+
```
153+
154+
## Prompt/Semantic Functions Behavior
155+
156+
When Prompt Functions are invoked using the Streaming API, they will attempt to use the Connectors streaming implementation.
157+
The connector will be responsible to provide the specialized type of `StreamingContent` and even if the underlying backend API don't support streaming the output will be one streamingcontent with the whole data.
158+
159+
## Method/Native Functions Behavior
160+
161+
Method Functions will support `StreamingContent` automatically with as a `StreamingMethodContent` wrapping the object returned in the iterator.
162+
163+
```csharp
164+
public sealed class StreamingMethodContent : StreamingContent
165+
{
166+
public override int ChoiceIndex => 0;
167+
168+
/// Method object value that represents the content chunk
169+
public object Value { get; }
170+
171+
/// Default implementation
172+
public override byte[] ToByteArray()
173+
{
174+
if (this.Value is byte[])
175+
{
176+
// If the method value is byte[] we return it directly
177+
return (byte[])this.Value;
178+
}
179+
180+
// By default if a native value is not byte[] we output the UTF8 string representation of the value
181+
return Encoding.UTF8.GetBytes(this.Value?.ToString());
182+
}
183+
184+
/// <inheritdoc/>
185+
public override string ToString()
186+
{
187+
return this.Value.ToString();
188+
}
189+
190+
/// <summary>
191+
/// Initializes a new instance of the <see cref="StreamingMethodContent"/> class.
192+
/// </summary>
193+
/// <param name="innerContent">Underlying object that represents the chunk</param>
194+
public StreamingMethodContent(object innerContent) : base(innerContent)
195+
{
196+
this.Value = innerContent;
197+
}
198+
}
199+
```
200+
201+
If a MethodFunction is returning an `IAsyncEnumerable` each enumerable result will be automatically wrapped in the `StreamingMethodContent` keeping the streaming behavior and the overall abstraction consistent.
202+
203+
When a MethodFunction is not an `IAsyncEnumerable`, the complete result will be wrapped in a `StreamingMethodContent` and will be returned as a single item.
204+
205+
## Pros
206+
207+
1. All the User Experience Goal section options will be possible.
208+
2. Kernel and Functions implementations will be able to stream data of any type, not limited to text
209+
3. The sk developer will be able to provide the streaming content type it expects from the `GetStreamingContentAsync<T>` method.
210+
4. Sk developer will be able to get streaming from the Kernel, Functions and Connectors with the same result type.
211+
212+
## Cons
213+
214+
1. If the sk developer wants to use the specialized type of `StreamingContent` he will need to know what the connector is being used to use the correct **StreamingContent extension method** or to provide directly type in `<T>`.
215+
2. Connectors will have greater responsibility to support the correct special types of `StreamingContent`.
216+
217+
### Option 2 - Dedicated Streaming Interfaces (Returning a Class)
218+
219+
All changes from option 1 with the small difference below:
220+
221+
- The Kernel and SKFunction streaming APIs interfaces will return `StreamingFunctionResult<T>` which also implements `IAsyncEnumerable<T>`
222+
- Connectors streaming APIs interfaces will return `StreamingConnectorContent<T>` which also implements `IAsyncEnumerable<T>`
223+
224+
The `StreamingConnectorContent` class is needed for connectors as one way to pass any information relative to the request and not the chunk that can be used by the functions to fill `StreamingFunctionResult` metadata.
225+
226+
## User Experience Goal
227+
228+
Option 2 Biggest benefit:
229+
230+
```csharp
231+
// When the caller needs to know more about the streaming he can get the result reference before starting the streaming.
232+
var streamingResult = await kernel.RunStreamingAsync(function);
233+
// Do something with streamingResult properties
234+
235+
// Consuming the streamingResult requires an extra await:
236+
await foreach(StreamingContent chunk content in await streamingResult)
237+
```
238+
239+
Using the other operations will be quite similar (only needing an extra `await` to get the iterator)
240+
241+
```csharp
242+
// Getting a Raw Streaming data from Kernel
243+
await foreach(string update in await kernel.RunStreamingAsync<byte[]>(function, variables))
244+
245+
// Getting a String as Streaming data from Kernel
246+
await foreach(string update in await kernel.RunStreamingAsync<string>(function, variables))
247+
248+
// Getting a StreamingContent as Streaming data from Kernel
249+
await foreach(StreamingContent update in await kernel.RunStreamingAsync<StreamingContent>(variables, function))
250+
// OR
251+
await foreach(StreamingContent update in await kernel.RunStreamingAsync(function, variables)) // defaults to Generic above)
252+
{
253+
Console.WriteLine(update);
254+
}
255+
256+
```
257+
258+
StreamingConnectorResult is a class that can store information regarding the result before the stream is consumed as well as any underlying object (breaking glass) that the stream consumes at the connector level.
259+
260+
```csharp
261+
262+
public sealed class StreamingConnectorResult<T> : IAsyncEnumerable<T>
263+
{
264+
private readonly IAsyncEnumerable<T> _StreamingContentource;
265+
266+
public object? InnerResult { get; private set; } = null;
267+
268+
public StreamingConnectorResult(Func<IAsyncEnumerable<T>> streamingReference, object? innerConnectorResult)
269+
{
270+
this._StreamingContentource = streamingReference.Invoke();
271+
this.InnerResult = innerConnectorResult;
272+
}
273+
}
274+
275+
interface ITextCompletion + IChatCompletion
276+
{
277+
Task<StreamingConnectorResult<T>> GetStreamingContentAsync<T>();
278+
// Throw exception if T is not supported
279+
// Initially connectors
280+
}
281+
```
282+
283+
StreamingFunctionResult is a class that can store information regarding the result before the stream is consumed as well as any underlying object (breaking glass) that the stream consumes from Kernel and SKFunctions.
284+
285+
```csharp
286+
public sealed class StreamingFunctionResult<T> : IAsyncEnumerable<T>
287+
{
288+
internal Dictionary<string, object>? _metadata;
289+
private readonly IAsyncEnumerable<T> _streamingResult;
290+
291+
public string FunctionName { get; internal set; }
292+
public Dictionary<string, object> Metadata { get; internal set; }
293+
294+
/// <summary>
295+
/// Internal object reference. (Breaking glass).
296+
/// Each connector will have its own internal object representing the result.
297+
/// </summary>
298+
public object? InnerResult { get; private set; } = null;
299+
300+
/// <summary>
301+
/// Instance of <see cref="SKContext"/> used by the function.
302+
/// </summary>
303+
internal SKContext Context { get; private set; }
304+
305+
public StreamingFunctionResult(string functionName, SKContext context, Func<IAsyncEnumerable<T>> streamingResult, object? innerFunctionResult)
306+
{
307+
this.FunctionName = functionName;
308+
this.Context = context;
309+
this._streamingResult = streamingResult.Invoke();
310+
this.InnerResult = innerFunctionResult;
311+
}
312+
}
313+
314+
interface ISKFunction
315+
{
316+
// Extension generic method to get from type <T>
317+
Task<StreamingFunctionResult<T>> InvokeStreamingAsync<T>(...);
318+
}
319+
320+
static class KernelExtensions
321+
{
322+
public static async Task<StreamingFunctionResult<T>> RunStreamingAsync<T>(this Kernel kernel, ISKFunction skFunction, ContextVariables? variables, CancellationToken cancellationToken)
323+
{
324+
...
325+
}
326+
}
327+
```
328+
329+
## Pros
330+
331+
1. All benefits from Option 1 +
332+
2. Having StreamingFunctionResults allow sk developer to know more details about the result before consuming the stream, like:
333+
- Any metadata provided by the underlying API,
334+
- SKContext
335+
- Function Name and Details
336+
3. Experience using the Streaming is quite similar (need an extra await to get the result) to option 1
337+
4. APIs behave similarly to the non-streaming API (returning a result representation to get the value)
338+
339+
## Cons
340+
341+
1. All cons from Option 1 +
342+
2. Added complexity as the IAsyncEnumerable cannot be passed directly in the method result demanding a delegate approach to be adapted inside of the Results that implements the IAsyncEnumerator.
343+
3. Added complexity where IDisposable is needed to be implemented in the Results to dispose the response object and the caller would need to handle the disposal of the result.
344+
4. As soon the caller gets a `StreamingFunctionResult` a network connection will be kept open until the caller implementation consume it (Enumerate over the `IAsyncEnumerable`).
345+
346+
## Decision Outcome
347+
348+
Option 1 was chosen as the best option as small benefit of the Option 2 don't justify the complexity involved described in the Cons.
349+
350+
Was also decided that the Metadata related to a connector backend response can be added to the `StreamingContent.Metadata` property. This will allow the sk developer to get the metadata even without a `StreamingConnectorResult` or `StreamingFunctionResult`.

‎dotnet/samples/KernelSyntaxExamples/Example16_CustomLLM.cs‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System;
44
using System.Collections.Generic;
55
using System.Runtime.CompilerServices;
6+
using System.Text;
67
using System.Text.Json;
78
using System.Threading;
89
using System.Threading.Tasks;
@@ -47,6 +48,37 @@ public async IAsyncEnumerable<ITextStreamingResult> GetStreamingCompletionsAsync
4748
{
4849
yield return new MyTextCompletionStreamingResult();
4950
}
51+
52+
public async IAsyncEnumerable<T> GetStreamingContentAsync<T>(string prompt, AIRequestSettings? requestSettings = null, [EnumeratorCancellation] CancellationToken cancellationToken = default)
53+
{
54+
if (typeof(T) == typeof(MyStreamingContent))
55+
{
56+
yield return (T)(object)new MyStreamingContent("llm content update 1");
57+
yield return (T)(object)new MyStreamingContent("llm content update 2");
58+
}
59+
}
60+
}
61+
62+
public class MyStreamingContent : StreamingContent
63+
{
64+
public override int ChoiceIndex => 0;
65+
66+
public string Content { get; }
67+
68+
public MyStreamingContent(string content) : base(content)
69+
{
70+
this.Content = content;
71+
}
72+
73+
public override byte[] ToByteArray()
74+
{
75+
return Encoding.UTF8.GetBytes(this.Content);
76+
}
77+
78+
public override string ToString()
79+
{
80+
return this.Content;
81+
}
5082
}
5183

5284
public class MyTextCompletionStreamingResult : ITextStreamingResult, ITextResult

0 commit comments

Comments
 (0)