.Net: Kernel Streaming Support by rogerbarreto · Pull Request #2986 · microsoft/semantic-kernel

Roger Barreto (rogerbarreto) · 2023-09-26T09:19:40Z

Motivation and Context

As a long-awaited feature request, this new allows streaming when calling Kernel.RunAsync or SKFunctions.InvokeAsync.
Resolves .Net: Streaming support for semantic functions on IKernel #1649
Resolves .NET Allow ISKFunction to return IAsyncEnumerable #1298
Adjusted Completion abstract interface methods/name following a common pattern
Added streaming component interface specializations.

Description

⚠️ Breaking Change

Streaming behavior by default is false since streaming responses from providers like OpenAI/Azure OpenAI won't give additional data like (PromptFilterResults, Usage(Completion Tokens, Prompt Tokens, Total Tokens, ...).

To enable streaming behavior set Streaming = true in a AIRequestSettings specialized class like OpenAIRequestSettings, once this is set for text/chat based models like OpenAi's GPT you can get streaming data using GetValue<IAsyncEnumerable<string>>() in the result of Kernel.RunAsync or Function.InvokeAsync.

How to:

myFunction = kernel.CreateSemanticFunction($"your prompt",
        requestSettings: new OpenAIRequestSettings
        {
            Streaming = true,
        });

var result = await kernel.RunAsync(myFunction);
await foreach (string token in result.GetValue<IAsyncEnumerable<string>>())
{
    Console.Write(token);
}

ℹ️ Any attempt on using SKContext from FunctionResults will cancel the streaming behavior as those require the full content to be available.

New Streaming flag added to bool AIRequestSettings.Streaming abstraction, this will signal that the caller has intention on use streaming when available.

New streaming interfaces added:

ITextStreamingCompletion
IChatStreamingCompletion

Changes to the method names following the pattern (which service is it and what it is returning)

IText(Streaming)Completion
- Old: List<ITextResult> GetCompletionsAsync
  New: GetTextResultsAsync
- Old: IAsyncEnumerable<ITextStreamingResult> GetStreamingCompletionsAsync
  New: GetTextStreamingResultsAsync
IText(Streaming)Result
- Old: Task<string> GetCompletionAsync
  New: GetTextAsync
- Old: IAsyncEnumerable<ITextStreamingResult> GetStreamingCompletionsAsync
  New: GetTextStreamingResultsAsync
IChat(Streaming)Completion
- Old: Task<IReadOnlyList<IChatResult>> GetChatCompletionsAsync
  New: GetChatResultsAsync
- Old: IAsyncEnumerable<IChatStreamingResult> GetStreamingChatCompletionsAsync
  New: GetChatStreamingResultsAsync

Services following this abstraction signales to Kernel it can use streaming when requested.

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

…AI streaming

Lisa Harrylock (hario90) · 2023-09-26T15:52:01Z

+                    MaxTokens = 1000
+                });
+
+        await foreach (string token in (await kernel.RunAsync(fixedFunction)).GetValue<IAsyncEnumerable<string>>()!)


nit: I would break this up the way you did in Example 32 for better readability, like:

Suggested change

await foreach (string token in (await kernel.RunAsync(fixedFunction)).GetValue<IAsyncEnumerable<string>>()!)

var result = await kernel.RunAsync(fixedFunction);

await foreach (string token in result.GetValue<IAsyncEnumerable<string>>()!)

Lisa Harrylock (hario90) · 2023-09-27T16:59:59Z

+        Console.WriteLine("======== Azure OpenAI - Multiple Text Completion - Raw Streaming ========");

-        var chatCompletion = new AzureChatCompletion(
+        var textCompletion = new AzureChatCompletion(


The type of completion didn't change to chat completion. - did you intend to change the type as well? If so - I was wondering if it makes sense to do this given that the text completion models are getting deprecated.

Lisa Harrylock (hario90) · 2023-09-27T17:08:49Z

+
+namespace Microsoft.SemanticKernel.Connectors.AI.OpenAI.AzureSdk;
+
+/// <summary> Represents a singular result of a chat completion.</summary>


Is this description accurate?

Lisa Harrylock (hario90) · 2023-09-27T17:09:10Z

+/// <summary> Represents a singular result of a chat completion.</summary>
+public class StreamingTextModelResult
+{
+    /// <summary> A unique identifier associated with this chat completion response. </summary>


Suggested change

/// <summary> A unique identifier associated with this chat completion response. </summary>

/// <summary> A unique identifier associated with this text completion response. </summary>

Lisa Harrylock (hario90) · 2023-09-27T17:12:41Z

    }

+    /// <summary>
+    /// Retrieves a typed <see cref="Completions"/> OpenAI / AzureOpenAI result from text completion prompt.


It seems confusing that this returns the AzureOpenAI result type. Is this what you meant?

/// Retrieves a <see cref="StreamingTextModelResult "/> from the model result.

Lisa Harrylock (hario90) · 2023-09-27T17:17:24Z

+    {
+        get
+        {
+            if (this.ExtensionData.TryGetValue("results_per_prompt", out var value))


suggestion: use a const for the string literal

Dmytro Struk (dmytrostruk) · 2023-09-28T10:05:17Z

+        var prompt = "Write one paragraph why AI is awesome";
+        var consoleLinesPerResult = 12;


nit: can be const

Dmytro Struk (dmytrostruk) · 2023-09-28T10:17:47Z

+    /// </summary>
+    public StreamingChatChoice Choice { get; }
+
+    /// <summary> Initializes a new instance of ChatModelResult. </summary>


Suggested change

/// <summary> Initializes a new instance of ChatModelResult. </summary>

/// <summary> Initializes a new instance of <see cref="StreamingChatModelResult"/>. </summary>

Dmytro Struk (dmytrostruk) · 2023-09-28T10:19:56Z

+    /// </summary>
+    public StreamingChoice Choice { get; }
+
+    /// <summary> Initializes a new instance of StreamingTextModelResult. </summary>


Suggested change

/// <summary> Initializes a new instance of StreamingTextModelResult. </summary>

/// <summary> Initializes a new instance of <see cref="StreamingTextModelResult"/>. </summary>

Dmytro Struk (dmytrostruk) · 2023-09-28T10:28:31Z

+    /// <summary>
+    /// Enables streaming behavior when available
+    /// </summary>
+    public bool? Streaming { get; set; } = false;


I would suggest to rename this property to clearly show that it's boolean flag (e.g. IsStreamingEnabled or UseStreaming). It will be easier to understand how to use it (settings.IsStreamingEnabled = true is more clear than settings.Streaming = ?).

Dmytro Struk (dmytrostruk) · 2023-09-28T10:29:52Z

+namespace Microsoft.SemanticKernel.AI.ChatCompletion;
+
+/// <summary>
+/// Provides extension methods for the IChatCompletion interface.


Suggested change

/// Provides extension methods for the IChatCompletion interface.

/// Provides extension methods for the <see cref="IChatStreamingCompletion"/> interface.

Dmytro Struk (dmytrostruk) · 2023-09-28T13:01:43Z

-            result = new FunctionResult(this.Name, this.PluginName, context, completion);
-
-            var modelResults = completionResults.Select(c => c.ModelResult).ToArray();
+            if (requestSettings?.Streaming == true &&


I would suggest to move streaming and non-streaming logic to separate private methods to keep method RunPromptAsync as simple as possible.

Dmytro Struk (dmytrostruk) · 2023-09-28T13:28:07Z

+    {
+        get
+        {
+            if (this.ExtensionData.TryGetValue("results_per_prompt", out var value))


If this property is applicable to general SemanticFunction logic (Method GetResultsCountOrDefault), we can:

Move ResultsPerPrompt parameter to base AIRequestSettings with default value 1 - although it will mean that this parameter is applicable to every AI connector, which is not the case.

Place logic for multiple results only where it's applicable - move it from SemanticFunction to OpenAI Connector.

Remove this property from OpenAIRequestSettings completely and place it as parameter of function. (I had an idea to replace SKContext as input parameter for function/kernel with FunctionRequest/KernelRequest, and property ResultsPerPrompt can be good first candidate for request models, but this is definitely separate scope of work and PR.)

Main point here, that we should not use ExtensionData as a way of storing general settings for functions and accessing them from Kernel/Function logic. As far as I understand, it should be used by Connectors only.

Dmytro Struk (dmytrostruk) · 2023-09-28T13:35:47Z


                functionResult = await skFunction.InvokeAsync(context, cancellationToken: cancellationToken).ConfigureAwait(false);

-                context = functionResult.Context;


This is breaking change with unknown outcomes. That's the reason why property Context still exists in FunctionResult model internally. Because, some functions may update context inside (since context is mutable at the moment), but some of them can return new cloned context instance with updated parameters (that's probably the way how it should be), which means that next function in pipeline won't receive updated variables if we remove this line.

We should investigate and test all available scenarios to understand if we can safely do this change and if we need to do it at all.

Dmytro Struk (dmytrostruk) · 2023-09-28T13:42:15Z

        var templateConfig = new PromptTemplateConfig();
+        if (useStream is not null)
+        {
+            templateConfig.Completion = new() { Streaming = useStream };


As mentioned above, name Streaming here doesn't show that it's boolean flag, that's why this test contains parameter useStream. I would prefer to use the same naming in settings and tests, like UseStreaming or IsStreamingEnabled.

Dmytro Struk (dmytrostruk) · 2023-09-28T13:45:43Z

+        var functionResult2 = kernelResult.FunctionResults.First(l => l.FunctionName == "Function2" && l.PluginName == PluginName);
+        var functionResult3 = kernelResult.FunctionResults.First(l => l.FunctionName == "Function3" && l.PluginName == PluginName);
+
+        var function3Result = functionResult3.GetValue<string>();


Do we need this result in separate variable?

Roger Barreto (rogerbarreto) · 2023-11-07T22:49:44Z

Dropping this change for now, as I will be working in potentially other decision regarding the Streaming approach for v1

Roger Barreto (rogerbarreto) added 8 commits September 25, 2023 15:56

Semantic functions defaulting to streaming

bbceac7

All integration tests passing

fe248b1

Removing unused code

197a5f6

Warnings fix

d084799

Adding streaming examples

b52c2e1

Adding more examples and streaming flag, and new ModelResuts for Open…

46f4b14

…AI streaming

Warnings fix

0a94021

Adjusting UT for the optional Streaming flag

f3eb459

Roger Barreto (rogerbarreto) requested a review from a team as a code owner September 26, 2023 09:19

Shawn Callegari (shawncal) added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel kernel.core labels Sep 26, 2023

github-actions Bot changed the title ~~Kernel Streaming Support - Proof of Concept~~ .Net: Kernel Streaming Support - Proof of Concept Sep 26, 2023

Roger Barreto (rogerbarreto) added 7 commits September 26, 2023 10:32

Removing unintended changes

6a5f332

Merge Fix

6e73371

Bug identified

9d67e8d

WIP

209282e

Splitting interfaces regarding Streaming support

9440d44

Warning fixes, UT + IT passing

6d3c737

Merge branch 'main' into features/kernel-streaming-poc1

974dc4d

Roger Barreto (rogerbarreto) changed the title ~~.Net: Kernel Streaming Support - Proof of Concept~~ .Net: Kernel Streaming Support Sep 26, 2023

Roger Barreto (rogerbarreto) added the PR: ready for review All feedback addressed, ready for reviews label Sep 26, 2023

Roger Barreto (rogerbarreto) self-assigned this Sep 26, 2023

Roger Barreto (rogerbarreto) added this to the v1 milestone Sep 26, 2023

This was linked to issues Sep 26, 2023

.Net: Streaming support for semantic functions on IKernel #1649

Closed

.NET Allow ISKFunction to return IAsyncEnumerable #1298

Closed

Roger Barreto (rogerbarreto) added the PR: breaking change Pull requests that introduce breaking changes label Sep 26, 2023

Lisa Harrylock (hario90) reviewed Sep 27, 2023

View reviewed changes

Dmytro Struk (dmytrostruk) reviewed Sep 28, 2023

View reviewed changes

Dmytro Struk (dmytrostruk) mentioned this pull request Sep 28, 2023

.Net: Token usage on all interactions with SK where LLM is engaged #2987

Closed

Roger Barreto (rogerbarreto) closed this Nov 7, 2023

Roger Barreto (rogerbarreto) deleted the features/kernel-streaming-poc1 branch December 5, 2023 13:43

	await foreach (string token in (await kernel.RunAsync(fixedFunction)).GetValue<IAsyncEnumerable<string>>()!)
	var result = await kernel.RunAsync(fixedFunction);
	await foreach (string token in result.GetValue<IAsyncEnumerable<string>>()!)


		namespace Microsoft.SemanticKernel.Connectors.AI.OpenAI.AzureSdk;

		/// <summary> Represents a singular result of a chat completion.</summary>

	/// <summary> A unique identifier associated with this chat completion response. </summary>
	/// <summary> A unique identifier associated with this text completion response. </summary>

		var prompt = "Write one paragraph why AI is awesome";
		var consoleLinesPerResult = 12;

	/// <summary> Initializes a new instance of ChatModelResult. </summary>
	/// <summary> Initializes a new instance of <see cref="StreamingChatModelResult"/>. </summary>

	/// Provides extension methods for the IChatCompletion interface.
	/// Provides extension methods for the <see cref="IChatStreamingCompletion"/> interface.


		functionResult = await skFunction.InvokeAsync(context, cancellationToken: cancellationToken).ConfigureAwait(false);

		context = functionResult.Context;

Conversation

Roger Barreto (rogerbarreto) commented Sep 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Description

Contribution Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dmytro Struk (dmytrostruk) Sep 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Roger Barreto (rogerbarreto) commented Nov 7, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Roger Barreto (rogerbarreto) commented Sep 26, 2023 •

edited

Loading

Dmytro Struk (dmytrostruk) Sep 28, 2023 •

edited

Loading