Skip to content

Commit 516d908

Browse files
.Net: Option #3 - Refactor to support generic LLM request settings (use AIRequestSettings) (#2829)
### Motivation and Context Currently the SK core uses a model for LLM request settings which is OpenAI specific. This is a refactor to remove this assumption and to allow SK to be LLM agnostic. Resolves: #2735 ### Description 1. Remove `ChatRequestSettings` and `CompleteRequestSettings` and use new class `AIRequestSettings` instead 1. Create default OpenAI request settings class `OpenAIRequestSetings` which extends `AIRequestSettings`. 1. Fix up planner, unit tests and integration tests 1. Add a new Kernel syntax example showing all options to configure LLM request settings ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄
1 parent 9d638e5 commit 516d908

71 files changed

Lines changed: 1398 additions & 746 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
---
2+
# These are optional elements. Feel free to remove any of them.
3+
status: proposed
4+
date: 2023-=9-15
5+
deciders: shawncal
6+
consulted: stoub, lemiller, dmytrostruk
7+
informed:
8+
---
9+
# Refactor to support generic LLM request settings
10+
11+
## Context and Problem Statement
12+
13+
The Semantic Kernel abstractions package includes a number of classes ([CompleteRequestSettings](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/AI/TextCompletion/CompleteRequestSettings.cs), [ChatRequestSettings](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/AI/ChatCompletion/ChatRequestSettings.cs) [PromptTemplateConfig.CompletionConfig](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel.Abstractions/SemanticFunctions/PromptTemplateConfig.cs#L18C1-L82C6)) which are used to support:
14+
15+
1. Passing LLM request settings when invoking an AI service
16+
2. Deserialization of LLM requesting settings when loading the `config.json` associated with a Semantic Function
17+
18+
The problem with these classes is they include OpenAI specific properties only. A developer can only pass OpenAI specific requesting settings which means:
19+
20+
1. Settings may be passed that have no effect e.g., passing `MaxTokens` to Huggingface
21+
2. Settings that do not overlap with the OpenAI properties cannot be sent e.g., Oobabooga supports additional parameters e.g., `do_sample`, `typical_p`, ...
22+
23+
Link to issue raised by the implementer of the Oobabooga AI service: <https://github.com/microsoft/semantic-kernel/issues/2735>
24+
25+
## Decision Drivers
26+
27+
* Semantic Kernel abstractions must be AI Service agnostic i.e., remove OpenAI specific properties.
28+
* Solution must continue to support loading Semantic Function configuration (which includes AI request settings) from `config.json`.
29+
* Provide good experience for developers e.g., must be able to program with type safety, intellisense, etc.
30+
* Provide a good experience for implementors of AI services i.e., should be clear how to define the appropriate AI Request Settings abstraction for the service they are supporting.
31+
* Semantic Kernel implementation and sample code should avoid specifying OpenAI specific request settings in code that is intended to be used with multiple AI services.
32+
* Semantic Kernel implementation and sample code must be clear if an implementation is intended to be OpenAI specific.
33+
34+
## Considered Options
35+
36+
* Use `dynamic` to pass request settings
37+
* Use `object` to pass request settings
38+
* Define a base class for AI request settings which all implementations must extend
39+
40+
Note: Using generics was discounted during an earlier investigation which Dmytro conducted.
41+
42+
## Decision Outcome
43+
44+
**Proposed:** Define a base class for AI request settings which all implementations must extend.
45+
46+
## Pros and Cons of the Options
47+
48+
### Use `dynamic` to pass request settings
49+
50+
The `IChatCompletion` interface would look like this:
51+
52+
```csharp
53+
public interface IChatCompletion : IAIService
54+
{
55+
ChatHistory CreateNewChat(string? instructions = null);
56+
57+
Task<IReadOnlyList<IChatResult>> GetChatCompletionsAsync(
58+
ChatHistory chat,
59+
dynamic? requestSettings = null,
60+
CancellationToken cancellationToken = default);
61+
62+
IAsyncEnumerable<IChatStreamingResult> GetStreamingChatCompletionsAsync(
63+
ChatHistory chat,
64+
dynamic? requestSettings = null,
65+
CancellationToken cancellationToken = default);
66+
}
67+
```
68+
69+
Developers would have the following options to specify the requesting settings for a semantic function:
70+
71+
```csharp
72+
// Option 1: Use an anonymous type
73+
await kernel.InvokeSemanticFunctionAsync("Hello AI, what can you do for me?", requestSettings: new { MaxTokens = 256, Temperature = 0.7 });
74+
75+
// Option 2: Use an OpenAI specific class
76+
await kernel.InvokeSemanticFunctionAsync(prompt, requestSettings: new OpenAIRequestSettings() { MaxTokens = 256, Temperature = 0.7 });
77+
78+
// Option 3: Load prompt template configuration from a JSON payload
79+
string configPayload = @"{
80+
""schema"": 1,
81+
""description"": ""Say hello to an AI"",
82+
""type"": ""completion"",
83+
""completion"": {
84+
""max_tokens"": 60,
85+
""temperature"": 0.5,
86+
""top_p"": 0.0,
87+
""presence_penalty"": 0.0,
88+
""frequency_penalty"": 0.0
89+
}
90+
}";
91+
var templateConfig = JsonSerializer.Deserialize<PromptTemplateConfig>(configPayload);
92+
var func = kernel.CreateSemanticFunction(prompt, config: templateConfig!, "HelloAI");
93+
await kernel.RunAsync(func);
94+
```
95+
96+
PR: <https://github.com/microsoft/semantic-kernel/pull/2807>
97+
98+
* Good, SK abstractions contain no references to OpenAI specific request settings
99+
* Neutral, because anonymous types can be used which allows a developer to pass in properties that may be supported by multiple AI services e.g., `temperature` or combine properties for different AI services e.g., `max_tokens` (OpenAI) and `max_new_tokens` (Oobabooga).
100+
* Bad, because it's not clear to developers what they should pass when creating a semantic function
101+
* Bad, because it's not clear to implementors of a chat/text completion service what they should accept or how to add service specific properties.
102+
* Bad, there is no compiler type checking for code paths where the dynamic argument has not been resolved which will impact code quality. Type issues manifest as `RuntimeBinderException`'s and may be difficult to troubleshoot. Special care needs to be taken with return types e.g., may be necessary to specify an explicit type rather than just `var` again to avoid errors such as `Microsoft.CSharp.RuntimeBinder.RuntimeBinderException : Cannot apply indexing with [] to an expression of type 'object'`
103+
104+
### Use `object` to pass request settings
105+
106+
The `IChatCompletion` interface would look like this:
107+
108+
```csharp
109+
public interface IChatCompletion : IAIService
110+
{
111+
ChatHistory CreateNewChat(string? instructions = null);
112+
113+
Task<IReadOnlyList<IChatResult>> GetChatCompletionsAsync(
114+
ChatHistory chat,
115+
object? requestSettings = null,
116+
CancellationToken cancellationToken = default);
117+
118+
IAsyncEnumerable<IChatStreamingResult> GetStreamingChatCompletionsAsync(
119+
ChatHistory chat,
120+
object? requestSettings = null,
121+
CancellationToken cancellationToken = default);
122+
}
123+
```
124+
125+
The calling pattern is the same as for the `dynamic` case i.e. use either an anonymous type, an AI service specific class e.g., `OpenAIRequestSettings` or load from JSON.
126+
127+
PR: <https://github.com/microsoft/semantic-kernel/pull/2819>
128+
129+
* Good, SK abstractions contain no references to OpenAI specific request settings
130+
* Neutral, because anonymous types can be used which allows a developer to pass in properties that may be supported by multiple AI services e.g., `temperature` or combine properties for different AI services e.g., `max_tokens` (OpenAI) and `max_new_tokens` (Oobabooga).
131+
* Bad, because it's not clear to developers what they should pass when creating a semantic function
132+
* Bad, because it's not clear to implementors of a chat/text completion service what they should accept or how to add service specific properties.
133+
* Bad, code is needed to perform type checks and explicit casts. The situation is slightly better than for the `dynamic` case.
134+
135+
### Define a base class for AI request settings which all implementations must extend
136+
137+
The `IChatCompletion` interface would look like this:
138+
139+
```csharp
140+
public interface IChatCompletion : IAIService
141+
{
142+
ChatHistory CreateNewChat(string? instructions = null);
143+
144+
Task<IReadOnlyList<IChatResult>> GetChatCompletionsAsync(
145+
ChatHistory chat,
146+
AIRequestSettings? requestSettings = null,
147+
CancellationToken cancellationToken = default);
148+
149+
IAsyncEnumerable<IChatStreamingResult> GetStreamingChatCompletionsAsync(
150+
ChatHistory chat,
151+
AIRequestSettings? requestSettings = null,
152+
CancellationToken cancellationToken = default);
153+
}
154+
```
155+
156+
`AIRequestSettings` is defined as follows:
157+
158+
```csharp
159+
public class AIRequestSettings
160+
{
161+
/// <summary>
162+
/// Service identifier.
163+
/// </summary>
164+
[JsonPropertyName("service_id")]
165+
[JsonPropertyOrder(1)]
166+
public string? ServiceId { get; set; } = null;
167+
168+
/// <summary>
169+
/// Extra properties
170+
/// </summary>
171+
[JsonExtensionData]
172+
public Dictionary<string, object>? ExtensionData { get; set; }
173+
}
174+
```
175+
176+
Developers would have the following options to specify the requesting settings for a semantic function:
177+
178+
```csharp
179+
// Option 1: Invoke the semantic function and pass an OpenAI specific instance
180+
var result = await kernel.InvokeSemanticFunctionAsync(prompt, requestSettings: new OpenAIRequestSettings() { MaxTokens = 256, Temperature = 0.7 });
181+
Console.WriteLine(result.Result);
182+
183+
// Option 2: Load prompt template configuration from a JSON payload
184+
string configPayload = @"{
185+
""schema"": 1,
186+
""description"": ""Say hello to an AI"",
187+
""type"": ""completion"",
188+
""completion"": {
189+
""max_tokens"": 60,
190+
""temperature"": 0.5,
191+
""top_p"": 0.0,
192+
""presence_penalty"": 0.0,
193+
""frequency_penalty"": 0.0
194+
}
195+
}";
196+
var templateConfig = JsonSerializer.Deserialize<PromptTemplateConfig>(configPayload);
197+
var func = kernel.CreateSemanticFunction(prompt, config: templateConfig!, "HelloAI");
198+
199+
await kernel.RunAsync(func);
200+
```
201+
202+
It would also be possible to use the following pattern:
203+
204+
```csharp
205+
this._summarizeConversationFunction = kernel.CreateSemanticFunction(
206+
SemanticFunctionConstants.SummarizeConversationDefinition,
207+
skillName: nameof(ConversationSummarySkill),
208+
description: "Given a section of a conversation, summarize conversation.",
209+
requestSettings: new AIRequestSettings()
210+
{
211+
ExtensionData = new Dictionary<string, object>()
212+
{
213+
{ "Temperature", 0.1 },
214+
{ "TopP", 0.5 },
215+
{ "MaxTokens", MaxTokens }
216+
}
217+
});
218+
219+
```
220+
221+
The caveat with this pattern is, assuming a more specific implementation of `AIRequestSettings` uses JSON serialization/deserialization to hydrate an instance from the base `AIRequestSettings`, this will only work if all properties are supported by the default JsonConverter e.g.,
222+
223+
* If we have `MyAIRequestSettings` which includes a `Uri` property. The implementation of `MyAIRequestSettings` would make sure to load a URI converter so that it can serialize/deserialize the settings correctly.
224+
* If the settings for `MyAIRequestSettings` are sent to an AI service which relies on the default JsonConverter then a `NotSupportedException` exception will be thrown.
225+
226+
PR: <https://github.com/microsoft/semantic-kernel/pull/2829>
227+
228+
* Good, SK abstractions contain no references to OpenAI specific request settings
229+
* Good, because it is clear to developers what they should pass when creating a semantic function and it is easy to discover what service specific request setting implementations exist.
230+
* Good, because it is clear to implementors of a chat/text completion service what they should accept and how to extend the base abstraction to add service specific properties.
231+
* Neutral, because `ExtensionData` can be used which allows a developer to pass in properties that may be supported by multiple AI services e.g., `temperature` or combine properties for different AI services e.g., `max_tokens` (OpenAI) and `max_new_tokens` (Oobabooga).

‎dotnet/SK-dotnet.sln‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
4040
EndProject
4141
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SemanticKernel.UnitTests", "src\SemanticKernel.UnitTests\SemanticKernel.UnitTests.csproj", "{37E39C68-5A40-4E63-9D3C-0C66AD98DFCB}"
4242
EndProject
43-
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "functions", "function", "{9ECD1AA0-75B3-4E25-B0B5-9F0945B64974}"
43+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "functions", "functions", "{9ECD1AA0-75B3-4E25-B0B5-9F0945B64974}"
4444
EndProject
4545
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Functions.UnitTests", "src\Functions\Functions.UnitTests\Functions.UnitTests.csproj", "{107156B4-5A8B-45C7-97A2-4544D7FA19DE}"
4646
EndProject

‎dotnet/samples/KernelSyntaxExamples/Example05_InlineFunctionDefinition.cs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System;
44
using System.Threading.Tasks;
55
using Microsoft.SemanticKernel;
6+
using Microsoft.SemanticKernel.Connectors.AI.OpenAI;
67
using RepoUtils;
78

89
// ReSharper disable once InconsistentNaming
@@ -48,15 +49,15 @@ Be creative and be funny. Let your imagination run wild.
4849
Event: {{$input}}
4950
";
5051

51-
var excuseFunction = kernel.CreateSemanticFunction(FunctionDefinition, maxTokens: 100, temperature: 0.4, topP: 1);
52+
var excuseFunction = kernel.CreateSemanticFunction(FunctionDefinition, requestSettings: new OpenAIRequestSettings() { MaxTokens = 100, Temperature = 0.4, TopP = 1 });
5253

5354
var result = await kernel.RunAsync("I missed the F1 final race", excuseFunction);
5455
Console.WriteLine(result);
5556

5657
result = await kernel.RunAsync("sorry I forgot your birthday", excuseFunction);
5758
Console.WriteLine(result);
5859

59-
var fixedFunction = kernel.CreateSemanticFunction($"Translate this date {DateTimeOffset.Now:f} to French format", maxTokens: 100);
60+
var fixedFunction = kernel.CreateSemanticFunction($"Translate this date {DateTimeOffset.Now:f} to French format", requestSettings: new OpenAIRequestSettings() { MaxTokens = 100 });
6061

6162
result = await kernel.RunAsync(fixedFunction);
6263
Console.WriteLine(result);

‎dotnet/samples/KernelSyntaxExamples/Example06_TemplateLanguage.cs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System;
44
using System.Threading.Tasks;
55
using Microsoft.SemanticKernel;
6+
using Microsoft.SemanticKernel.Connectors.AI.OpenAI;
67
using Microsoft.SemanticKernel.Plugins.Core;
78
using Microsoft.SemanticKernel.TemplateEngine.Prompt;
89
using RepoUtils;
@@ -55,7 +56,7 @@ Is it weekend time (weekend/not weekend)?
5556
Console.WriteLine(renderedPrompt);
5657

5758
// Run the prompt / semantic function
58-
var kindOfDay = kernel.CreateSemanticFunction(FunctionDefinition, maxTokens: 150);
59+
var kindOfDay = kernel.CreateSemanticFunction(FunctionDefinition, requestSettings: new OpenAIRequestSettings() { MaxTokens = 100 });
5960

6061
// Show the result
6162
Console.WriteLine("--- Semantic Function result");

‎dotnet/samples/KernelSyntaxExamples/Example07_BingAndGoogleSkills.cs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System;
44
using System.Threading.Tasks;
55
using Microsoft.SemanticKernel;
6+
using Microsoft.SemanticKernel.Connectors.AI.OpenAI;
67
using Microsoft.SemanticKernel.Plugins.Web;
78
using Microsoft.SemanticKernel.Plugins.Web.Bing;
89
using Microsoft.SemanticKernel.Plugins.Web.Google;
@@ -137,7 +138,7 @@ [END OF EXAMPLES]
137138
var questions = "Who is the most followed person on TikTok right now? What's the exchange rate EUR:USD?";
138139
Console.WriteLine(questions);
139140

140-
var oracle = kernel.CreateSemanticFunction(SemanticFunction, maxTokens: 200, temperature: 0, topP: 1);
141+
var oracle = kernel.CreateSemanticFunction(SemanticFunction, requestSettings: new OpenAIRequestSettings() { MaxTokens = 150, Temperature = 0, TopP = 1 });
141142

142143
var answer = await kernel.RunAsync(oracle, new(questions)
143144
{

‎dotnet/samples/KernelSyntaxExamples/Example10_DescribeAllSkillsAndFunctions.cs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
using System;
44
using System.Threading.Tasks;
55
using Microsoft.SemanticKernel;
6+
using Microsoft.SemanticKernel.Connectors.AI.OpenAI;
67
using Microsoft.SemanticKernel.Plugins.Core;
78
using Microsoft.SemanticKernel.SkillDefinition;
89
using RepoUtils;
@@ -39,15 +40,15 @@ public static Task RunAsync()
3940
kernel.ImportSemanticSkillFromDirectory(folder, "SummarizeSkill");
4041

4142
// Define a semantic function inline, without naming
42-
var sFun1 = kernel.CreateSemanticFunction("tell a joke about {{$input}}", maxTokens: 150);
43+
var sFun1 = kernel.CreateSemanticFunction("tell a joke about {{$input}}", requestSettings: new OpenAIRequestSettings() { MaxTokens = 150 });
4344

4445
// Define a semantic function inline, with skill name
4546
var sFun2 = kernel.CreateSemanticFunction(
4647
"write a novel about {{$input}} in {{$language}} language",
4748
skillName: "Writing",
4849
functionName: "Novel",
4950
description: "Write a bedtime story",
50-
maxTokens: 150);
51+
requestSettings: new OpenAIRequestSettings() { MaxTokens = 150 });
5152

5253
var functions = kernel.Skills.GetFunctionViews();
5354

‎dotnet/samples/KernelSyntaxExamples/Example15_TextMemorySkill.cs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
using System.Threading;
55
using System.Threading.Tasks;
66
using Microsoft.SemanticKernel;
7+
using Microsoft.SemanticKernel.Connectors.AI.OpenAI;
78
using Microsoft.SemanticKernel.Connectors.AI.OpenAI.TextEmbedding;
89
using Microsoft.SemanticKernel.Connectors.Memory.AzureCognitiveSearch;
910
using Microsoft.SemanticKernel.Connectors.Memory.Chroma;
@@ -281,7 +282,7 @@ END FACTS
281282
Answer:
282283
";
283284

284-
var aboutMeOracle = kernel.CreateSemanticFunction(RecallFunctionDefinition, maxTokens: 100);
285+
var aboutMeOracle = kernel.CreateSemanticFunction(RecallFunctionDefinition, requestSettings: new OpenAIRequestSettings() { MaxTokens = 100 });
285286

286287
result = await kernel.RunAsync(aboutMeOracle, new()
287288
{

‎dotnet/samples/KernelSyntaxExamples/Example16_CustomLLM.cs‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
using System.Threading;
88
using System.Threading.Tasks;
99
using Microsoft.SemanticKernel;
10+
using Microsoft.SemanticKernel.AI;
1011
using Microsoft.SemanticKernel.AI.TextCompletion;
12+
using Microsoft.SemanticKernel.Connectors.AI.OpenAI;
1113
using Microsoft.SemanticKernel.Orchestration;
1214
using RepoUtils;
1315

@@ -27,15 +29,15 @@
2729
*/
2830
public class MyTextCompletionService : ITextCompletion
2931
{
30-
public Task<IReadOnlyList<ITextResult>> GetCompletionsAsync(string text, CompleteRequestSettings requestSettings, CancellationToken cancellationToken = default)
32+
public Task<IReadOnlyList<ITextResult>> GetCompletionsAsync(string text, AIRequestSettings? requestSettings, CancellationToken cancellationToken = default)
3133
{
3234
return Task.FromResult<IReadOnlyList<ITextResult>>(new List<ITextResult>
3335
{
3436
new MyTextCompletionStreamingResult()
3537
});
3638
}
3739

38-
public async IAsyncEnumerable<ITextStreamingResult> GetStreamingCompletionsAsync(string text, CompleteRequestSettings requestSettings, [EnumeratorCancellation] CancellationToken cancellationToken = default)
40+
public async IAsyncEnumerable<ITextStreamingResult> GetStreamingCompletionsAsync(string text, AIRequestSettings? requestSettings, [EnumeratorCancellation] CancellationToken cancellationToken = default)
3941
{
4042
yield return new MyTextCompletionStreamingResult();
4143
}
@@ -125,7 +127,7 @@ private static async Task CustomTextCompletionAsync()
125127
Console.WriteLine("======== Custom LLM - Text Completion - Raw ========");
126128
var completionService = new MyTextCompletionService();
127129

128-
var result = await completionService.CompleteAsync("I missed the training session this morning", new CompleteRequestSettings());
130+
var result = await completionService.CompleteAsync("I missed the training session this morning");
129131

130132
Console.WriteLine(result);
131133
}
@@ -143,7 +145,7 @@ private static async Task CustomTextCompletionStreamAsync()
143145

144146
private static async Task TextCompletionStreamAsync(string prompt, ITextCompletion textCompletion)
145147
{
146-
var requestSettings = new CompleteRequestSettings()
148+
var requestSettings = new OpenAIRequestSettings()
147149
{
148150
MaxTokens = 100,
149151
FrequencyPenalty = 0,

0 commit comments

Comments
 (0)