Skip to content

Commit 274df9a

Browse files
authored
.Net: Add StateExtensions ADR, Abstractions, update AgentThread and add Unit tests. (#11632)
### Motivation and Context #10100 #10712 ### Description This is the first PR of a number of PRs and adds 1. A base abstraction for creating a conversation state extension 2. A class to manage multiple extensions 3. Updates to the AgentThread to use extensions 4. Unit tests 5. ADR document In follow up PRs I'll add: - Updates to Agents to use extensions on AgentThread - Individual Extension implementations ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
1 parent c36864a commit 274df9a

13 files changed

Lines changed: 1077 additions & 2 deletions
Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
---
2+
# These are optional elements. Feel free to remove any of them.
3+
status: accepted
4+
contact: westey-m
5+
date: 2025-04-17
6+
deciders: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
7+
consulted: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
8+
informed: westey-m, markwallace-microsoft, alliscode, TaoChenOSU, moonbox3, crickman
9+
---
10+
11+
# Agents with Memory
12+
13+
## What do we mean by Memory?
14+
15+
By memory we mean the capability to remember information and skills that are learned during
16+
a conversation and re-use those later in the same conversation or later in a subsequent conversation.
17+
18+
## Context and Problem Statement
19+
20+
Today we support multiple agent types with different characteristics:
21+
22+
1. In process vs remote.
23+
1. Remote agents that store and maintain conversation state in the service vs those that require the caller to provide conversation state on each invocation.
24+
25+
We need to support advanced memory capabilities across this range of agent types.
26+
27+
### Memory Scope
28+
29+
Another aspect of memory that is important to consider is the scope of different memory types.
30+
Most agent implementations have instructions and skills but the agent is not tied to a single conversation.
31+
On each invocation of the agent, the agent is told which conversation to participate in, during that invocation.
32+
33+
Memories about a user or about a conversation with a user is therefore extracted from one of these conversation and recalled
34+
during the same or another conversation with the same user.
35+
These memories will typically contain information that the user would not like to share with other users of the system.
36+
37+
Other types of memories also exist which are not tied to a specific user or conversation.
38+
E.g. an Agent may learn how to do something and be able to do that in many conversations with different users.
39+
With these type of memories there is of cousrse risk in leaking personal information between different users which is important to guard against.
40+
41+
### Packaging memory capabilities
42+
43+
All of the above memory types can be supported for any agent by attaching software components to conversation threads.
44+
This is achieved via a simple mechanism of:
45+
46+
1. Inspecting and using messages as they are passed to and from the agent.
47+
1. Passing additional context to the agent per invocation.
48+
49+
With our current `AgentThread` implementation, when an agent is invoked, all input and output messages are already passed to the `AgentThread`
50+
and can be made available to any components attached to the `AgentThread`.
51+
Where agents are remote/external and manage conversation state in the service, passing the messages to the `AgentThread` may not have any
52+
affect on the thread in the service. This is OK, since the service will have already updated the thread during the remote invocation.
53+
It does however, still allow us to subscribe to messages in any attached components.
54+
55+
For the second requirement of getting additional context per invocation, the agent may ask the thread passed to it, to in turn ask
56+
each of the components attached to it, to provide context to pass to the Agent.
57+
This enables the component to provide memories that it contains to the Agent as needed.
58+
59+
Different memory capabilities can be built using separate components. Each component would have the following characteristics:
60+
61+
1. May store some context that can be provided to the agent per invocation.
62+
1. May inspect messages from the conversation to learn from the conversation and build its context.
63+
1. May register plugins to allow the agent to directly store, retrieve, update or clear memories.
64+
65+
### Suspend / Resume
66+
67+
Building a service to host an agent comes with challenges.
68+
It's hard to build a stateful service, but service consumers expect an experience that looks stateful from the outside.
69+
E.g. on each invocation, the user expects that the service can continue a conversation they are having.
70+
71+
This means that where the the service is exposing a local agent with local conversation state management (e.g. via `ChatHistory`)
72+
that conversation state needs to be loaded and persisted for each invocation of the service.
73+
74+
It also means that any memory components that may have some in-memory state will need to be loaded and persisted too.
75+
76+
For cases like this, the `OnSuspend` and `OnResume` methods allow notification of the components that they need to save or reload their state.
77+
It is up to each of these components to decide how and where to save state to or load state from.
78+
79+
## Proposed interface for Memory Components
80+
81+
The types of events that Memory Components require are not unique to memory, and can be used to package up other capabilities too.
82+
The suggestion is therefore to create a more generally named type that can be used for other scenarios as well and can even
83+
be used for non-agent scenarios too.
84+
85+
This type should live in the `Microsoft.SemanticKernel.Abstractions` nuget, since these components can be used by systems other than just agents.
86+
87+
```csharp
88+
namespace Microsoft.SemanticKernel;
89+
90+
public abstract class ConversationStateExtension
91+
{
92+
public virtual IReadOnlyCollection<AIFunction> AIFunctions => Array.Empty<AIFunction>();
93+
94+
public virtual Task OnThreadCreatedAsync(string? threadId, CancellationToken cancellationToken = default);
95+
public virtual Task OnThreadDeleteAsync(string? threadId, CancellationToken cancellationToken = default);
96+
97+
// OnThreadCheckpointAsync not included in initial release, maybe in future.
98+
public virtual Task OnThreadCheckpointAsync(string? threadId, CancellationToken cancellationToken = default);
99+
100+
public virtual Task OnNewMessageAsync(string? threadId, ChatMessage newMessage, CancellationToken cancellationToken = default);
101+
public abstract Task<string> OnModelInvokeAsync(ICollection<ChatMessage> newMessages, CancellationToken cancellationToken = default);
102+
103+
public virtual Task OnSuspendAsync(string? threadId, CancellationToken cancellationToken = default);
104+
public virtual Task OnResumeAsync(string? threadId, CancellationToken cancellationToken = default);
105+
}
106+
```
107+
108+
> TODO: Decide about the correct namespace for `ConversationStateExtension`
109+
110+
## Managing multiple components
111+
112+
To manage multiple components I propose that we have a `ConversationStateExtensionsManager`.
113+
This class allows registering components and delegating new message notifications, ai invocation calls, etc. to the contained components.
114+
115+
## Integrating with agents
116+
117+
I propose to add a `ConversationStateExtensionsManager` to the `AgentThread` class, allowing us to attach components to any `AgentThread`.
118+
119+
When an `Agent` is invoked, we will call `OnModelInvokeAsync` on each component via the `ConversationStateExtensionsManager` to get
120+
a combined set of context to pass to the agent for this invocation. This will be internal to the `Agent` class and transparent to the user.
121+
122+
```csharp
123+
var additionalInstructions = await currentAgentThread.OnModelInvokeAsync(messages, cancellationToken).ConfigureAwait(false);
124+
```
125+
126+
## Usage examples
127+
128+
### Multiple threads using the same memory component
129+
130+
```csharp
131+
// Create a vector store for storing memories.
132+
var vectorStore = new InMemoryVectorStore();
133+
// Create a memory store that is tired to a "Memories" collection in the vector store and stores memories under the "user/12345" namespace.
134+
using var textMemoryStore = new VectorDataTextMemoryStore<string>(vectorStore, textEmbeddingService, "Memories", "user/12345", 1536);
135+
136+
// Create a memory component to will pull user facts from the conversation, store them in the vector store
137+
// and pass them to the agent as additional instructions.
138+
var userFacts = new UserFactsMemoryComponent(this.Fixture.Agent.Kernel, textMemoryStore);
139+
140+
// Create a thread and attach a Memory Component.
141+
var agentThread1 = new ChatHistoryAgentThread();
142+
agentThread1.ThreadExtensionsManager.Add(userFacts);
143+
var asyncResults1 = agent.InvokeAsync("Hello, my name is Caoimhe.", agentThread1);
144+
145+
// Create a second thread and attach a Memory Component.
146+
var agentThread2 = new ChatHistoryAgentThread();
147+
agentThread2.ThreadExtensionsManager.Add(userFacts);
148+
var asyncResults2 = agent.InvokeAsync("What is my name?.", agentThread2);
149+
// Expected response contains Caoimhe.
150+
```
151+
152+
### Using a RAG component
153+
154+
```csharp
155+
// Create Vector Store and Rag Store/Component
156+
var vectorStore = new InMemoryVectorStore();
157+
using var ragStore = new TextRagStore<string>(vectorStore, textEmbeddingService, "Memories", 1536, "group/g2");
158+
var ragComponent = new TextRagComponent(ragStore, new TextRagComponentOptions());
159+
160+
// Upsert docs into vector store.
161+
await ragStore.UpsertDocumentsAsync(
162+
[
163+
new TextRagDocument("The financial results of Contoso Corp for 2023 is as follows:\nIncome EUR 174 000 000\nExpenses EUR 152 000 000")
164+
{
165+
SourceName = "Contoso 2023 Financial Report",
166+
SourceReference = "https://www.consoso.com/reports/2023.pdf",
167+
Namespaces = ["group/g2"]
168+
}
169+
]);
170+
171+
// Create a new agent thread and register the Rag component
172+
var agentThread = new ChatHistoryAgentThread();
173+
agentThread.ThreadExtensionsManager.RegisterThreadExtension(ragComponent);
174+
175+
// Inovke the agent.
176+
var asyncResults1 = agent.InvokeAsync("What was the income of Contoso for 2023", agentThread);
177+
// Expected response contains the 174M income from the document.
178+
```
179+
180+
## Decisions to make
181+
182+
### Extension base class name
183+
184+
1. ConversationStateExtension
185+
1. Long
186+
1. MemoryComponent
187+
1. Too specific
188+
189+
Chose ConversationStateExtension.
190+
191+
### Location for abstractions
192+
193+
1. Microsoft.SemanticKernel.<baseclass>
194+
1. Microsoft.SemanticKernel.Memory.<baseclass>
195+
1. Microsoft.SemanticKernel.Memory.<baseclass> (in separate nuget)
196+
197+
Chose Microsoft.SemanticKernel.<baseclass>.
198+
199+
### Location for memory components
200+
201+
1. A nuget for each component
202+
1. Microsoft.SemanticKernel.Core nuget
203+
1. Microsoft.SemanticKernel.Memory nuget
204+
1. Microsoft.SemanticKernel.ConversationStateExtensions nuget
205+
206+
Chose Microsoft.SemanticKernel.Core nuget

‎dotnet/docs/EXPERIMENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ You can use the following diagnostic IDs to ignore warnings or errors for a part
2525
| SKEXP0100 | Advanced Semantic Kernel features |
2626
| SKEXP0110 | Semantic Kernel Agents |
2727
| SKEXP0120 | Native-AOT |
28+
| SKEXP0130 | Conversation State |
2829

2930
## Experimental Features Tracking
3031

‎dotnet/src/Agents/Abstractions/AgentThread.cs‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Copyright (c) Microsoft. All rights reserved.
22

33
using System;
4+
using System.Diagnostics.CodeAnalysis;
45
using System.Threading;
56
using System.Threading.Tasks;
67

@@ -26,6 +27,52 @@ public abstract class AgentThread
2627
/// </summary>
2728
public virtual bool IsDeleted { get; protected set; } = false;
2829

30+
/// <summary>
31+
/// Gets or sets the container for conversation state part components that manages their lifecycle and interactions.
32+
/// </summary>
33+
[Experimental("SKEXP0110")]
34+
public virtual ConversationStatePartsManager StateParts { get; init; } = new ConversationStatePartsManager();
35+
36+
/// <summary>
37+
/// Called when the current conversion is temporarily suspended and any state should be saved.
38+
/// </summary>
39+
/// <param name="cancellationToken">The <see cref="CancellationToken"/> to monitor for cancellation requests. The default is <see cref="CancellationToken.None"/>.</param>
40+
/// <returns>An async task.</returns>
41+
/// <remarks>
42+
/// In a service that hosts an agent, that is invoked via calls to the service, this might be at the end of each service call.
43+
/// In a client application, this might be when the user closes the chat window or the application.
44+
/// </remarks>
45+
[Experimental("SKEXP0110")]
46+
public virtual Task OnSuspendAsync(CancellationToken cancellationToken = default)
47+
{
48+
return this.StateParts.OnSuspendAsync(this.Id, cancellationToken);
49+
}
50+
51+
/// <summary>
52+
/// Called when the current conversion is resumed and any state should be restored.
53+
/// </summary>
54+
/// <param name="cancellationToken">The <see cref="CancellationToken"/> to monitor for cancellation requests. The default is <see cref="CancellationToken.None"/>.</param>
55+
/// <returns>An async task.</returns>
56+
/// <remarks>
57+
/// In a service that hosts an agent, that is invoked via calls to the service, this might be at the start of each service call where a previous conversation is being continued.
58+
/// In a client application, this might be when the user re-opens the chat window to resume a conversation after having previously closed it.
59+
/// </remarks>
60+
[Experimental("SKEXP0110")]
61+
public virtual Task OnResumeAsync(CancellationToken cancellationToken = default)
62+
{
63+
if (this.IsDeleted)
64+
{
65+
throw new InvalidOperationException("This thread has been deleted and cannot be used anymore.");
66+
}
67+
68+
if (this.Id is null)
69+
{
70+
throw new InvalidOperationException("This thread cannot be resumed, since it has not been created.");
71+
}
72+
73+
return this.StateParts.OnResumeAsync(this.Id, cancellationToken);
74+
}
75+
2976
/// <summary>
3077
/// Creates the thread and returns the thread id.
3178
/// </summary>
@@ -45,6 +92,10 @@ protected internal virtual async Task CreateAsync(CancellationToken cancellation
4592
}
4693

4794
this.Id = await this.CreateInternalAsync(cancellationToken: cancellationToken).ConfigureAwait(false);
95+
96+
#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
97+
await this.StateParts.OnThreadCreatedAsync(this.Id!, cancellationToken).ConfigureAwait(false);
98+
#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
4899
}
49100

50101
/// <summary>
@@ -65,6 +116,10 @@ public virtual async Task DeleteAsync(CancellationToken cancellationToken = defa
65116
throw new InvalidOperationException("This thread cannot be deleted, since it has not been created.");
66117
}
67118

119+
#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
120+
await this.StateParts.OnThreadDeleteAsync(this.Id!, cancellationToken).ConfigureAwait(false);
121+
#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
122+
68123
await this.DeleteInternalAsync(cancellationToken).ConfigureAwait(false);
69124

70125
this.IsDeleted = true;
@@ -92,6 +147,10 @@ internal virtual async Task OnNewMessageAsync(ChatMessageContent newMessage, Can
92147
await this.CreateAsync(cancellationToken).ConfigureAwait(false);
93148
}
94149

150+
#pragma warning disable SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
151+
await this.StateParts.OnNewMessageAsync(this.Id, newMessage, cancellationToken).ConfigureAwait(false);
152+
#pragma warning restore SKEXP0110 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
153+
95154
await this.OnNewMessageInternalAsync(newMessage, cancellationToken).ConfigureAwait(false);
96155
}
97156

‎dotnet/src/Agents/UnitTests/Agents.UnitTests.csproj‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
<IsTestProject>true</IsTestProject>
99
<IsPackable>false</IsPackable>
1010
<LangVersion>12</LangVersion>
11-
<NoWarn>$(NoWarn);CA2007,CA1812,CA1861,CA1063,CS0618,VSTHRD111,SKEXP0001,SKEXP0050,SKEXP0110;OPENAI001</NoWarn>
11+
<NoWarn>$(NoWarn);CA2007,CA1812,CA1861,CA1063,CS0618,VSTHRD111,SKEXP0001,SKEXP0050,SKEXP0110,SKEXP0130;OPENAI001</NoWarn>
1212
</PropertyGroup>
1313

1414
<ItemGroup>

0 commit comments

Comments
 (0)