Skip to content

Commit a7a56e5

Browse files
esttenoriocrickman
andauthored
.Net: Processes State Management Part 2 (#9481)
### Description - Moving `ProcessStateMetadata` logic to factory/extension to try to keep clean existing core components - Adding check of step name when adding new steps/processes to process - names must be unique so when applying saved state it propagates properly. Currently there are issues if multiple steps have the same name - State versioning initial support: - Processes with 1:1 step match previous and new version (step name change and/or step logic change) - minor changes on process steps and not in root process itself Fixing #9358 #### Out of scope but will be addressed in next PRs - Processes with N:M steps (process flow changed and potentially also step logic changed) - New samples showcasing how to make use of N:M mapping ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Chris <[email protected]>
1 parent 76e904c commit a7a56e5

35 files changed

Lines changed: 683 additions & 186 deletions

‎dotnet/samples/GettingStartedWithProcesses/README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,56 @@ flowchart RL
271271
end
272272
```
273273

274+
## Concepts
275+
276+
### Components
277+
278+
- **Process:** A sequence of steps designed to achieve a specific goal. These steps are interconnected in such a way that they can communicate by sending and receiving events. The connections between the steps are established during the process creation.
279+
- **Steps:** Individual activities within a process, each with defined inputs and outputs, contributing to the overall objective. Existing processes can be utilized as steps within another process. There are two main types of steps:
280+
- _Stateless Steps_: These steps do not retain any information between executions. They operate independently without the need to store state data.
281+
- _Stateful Steps_: These steps maintain a state that can be persisted, allowing the state to be reused and updated in subsequent runs of the process. The state of these steps can be stored and serialized.
282+
283+
In general, both processes and steps are designed to be reusable across different processes.
284+
285+
### Versioning
286+
287+
Once stateful steps/processes have been deployed, versioning becomes a crucial aspect to understand.
288+
It enables you to tweak and improve processes while maintaining the ability to read step states generated by previous versions of the steps.
289+
290+
Stateful processes involve steps that maintain state information.
291+
When these processes are updated, it's important to manage versioning effectively to ensure continuity and compatibility with previously saved states.
292+
293+
There are two primary scenarios to consider when addressing process state versioning:
294+
295+
1. **Minor SK Process Improvements/Changes:**
296+
297+
In this scenario, the root process remains conceptually the same, but with some modifications:
298+
299+
- _Step Renaming:_ Some step names may have been changed.
300+
- _Step Version Updates:_ New versions of one or more steps used by the root process or any steps in a subprocess may be introduced.
301+
302+
**Considerations:**
303+
304+
- Ensure backward compatibility by mapping old step names to new step names.
305+
- Validate that the new step versions can read and interpret the state data generated by previous versions.
306+
307+
**Related Samples:**
308+
309+
- `Step03a_FoodPreparation.cs/RunStatefulFriedFishV2ProcessWithLowStockV1StateFromFileAsync`
310+
- `Step03a_FoodPreparation.cs/RunStatefulFishSandwichV2ProcessWithLowStockV1StateFromFileAsync`
311+
312+
2. **Major SK Process Improvements/Changes:**
313+
314+
This scenario involves significant modifications to the root process, which may include:
315+
316+
- _Step Refactoring_: Multiple steps may be refactored and replaced. However, some properties of the replaced steps can be used to set properties of the new steps.
317+
- _Custom Mappings:_ Custom equivalent mappings may be required to translate the previous stored state to the current process state.
318+
319+
**Considerations:**
320+
321+
- Develop a detailed mapping strategy to align old and new process states.
322+
- Implement and test custom mappings to ensure data integrity and process continuity.
323+
274324
## Running Examples with Filters
275325
Examples may be explored and ran within _Visual Studio_ using _Test Explorer_.
276326

‎dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FishAndChipsProcess.cs‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ public static class ProcessEvents
1717
{
1818
public const string PrepareFishAndChips = nameof(PrepareFishAndChips);
1919
public const string FishAndChipsReady = nameof(FishAndChipsReady);
20+
public const string FishAndChipsIngredientOutOfStock = nameof(FishAndChipsIngredientOutOfStock);
2021
}
2122

2223
public static ProcessBuilder CreateProcess(string processName = "FishAndChipsProcess")
@@ -48,6 +49,35 @@ public static ProcessBuilder CreateProcess(string processName = "FishAndChipsPro
4849
return processBuilder;
4950
}
5051

52+
public static ProcessBuilder CreateProcessWithStatefulSteps(string processName = "FishAndChipsWithStatefulStepsProcess")
53+
{
54+
var processBuilder = new ProcessBuilder(processName);
55+
var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulStepsV1());
56+
var makePotatoFriesStep = processBuilder.AddStepFromProcess(PotatoFriesProcess.CreateProcessWithStatefulSteps());
57+
var addCondimentsStep = processBuilder.AddStepFromType<AddFishAndChipsCondimentsStep>();
58+
// An additional step that is the only one that emits an public event in a process can be added to maintain event names unique
59+
var externalStep = processBuilder.AddStepFromType<ExternalFishAndChipsStep>();
60+
61+
processBuilder
62+
.OnInputEvent(ProcessEvents.PrepareFishAndChips)
63+
.SendEventTo(makeFriedFishStep.WhereInputEventIs(FriedFishProcess.ProcessEvents.PrepareFriedFish))
64+
.SendEventTo(makePotatoFriesStep.WhereInputEventIs(PotatoFriesProcess.ProcessEvents.PreparePotatoFries));
65+
66+
makeFriedFishStep
67+
.OnEvent(FriedFishProcess.ProcessEvents.FriedFishReady)
68+
.SendEventTo(new ProcessFunctionTargetBuilder(addCondimentsStep, parameterName: "fishActions"));
69+
70+
makePotatoFriesStep
71+
.OnEvent(PotatoFriesProcess.ProcessEvents.PotatoFriesReady)
72+
.SendEventTo(new ProcessFunctionTargetBuilder(addCondimentsStep, parameterName: "potatoActions"));
73+
74+
addCondimentsStep
75+
.OnEvent(AddFishAndChipsCondimentsStep.OutputEvents.CondimentsAdded)
76+
.SendEventTo(new ProcessFunctionTargetBuilder(externalStep));
77+
78+
return processBuilder;
79+
}
80+
5181
private sealed class AddFishAndChipsCondimentsStep : KernelProcessStep
5282
{
5383
public static class Functions

‎dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FishSandwichProcess.cs‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,10 +46,38 @@ public static ProcessBuilder CreateProcess(string processName = "FishSandwichPro
4646
return processBuilder;
4747
}
4848

49-
public static ProcessBuilder CreateProcessWithStatefulSteps(string processName = "FishSandwichWithStatefulStepsProcess")
49+
public static ProcessBuilder CreateProcessWithStatefulStepsV1(string processName = "FishSandwichWithStatefulStepsProcess")
5050
{
51-
var processBuilder = new ProcessBuilder(processName);
52-
var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulSteps());
51+
var processBuilder = new ProcessBuilder(processName) { Version = "FishSandwich.V1" };
52+
var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulStepsV1());
53+
var addBunsStep = processBuilder.AddStepFromType<AddBunsStep>();
54+
var addSpecialSauceStep = processBuilder.AddStepFromType<AddSpecialSauceStep>();
55+
// An additional step that is the only one that emits an public event in a process can be added to maintain event names unique
56+
var externalStep = processBuilder.AddStepFromType<ExternalFriedFishStep>();
57+
58+
processBuilder
59+
.OnInputEvent(ProcessEvents.PrepareFishSandwich)
60+
.SendEventTo(makeFriedFishStep.WhereInputEventIs(FriedFishProcess.ProcessEvents.PrepareFriedFish));
61+
62+
makeFriedFishStep
63+
.OnEvent(FriedFishProcess.ProcessEvents.FriedFishReady)
64+
.SendEventTo(new ProcessFunctionTargetBuilder(addBunsStep));
65+
66+
addBunsStep
67+
.OnEvent(AddBunsStep.OutputEvents.BunsAdded)
68+
.SendEventTo(new ProcessFunctionTargetBuilder(addSpecialSauceStep));
69+
70+
addSpecialSauceStep
71+
.OnEvent(AddSpecialSauceStep.OutputEvents.SpecialSauceAdded)
72+
.SendEventTo(new ProcessFunctionTargetBuilder(externalStep));
73+
74+
return processBuilder;
75+
}
76+
77+
public static ProcessBuilder CreateProcessWithStatefulStepsV2(string processName = "FishSandwichWithStatefulStepsProcess")
78+
{
79+
var processBuilder = new ProcessBuilder(processName) { Version = "FishSandwich.V2" };
80+
var makeFriedFishStep = processBuilder.AddStepFromProcess(FriedFishProcess.CreateProcessWithStatefulStepsV2("FriedFishStep"), aliases: ["FriedFishWithStatefulStepsProcess"]);
5381
var addBunsStep = processBuilder.AddStepFromType<AddBunsStep>();
5482
var addSpecialSauceStep = processBuilder.AddStepFromType<AddSpecialSauceStep>();
5583
// An additional step that is the only one that emits an public event in a process can be added to maintain event names unique

‎dotnet/samples/GettingStartedWithProcesses/Step03/Processes/FriedFishProcess.cs‎

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Copyright (c) Microsoft. All rights reserved.
22

33
using Microsoft.SemanticKernel;
4+
using Microsoft.SemanticKernel.Process;
45
using Step03.Models;
56
using Step03.Steps;
67
namespace Step03.Processes;
@@ -16,7 +17,7 @@ public static class ProcessEvents
1617
// When multiple processes use the same final step, the should event marked as public
1718
// so that the step event can be used as the output event of the process too.
1819
// In these samples both fried fish and potato fries end with FryStep success
19-
public const string FriedFishReady = nameof(FryFoodStep.OutputEvents.FriedFoodReady);
20+
public const string FriedFishReady = FryFoodStep.OutputEvents.FriedFoodReady;
2021
}
2122

2223
/// <summary>
@@ -30,7 +31,7 @@ public static ProcessBuilder CreateProcess(string processName = "FriedFishProces
3031
var processBuilder = new ProcessBuilder(processName);
3132

3233
var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsStep>();
33-
var chopStep = processBuilder.AddStepFromType<CutFoodStep>("chopStep");
34+
var chopStep = processBuilder.AddStepFromType<CutFoodStep>();
3435
var fryStep = processBuilder.AddStepFromType<FryFoodStep>();
3536

3637
processBuilder
@@ -52,19 +53,48 @@ public static ProcessBuilder CreateProcess(string processName = "FriedFishProces
5253
return processBuilder;
5354
}
5455

56+
public static ProcessBuilder CreateProcessWithStatefulStepsV1(string processName = "FriedFishWithStatefulStepsProcess")
57+
{
58+
// It is recommended to specify process version in case this process is used as a step by another process
59+
var processBuilder = new ProcessBuilder(processName) { Version = "FriedFishProcess.v1" }; ;
60+
61+
var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsWithStockStep>();
62+
var chopStep = processBuilder.AddStepFromType<CutFoodStep>();
63+
var fryStep = processBuilder.AddStepFromType<FryFoodStep>();
64+
65+
processBuilder
66+
.OnInputEvent(ProcessEvents.PrepareFriedFish)
67+
.SendEventTo(new ProcessFunctionTargetBuilder(gatherIngredientsStep));
68+
69+
gatherIngredientsStep
70+
.OnEvent(GatherFriedFishIngredientsWithStockStep.OutputEvents.IngredientsGathered)
71+
.SendEventTo(new ProcessFunctionTargetBuilder(chopStep, functionName: CutFoodWithSharpeningStep.Functions.ChopFood));
72+
73+
chopStep
74+
.OnEvent(CutFoodWithSharpeningStep.OutputEvents.ChoppingReady)
75+
.SendEventTo(new ProcessFunctionTargetBuilder(fryStep));
76+
77+
fryStep
78+
.OnEvent(FryFoodStep.OutputEvents.FoodRuined)
79+
.SendEventTo(new ProcessFunctionTargetBuilder(gatherIngredientsStep));
80+
81+
return processBuilder;
82+
}
83+
5584
/// <summary>
5685
/// For a visual reference of the FriedFishProcess with stateful steps check this
5786
/// <see href="https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/GettingStartedWithProcesses/README.md#fried-fish-preparation-with-knife-sharpening-and-ingredient-stock-process" >diagram</see>
5887
/// </summary>
5988
/// <param name="processName">name of the process</param>
6089
/// <returns><see cref="ProcessBuilder"/></returns>
61-
public static ProcessBuilder CreateProcessWithStatefulSteps(string processName = "FriedFishWithStatefulStepsProcess")
90+
public static ProcessBuilder CreateProcessWithStatefulStepsV2(string processName = "FriedFishWithStatefulStepsProcess")
6291
{
63-
var processBuilder = new ProcessBuilder(processName);
92+
// It is recommended to specify process version in case this process is used as a step by another process
93+
var processBuilder = new ProcessBuilder(processName) { Version = "FriedFishProcess.v2" };
6494

65-
var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsWithStockStep>();
66-
var chopStep = processBuilder.AddStepFromType<CutFoodWithSharpeningStep>("chopStep");
67-
var fryStep = processBuilder.AddStepFromType<FryFoodStep>();
95+
var gatherIngredientsStep = processBuilder.AddStepFromType<GatherFriedFishIngredientsWithStockStep>(name: "gatherFishIngredientStep", aliases: ["GatherFriedFishIngredientsWithStockStep"]);
96+
var chopStep = processBuilder.AddStepFromType<CutFoodWithSharpeningStep>(name: "chopFishStep", aliases: ["CutFoodStep"]);
97+
var fryStep = processBuilder.AddStepFromType<FryFoodStep>(name: "fryFishStep", aliases: ["FryFoodStep"]);
6898

6999
processBuilder
70100
.OnInputEvent(ProcessEvents.PrepareFriedFish)
@@ -97,11 +127,13 @@ public static ProcessBuilder CreateProcessWithStatefulSteps(string processName =
97127
return processBuilder;
98128
}
99129

130+
[KernelProcessStepMetadata("GatherFishIngredient.V1")]
100131
private sealed class GatherFriedFishIngredientsStep : GatherIngredientsStep
101132
{
102133
public GatherFriedFishIngredientsStep() : base(FoodIngredients.Fish) { }
103134
}
104135

136+
[KernelProcessStepMetadata("GatherFishIngredient.V2")]
105137
private sealed class GatherFriedFishIngredientsWithStockStep : GatherIngredientsWithStockStep
106138
{
107139
public GatherFriedFishIngredientsWithStockStep() : base(FoodIngredients.Fish) { }

‎dotnet/samples/GettingStartedWithProcesses/Step03/ProcessesStates/FishSandwichStateProcessSuccess.json‎

Lines changed: 26 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4,39 +4,44 @@
44
"stepsState": {
55
"GatherFriedFishIngredientsWithStockStep": {
66
"state": {
7-
"IngredientsStock": 3
7+
"IngredientsStock": 4
88
},
9-
"id": "cfc7d51f3bfa455c980d282dcdd85b13",
10-
"name": "GatherFriedFishIngredientsWithStockStep"
9+
"id": "32301d167a604f7bb1b69663d8feafe6",
10+
"name": "GatherFriedFishIngredientsWithStockStep",
11+
"versionInfo": "GatherFishIngredient.V2"
1112
},
12-
"chopStep": {
13-
"state": {
14-
"KnifeSharpness": 2
15-
},
16-
"id": "daf4a782df094deeb111ae701431ddbb",
17-
"name": "chopStep"
13+
"CutFoodStep": {
14+
"id": "3be68c614bc44bedb7bce0f002110968",
15+
"name": "CutFoodStep",
16+
"versionInfo": "CutFoodStep.V1"
1817
},
1918
"FryFoodStep": {
20-
"id": "e508ffe1d4714097b9716db1fd48b587",
21-
"name": "FryFoodStep"
19+
"id": "5f80945fafec4a0bbeccf3ff23a8c1a1",
20+
"name": "FryFoodStep",
21+
"versionInfo": "FryFoodStep.V1"
2222
}
2323
},
24-
"id": "a201334a1f534e9db9e1d46dce8345a8",
25-
"name": "FriedFishWithStatefulStepsProcess"
24+
"id": "c9d2e56dcc5a4b5ea73e05ae53635c90",
25+
"name": "FriedFishWithStatefulStepsProcess",
26+
"versionInfo": "FriedFishProcess.v1"
2627
},
2728
"AddBunsStep": {
28-
"id": "8a9b2d66e0594ee898d1c94c8bc07d0e",
29-
"name": "AddBunsStep"
29+
"id": "ef113fc874b0473c9c275827effe8dd8",
30+
"name": "AddBunsStep",
31+
"versionInfo": "v1"
3032
},
3133
"AddSpecialSauceStep": {
32-
"id": "6b0e92097cb74f5cbac2a71473a4e9c2",
33-
"name": "AddSpecialSauceStep"
34+
"id": "65a35f6dec6e45cab9e1b4df7d43a6bd",
35+
"name": "AddSpecialSauceStep",
36+
"versionInfo": "v1"
3437
},
3538
"ExternalFriedFishStep": {
36-
"id": "59ebd4724684469ab19d86d281c205e3",
37-
"name": "ExternalFriedFishStep"
39+
"id": "ab186a81480a45249cfff9e0c56319b8",
40+
"name": "ExternalFriedFishStep",
41+
"versionInfo": "v1"
3842
}
3943
},
40-
"id": "38e8f477-c022-41fc-89f1-3dd3509d0e83",
41-
"name": "FishSandwichWithStatefulStepsProcess"
44+
"id": "0de6d539-c1bf-44ad-816d-650231cd2034",
45+
"name": "FishSandwichWithStatefulStepsProcess",
46+
"versionInfo": "FishSandwich.V1"
4247
}

‎dotnet/samples/GettingStartedWithProcesses/Step03/ProcessesStates/FishSandwichStateProcessSuccessLowStock.json‎

Lines changed: 25 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -6,37 +6,42 @@
66
"state": {
77
"IngredientsStock": 1
88
},
9-
"id": "cfc7d51f3bfa455c980d282dcdd85b13",
10-
"name": "GatherFriedFishIngredientsWithStockStep"
9+
"id": "32301d167a604f7bb1b69663d8feafe6",
10+
"name": "GatherFriedFishIngredientsWithStockStep",
11+
"versionInfo": "GatherFishIngredient.V2"
1112
},
12-
"chopStep": {
13-
"state": {
14-
"KnifeSharpness": 2
15-
},
16-
"id": "daf4a782df094deeb111ae701431ddbb",
17-
"name": "chopStep"
13+
"CutFoodStep": {
14+
"id": "3be68c614bc44bedb7bce0f002110968",
15+
"name": "CutFoodStep",
16+
"versionInfo": "CutFoodStep.V1"
1817
},
1918
"FryFoodStep": {
20-
"id": "e508ffe1d4714097b9716db1fd48b587",
21-
"name": "FryFoodStep"
19+
"id": "5f80945fafec4a0bbeccf3ff23a8c1a1",
20+
"name": "FryFoodStep",
21+
"versionInfo": "FryFoodStep.V1"
2222
}
2323
},
24-
"id": "a201334a1f534e9db9e1d46dce8345a8",
25-
"name": "FriedFishWithStatefulStepsProcess"
24+
"id": "c9d2e56dcc5a4b5ea73e05ae53635c90",
25+
"name": "FriedFishWithStatefulStepsProcess",
26+
"versionInfo": "FriedFishProcess.v1"
2627
},
2728
"AddBunsStep": {
28-
"id": "8a9b2d66e0594ee898d1c94c8bc07d0e",
29-
"name": "AddBunsStep"
29+
"id": "ef113fc874b0473c9c275827effe8dd8",
30+
"name": "AddBunsStep",
31+
"versionInfo": "v1"
3032
},
3133
"AddSpecialSauceStep": {
32-
"id": "6b0e92097cb74f5cbac2a71473a4e9c2",
33-
"name": "AddSpecialSauceStep"
34+
"id": "65a35f6dec6e45cab9e1b4df7d43a6bd",
35+
"name": "AddSpecialSauceStep",
36+
"versionInfo": "v1"
3437
},
3538
"ExternalFriedFishStep": {
36-
"id": "59ebd4724684469ab19d86d281c205e3",
37-
"name": "ExternalFriedFishStep"
39+
"id": "ab186a81480a45249cfff9e0c56319b8",
40+
"name": "ExternalFriedFishStep",
41+
"versionInfo": "v1"
3842
}
3943
},
40-
"id": "38e8f477-c022-41fc-89f1-3dd3509d0e83",
41-
"name": "FishSandwichWithStatefulStepsProcess"
44+
"id": "0de6d539-c1bf-44ad-816d-650231cd2034",
45+
"name": "FishSandwichWithStatefulStepsProcess",
46+
"versionInfo": "FishSandwich.V1"
4247
}

0 commit comments

Comments
 (0)