Skip to content

Commit e28682a

Browse files
Copilotlokitoth
andauthored
1 parent 66a09a7 commit e28682a

2 files changed

Lines changed: 417 additions & 0 deletions

File tree

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# Issue #5350 — Root-cause validation plan
2+
3+
**Issue:** [`.NET: [Bug]: Checkpoint round-trip loses ToolApprovalRequestContent.ToolCall concrete type (FunctionCallContent → base ToolCallContent), breaking FICC approval resume`](https://github.com/microsoft/agent-framework/issues/5350)
4+
5+
**Status of repro:** A focused repro test class
6+
`dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/ToolApprovalRequestCheckpointReproTests.cs`
7+
was added covering six progressively more end-to-end variants of the path the issue
8+
describes. **All six tests pass** on `main`, i.e. *the bug as described does not
9+
reproduce* at any of the layers exercised here:
10+
11+
| # | Test | What it exercises | Result |
12+
|---|-------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|
13+
| 1 | `Repro_5350_ToolApprovalRequestContent_DirectJsonMarshallerRoundtrip_PreservesFunctionCallContent` | `JsonMarshaller.Marshal/Marshal<T>` on `ToolApprovalRequestContent` directly (same options chain as `CheckpointManager.CreateJson`) | Pass |
14+
| 2 | `Repro_5350_ToolApprovalRequestContent_WrappedInPortableValue_PreservesFunctionCallContent` | Same, wrapped in a `PortableValue` (as `PortableMessageEnvelope` would store it) | Pass |
15+
| 3 | `Repro_5350_ToolApprovalRequestContent_AsExternalRequestData_PreservesFunctionCallContent` | Same, as the `Data` payload of an `ExternalRequest` | Pass |
16+
| 4 | `Repro_5350_DirectJsonMarshallerRoundtrip_IsDeterministic` | 25× repetition of #1 to rule out flakiness / JIT order | Pass |
17+
| 5 | `Repro_5350_CaptureWireFormat_ForInspection` | Captures the on-the-wire shape so we can compare against the OP's SQL row | Pass |
18+
| 6 | `Repro_5350_EndToEnd_JsonCheckpointResume_PreservesFunctionCallContentAsync` | Full `CheckpointManager.CreateJson(InMemoryJsonStore) → RunStreamingAsync → SuperStep checkpoint → ResumeStreamingAsync` cycle with a `RequestPort<TARC, TARR>` | Pass |
19+
20+
This is **consistent with [@lokitoth's second comment](https://github.com/microsoft/agent-framework/issues/5350#issuecomment-4379664401)**:
21+
22+
> Looking at the MEAI types, it does have `[JsonDerivedType(typeof(FunctionCallContent), "functionCall")]` set on it, and has had it that way for some time, so it is unlikely to be the root cause. […] `JsonMarshaller` […] takes an optional `JsonSerializationOptions` provided by the user […] but this is used only if the internal one (via `WorkflowsJsonUtilities`, which chains to `AgentAbstractionsJsonUtilities`, which then chains to the `AIJsonUtilities` class) [is missing the type].
23+
24+
Inspection of the actual wire format produced by `JsonMarshaller` for a
25+
`ToolApprovalRequestContent` containing a `FunctionCallContent` confirms the
26+
discriminator is emitted:
27+
28+
```jsonc
29+
// TARC at top level
30+
{
31+
"toolCall": {
32+
"$type": "functionCall", // ← polymorphism discriminator present
33+
"name": "DoTheThing",
34+
"arguments": { "x": 42 },
35+
"informationalOnly": false,
36+
"callId": "call-1"
37+
},
38+
"requestId": "req-1"
39+
}
40+
41+
// TARC inside a PortableValue (= shape stored by PortableMessageEnvelope)
42+
{
43+
"typeId": {
44+
"assemblyName": "Microsoft.Extensions.AI.Abstractions, Version=10.5.0.0, …",
45+
"typeName": "Microsoft.Extensions.AI.ToolApprovalRequestContent"
46+
},
47+
"value": {
48+
"toolCall": {
49+
"$type": "functionCall",
50+
…
51+
},
52+
"requestId": "req-1"
53+
}
54+
}
55+
```
56+
57+
So the OP's stated root-cause hypothesis — *"`AIContent`/`ToolCallContent`/
58+
`FunctionCallContent` are missing `[JsonPolymorphic]`/`[JsonDerivedType]`, or
59+
`CheckpointManager.CreateJson` does not pull from `AIJsonUtilities.DefaultOptions`"*
60+
— is **not** what is producing the failure described in the issue. The annotations
61+
exist, the resolver chain wires them through, and the discriminator does survive
62+
both the direct `JsonMarshaller` round-trip and a real `Run → checkpoint → Resume`
63+
cycle for a `RequestPort` whose request type is `ToolApprovalRequestContent`.
64+
65+
The remaining sections lay out, in priority order, the work needed to identify
66+
the *actual* root cause. The plan deliberately keeps the failing repro from
67+
above as the baseline ("this is what works") and walks outward from it toward
68+
the OP's reported scenario, varying one dimension at a time.
69+
70+
## Plan
71+
72+
### Track A — Reproduce in a configuration closer to the OP's pattern "B"
73+
74+
The OP's repro path differs from the new tests in three concrete ways. Each is a
75+
candidate root cause; the test matrix below isolates them. Each row is "add a
76+
test that reproduces the OP's symptom (`postResume.ToolCall is not FunctionCallContent`)".
77+
78+
| Step | Variable that changes vs. the passing tests in this repo | Why it matters | Pass criteria |
79+
|------|--------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
80+
| A1 | Use `ChatClientAgent` + `ApprovalRequiredAIFunction` bound into a `WorkflowBuilder` (the GroupChatToolApproval "pattern B" path), with a fake `IChatClient` that emits a single `FunctionCallContent`. | This is the only major piece of the OP's setup that the current tests do not exercise. The TARC in this path is *generated* by `FunctionInvokingChatClient` and flows through the `AIAgentHostExecutor`. | Test fails (TARC.ToolCall comes back as base type) → root cause is in the agent-host/FICC bridge, not the marshaller. |
81+
| A2 | Same as A1 but with the request payload also surfaced as part of a `ChatMessage.Contents`-style transport (whatever the agent host actually serializes through). Inspect the on-the-wire JSON for the inner `toolCall` and look for an absent or differently-named `$type`. | If the TARC is round-tripped as `AIContent`/`ChatMessage` instead of as itself, the polymorphism is two-deep (`AIContent` → TARC, TARC → ToolCall). It is plausible that one branch resolves but the other doesn't. | Find missing `$type` in serialized payload. |
82+
| A3 | Same as the passing test #6 in this file, but with the SQL Server-style store that round-trips the `JsonElement` through `string` and back (e.g. `element.GetRawText()` → `JsonDocument.Parse`). | The OP uses Dapper + SQL Server. If the column is `nvarchar` and the round-trip preserves ordering, this should be identity-preserving — but if the OP uses `jsonb`-like storage that reorders metadata properties, the `$type` discriminator can be moved out of "first" position, which then requires `AllowOutOfOrderMetadataProperties = true`. Note `JsonMarshaller` only propagates that one flag from the user's `customOptions`. | Test fails when ordering is permuted before deserialization. Confirms the storage layer is reordering metadata. |
83+
| A4 | Same as #6 but with a non-default `JsonSerializerOptions` passed as `customOptions` to `CheckpointManager.CreateJson`, where the user-supplied options do **not** include the polymorphism resolver and `JsonMarshaller`'s `LookupTypeInfo` falls back to them for some type. | `JsonMarshaller.LookupTypeInfo` only goes to the external options when the internal chain doesn't know about the type. For most cases this won't trigger, but it's worth confirming that supplying a custom `JsonSerializerOptions` does not silently displace the internal chain. | Either the external options are never consulted for `AIContent` (good), or there is a sneak path where they are (regression). |
84+
85+
### Track B — Validate the wire format the OP actually persists
86+
87+
Once Track A has reproduced (or has clearly failed to reproduce) the symptom,
88+
ask the OP for one of the following, in order of preference:
89+
90+
1. The raw `JsonElement.GetRawText()` they pass to their SQL store on commit,
91+
and the raw string they read back on `RetrieveCheckpointAsync` — for the
92+
exact checkpoint that contains the failing `ToolApprovalRequestContent`.
93+
This tells us in a single round-trip whether:
94+
- the `$type` discriminator is present on commit (rules out write-side bug),
95+
- the `$type` discriminator survives the SQL round-trip (rules out store bug),
96+
- the discriminator is in metadata-first position (rules out the
97+
`AllowOutOfOrderMetadataProperties` story).
98+
2. A repro PR/gist that runs against `CheckpointManager.CreateJson(...)` with
99+
an in-memory `JsonCheckpointStore` shim that mirrors how their SQL subclass
100+
passes bytes through. The OP already offered this in the issue body
101+
(*"I can produce a minimal standalone repro against a fake `IChatClient` if useful — let me know."*) — taking them up on it short-circuits a lot of guessing.
102+
103+
### Track C — Defense-in-depth fixes worth landing regardless of root cause
104+
105+
These are addressed in the OP's "Asks" section (asks 2–4) and are useful
106+
documentation/ergonomics improvements even if the root cause turns out to be in
107+
the OP's store implementation:
108+
109+
1. **Doc-only**: in the XML docs for `CheckpointManager.CreateJson` and on the
110+
`JsonCheckpointStore` base class, explicitly state that the internal options
111+
chain through `WorkflowsJsonUtilities → AgentAbstractionsJsonUtilities →
112+
AIJsonUtilities`, and that user-supplied `customOptions` are consulted only
113+
when the internal chain has no `TypeInfo` for a requested type.
114+
2. **Doc-only**: document the contract that any user `JsonCheckpointStore`
115+
subclass MUST preserve the exact byte sequence of the `JsonElement` it is
116+
given. If the underlying store reorders JSON metadata, the consumer must opt
117+
in by setting `AllowOutOfOrderMetadataProperties = true` on a
118+
`JsonSerializerOptions` passed as `customOptions` to
119+
`CheckpointManager.CreateJson(...)`.
120+
3. **Optional, only if Track A reproduces**: add a regression test mirroring
121+
the failing path under `dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests`
122+
so the fix is locked in.
123+
124+
### Track D — Reject the OP's hypothesis (if not already done)
125+
126+
The combination of:
127+
128+
- the polymorphism annotations existing in `Microsoft.Extensions.AI` for some time
129+
(per @lokitoth),
130+
- the resolver chain in `WorkflowsJsonUtilities.CreateDefaultOptions()` already
131+
putting `AgentAbstractionsJsonUtilities.DefaultOptions.TypeInfoResolver` first
132+
(which itself puts `AIJsonUtilities.DefaultOptions.TypeInfoResolver` first),
133+
- the wire format captured above showing `"$type": "functionCall"` is present, and
134+
- the full `Run → checkpoint → Resume` test in this PR passing,
135+
136+
is sufficient to **disprove** the OP's stated hypothesis. Once Track A or
137+
Track B identifies the actual cause, the issue should be updated with a brief
138+
explanation of why the original guess was incorrect, so future readers don't
139+
re-tread it.
140+
141+
## Pointers for the next investigator
142+
143+
- Repro tests:
144+
`dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/ToolApprovalRequestCheckpointReproTests.cs`
145+
- Marshaller under test:
146+
`dotnet/src/Microsoft.Agents.AI.Workflows/Checkpointing/JsonMarshaller.cs`
147+
- Where `CheckpointManager.CreateJson` enters that path:
148+
`dotnet/src/Microsoft.Agents.AI.Workflows/CheckpointManager.cs`
149+
- Options chain (TARC ⇒ resolver):
150+
`dotnet/src/Microsoft.Agents.AI.Workflows/WorkflowsJsonUtilities.cs`
151+
→ `dotnet/src/Microsoft.Agents.AI.Abstractions/AgentAbstractionsJsonUtilities.cs`
152+
→ `Microsoft.Extensions.AI.AIJsonUtilities.DefaultOptions`
153+
- `PortableValue`-side serialization (this is where `value.Value.GetType()` is
154+
used on write, which is the most plausible *internal* place a polymorphism
155+
bug could hide, but the wire capture above shows it is not the cause in the
156+
TARC-as-RequestPort scenario):
157+
`dotnet/src/Microsoft.Agents.AI.Workflows/Checkpointing/PortableValueConverter.cs`
158+
- Likely next code to inspect for Track A1/A2 (agent-host serialization path):
159+
`dotnet/src/Microsoft.Agents.AI.Workflows/AIAgentHostExecutor*.cs` and
160+
whatever turns the FICC-generated TARC into something
161+
`AIAgentHostExecutor` stores in its state bag.

0 commit comments

Comments
 (0)