|
| 1 | +# Issue #5350 — Root-cause validation plan |
| 2 | + |
| 3 | +**Issue:** [`.NET: [Bug]: Checkpoint round-trip loses ToolApprovalRequestContent.ToolCall concrete type (FunctionCallContent → base ToolCallContent), breaking FICC approval resume`](https://github.com/microsoft/agent-framework/issues/5350) |
| 4 | + |
| 5 | +**Status of repro:** A focused repro test class |
| 6 | +`dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/ToolApprovalRequestCheckpointReproTests.cs` |
| 7 | +was added covering six progressively more end-to-end variants of the path the issue |
| 8 | +describes. **All six tests pass** on `main`, i.e. *the bug as described does not |
| 9 | +reproduce* at any of the layers exercised here: |
| 10 | + |
| 11 | +| # | Test | What it exercises | Result | |
| 12 | +|---|-------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| |
| 13 | +| 1 | `Repro_5350_ToolApprovalRequestContent_DirectJsonMarshallerRoundtrip_PreservesFunctionCallContent` | `JsonMarshaller.Marshal/Marshal<T>` on `ToolApprovalRequestContent` directly (same options chain as `CheckpointManager.CreateJson`) | Pass | |
| 14 | +| 2 | `Repro_5350_ToolApprovalRequestContent_WrappedInPortableValue_PreservesFunctionCallContent` | Same, wrapped in a `PortableValue` (as `PortableMessageEnvelope` would store it) | Pass | |
| 15 | +| 3 | `Repro_5350_ToolApprovalRequestContent_AsExternalRequestData_PreservesFunctionCallContent` | Same, as the `Data` payload of an `ExternalRequest` | Pass | |
| 16 | +| 4 | `Repro_5350_DirectJsonMarshallerRoundtrip_IsDeterministic` | 25× repetition of #1 to rule out flakiness / JIT order | Pass | |
| 17 | +| 5 | `Repro_5350_CaptureWireFormat_ForInspection` | Captures the on-the-wire shape so we can compare against the OP's SQL row | Pass | |
| 18 | +| 6 | `Repro_5350_EndToEnd_JsonCheckpointResume_PreservesFunctionCallContentAsync` | Full `CheckpointManager.CreateJson(InMemoryJsonStore) → RunStreamingAsync → SuperStep checkpoint → ResumeStreamingAsync` cycle with a `RequestPort<TARC, TARR>` | Pass | |
| 19 | + |
| 20 | +This is **consistent with [@lokitoth's second comment](https://github.com/microsoft/agent-framework/issues/5350#issuecomment-4379664401)**: |
| 21 | + |
| 22 | +> Looking at the MEAI types, it does have `[JsonDerivedType(typeof(FunctionCallContent), "functionCall")]` set on it, and has had it that way for some time, so it is unlikely to be the root cause. […] `JsonMarshaller` […] takes an optional `JsonSerializationOptions` provided by the user […] but this is used only if the internal one (via `WorkflowsJsonUtilities`, which chains to `AgentAbstractionsJsonUtilities`, which then chains to the `AIJsonUtilities` class) [is missing the type]. |
| 23 | +
|
| 24 | +Inspection of the actual wire format produced by `JsonMarshaller` for a |
| 25 | +`ToolApprovalRequestContent` containing a `FunctionCallContent` confirms the |
| 26 | +discriminator is emitted: |
| 27 | + |
| 28 | +```jsonc |
| 29 | +// TARC at top level |
| 30 | +{ |
| 31 | + "toolCall": { |
| 32 | + "$type": "functionCall", // ← polymorphism discriminator present |
| 33 | + "name": "DoTheThing", |
| 34 | + "arguments": { "x": 42 }, |
| 35 | + "informationalOnly": false, |
| 36 | + "callId": "call-1" |
| 37 | + }, |
| 38 | + "requestId": "req-1" |
| 39 | +} |
| 40 | + |
| 41 | +// TARC inside a PortableValue (= shape stored by PortableMessageEnvelope) |
| 42 | +{ |
| 43 | + "typeId": { |
| 44 | + "assemblyName": "Microsoft.Extensions.AI.Abstractions, Version=10.5.0.0, …", |
| 45 | + "typeName": "Microsoft.Extensions.AI.ToolApprovalRequestContent" |
| 46 | + }, |
| 47 | + "value": { |
| 48 | + "toolCall": { |
| 49 | + "$type": "functionCall", |
| 50 | + … |
| 51 | + }, |
| 52 | + "requestId": "req-1" |
| 53 | + } |
| 54 | +} |
| 55 | +``` |
| 56 | + |
| 57 | +So the OP's stated root-cause hypothesis — *"`AIContent`/`ToolCallContent`/ |
| 58 | +`FunctionCallContent` are missing `[JsonPolymorphic]`/`[JsonDerivedType]`, or |
| 59 | +`CheckpointManager.CreateJson` does not pull from `AIJsonUtilities.DefaultOptions`"* |
| 60 | +— is **not** what is producing the failure described in the issue. The annotations |
| 61 | +exist, the resolver chain wires them through, and the discriminator does survive |
| 62 | +both the direct `JsonMarshaller` round-trip and a real `Run → checkpoint → Resume` |
| 63 | +cycle for a `RequestPort` whose request type is `ToolApprovalRequestContent`. |
| 64 | + |
| 65 | +The remaining sections lay out, in priority order, the work needed to identify |
| 66 | +the *actual* root cause. The plan deliberately keeps the failing repro from |
| 67 | +above as the baseline ("this is what works") and walks outward from it toward |
| 68 | +the OP's reported scenario, varying one dimension at a time. |
| 69 | + |
| 70 | +## Plan |
| 71 | + |
| 72 | +### Track A — Reproduce in a configuration closer to the OP's pattern "B" |
| 73 | + |
| 74 | +The OP's repro path differs from the new tests in three concrete ways. Each is a |
| 75 | +candidate root cause; the test matrix below isolates them. Each row is "add a |
| 76 | +test that reproduces the OP's symptom (`postResume.ToolCall is not FunctionCallContent`)". |
| 77 | + |
| 78 | +| Step | Variable that changes vs. the passing tests in this repo | Why it matters | Pass criteria | |
| 79 | +|------|--------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------| |
| 80 | +| A1 | Use `ChatClientAgent` + `ApprovalRequiredAIFunction` bound into a `WorkflowBuilder` (the GroupChatToolApproval "pattern B" path), with a fake `IChatClient` that emits a single `FunctionCallContent`. | This is the only major piece of the OP's setup that the current tests do not exercise. The TARC in this path is *generated* by `FunctionInvokingChatClient` and flows through the `AIAgentHostExecutor`. | Test fails (TARC.ToolCall comes back as base type) → root cause is in the agent-host/FICC bridge, not the marshaller. | |
| 81 | +| A2 | Same as A1 but with the request payload also surfaced as part of a `ChatMessage.Contents`-style transport (whatever the agent host actually serializes through). Inspect the on-the-wire JSON for the inner `toolCall` and look for an absent or differently-named `$type`. | If the TARC is round-tripped as `AIContent`/`ChatMessage` instead of as itself, the polymorphism is two-deep (`AIContent` → TARC, TARC → ToolCall). It is plausible that one branch resolves but the other doesn't. | Find missing `$type` in serialized payload. | |
| 82 | +| A3 | Same as the passing test #6 in this file, but with the SQL Server-style store that round-trips the `JsonElement` through `string` and back (e.g. `element.GetRawText()` → `JsonDocument.Parse`). | The OP uses Dapper + SQL Server. If the column is `nvarchar` and the round-trip preserves ordering, this should be identity-preserving — but if the OP uses `jsonb`-like storage that reorders metadata properties, the `$type` discriminator can be moved out of "first" position, which then requires `AllowOutOfOrderMetadataProperties = true`. Note `JsonMarshaller` only propagates that one flag from the user's `customOptions`. | Test fails when ordering is permuted before deserialization. Confirms the storage layer is reordering metadata. | |
| 83 | +| A4 | Same as #6 but with a non-default `JsonSerializerOptions` passed as `customOptions` to `CheckpointManager.CreateJson`, where the user-supplied options do **not** include the polymorphism resolver and `JsonMarshaller`'s `LookupTypeInfo` falls back to them for some type. | `JsonMarshaller.LookupTypeInfo` only goes to the external options when the internal chain doesn't know about the type. For most cases this won't trigger, but it's worth confirming that supplying a custom `JsonSerializerOptions` does not silently displace the internal chain. | Either the external options are never consulted for `AIContent` (good), or there is a sneak path where they are (regression). | |
| 84 | + |
| 85 | +### Track B — Validate the wire format the OP actually persists |
| 86 | + |
| 87 | +Once Track A has reproduced (or has clearly failed to reproduce) the symptom, |
| 88 | +ask the OP for one of the following, in order of preference: |
| 89 | + |
| 90 | +1. The raw `JsonElement.GetRawText()` they pass to their SQL store on commit, |
| 91 | + and the raw string they read back on `RetrieveCheckpointAsync` — for the |
| 92 | + exact checkpoint that contains the failing `ToolApprovalRequestContent`. |
| 93 | + This tells us in a single round-trip whether: |
| 94 | + - the `$type` discriminator is present on commit (rules out write-side bug), |
| 95 | + - the `$type` discriminator survives the SQL round-trip (rules out store bug), |
| 96 | + - the discriminator is in metadata-first position (rules out the |
| 97 | + `AllowOutOfOrderMetadataProperties` story). |
| 98 | +2. A repro PR/gist that runs against `CheckpointManager.CreateJson(...)` with |
| 99 | + an in-memory `JsonCheckpointStore` shim that mirrors how their SQL subclass |
| 100 | + passes bytes through. The OP already offered this in the issue body |
| 101 | + (*"I can produce a minimal standalone repro against a fake `IChatClient` if useful — let me know."*) — taking them up on it short-circuits a lot of guessing. |
| 102 | + |
| 103 | +### Track C — Defense-in-depth fixes worth landing regardless of root cause |
| 104 | + |
| 105 | +These are addressed in the OP's "Asks" section (asks 2–4) and are useful |
| 106 | +documentation/ergonomics improvements even if the root cause turns out to be in |
| 107 | +the OP's store implementation: |
| 108 | + |
| 109 | +1. **Doc-only**: in the XML docs for `CheckpointManager.CreateJson` and on the |
| 110 | + `JsonCheckpointStore` base class, explicitly state that the internal options |
| 111 | + chain through `WorkflowsJsonUtilities → AgentAbstractionsJsonUtilities → |
| 112 | + AIJsonUtilities`, and that user-supplied `customOptions` are consulted only |
| 113 | + when the internal chain has no `TypeInfo` for a requested type. |
| 114 | +2. **Doc-only**: document the contract that any user `JsonCheckpointStore` |
| 115 | + subclass MUST preserve the exact byte sequence of the `JsonElement` it is |
| 116 | + given. If the underlying store reorders JSON metadata, the consumer must opt |
| 117 | + in by setting `AllowOutOfOrderMetadataProperties = true` on a |
| 118 | + `JsonSerializerOptions` passed as `customOptions` to |
| 119 | + `CheckpointManager.CreateJson(...)`. |
| 120 | +3. **Optional, only if Track A reproduces**: add a regression test mirroring |
| 121 | + the failing path under `dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests` |
| 122 | + so the fix is locked in. |
| 123 | + |
| 124 | +### Track D — Reject the OP's hypothesis (if not already done) |
| 125 | + |
| 126 | +The combination of: |
| 127 | + |
| 128 | +- the polymorphism annotations existing in `Microsoft.Extensions.AI` for some time |
| 129 | + (per @lokitoth), |
| 130 | +- the resolver chain in `WorkflowsJsonUtilities.CreateDefaultOptions()` already |
| 131 | + putting `AgentAbstractionsJsonUtilities.DefaultOptions.TypeInfoResolver` first |
| 132 | + (which itself puts `AIJsonUtilities.DefaultOptions.TypeInfoResolver` first), |
| 133 | +- the wire format captured above showing `"$type": "functionCall"` is present, and |
| 134 | +- the full `Run → checkpoint → Resume` test in this PR passing, |
| 135 | + |
| 136 | +is sufficient to **disprove** the OP's stated hypothesis. Once Track A or |
| 137 | +Track B identifies the actual cause, the issue should be updated with a brief |
| 138 | +explanation of why the original guess was incorrect, so future readers don't |
| 139 | +re-tread it. |
| 140 | + |
| 141 | +## Pointers for the next investigator |
| 142 | + |
| 143 | +- Repro tests: |
| 144 | + `dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/ToolApprovalRequestCheckpointReproTests.cs` |
| 145 | +- Marshaller under test: |
| 146 | + `dotnet/src/Microsoft.Agents.AI.Workflows/Checkpointing/JsonMarshaller.cs` |
| 147 | +- Where `CheckpointManager.CreateJson` enters that path: |
| 148 | + `dotnet/src/Microsoft.Agents.AI.Workflows/CheckpointManager.cs` |
| 149 | +- Options chain (TARC ⇒ resolver): |
| 150 | + `dotnet/src/Microsoft.Agents.AI.Workflows/WorkflowsJsonUtilities.cs` |
| 151 | + → `dotnet/src/Microsoft.Agents.AI.Abstractions/AgentAbstractionsJsonUtilities.cs` |
| 152 | + → `Microsoft.Extensions.AI.AIJsonUtilities.DefaultOptions` |
| 153 | +- `PortableValue`-side serialization (this is where `value.Value.GetType()` is |
| 154 | + used on write, which is the most plausible *internal* place a polymorphism |
| 155 | + bug could hide, but the wire capture above shows it is not the cause in the |
| 156 | + TARC-as-RequestPort scenario): |
| 157 | + `dotnet/src/Microsoft.Agents.AI.Workflows/Checkpointing/PortableValueConverter.cs` |
| 158 | +- Likely next code to inspect for Track A1/A2 (agent-host serialization path): |
| 159 | + `dotnet/src/Microsoft.Agents.AI.Workflows/AIAgentHostExecutor*.cs` and |
| 160 | + whatever turns the FICC-generated TARC into something |
| 161 | + `AIAgentHostExecutor` stores in its state bag. |
0 commit comments