Python: [Feature]: persist AG-UI thread snapshots incrementally (and handle client disconnect), not only at run completion #7215

Activity

1. Anton Sokolovskyi (antsok)

   added the

   Feature

   issue type on Jul 20, 2026
2. github-project-automation

   added this to Agent Frameworkon Jul 20, 2026
3. Giles Odigwe (giles17)

   added

   triageUsage: [Issues], Target: All issues that still need to be triagedUsage: [Issues], Target: All issues that still need to be triaged

   on Jul 20, 2026
4. Evan Mattson (moonbox3)

   added

   pythonUsage: [Issues, PRs], Target: PythonUsage: [Issues, PRs], Target: Python

   ag-uiUsage: [Issues, PRs], Target: AG-UI protocol integrationUsage: [Issues, PRs], Target: AG-UI protocol integration

   and removed

   triageUsage: [Issues], Target: All issues that still need to be triagedUsage: [Issues], Target: All issues that still need to be triaged

   on Jul 22, 2026
5. Evan Mattson (moonbox3)

   self-assigned this

   on Jul 22, 2026
6. github-actions

   changed the title [-][Feature]: persist AG-UI thread snapshots incrementally (and handle client disconnect), not only at run completion[/-] [+]Python: [Feature]: persist AG-UI thread snapshots incrementally (and handle client disconnect), not only at run completion[/+] on Jul 22, 2026
7. Evan Mattson (moonbox3)

   added

   harness[Issues, PRs], Target: harness-level items[Issues, PRs], Target: harness-level items

   on Aug 3, 2026
8. antsok commented on Aug 22, 2026

   

   Anton Sokolovskyi (antsok)

   on Aug 22, 2026

   ContributorAuthor

   More actions

   Since filing this we have hit a concrete data-loss defect in the same area, and it does not depend on the feature this issue asks for. Would you consider reclassifying this as a bug — or splitting it, keeping the incremental-persistence request as a feature and tracking the following separately?

   The failed behaviour

   A completed run's stored transcript can be silently and permanently replaced by a shorter one. No error, no warning, and the loss is only visible later, as a conversation that is missing turns.

   The AG-UI thread snapshot is written as the request's message list plus the current run's output. The stored messages are never consulted:

   # _agent_run.py — the request's list is the only input
   messages, snapshot_messages = normalize_agui_input_messages(raw_messages, ...)
   
   # end of run — that list, plus this run's output, is what gets persisted
   persisted_messages = latest_messages_snapshot
   await snapshot_session.save(messages=persisted_messages, ...)

   So whenever a client posts with a transcript shorter than the stored one, the shorter list becomes the thread's record.

   What makes this look unintended rather than by design is the asymmetry with state, which is merged in the same file:

   # _snapshot_session.py — effective_state()
   state.update(self._stored.state)   # stored first
   state.update(request_state)        # request overlays it, key by key

   State reconciles stored with request. Messages replace outright.

   The rest of the design reads as assuming exactly two client positions — hydrate_events serves the stored snapshot when a request carries no messages, and the request is treated as authoritative when it carries some. A client that carries some messages but is behind the stored snapshot is a third position the protocol has no way to express, and the server resolves it by trusting the client.

   Minimal reproduction (no custom host, no disconnect, no error path)

   1. Open thread T in client A and run a turn to completion. The stored snapshot now holds turns 1..N.
   2. Have a second client B on the same thread whose transcript ends at turn N-1 — a second tab, a second device, or the same tab opened before that turn.
   3. Post any message from B.
   4. When that run completes, the stored snapshot is B's shorter list plus the new turn. Turn N is gone from the thread, and reloading in either client shows the shortened conversation.

   No crash and no dropped connection are required; the ordinary success path is sufficient. A dropped connection makes it easier to reach, because a client that misses events is exactly a client whose transcript is behind, but it is not a precondition.

   Impact

   Completed agent work disappears from a thread with no error surfaced to the user or the application. It is worse the longer a run is, since the window in which another surface can be behind grows with the run — which is what makes it hurt most for the long tool-using runs this framework is built for.

   How it actually reached us: a mobile client and a long run

   The two-client reproduction above is the minimal form. What put us here was a phone, and it is worth describing because it is the ordinary case rather than a contrived one.

   A user started a long analysis from Android Chrome and switched away. The tab was backgrounded and eventually discarded. On mobile this does not produce a clean, immediate disconnect: the browser stops running the page's JavaScript well before the server learns the peer is gone, and when the fetch is eventually aborted the client finalises the run on its own side. So the client's transcript stops at whatever it had already processed, while the server carries on and commits the completed run. Both sides believe they are consistent; neither is.

   The user came back nearly two hours later to a transcript frozen at the moment the tab went away, and sent a message asking why it had stopped. That message carried the frozen transcript. When the run ended, the thread's stored record became the frozen list plus that turn — and roughly forty-four minutes of completed work (224 model round trips, 387 tool calls, ~22M input tokens, two delegated sub-agents, files written to durable storage) stopped being part of the conversation. It had been in the store for over an hour before the next request replaced it.

   That last turn happened to fail on a provider error, so in our case the write came through our own crash-recovery path rather than the normal one. It changes nothing about the outcome: both paths persist the request's list plus the run's output, so a turn that succeeded would have produced exactly the same record. We mention it only so the failure is not mistaken for an error-handling defect — it is the ordinary save.

   Nothing in that sequence is exotic: background a tab on a phone during a long agent run, come back, and type. The only reason the work existed to be lost is that our host detaches runs from the request so a suspended tab does not abandon an analysis in progress. That is our addition and we are not asking you to adopt it — but it is worth being precise about what it does and does not explain. The detach is why there was completed work in the store; it is not why the store lost it. The loss happens in the save path, which never consults what it is overwriting, and the minimal reproduction above reaches it on stock with no detaching, no disconnect and no error at all.

   The mobile angle also bears on the original request in this issue: without detaching, the same suspended tab simply kills the run, and because the end-of-run save sits inline before RUN_FINISHED rather than in a finally, everything it streamed is discarded. Either way, a phone going to sleep costs the user the run — once by losing it in flight, once by overwriting it afterwards.

   Precedent in this repo

   PR #7242 (for #7211) taught save_messages to reconcile its input against what the store already holds, via get_message_identity / filter_new_messages, instead of appending blindly. That is the same move — a store reconciling rather than trusting its input — applied to the history provider. The AG-UI snapshot store has no equivalent.

   Possible directions

   Either would close it:

   • Reconcile on save. Treat a save that drops message ids the stored snapshot already holds as a merge rather than a replacement, mirroring what effective_state already does for state.
   • Let the request declare what it has. A watermark or version on the inbound transcript would let the server detect staleness instead of assuming its absence, which also gives clients a way to notice they are behind.

   On the original scope

   The incremental-persistence and disconnect-handling request in this issue still stands on its own and is genuinely feature-shaped, so splitting is fine if that is cleaner for triage. Worth noting for that half: checkpoint appears throughout the workflow path and not at all in the agent run path, so the durability concept already exists in the framework — the ask is to extend it across that boundary rather than to invent it.

   Environment: agent-framework-core: 1.14.0
9. Evan Mattson (moonbox3)

   assigned

   Eduard van Valkenburg (eavanvalkenburg)

   and unassigned

   Evan Mattson (moonbox3)

   on Sep 8, 2026
10. Eduard van Valkenburg (eavanvalkenburg)

    closed this as completedin #8672on Sep 29, 2026
11. github-project-automation

    moved this to Done in Agent Frameworkon Sep 29, 2026