Python: AG-UI + HistoryProvider: compose history+input without duplicating turns (UI vs LLM context authorities) #8135

Summary

When hosting AG-UI with a durable HistoryProvider (UI transcript vs LLM working memory as two authorities), the framework should dedupe when assembling model context (history + run input). Today that composition is a blind concatenate; hosts must invent wrappers (e.g. “latest user turn only”) to avoid stacking the same turns twice.

Two authorities (intended split)

This is a common production shape (also reflected in the ag_ui_assistant_ui_chat sample, which adds a host-level LatestUserTurnAgent to truncate AG-UI input before Agent.run).

What the framework does today

1. AG-UI adapter (run_agent_stream) passes the reconstructed AG-UI message list straight into agent.run(messages, …) — no “latest turn only” truncation inside agent_framework_ag_ui.
2. Agent core builds model messages as SessionContext.get_messages(include_input=True) → context (history) then extend(input_messages) with no filter_new_messages.
3. filter_new_messages exists and is used on persist (save_messages) to avoid superlinear growth when replaying transcripts — it is not applied when composing what the model sees.
4. AG-UI session continuation deliberately excludes HistoryProvider source ids from serialized continuation state, so the default examples (no durable history) stay correct: empty history + full AG-UI messages each request. That default does not cover hosts that attach a durable, load_messages=True HistoryProvider.

Failure mode

With durable HistoryProvider + AG-UI full (or turn-sized) input:

HistoryProvider:  [user, assistant(tool_call), …]   # already saved (e.g. after an interrupt turn)
run input:        [user, assistant(tool_call), tool(result), …]  # same turn again from AG-UI / approval resolve
model sees:       history + input  → duplicated user / tool-call turns

HITL / approval resume makes this especially visible: the interrupt turn is often already in history, while Allow continues with the same user + assistant(tool_call) plus the new tool result.

Write-path dedupe may keep the DB clean; the prompt still stacks.

Expected behavior

When AG-UI (or any caller) coexists with a HistoryProvider that loads prior turns:

• Model context should be equivalent to history + filter_new_messages(history, input) (or an equivalent overlap/suffix algorithm), so turns already present in history are not appended again from input_messages.
• Ideally this lives in one place (e.g. get_messages(include_input=True) and/or PerServiceCallHistoryPersistingMiddleware reconstruction), so every host does not reimplement truncation wrappers.
• Persist behavior can stay as today (filter_new_messages on save).

Optional / related: document the two-authority model explicitly for AG-UI hosts (snapshot = UI, HistoryProvider = LLM), and that relying on “empty history because continuation excludes history keys” is insufficient once a durable provider is attached.

Suggested direction (non-prescriptive)

• Apply the existing filter_new_messages(existing=history, incoming=input) (or shared overlap logic) at compose time when include_input=True and history context is non-empty.
• Add a regression test: durable HistoryProvider already contains [u, a]; run input is [u, a, tool]; messages sent to the chat client are [u, a, tool] once — not [u, a, u, a, tool].
• Keep default AG-UI examples (no durable history) unchanged.

Environment

• Package: agent-framework / agent-framework-ag-ui (Python)
• Related host workaround pattern: sample python/samples/05-end-to-end/ag_ui_assistant_ui_chat (LatestUserTurnAgent truncates so HistoryProvider is not duplicated into the prompt)

Related

• assistant-ui client resume re-sending full messages on Allow (worsens snapshot duplication on the UI path): fix(react-ag-ui): interrupt resume (Allow/Deny) should not re-send full message history assistant-ui/assistant-ui#7011
  That issue is about snapshot duplication; this issue is about LLM context duplication when HistoryProvider loads across turns.