Reduce repeated full-context injection for agent turns #3242

Activity

1. songshikang0111 commented on Jul 29, 2026

   

   songshikang0111

   on Jul 29, 2026 – with ChatGPT Codex Connector

   More actions

   Additional Codex ACP evidence and implementation direction

   I captured one session/prompt produced by Buzz for a local Codex agent. The JSON payload was 14,996 bytes and contained five text blocks:

   • [Base]
   • [System]
   • [Agent Memory — core]
   • [Context]
   • [Buzz event]

   No private channel/user/event identifiers are needed to reproduce the structural problem. Under ACP v1, session/prompt.params.prompt is the user message, so [System] is only a textual label in that payload; it is not delivered with system/developer precedence.

   This matches Buzz's current compatibility path:

   • queue.rs::format_prompt omits Base/System from the turn only when the negotiated protocol supports session-level system prompts.
   • pool.rs::has_system_prompt_support and session_new_system_prompt use the session-level path for protocol v2, while legacy v1 gets the static material prepended again.
   • The current Codex ACP compatibility path negotiates ACP v1, so it takes the repeated user-message path.

   Suggested staged fix

   1. Classify prompt material by lifetime. Treat Buzz Base, channel System/persona, and workspace guidance as session-static. Keep Context, Buzz event, reply target, and changed memory/canvas data turn-dynamic.
   2. Negotiate persistent-instruction support at session creation. ACP v1 has no standard system-prompt field on session/new, so this should be an explicit Buzz/adapter capability or namespaced extension, not an assumed standard field.
   3. Translate that capability in each adapter. For Codex, map the static Buzz instructions to Codex app-server's thread/start.developerInstructions (or the equivalent developer_instructions session config). Prefer additive developer instructions; do not replace Codex's built-in base instructions.
   4. Retain today's per-turn injection only as a fallback for agents that do not acknowledge persistent instructions.
   5. Handle compaction/session rotation explicitly. Reinstall static instructions when a new underlying thread/session is created, or when the adapter reports that persistent instructions were lost. This preserves the safety motivation behind the legacy reinjection behavior.
   6. Add regression tests and telemetry. Record negotiated protocol/capability plus total, static, and dynamic prompt bytes. Test that capable sessions receive static instructions once, subsequent turns exclude them, and legacy sessions still receive the fallback.

   The intended split would be approximately:

   session creation: Base + System + workspace guidance
   session/prompt:   Context + Buzz event + changed memory/canvas only
   

   This is not a claim that the underlying model will never see the static instructions again—the adapter/runtime may retain, resend, or cache them internally. The concrete wins are eliminating repeated Buzz-owned user-message content, restoring the correct instruction authority for [System], and making the behavior measurable.
2. PavelTajdus commented on Jul 29, 2026

   

   PavelTajdus

   on Jul 29, 2026

   More actions

   Additional evidence: repeated dynamic thread context

   Moving Base/System to session creation will remove the largest static duplication, but the dynamic [Context] block can still resend an almost complete thread on every short follow-up.

   In one real multi-turn Buzz session, one-line operational messages repeatedly carried 13 prior messages that were already present in the underlying ACP conversation. After the main task was complete, subsequent turns continued to resend detailed merge, test, and cleanup history. No private channel, user, or event identifiers are needed to reproduce this behavior.

   Consider tracking which thread events have already been delivered to the underlying session:

   • Normal turns: send only the triggering event, reply metadata, and newly added thread events.
   • Compaction or session rotation: reinstall a compact thread summary plus any still-relevant open state, rather than replaying the full raw thread.
   • Updated workspace guidance: deliver the changed guidance once with explicit replacement semantics, instead of leaving several full versions in active conversation history.

   This is separate from the static Base/System fix: without thread-context deduplication, prompt size can continue growing with the conversation even after session-static instructions are handled correctly.
3. mgoulart commented on Aug 2, 2026

   

   mgoulart

   on Aug 2, 2026

   More actions

   Measured token cost on the protocol-v1 Claude Code path

   The report notes that no token-usage or prompt-cache telemetry was available, so the cost impact was an observed risk rather than a measured value. Here are the numbers from the Anthropic API's own usage accounting, which confirm it and put a size on it.

   Setup: Buzz Desktop 0.5.3 (macOS), managed Claude Code agents via buzz-acp, claude-code 2.1.220. 17 sessions, 1,305 API calls, 2026-07-28 → 2026-08-01. Numbers come from usage on each assistant message in Claude Code's own session logs, so cache_creation_input_tokens vs cache_read_input_tokens is the API's accounting, not an estimate. Per #3126, Claude Code negotiates ACP v1, so these runs take prepend_base_for_legacy — this is the repeated-static path, not the v2 one.

   Structure of each turn

   Every prompt is one user message with three parts, all three re-sent in full on every turn:

   Part	Size
   [Base]	12,736–13,707 chars (~3.2–3.4k tok), varying only by harness version
   # Identity / # Role	up to 2,534 chars (absent in 2 of the 17)
   [Thread Context (N of M messages, truncated)]	0–23,889 chars

   The static block is billed as a cache write every turn, not a cache hit

   This is the part I think is worth adding to the report. Because the block is appended at the tail of a growing conversation rather than sitting in a cached prefix, its byte-identical content buys nothing. Three consecutive human turns in one session:

   INJECT 32,353 chars (~8,088 tok)  → cache_create=21,854  cache_read=18,269
   INJECT 33,729 chars (~8,432 tok)  → cache_create=13,120  cache_read=45,536
   INJECT 38,966 chars (~9,741 tok)  → cache_create=15,547  cache_read=92,666
   

   Across all 17 sessions, [Base] alone accounts for 539,546 chars re-sent verbatim (~135k tokens), none of it served from cache.

   Cost, once re-reads are counted

   The injection is then re-read by every subsequent API call in the turn, so the raw size understates it. Attributing each injection to the calls that carry it:

   	API calls	billed input	injected	carried	%
   worst session	61	5.0M	26.3k	1.11M	22.3%
   2nd	65	5.2M	22.9k	988k	19.1%
   3rd	17	783k	7.8k	132k	16.9%
   largest session	305	41.1M	42.4k	3.77M	9.2%
   all 17	1,305	136.7M	210k	12.2M	8.9%

   Long tool-heavy sessions dilute the ratio; short conversational ones are dominated by it. A three-turn session where the agent did little real work paid 22%.

   Confirming the thread-context half

   @PavelTajdus's "13 prior messages" observation reproduces exactly, and the header states it: consecutive turns in one session read

   [Thread Context (13 of 19 messages, truncated)]
   [Thread Context (13 of 23 messages, truncated)]
   [Thread Context (13 of 26 messages, truncated)]
   

   Two details that may matter for the fix:

   • The cap is on message count, not size. Those 13 messages were 23,889 chars (~6k tokens) in one session, because the participants were agents posting long reports; another session's 11 messages were 12,762 chars. A count-based cap gives no bound on the tokens the block costs.
   • The window is renumbered [1]–[13] each turn, so the model can't tell which entries it has already been shown. That makes the delivered-events tracking @PavelTajdus proposes strictly more useful than raising or lowering the cap.

   BUZZ_ACP_CONTEXT_MESSAGE_LIMIT tunes the count (0 disables, max 100) but is count-based and doesn't address the re-sending.

   Suggested telemetry

   The original report asks for per-turn input / cached input / cache-write / output. The one field I'd add: whether the static prefix was served from cache. In these runs the answer is always no, and that single bit distinguishes "the block is large" from "the block is large and uncacheable where it sits" — the second is what makes it worth fixing ahead of the v2 pin landing.
4. loganrenz commented on Aug 4, 2026

   

   loganrenz

   on Aug 4, 2026

   More actions

   Additional current-version evidence from a real implementation/review task on Buzz Desktop v0.5.4 (macOS, managed Claude Code agent):

   • One long-lived implementer session reached 320,644 / 1,000,000 displayed usage tokens during a bounded two-file draft-PR task.
   • After the agent stopped producing new tool activity, file changes, or a delivered reply for more than five minutes, the Desktop Restart action created a fresh session. Its first visible usage reading was 84,024 / 1,000,000, later 92,336 after completing the repair.
   • The restarted session recovered from the existing worktree and finished the exact-SHA handoff successfully, so session rotation was operationally viable; it was just manual and late.
   • The task also required several short follow-up wakeups between implementation, validation, push, and reviewer-repair phases. Re-sending full static and thread context on each wakeup magnifies the cost described in this issue.

   This does not isolate how much of the 320,644 total was Buzz-owned repeated context versus necessary repository/tool output, so I am not attributing the entire value to this bug. It does show that the active-context pressure remains visible on v0.5.4 and supports two acceptance points already discussed here: per-turn static/dynamic prompt telemetry, and an explicit task/session rotation path that restores compact state from worktree + exact SHA rather than waiting for a near-million-token session to degrade.

   No private message content, identifiers, or credential values are included.
5. InsightStream-Dev commented on Aug 6, 2026

   

   InsightStream-Dev

   on Aug 6, 2026

   More actions

   Console-validated reproduction on the Claude Code path, plus the floor cost of a trivial DM

   Independent confirmation of @mgoulart's measurements, with two additions: totals validated to-the-token against the Anthropic Console usage dashboard, and a measured floor cost for the lightest possible interaction — a one-line DM.

   Setup: Buzz Desktop 0.5.3 (macOS), managed Claude Code agents via @agentclientprotocol/claude-agent-acp 0.64.2 (protocol-v1 / prepend_base_for_legacy path per #3126), claude CLI 2.1.220→2.1.223, model claude-opus-5. 16 sessions, 730 API calls, 2026-08-03 → 2026-08-06 (UTC). Default triggers (mentions/DMs); heartbeats disabled.

   A methodological caveat for anyone measuring from Claude Code session logs

   In our capture, the usage-bearing records under ~/.claude/projects/**/*.jsonl contained duplicates: 1,417 raw records resolved to 730 distinct message.ids (~2× on the Buzz sdk-ts sessions). After deduplicating by message.id, our per-hour totals matched the Console usage dashboard exactly — e.g. for the 2026-08-06 00:00 UTC hour, transcripts said 20,874,592 in / 119,549 out and Console said the same, with a day-level residual of zero. Raw (undeduplicated) sums overstate by roughly 2×. Worth checking whether the 17-session figures upthread were deduplicated the same way.

   Totals (deduplicated, Console-reconciled)

   UTC date	input	output	cache write	cache read	total
   2026-08-03	726	119,021	548,195	12,244,236	12,912,178
   2026-08-04	28	14,051	178,585	1,119,058	1,311,722
   2026-08-05	497	186,027	747,524	58,958,857	59,892,905
   2026-08-06	554	198,091	1,116,294	35,018,499	36,333,438
   total	1,805	517,190	2,590,598	107,340,650	110,450,243

   97.2% of all billed tokens are cache reads. This is the amplification mechanism the report describes, measured end-to-end: whatever lands in the conversation — including the re-injected static blocks — is re-read by every subsequent API call in every subsequent turn. On this workload that multiplier is ~41× (107.3M cache reads against 2.6M cache writes). For a subscription user whose CLI has silently fallen back to API billing (#3831 is the adjacent failure), this surfaced as two real invoices totaling $88 for three days of light-to-moderate agent use.

   The floor: one one-line DM costs ~113K input-side tokens

   The cheapest interaction we measured — DM to an idle agent, "what model are you currently using?", fresh session — took 3 API calls (the agent used buzz CLI tools to compose and send its reply):

   call	input	output	cache write	cache read
   1 (session start)	2	46	21,831	15,282
   2	2	290	712	37,113
   3	2	71	410	37,825
   total	6	407	22,953	90,220

   Two things line up with the report:

   • The ~22K cache write on call 1 is the injected stack ([Base] + system prompt + skills + [Context]) being written fresh — consistent with @mgoulart's finding that the tail-appended static block is billed as a cache write rather than served as a cache hit.
   • Calls 2–3 each re-read ~37K tokens to deliver a one-line answer. Every injected token is paid once as a write and then again on each of the turn's remaining calls — so short conversational turns have the worst injected-content ratio, matching the 22%-of-a-short-session figure upthread.

   Why this pushes toward session-prefix anchoring rather than context caps

   The per-channel long-lived session is Buzz's core feature — the fix that preserves it is the one proposed here: move the static blocks to a stable cached prefix at session/new (blocked on the v2 path in #3126 for Claude/Codex) and send only per-event deltas, rather than capping or rotating context. Count-based caps (BUZZ_ACP_CONTEXT_MESSAGE_LIMIT) don't bound tokens, and rotation throws away the shared context that makes multi-agent channels useful. On our numbers, a cached prefix would convert the dominant cost (repeated writes + re-reads of byte-identical content at Opus rates) into 0.1×-rate cache hits without changing agent behavior at all.

   👍React with 👍1Reacted by Alan Redzepagic
6. eskimm commented on Aug 7, 2026

   

   eskimm

   on Aug 7, 2026

   More actions

   This is the single biggest barrier to use Buzz as daily driver,

   👍React with 👍1Reacted by CWD
7. tlongwell-block commented on Aug 14, 2026

   

   tlongwell-block

   on Aug 14, 2026

   Collaborator

   More actions

   👀
8. tlongwell-block commented on Aug 14, 2026

   

   tlongwell-block

   on Aug 14, 2026

   Collaborator

   More actions

   This should be fixed in claude code since #4395

   And more generally #5423
9. tlongwell-block commented on Aug 14, 2026

   

   tlongwell-block

   on Aug 14, 2026

   Collaborator

   More actions

   Please update to any release since v0.5.9 to get this fixed! Thanks for the reports here, folks!
10. tlongwell-block

    closed this as completedon Aug 14, 2026