Repository navigation
Cache‐Aligned Summarization
bash-agent is an AI agent built for long-running autonomous work. Each conversation turn—user messages, model responses, tool call results—is recorded as part of the conversation history.
The problem: history never shrinks on its own. As the conversation grows, two real-world issues emerge:
- Finite context window: Every LLM has a maximum context length; once history exceeds that limit, the model stops working properly.
- Rising costs: Each API call re-submits the entire history. Even if large portions are duplicated, the provider charges full input price for every token.
To stay within these limits, bash-agent periodically compacts the history—distilling long exchanges into a concise summary that replaces the raw old messages in future requests.
A naive approach: when the history nears the limit, send a dedicated request asking the model to “summarize the conversation above”. But this summary request itself isn’t free; you still have to resend the messages being compacted.
If the structure of this summary request differs from the normal chat requests, those historical messages won’t hit the API’s prompt cache—they’ll all be charged at the full input price.
A quick calculation: compacting 45,000 tokens of history with a dedicated uncached request costs roughly $0.143 (Claude pricing). That’s like buying an expensive energy-saving bulb whose price exceeds the electricity it saves.
Many LLM APIs (Anthropic Claude, OpenAI) support Prompt Caching. The principle is straightforward:
sequenceDiagram
participant Client as bash-agent
participant API as LLM API Server
Note over Client,API: Normal chat turn 1
Client->>API: Request A: system prompt + tools + message 1
API->>API: Prefix is brand new — charged at full price
API-->>Client: Response A
Note over Client,API: Normal chat turn 2
Client->>API: Request B: system prompt + tools + message 1 + message 2
API->>API: Prefix “system prompt + tools” matches Request A<br/>charged at the heavily discounted cache price (~10%)
API-->>Client: Response B
In bash-agent, every normal chat request starts with the same system prompt and tool definitions, so this shared prefix consistently hits the cache across turns—keeping costs extremely low.
The idea follows naturally: make the summary request look as much like a normal chat request as possible, so it “rides” the cache that already exists.
The implementation boils down to one rule: don’t alter the request structure; simply append a short “please summarize” instruction at the end of the normal conversation messages.
This means the token makeup of a summary request looks like this:
pie title Token billing breakdown of a summary request
"Cache hit (system prompt + tools + old summary)" : 5000
"Cache hit (historical messages being compacted)" : 39500
"New instruction (charged at full price)" : 500
- The first two parts were already sent in earlier normal requests, so the API serves them from the cache at the drastically lower cache price (usually ~10% of the full input price).
- Only the appended summary instruction is new and must be paid at the full input price—but it’s usually just a few hundred tokens.
Crucially, the system prompt is never changed. Modifying the system prompt would break the shared prefix, invalidating the cache from the very first token and wiping out all savings.
Compacting 45,000 tokens as an example:
flowchart LR
subgraph A["Traditional (no alignment)"]
direction LR
A1["45,000 tokens all at full price"] --> A2["Cost ≈ $0.143"]
end
subgraph B["Cache-Aligned"]
direction LR
B1["44,500 tokens from cache<br/>(cache price)"] --> B2["Cost ≈ $0.015"]
B3["500 new instruction tokens<br/>(full price)"] --> B2
end
Cache alignment cuts the cost of a single compaction by roughly 90%. This means bash-agent can compact aggressively and often, keeping the context window healthy without worrying that the compaction itself will eat the savings.
Once the compaction finishes, the generated summary text is saved. In every subsequent turn, it’s injected at the very beginning of the prompt, so the model sees “what happened before” without the full, verbose history.
flowchart LR
A["History too long<br/>compaction needed"] --> B["Send cache-aligned summary request"]
B --> C["Generate summary text"]
C --> D["Save as summary.txt"]
D --> E["Inject summary at start<br/>of future turns"]
E --> F["Context window stays healthy"]
The summary itself becomes part of the shared prefix for future requests, continuing to benefit from the same prompt caching—no additional cache-breakage cost is introduced.
Cache-Aligned Summarization is bash-agent’s strategy for cost-efficient context compaction. It leverages the LLM API’s existing prompt caching with a simple, elegant approach:
- Keep the request structure unchanged — the summary request shares the exact same prefix as normal chats.
- Only append a brief instruction — the only full-price tokens are the short “please summarize” line.
- ~90% cost reduction — compaction becomes cheap enough to run frequently without hesitation.
- The system prompt stays stable — preserving the cache chain and maximizing long-term savings.
With this design, bash-agent can maintain a manageable context size during long, multi-turn autonomous tasks, without letting the cost of compaction eat up the benefits.
For more details see the bash-agent project and the original README section.