Research Finding

Source: Factory.ai blog (2025), ICLR 2025 evaluation work
Applicability: High | Complexity: Simple

Problem

Zeph's current context summaries are unstructured prose generated by the LLM. Critical facts (file paths, decisions, next steps) can silently disappear during compression with no way to detect the loss post-hoc.

Proposed Approach

Maintain a typed schema for conversation summaries with mandatory sections:

• Session intent (what the user is trying to accomplish)
• Files modified (list of files changed and how)
• Decisions made (architectural/design choices)
• Open questions (unresolved items)
• Next steps (what the agent should do on resume)

Because each section must be populated, the LLM compressor cannot skip critical categories. Stored in existing SQLite summaries table as structured JSON or TOML block alongside the prose summary.

Evaluation

Function-first metric: after summarization, inject the summary as new context and ask the agent a factual question derived from the original turn. Pass = agent answers correctly. This directly measures information retention, unlike ROUGE/embedding similarity.

Integration Points

• crates/zeph-memory: update create_summary() to use structured prompt and JSON schema
• [memory] config: optional structured_summaries = true flag
• Debug dump: include pre/post summary token counts and section completeness check
• CLI: no change needed (summarization is background/automatic)

Reference

• Factory.ai: https://factory.ai/news/compressing-context
• ICLR 2025 evaluation: https://factory.ai/news/evaluating-compression