This document defines the first four scenarios for evaluating durable project memory in coding agents.
The benchmark defines the scenarios. Tests A, B, C, and D have recorded manual executions. Test C has a protocol, starter fixture, three preserved inconclusive attempts, and a completed revision-2 audit result and revision-3 clean rerun. Revision 3 passes all three provenance audits after addressing the observed explanation and read-only containment defects. Test D has a recorded PASS from a real Prospector repository-history replay. Cross-scenario scoring methodology, automated evaluation, and broader datasets remain future work.
Session 1 discovers an architectural fact about the project.
The session ends.
Session 2 receives a related task that requires the same architectural knowledge.
Does the agent investigate the same thing again?
An agent records:
Authentication lives in module A.
Later, the architecture changes and authentication moves to module B.
Can the memory system represent that the original conclusion is no longer current?
An agent retrieves a remembered architectural fact from project memory.
Can the developer trace where that conclusion came from?
Current execution is defined by the Test C revision-3 protocol, using the source fixture. The first attempt was inconclusive: capture hit an account usage limit before committing memory, so no provenance audits ran. The second attempt also hit the usage limit, before writing memory. The third attempt completed but was ineligible: its handler claim was compound and its commit hook failed inside the sandbox. Setup invalidity makes the aggregate inconclusive; the known claim defect is retained. No audits ran and no capture was hand-repaired. The revision-2 result repaired the capture environment, produced and committed an atomic claim, then ran all three audits. It remains inconclusive: slots 1/2 fail C3 and slot 3 is unevaluable after an audit Git-index metadata change. No audit is replaced. The separate revision-3 result passes: it records an eligible fresh capture and three passing read-only audits.
A previous claim stored in project memory becomes wrong or outdated.
A newer conclusion replaces it.
Can we see both the old conclusion and what superseded it?
The recorded Test D PASS is a real Prospector repository-history replay. It demonstrated preservation and reconstruction of a multi-generation correction chain in that repository.
GraphKeeper Memory Bench v0.1 currently defines only these four scenarios:
- Repeated investigation
- Stale memory
- Provenance
- Correction history
Test A has recorded pre-fix FAIL and post-fix PASS results. Test B has recorded a pre-fix FAIL and a post-fix PASS. The separate open-run lifecycle defect observed in that pass has a subsequent close-command PASS. Test C's latest result has a valid committed atomic capture and all three audit slots accounted for, but is inconclusive (two C3 failures and one isolation-unevaluable slot). The separate revision-3 experiment passes with an eligible fresh capture and 3/3 passing audit slots. Test D has a recorded PASS: a real Prospector repository-history replay demonstrated preservation and reconstruction of a multi-generation correction chain. These are manual evidence records, not an automated benchmark harness; cross-scenario scoring methodology, automated evaluation, and broader datasets remain future work.