Skip to content

Commit 6ff63c5

Browse files
authored
feat(eval): context-retention evaluation harness -- PR1 of 2 (TKT-319)
Squash of feat/tkt-319-context-retention-eval. See PR #12 for full details.
1 parent 441ad18 commit 6ff63c5

20 files changed

Lines changed: 1236 additions & 0 deletions

‎FORK.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ retires it without reading every PR.
5555
| V1 compatibility shim: `time_archived` retained as a mirror of `lifecycle === 'archived'`, and `PATCH /session/:id { time: { archived } }` rewritten as an adapter over the lifecycle service ([#4](https://github.com/Draugur-AI/opencode/pull/4)) | Old clients must keep working through the migration window, and two paths writing the same state independently is how they drift apart — so the old route translates rather than writing the column itself. Validation post, "Compatibility is a differential test" | n/a — shim, not proposed upstream | `session.time_archived` kept beyond its usefulness to the current domain | none beyond the row above | **remove** — delete the column and the V1 route together once telemetry shows the V1 archive path unused for a full release window. Known limit until then: a **trashed** session still appears in a V1 active list, because V1 filters on `time_archived` and has no representation for trash; it disappears from V1 on purge. Revisit when slice 2 ships the Trash UI |
5656
| Durable agent intent: `SessionGoal`/`SessionLedger` domains — `Goal.Updated`/`Goal.StatusChanged`/`Ledger.Added`/`Ledger.Superseded` events, `session_goal`/`session_ledger` tables, session-scoped `GoalContext`/`LedgerContext` composed directly in the runner alongside skill/reference guidance, `goal_get`/`goal_update_progress`/`ledger_add` tools, `session.goal`/`session.ledger` route groups ([#7](https://github.com/Draugur-AI/opencode/pull/7)) | Upstream has no durable goal or working-ledger concept at all, so a long session's binding objective and accumulated decisions exist only as prose the compaction summary may or may not preserve. Design post, "Durable agent intent must survive compaction"; build post, "Durable intent: goal, ledger, context, and recovery" | not yet filed | Adds `session_goal` (one row per session, atomic-replace) and `session_ledger` (append-only, superseded not deleted) with FK cascade to `session`; four new durable event types, additive — no existing decoder changed | `20260810151354_session_goal_ledger`: additive DDL only, no backfill (no prior rows to migrate). Idempotent by construction | **upstream** — this domain is exactly what the design/build posts argue upstream needs; file the issue before the first sync that touches session context assembly |
5757
| History search & compaction telemetry: `session_transcript_search` FTS5 projection kept in sync by the projector, `history_search`/`history_get` tools, `session.history.search`/`.get` routes; `SessionEvent.Compaction.Ended` versioned to durable v2 (per-compaction telemetry) with `EndedV1` kept as a superseded decoder for old rows — the first event in this fork to keep an old decoder alongside its successor rather than deleting it ([#10](https://github.com/Draugur-AI/opencode/pull/10)) | Upstream has no full-history search surface and no compaction telemetry, so an identifier a compaction summary drops is unrecoverable and a compaction's cost/effect is invisible. Design post, "Full-history search" and "Compaction telemetry"; build post, "versioned successor, old decoder kept" | not yet filed | Adds `session_transcript_search` (FTS5, no FK — deleted explicitly in `purge.ts`); `Compaction.Ended` gains 8 telemetry fields (additive, v2), `EndedV1` is the kept v1 decoder (durable-only, never in the general event union). `Event.define` gained an optional `identifier` param — the versioned-successor pattern is the first case where two definitions share one `type` literal, which without a distinct OpenAPI identifier collided into duplicate generated schemas; `packages/sdk/js/script/build.ts`'s dedup pass was generalized from a single known-unreachable-name check to a structural (byte-equality) fixed-point pass to collapse them | `20260810170000_session_transcript_search`: additive FTS5 virtual table, no backfill. Also closed a fresh-DB migration gap for any hand-written (non-drizzle) schema object via `HAND_MAINTAINED_SCHEMA_ADDITIONS` in `migration.ts` | **upstream** — history search and compaction telemetry are exactly what the design/build posts argue upstream needs; file the issue before the first sync that touches compaction or session event versioning |
58+
| Context-retention evaluation harness: new `@opencode-ai/eval` package — a scripted-model V2 app-graph builder, a per-instance fake `LLMClient.Service` capturing every composed `LLMRequest`, injected/real compaction epoch drivers, restart-survival via `Database.layerFromPath` reopen, per-source token/byte accounting, and two fixtures (omitted-identifier-recovery, constraint-survival) run in both real and pre-slice-4-approximation (`Layer.mock` on `SessionGoal`/`SessionLedger.context()`) modes, with a `bun run gate` CLI report ([#12](https://github.com/Draugur-AI/opencode/pull/12), PR1 of 2 — PR2 adds the real second-binary/real-model release gate) | Upstream has no outcome-evaluation harness for compaction reliability at all, and the design/validation posts are explicit that schema/service tests passing is not evidence the agent actually retains anything — "score decisions, not summary resemblance." Design post, "Context-retention evaluation"; validation post, "Agent reliability requires outcome evaluations" | not yet filed | none — pure new tooling package, no schema/table changes | none | **keep** — this harness is fork-specific by construction (it tests `SessionGoal`/`SessionLedger`/history-search, none of which exist upstream); it stays fork-only even if the domains it tests eventually get proposed upstream |
5859

5960
`e2e` (in `test.yml`), `nix-eval.yml`, `pr-management.yml`'s `check-duplicates` job, and most of
6061
the release/publish/deploy/beta/docs/notify/storybook/triage/stats workflows are **not** in this

‎bun.lock‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/eval/package.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"$schema": "https://json.schemastore.org/package.json",
3+
"name": "@opencode-ai/eval",
4+
"private": true,
5+
"type": "module",
6+
"exports": {
7+
".": "./src/index.ts"
8+
},
9+
"scripts": {
10+
"test": "bun test --timeout 30000 --only-failures",
11+
"typecheck": "tsgo --noEmit",
12+
"gate": "bun run script/cli.ts"
13+
},
14+
"dependencies": {
15+
"@opencode-ai/core": "workspace:*",
16+
"@opencode-ai/llm": "workspace:*",
17+
"@opencode-ai/schema": "workspace:*",
18+
"effect": "catalog:"
19+
},
20+
"devDependencies": {
21+
"@tsconfig/bun": "catalog:",
22+
"@types/bun": "catalog:",
23+
"@typescript/native-preview": "catalog:"
24+
}
25+
}

‎packages/eval/script/cli.ts‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/env bun
2+
import { fileURLToPath } from "url"
3+
import fs from "node:fs"
4+
import os from "node:os"
5+
import path from "node:path"
6+
import { Fixture } from "../src/fixture"
7+
import { Report } from "../src/report"
8+
import { OmittedIdentifierFixture } from "../src/fixtures/omitted-identifier"
9+
import { ConstraintSurvivalFixture } from "../src/fixtures/constraint-survival"
10+
11+
const dir = fileURLToPath(new URL("..", import.meta.url))
12+
process.chdir(dir)
13+
14+
const tmpdir = () => fs.mkdtempSync(path.join(os.tmpdir(), "opencode-eval-"))
15+
16+
const fixtures = [
17+
{ name: "omitted-identifier-recovery", run: OmittedIdentifierFixture.run },
18+
{ name: "constraint-survival", run: ConstraintSurvivalFixture.run },
19+
]
20+
21+
const main = async () => {
22+
const reports: Report.FixtureReport[] = []
23+
for (const fixture of fixtures) {
24+
const checkpoints = await Fixture.run(fixture.run, { databasePath: `${tmpdir()}/eval.db`, mode: "real" })
25+
reports.push({ name: fixture.name, mode: "real", checkpoints })
26+
}
27+
// The B-mode approximation run exists to validate the toggle itself is meaningful (a run that
28+
// "passes" identically in both modes would mean the fixture isn't testing slice-4/5 at all) --
29+
// not every fixture needs it, but constraint-survival is the one built to show the delta.
30+
const bModeCheckpoints = await Fixture.run(ConstraintSurvivalFixture.run, {
31+
databasePath: `${tmpdir()}/eval.db`,
32+
mode: "pre-slice-4-approximation",
33+
})
34+
reports.push({ name: "constraint-survival", mode: "pre-slice-4-approximation", checkpoints: bModeCheckpoints })
35+
36+
console.log(Report.render(reports))
37+
38+
const anyFail = reports.some((report) => report.mode === "real" && report.checkpoints.some((c) => !c.pass))
39+
if (anyFail) process.exit(1)
40+
}
41+
42+
await main()
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
export * as CompactionEpoch from "./compaction-epoch"
2+
3+
import { DateTime, Effect } from "effect"
4+
import { EventV2 } from "@opencode-ai/core/event"
5+
import { SessionV2 } from "@opencode-ai/core/session"
6+
import { SessionEvent } from "@opencode-ai/core/session/event"
7+
import { SessionMessage } from "@opencode-ai/core/session/message"
8+
9+
/**
10+
* Injects one durable compaction epoch (Started + Ended) directly, without driving a real turn
11+
* or LLM call. This is the WORKHORSE for most fixtures -- fast, deterministic, and exercises the
12+
* same SessionContextEpoch.prepare() replacement logic every real compaction does. It bypasses
13+
* the actual size-trigger and summarizer call, though, so it is labeled a deterministic
14+
* approximation everywhere its results are reported (report.ts) -- the anchor fixture
15+
* (compaction-anchor.test.ts) is what validates this approximation is honest.
16+
*/
17+
export const inject = Effect.fn("CompactionEpoch.inject")(function* (input: {
18+
readonly sessionID: SessionV2.ID
19+
readonly summary: string
20+
readonly recent?: string
21+
readonly sourceSeqStart?: number
22+
readonly sourceSeqEnd?: number
23+
}) {
24+
const events = yield* EventV2.Service
25+
const messageID = SessionMessage.ID.create()
26+
const startedAt = yield* DateTime.now
27+
yield* events.publish(SessionEvent.Compaction.Started, {
28+
sessionID: input.sessionID,
29+
messageID,
30+
timestamp: startedAt,
31+
reason: "manual",
32+
})
33+
const endedAt = yield* DateTime.now
34+
const recent = input.recent ?? ""
35+
yield* events.publish(SessionEvent.Compaction.Ended, {
36+
sessionID: input.sessionID,
37+
messageID,
38+
timestamp: endedAt,
39+
reason: "manual",
40+
text: input.summary,
41+
recent,
42+
tokensBefore: 0,
43+
retainedTailMessages: 0,
44+
retainedTailTokens: 0,
45+
summaryBytes: Buffer.byteLength(input.summary, "utf8"),
46+
summaryTokens: Math.round(input.summary.length / 4),
47+
durationMs: 0,
48+
sourceSeqStart: input.sourceSeqStart ?? 0,
49+
sourceSeqEnd: input.sourceSeqEnd ?? 0,
50+
})
51+
return messageID
52+
})
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
export * as ContextAccounting from "./context-accounting"
2+
3+
import { Token } from "@opencode-ai/core/util/token"
4+
import type { LLMRequest } from "@opencode-ai/llm"
5+
6+
/**
7+
* Per-source token/byte accounting for one composed LLMRequest -- the spec's "record context
8+
* bytes and tokens contributed by goal, ledger, profile, recent tail, and retrieved history"
9+
* requirement, and the harness's primary defense against "reliability bought by injecting the
10+
* full transcript" (making history_search results balloon the tail would show up here as a
11+
* jump in `tail`, not `retrievedHistory`, since a tool result IS part of the message tail).
12+
*
13+
* `system` in an LLMRequest is a flat array of parts with no source tags, so attribution here is
14+
* by TEXT MATCH against durable-state renderers' own delimiters (`<session_goal>`,
15+
* `<session_ledger>`) rather than by construction -- the harness observes the wire artifact, the
16+
* same discipline the rest of this fork uses (verify the artifact, not the mechanism that built
17+
* it). A goal/ledger renderer that changes its wrapper tag needs this updated alongside it; there
18+
* is no schema-level guarantee tying the two together.
19+
*/
20+
export interface Breakdown {
21+
readonly goal: { readonly bytes: number; readonly tokens: number }
22+
readonly ledger: { readonly bytes: number; readonly tokens: number }
23+
readonly otherSystem: { readonly bytes: number; readonly tokens: number }
24+
readonly tail: { readonly bytes: number; readonly tokens: number }
25+
readonly tools: { readonly bytes: number; readonly tokens: number }
26+
readonly total: { readonly bytes: number; readonly tokens: number }
27+
}
28+
29+
const sizeOf = (text: string) => ({ bytes: Buffer.byteLength(text, "utf8"), tokens: Token.estimate(text) })
30+
31+
const add = (a: { bytes: number; tokens: number }, b: { bytes: number; tokens: number }) => ({
32+
bytes: a.bytes + b.bytes,
33+
tokens: a.tokens + b.tokens,
34+
})
35+
36+
export const breakdown = (request: LLMRequest): Breakdown => {
37+
let goal = { bytes: 0, tokens: 0 }
38+
let ledger = { bytes: 0, tokens: 0 }
39+
let otherSystem = { bytes: 0, tokens: 0 }
40+
41+
for (const part of request.system) {
42+
const size = sizeOf(part.text)
43+
if (part.text.includes("<session_goal>")) goal = add(goal, size)
44+
else if (part.text.includes("<session_ledger>")) ledger = add(ledger, size)
45+
else otherSystem = add(otherSystem, size)
46+
}
47+
48+
const tail = sizeOf(JSON.stringify(request.messages))
49+
const tools = sizeOf(JSON.stringify(request.tools ?? []))
50+
const total = [goal, ledger, otherSystem, tail, tools].reduce(add, { bytes: 0, tokens: 0 })
51+
52+
return { goal, ledger, otherSystem, tail, tools, total }
53+
}

‎packages/eval/src/fake-llm.ts‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
export * as FakeLLM from "./fake-llm"
2+
3+
import {
4+
LLMClient,
5+
LLMEvent,
6+
type LLMClientShape,
7+
type LLMClientService,
8+
type LLMError,
9+
type LLMRequest,
10+
} from "@opencode-ai/llm"
11+
import { Layer, Stream } from "effect"
12+
13+
/**
14+
* A per-instance fake LLMClient.Service, scripted per call. Modeled on the module-level fake
15+
* client in packages/core/test/session-runner.test.ts, but instantiable multiple times with
16+
* independent state -- an eval run drives many sessions (fixtures x modes x epochs) and each
17+
* needs its own captured-request log and its own response queue, not one shared mutable module.
18+
*/
19+
export interface Instance {
20+
readonly layer: Layer.Layer<LLMClientService>
21+
/** Every LLMRequest this client has streamed a response for, in call order. */
22+
readonly requests: LLMRequest[]
23+
/** Queue one scripted response for the next stream() call. FIFO; each call consumes one entry. */
24+
readonly push: (events: readonly LLMEvent[]) => void
25+
/** Clear captured requests without touching the queued script. */
26+
readonly resetRequests: () => void
27+
}
28+
29+
export const make = (): Instance => {
30+
const requests: LLMRequest[] = []
31+
const queue: (readonly LLMEvent[])[] = []
32+
33+
const layer = Layer.succeed(
34+
LLMClient.Service,
35+
LLMClient.Service.of({
36+
prepare: () => {
37+
throw new Error("FakeLLM.Instance: prepare() is not scripted")
38+
},
39+
stream: ((request: LLMRequest) => {
40+
requests.push(request)
41+
return Stream.fromIterable(queue.shift() ?? [])
42+
}) as unknown as LLMClientShape["stream"],
43+
generate: () => {
44+
throw new Error("FakeLLM.Instance: generate() is not scripted")
45+
},
46+
}),
47+
)
48+
49+
return {
50+
layer,
51+
requests,
52+
push: (events) => queue.push(events),
53+
resetRequests: () => {
54+
requests.length = 0
55+
},
56+
}
57+
}
58+
59+
/** A single assistant turn that just answers with plain text and stops. */
60+
export const textTurn = (text: string, id = "eval_text"): LLMEvent[] => [
61+
LLMEvent.stepStart({ index: 0 }),
62+
LLMEvent.textStart({ id }),
63+
LLMEvent.textDelta({ id, text }),
64+
LLMEvent.textEnd({ id }),
65+
LLMEvent.stepFinish({ index: 0, reason: "stop" }),
66+
LLMEvent.finish({ reason: "stop" }),
67+
]
68+
69+
/**
70+
* A single assistant turn that calls one provider-executed tool and stops. The runner loops back
71+
* for a follow-up model call once the tool result is fed back in -- callers must also push a
72+
* subsequent response (even an empty `[]`, or another textTurn) for that follow-up.
73+
*/
74+
export const toolCallTurn = (callID: string, name: string, input: unknown): LLMEvent[] => [
75+
LLMEvent.stepStart({ index: 0 }),
76+
LLMEvent.toolCall({ id: callID, name, input }),
77+
LLMEvent.stepFinish({ index: 0, reason: "tool-calls" }),
78+
LLMEvent.finish({ reason: "tool-calls" }),
79+
]
80+
81+
export type { LLMError }

‎packages/eval/src/fixture.ts‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
export * as Fixture from "./fixture"
2+
3+
import { Effect } from "effect"
4+
import { EvalGraph } from "./graph"
5+
import type { ContextAccounting } from "./context-accounting"
6+
7+
export interface CheckpointResult {
8+
readonly name: string
9+
readonly pass: boolean
10+
readonly detail?: string
11+
readonly accounting?: ContextAccounting.Breakdown
12+
}
13+
14+
export interface Result {
15+
readonly fixture: string
16+
readonly mode: "real" | "pre-slice-4-approximation"
17+
readonly checkpoints: readonly CheckpointResult[]
18+
}
19+
20+
/**
21+
* A fixture is a self-contained scenario: given a fresh graph handle and session, drive turns
22+
* and checkpoints, and return the scored results. Fixtures are plain functions (not a declarative
23+
* DSL) so the same Effect/FakeLLM primitives every other harness piece already uses work here
24+
* too -- no second scripting language to keep in sync with the runner's real behavior.
25+
*/
26+
// `any` here mirrors packages/core/test/lib/effect.ts's own Body<A,E,R>: a fixture's error and
27+
// requirement types vary per scenario, and every fixture is always run via `Fixture.run` below,
28+
// which discharges them with Effect.provide(handle.layer) -- the precise types aren't load-bearing
29+
// at this boundary the way they are inside a fixture's own implementation.
30+
export type Fixture = (handle: EvalGraph.Handle) => Effect.Effect<readonly CheckpointResult[], any, any>
31+
32+
export const check = (name: string, pass: boolean, detail?: string): CheckpointResult => ({ name, pass, detail })
33+
34+
/** Builds a fresh graph (real, or B-mode if `mode` says so) and runs one fixture against it. */
35+
export const run = (
36+
fixture: Fixture,
37+
input: { readonly databasePath: string; readonly mode: "real" | "pre-slice-4-approximation" },
38+
): Promise<readonly CheckpointResult[]> => {
39+
const handle = EvalGraph.build({
40+
databasePath: input.databasePath,
41+
directory: EvalGraph.defaultDirectory,
42+
overrides: input.mode === "pre-slice-4-approximation" ? EvalGraph.preSlice4Overrides : undefined,
43+
})
44+
const provided = fixture(handle).pipe(Effect.scoped, Effect.provide(handle.layer)) as Effect.Effect<
45+
readonly CheckpointResult[],
46+
unknown
47+
>
48+
return Effect.runPromise(provided)
49+
}

0 commit comments

Comments
 (0)