Skip to content

feat(memory): tier1 retrieval-at-start — CLI-runner integration, LLM reranker, memory tier1 CLI#63

Merged
Agent-Aurelius merged 9 commits into
mainfrom
f1/tier1-on-benchagi-main
Jun 10, 2026
Merged

feat(memory): tier1 retrieval-at-start — CLI-runner integration, LLM reranker, memory tier1 CLI#63
Agent-Aurelius merged 9 commits into
mainfrom
f1/tier1-on-benchagi-main

Conversation

@Agent-Aurelius

Copy link
Copy Markdown

Re-cut of #62 based on BenchAGI main (the #62 branch was based on upstream main — ~4.8k upstream commits made it unmergeable here; this carries ONLY the six tier1 commits).

What

Tier-1 retrieval-at-start: at session cold start, search the agent's own memory index for the opening topic and prepend RETRIEVED-CONTEXT-TIER1.md ahead of MEMORY.md — flag-gated (memorySearch.query.tier1.enabled), fail-open, byte-bounded.

  • Observability: durable JSONL trace (~/.openclaw/logs/tier1-retrieval.jsonl) for every fire/miss.
  • CLI-runner integration: tier1 fires on the live CLI backend (cli-runner/prepare.ts), not just the embedded runner — fixes the wrong-runner gap for claude-cli agents.
  • LLM reranker (memorySearch.query.reranker): optional judge (OpenAI-compatible endpoint, e.g. local Ollama) re-orders candidates; refines but never drops retrieval; fails open to cosine order on timeout/error/non-JSON.
  • openclaw memory tier1 <query> --agent <id> [--json|--out <path>] [--session-key <k>]: harness-agnostic entrypoint so non-gateway surfaces (launcher hooks, web apps) get the identical retrieval contract. Exit 0 on every retrieval miss (fail-open); exit 1 only on usage error.

Config

"memorySearch": { "query": {
  "tier1": { "enabled": true },
  "reranker": { "enabled": true, "baseUrl": "http://127.0.0.1:11434", "model": "qwen2.5:7b-instruct", "timeoutMs": 6000, "minScore": 0.5, "topK": 8 }
} }

Verification

  • tsgo:prod clean; vitest 111/111 across tier1-retrieval.test.ts + memory-core cli.test.ts (reranker re-order/fail-open on HTTP error, timeout, non-JSON, emptied set; override clamps; CLI disabled-gate/fail-open/--out/--session-key).
  • Cherry-picked from the live-proven fork deployment; conflicts were import-region only + one rename, resolved against the deployed reference.

🤖 Generated with Claude Code

Agent-Aurelius and others added 6 commits June 9, 2026 14:45
On a cold (full-bootstrap), primary, non-heartbeat session start, search the
agent's own memory index for the opening prompt's topic and prepend a small,
byte-bounded RETRIEVED-CONTEXT-TIER1.md ahead of MEMORY.md. Fixes the cross-thread
cold-start miss (a fresh Slack thread about a prior topic started cold even with
MEMORY.md injected).

- src/agents/tier1-retrieval.ts: buildTier1RetrievalContextFile — flag gate, query
  clean, in-process host search (getActiveMemorySearchManager, sources:["memory"],
  tenant-scoped by agentId), hard-timeout fail-open, top-K + maxBytes bound.
- attempt.ts: one eligibility-gated injection block (not raw, bootstrapMode full,
  non-heartbeat, isPrimaryBootstrapRun) + a structured tier1-retrieval log line;
  Tier-0 budget accounting unchanged.
- config: agents.*.memorySearch.query.tier1 {enabled,maxResults,minScore,maxBytes,
  timeoutMs}, default OFF (ship dark); per-agent override (Aurelius first).
- tier1-retrieval.test.ts: 10 cases (disabled/no-signal/error/timeout/below-threshold/
  injected+bounded+ordered). tsgo:core clean; bootstrap+attempt suites green.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gateway sends agent stderr to /dev/null, so F1's `log.info("tier1-retrieval
...")` was unobservable — couldn't confirm it fired. Add recordTier1Diag() which
best-effort appends each outcome (query, hits, injected, latency, reason,
sessionKey, agentId) to ~/.openclaw/logs/tier1-retrieval.jsonl, independent of
subsystem log routing. Never throws. attempt.ts calls it alongside the log line.

Verify a fresh-thread session with: tail ~/.openclaw/logs/tier1-retrieval.jsonl

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…al root cause)

Agents running via provider=claude-cli use the CLI runner, NOT the embedded/API
runner (attempt.ts) where F1 was first wired — so it never fired on real turns.
Move the Tier-1 retrieval into the CLI runner's context assembly
(cli-runner/prepare.ts), gated on a populated cold-start contextFiles,
prepending the slice ahead of MEMORY.md.

Also:
- cleanTier1Query: extract the real user message from the Slack relay preamble
  (the "## Message (user-controlled text...)" boundary) — was embedding the whole
  ~8KB wrapper, which is junk + made bge-m3/Ollama throw.
- Tier1Diag.err: capture the caught search-error text in the trace.
- recordTier1Gate (temp instrumentation): log the eligibility breakdown per run.

Verified live: clean query, Ollama bge-m3 embed ~156ms, search runs.
Open: recall QUALITY (bge-m3 cosine too compressed to discriminate) -> follow-up
reranker+embedder lane.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…il-open)

After the cosine candidate set in buildTier1RetrievalContextFile, an optional LLM
judge (qwen2.5:7b-instruct on the carbon-white RTX 5090) scores each candidate for
"does this actually answer THIS question?", re-sorts by judge score, and drops below
query.reranker.minScore. Catches topically-adjacent hard negatives the embedder alone
can't separate (the 0.40-vs-0.38 problem). FAIL-OPEN: any judge error/timeout/empty
returns the cosine order unchanged - the rerank refines, never drops, retrieval.

- zod-schema.agent-runtime.ts: query.reranker {enabled,baseUrl,apiKey,model,timeoutMs,minScore,topK}
- memory-search.ts: resolve query.reranker into ResolvedMemorySearchConfig
- tier1-retrieval.ts: rerankTier1Candidates() + observability (kind:"rerank" diag line)

Verified live on the mini1 brain: cold-start turn fired the judge (before:2 -> after:1),
reason:"ok", both embedder+judge serving on the 5090.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Harness-agnostic entrypoint for tier1 retrieval-at-start: `openclaw memory
tier1 <query>` builds the same RETRIEVED-CONTEXT-TIER1.md slice the runners
inject at cold session start, so any external harness (or an operator) can
fetch it without going through a runner.

- Registration + runtime follow the existing `memory search` pattern
  (positional query or --query, --agent, --json, --out, --session-key,
  --max-results, --max-bytes).
- Honors the `query.tier1.enabled` flag gate exactly as the runners do:
  disabled -> `{ injected: false, reason: "disabled" }`, never an error.
  Fail-open contract: every retrieval miss (disabled / no-signal / timeout /
  error / below-threshold) exits 0; only a missing query is a usage error.
- Exposes buildTier1RetrievalContextFile through the established
  memory-core-host-runtime-core plugin-sdk surface.
- tier1 module: small additive caller overrides for maxResults/maxBytes,
  hard-capped at 8 hits / 8192 bytes.
- Brings the ported tier1/reranker module up to main's gates: adds the
  missing `query.reranker` member on the MemorySearchConfig input type,
  exports rerankTier1Candidates, and fixes sort/boolean-compare lint.

Co-Authored-By: Claude Fable 5 <[email protected]>
- rerankTier1Candidates: re-orders by judge score within topK, drops
  below-minScore candidates, limits to topK on success, and fails open on
  judge HTTP error, timeout (abort), non-JSON responses, an emptied set,
  and missing baseUrl/model or single-candidate input.
- buildTier1RetrievalContextFile: caller maxResults/maxBytes overrides are
  honored and clamped to the hard caps; config-enabled reranker re-orders
  the injected slice.
- `memory tier1` CLI: usage error on missing query (exit 1), disabled gate
  emits JSON without opening a manager, enabled path injects with
  memory-scoped search + per-command session key, --session-key override,
  fail-open exit 0 when the search throws, --out writes only when injected,
  and the default mode prints the body to stdout.

recordTier1Diag is intentionally not unit-tested: TIER1_DIAG_LOG resolves
os.homedir() at module load, and mocking the homedir globally is worse than
the missing coverage.

Co-Authored-By: Claude Fable 5 <[email protected]>
@Agent-Aurelius
Agent-Aurelius marked this pull request as ready for review June 9, 2026 23:22
Agent-Aurelius and others added 2 commits June 9, 2026 17:33
- type the reranker auth fetch spies with the fetch signature so
  mock.calls[0]?.[1] doesn't index an empty-args tuple (TS2493 under
  the test tsconfig)
- teach the secret-resolution coverage guard the merged-targets shape
  (const targetIds = get*(); then optional feature targets added) the
  tier1-reranker secretRef wiring introduced — the path still routes
  through resolveCommandSecretRefsViaGateway with get*()-sourced ids

Co-Authored-By: Claude Fable 5 <[email protected]>
@Agent-Aurelius
Agent-Aurelius merged commit 479578f into main Jun 10, 2026
92 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant