feat(memory): tier1 retrieval-at-start — CLI-runner integration, LLM reranker, memory tier1 CLI#62
Closed
Agent-Aurelius wants to merge 4793 commits into
Closed
feat(memory): tier1 retrieval-at-start — CLI-runner integration, LLM reranker, memory tier1 CLI#62Agent-Aurelius wants to merge 4793 commits into
Agent-Aurelius wants to merge 4793 commits into
Conversation
Fix MiniMax-M3 Anthropic-compatible requests so OpenClaw no longer sends the disabled-thinking payload that makes M3 return empty content. M3 defaults now stay on MiniMax's omitted/adaptive thinking path, explicit `/think off` is still respected, and MiniMax-M2.x keeps the disabled-thinking default that prevents reasoning_content leaks. Also wires the MiniMax thinking policy through bundled provider-policy loading so pre-runtime and configless embedded-agent paths resolve the same defaults. Thanks @IamVNIE for the live MiniMax API repro and initial patch.
Move Zalo hosted outbound media metadata and expiry into plugin state, add SDK chunked hosted media storage, and keep CI/type/lint gates green after rebase.
Decode Google Vertex authorized_user ADC OAuth token refresh responses from bytes so gzip-compressed token payloads still expose access_token. Adds a regression test for the compressed token response path while preserving plain JSON handling and the custom fetch seam. Proof: OPENCLAW_VITEST_MAX_WORKERS=1 node scripts/run-vitest.mjs extensions/google/transport-stream.test.ts; pnpm exec oxfmt --check extensions/google/vertex-adc.ts extensions/google/transport-stream.test.ts; pnpm tsgo:extensions; git diff --check origin/main...HEAD; autoreview --mode branch --base origin/main. PR CI check-test-types failure was reproduced on current origin/main 607bbe4 and is unrelated to this two-file Google provider change. Thanks @liaoandi for the fix and live Google Vertex ADC proof.
Decode HTML-entity escaped xAI and Venice tool-call arguments through the shared core compat path exactly once, preventing literal entities such as & from being over-decoded before tool execution and transcript persistence. Removes xAI's duplicate provider-local decoder and keeps regression coverage for the shared core wrapper, xAI stream wrapper, and Venice compat path. Thanks @yetval for the fix.
Preserve streamed Responses tool-call argument deltas when the final done event omits or sends empty arguments, fixing LM Studio argument-bearing tools from arriving as `{}`.
Fixes openclaw#90585.
Thanks @849261680.
Fixes openclaw#89090. Release note: prevent repeated visible message sends from bypassing loop blocking when delivery results include fresh message, file, poll, receipt, run, idempotency, or timestamp fields. Normalizes send-like result hashing for the core message tool, sessions_send, and provider-docked messaging tools while preserving stable routing and outcome facts. Verification: - node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts src/agents/tools/message-tool.test.ts - .agents/skills/autoreview/scripts/autoreview --mode local --parallel-tests "node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts src/agents/tools/message-tool.test.ts" - .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --parallel-tests "node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts src/agents/tools/message-tool.test.ts" - gh pr checks 89109 --watch --interval 30 Co-authored-by: openperf <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>
Bind LM Studio wizard prompter callbacks before storing them so class-backed gateway setup sessions keep their receiver and no longer crash when selecting LM Studio. Thanks @christineyan4. Co-authored-by: Christine Yan <[email protected]> Co-authored-by: Copilot <[email protected]>
Fixes openclaw#90891. Doctor now reports official managed plugin version drift from the daemon-local status path, using the probed running gateway version and suppressing the advisory when probe auth is skipped or unsafe. The status probe also avoids re-entering config-backed exec SecretRef credential resolution when exec refs are disabled. Verification: - `node scripts/run-vitest.mjs src/commands/agent-via-gateway.test.ts src/cli/daemon-cli/probe.test.ts src/cli/daemon-cli/status.gather.test.ts src/flows/doctor-health-contributions.test.ts src/commands/doctor-workspace-status.test.ts src/gateway/probe-auth.test.ts` - `.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main` - Crabbox delegated Blacksmith Testbox `tbx_01ktmwa5q0c2eb688dkbkw8v2b`: `OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 corepack pnpm check:changed`
On a cold (full-bootstrap), primary, non-heartbeat session start, search the
agent's own memory index for the opening prompt's topic and prepend a small,
byte-bounded RETRIEVED-CONTEXT-TIER1.md ahead of MEMORY.md. Fixes the cross-thread
cold-start miss (a fresh Slack thread about a prior topic started cold even with
MEMORY.md injected).
- src/agents/tier1-retrieval.ts: buildTier1RetrievalContextFile — flag gate, query
clean, in-process host search (getActiveMemorySearchManager, sources:["memory"],
tenant-scoped by agentId), hard-timeout fail-open, top-K + maxBytes bound.
- attempt.ts: one eligibility-gated injection block (not raw, bootstrapMode full,
non-heartbeat, isPrimaryBootstrapRun) + a structured tier1-retrieval log line;
Tier-0 budget accounting unchanged.
- config: agents.*.memorySearch.query.tier1 {enabled,maxResults,minScore,maxBytes,
timeoutMs}, default OFF (ship dark); per-agent override (Aurelius first).
- tier1-retrieval.test.ts: 10 cases (disabled/no-signal/error/timeout/below-threshold/
injected+bounded+ordered). tsgo:core clean; bootstrap+attempt suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gateway sends agent stderr to /dev/null, so F1's `log.info("tier1-retrieval
...")` was unobservable — couldn't confirm it fired. Add recordTier1Diag() which
best-effort appends each outcome (query, hits, injected, latency, reason,
sessionKey, agentId) to ~/.openclaw/logs/tier1-retrieval.jsonl, independent of
subsystem log routing. Never throws. attempt.ts calls it alongside the log line.
Verify a fresh-thread session with: tail ~/.openclaw/logs/tier1-retrieval.jsonl
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…al root cause) Agents running via provider=claude-cli use the CLI runner, NOT the embedded/API runner (attempt.ts) where F1 was first wired — so it never fired on real turns. Move the Tier-1 retrieval into the CLI runner's context assembly (cli-runner/prepare.ts), gated on a populated cold-start contextFiles, prepending the slice ahead of MEMORY.md. Also: - cleanTier1Query: extract the real user message from the Slack relay preamble (the "## Message (user-controlled text...)" boundary) — was embedding the whole ~8KB wrapper, which is junk + made bge-m3/Ollama throw. - Tier1Diag.err: capture the caught search-error text in the trace. - recordTier1Gate (temp instrumentation): log the eligibility breakdown per run. Verified live: clean query, Ollama bge-m3 embed ~156ms, search runs. Open: recall QUALITY (bge-m3 cosine too compressed to discriminate) -> follow-up reranker+embedder lane. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…il-open)
After the cosine candidate set in buildTier1RetrievalContextFile, an optional LLM
judge (qwen2.5:7b-instruct on the carbon-white RTX 5090) scores each candidate for
"does this actually answer THIS question?", re-sorts by judge score, and drops below
query.reranker.minScore. Catches topically-adjacent hard negatives the embedder alone
can't separate (the 0.40-vs-0.38 problem). FAIL-OPEN: any judge error/timeout/empty
returns the cosine order unchanged - the rerank refines, never drops, retrieval.
- zod-schema.agent-runtime.ts: query.reranker {enabled,baseUrl,apiKey,model,timeoutMs,minScore,topK}
- memory-search.ts: resolve query.reranker into ResolvedMemorySearchConfig
- tier1-retrieval.ts: rerankTier1Candidates() + observability (kind:"rerank" diag line)
Verified live on the mini1 brain: cold-start turn fired the judge (before:2 -> after:1),
reason:"ok", both embedder+judge serving on the 5090.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Harness-agnostic entrypoint for tier1 retrieval-at-start: `openclaw memory
tier1 <query>` builds the same RETRIEVED-CONTEXT-TIER1.md slice the runners
inject at cold session start, so any external harness (or an operator) can
fetch it without going through a runner.
- Registration + runtime follow the existing `memory search` pattern
(positional query or --query, --agent, --json, --out, --session-key,
--max-results, --max-bytes).
- Honors the `query.tier1.enabled` flag gate exactly as the runners do:
disabled -> `{ injected: false, reason: "disabled" }`, never an error.
Fail-open contract: every retrieval miss (disabled / no-signal / timeout /
error / below-threshold) exits 0; only a missing query is a usage error.
- Exposes buildTier1RetrievalContextFile through the established
memory-core-host-runtime-core plugin-sdk surface.
- tier1 module: small additive caller overrides for maxResults/maxBytes,
hard-capped at 8 hits / 8192 bytes.
- Brings the ported tier1/reranker module up to main's gates: adds the
missing `query.reranker` member on the MemorySearchConfig input type,
exports rerankTier1Candidates, and fixes sort/boolean-compare lint.
Co-Authored-By: Claude Fable 5 <[email protected]>
- rerankTier1Candidates: re-orders by judge score within topK, drops below-minScore candidates, limits to topK on success, and fails open on judge HTTP error, timeout (abort), non-JSON responses, an emptied set, and missing baseUrl/model or single-candidate input. - buildTier1RetrievalContextFile: caller maxResults/maxBytes overrides are honored and clamped to the hard caps; config-enabled reranker re-orders the injected slice. - `memory tier1` CLI: usage error on missing query (exit 1), disabled gate emits JSON without opening a manager, enabled path injects with memory-scoped search + per-command session key, --session-key override, fail-open exit 0 when the search throws, --out writes only when injected, and the default mode prints the body to stdout. recordTier1Diag is intentionally not unit-tested: TIER1_DIAG_LOG resolves os.homedir() at module load, and mocking the homedir globally is worse than the missing coverage. Co-Authored-By: Claude Fable 5 <[email protected]>
Author
|
Superseded by #63 — this branch is based on upstream main (~4.8k commits ahead of BenchAGI main → mergeStateStatus DIRTY, 13 conflicting files; the substrate can't land it). #63 carries the same six tier1 commits re-based onto BenchAGI main (tsgo clean, 111/111 tests). Keeping this branch around as the upstream-sync reference; close-or-keep is Cory's call on fork strategy. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Tier-1 retrieval-at-start: at the cold start of a fresh session/thread, search the agent's own memory index for the opening topic and prepend a small synthetic context file (
RETRIEVED-CONTEXT-TIER1.md) ahead of MEMORY.md — so a new thread about a prior topic starts already knowing the relevant prior decision instead of cold. Flag-gated (default OFF), bounded, tenant-safe, fail-open.This PR brings the branch from "embedded-runner only" to durable/portable:
~/.openclaw/logs/tier1-retrieval.jsonl) so firing is verifiable even when agent stderr is discarded.provider=claude-clirun throughsrc/agents/cli-runner/prepare.ts, not the embedded runner where tier1 was first wired, so it never fired on live turns. The retrieval now runs in the CLI runner's context assembly, gated on a cold-start bootstrap, prepending the slice ahead of MEMORY.md. Also hardens the query cleaner (extracts the real user message from relay preambles instead of embedding an ~8KB wrapper).openclaw memory tier1 <query>— new harness-agnostic entrypoint in the memory-core plugin CLI, following thememory searchpattern:--query,--agent,--json,--out,--session-key,--max-results,--max-bytes(caller overrides hard-capped at 8 hits / 8192 bytes). Honors thequery.tier1.enabledgate exactly as the runners do.Fail-open contract
Every retrieval miss — disabled, no signal, unavailable, timeout, error, below threshold, judge failure — degrades to the pre-tier1 behavior and the CLI exits 0. Only a missing query is a usage error (exit 1). The retrieval can refine a session start; it can never break one.
Tests
src/agents/tier1-retrieval.test.ts(21): query cleaning, body rendering, flag gate, timeout/error fail-open, score ordering, override clamps, and the reranker (re-ordering within topK, minScore filtering, topK truncation, fail-open on HTTP error / timeout / non-JSON / emptied set / missing config).extensions/memory-core/src/cli.test.ts(76, 8 new): usage error, disabled gate without opening a manager, enabled inject path with memory-scoped search,--session-key, fail-open exit 0 on search error,--outwrite-only-when-injected, default stdout body.Verified live from the built CLI: disabled gate returns
{"injected":false,...,"reason":"disabled"}exit 0; with a temp config + local embedder the full path returnsinjected:truewith the rendered slice.Config example
{ "agents": { "defaults": { "memorySearch": { "query": { "tier1": { "enabled": true, "maxResults": 4, "minScore": 0.45, "maxBytes": 1600, "timeoutMs": 1200 }, "reranker": { "enabled": true, "baseUrl": "http://127.0.0.1:11434", "model": "qwen2.5:7b-instruct", "timeoutMs": 6000, "minScore": 0.5, "topK": 8 } } } } } }🤖 Generated with Claude Code