Skip to content

Commit cc22cda

Browse files
anandgupta42claude
andauthored
feat(learn): learn team conventions from corrections and deliver them by retrieval (#1405)
* wip: learn command, RSI demo harness and research (experiment in progress) * wip: harness gate/detector fixes after first saas run * fix: learn UX — visible safety, distinct-feedback auto-remove, stdin feedback, guidance Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix: harden learn curator, promote validation and redaction after review - `validateCandidate` rejects unmanaged body lines, hand-edited frontmatter and duplicate ids; promote publishes the canonical serialization - `promote` and `rollback` delete `candidate.md` so a rolled-back version is not resurrected - `redactSecrets` `ASSIGNMENT` regex is linear; feedback and tool inputs are clipped before redaction - lint: NFKC + `\p{Cf}`/`\p{Cc}` stripping, line separators, markdown links, URL schemes, `//host`, bare domains, emails, session ids, more shell rules; text is stored normalized - CRLF playbooks parse; duplicate ids are re-id'd on parse - edit budget: 1 HELPFUL per bullet, 3 EDITs and 3 REMOVEs per reflection - distinct-feedback hash uses normalized content plus session or trajectory - model call has a `--timeout` abort signal; `--feedback -` on a TTY fails fast - `candidate.md`, `harmful.json` and `SKILL.md` are written via temp file and rename Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test: fix support-tickets verifier naming collapse; add RSI findings * docs: compare altimate-code learn with codex-engineer RSI signals * feat: capture learning signals automatically (user corrections, tool retries, review/ci) for learn - opt-in capture (`learn.capture` / `ALTIMATE_LEARN_CAPTURE=1`) via a Bus subscription started from instance bootstrap; local-only `.altimate-code/learn/signals.jsonl` with redaction, clipping and dedupe - conservative `correctionReason` classifier; first prompt of a session is never a correction - `tool_retry` signal when the same tool fails 3+ consecutive times (one per episode) - `learn signals`, `learn signal add`, `learn reflect --session` without `--feedback`, `learn reflect --pending`; signals are consumed only after a successful reflection - auto-reflect at the end of `run` (`learn.auto_reflect` / `ALTIMATE_LEARN_AUTO=1`, model `learn.model` / `ALTIMATE_LEARN_MODEL`); stages the candidate only, never promotes, never affects the exit code - reflector prompt: user feedback is in-chat corrections (expected behaviour), prior agent behaviour is the mistake Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix: recognise code-review phrasing as a correction; add corrections-only harness * feat: `skill publish --replace` updates your own same-name skill published from another checkout * docs: corrections-only RSI run results * docs: research on keeping learned lessons from bloating the context How coding-agent harnesses and memory frameworks bound always-on knowledge, evidence on instruction-count scaling and prompt caching, and a four-tier design (core, task-matched, archive, checks) for `altimate-code learn`. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test: rule-budget experiment (N=25/50/100, task-matched, pull) and results Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): retire contradicted lessons instead of keeping or losing them Measured: a playbook holding a stale lesson next to its replacement drops held-out pass to 1/9 (the agent follows the stale one), and a weak reflector that only marks stale lessons HARMFUL lets the curator auto-remove them, losing the convention (3/9 and 4/9 vs 9/9 with a strong reflector). - anchors: lessons overlap when they share a code identifier in backticks (affix-aware, generic SQL functions ignored) - curator: an overlapping ADD/EDIT must declare `supersedes` or `coexists`; explicit supersede replaces in place and runs before fuzzy dedupe; an overlapping ADD next to a lesson marked harmful in the same reflection supersedes it implicitly - replacement step: a lesson removed for contradiction without a replacement gets one narrow model call for the corrected rule (or NONE); failures are queued in pending-replacements.jsonl and retried (5 attempts), shown in `learn show` - coexist links persist in the playbook; promote refuses undeclared overlaps - reflector prompt: rewrite a contradicted lesson; HARMFUL alone loses it End to end (Gemini agent, outdated playbook, teammate corrections, 2 iterations): weak reflector 9/9, 9/9, 9/9 held-out vs 3/9, 4/9 on the previous code; strong reflector 9/9 unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test: conflict, drift and model-swap experiments for learned-lesson retirement - harness: --seed-playbook for the corrections loop, REVIEWER_MAX_TURNS env - budget/: applicable/overgeneral/conflict arms, stale seed, drift scripts - research: experiments 2 and 3 results Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): harden opt-in learning against races, leaks and lost state From an independent review of the full `learn` diff: - cross-process lock (Flock) around candidate, harmful, history, pending and signal state; reflection re-reads and re-curates under the lock; promote commits only the candidate hash it showed - redact credential CLI arguments, connection strings, emails and SSNs in every model input (digest, feedback, existing bullets); lint rejects them in lessons - lint rejects verification-weakening lessons (skip tests/review, `commit -n`) - trajectory export includes user prompts only with `--include-prompts`, redacted - drain capture at run end and instance disposal; quarantine malformed state - consume only signals that fit in the model request - reject/rollback discard pending recovery work - external user corrections reflect without a session - accurate advice for a publish rename that collides on update - test: bootstrap and run-end leave learn untouched when learning is disabled Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): close second-review gaps in redaction, lint, rebasing and locking - stale EDIT/REMOVE/HARMFUL on a bullet changed during reflection is rejected and its signals retried - redact before clipping; cover curl --user, Authorization Basic/Bearer, scheme://user:pass@ for any scheme - verification lint catches skip/disable of tests, CI, review and hooks, respecting negation - -p/-P treated as a password only for tools where it is one (mysql, sqlcmd, bcp...) - learn lock verifies its lease before every write; 10-minute stale threshold - bootstrap does not import capture when learning is disabled Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): restore redaction and lint coverage, keep feedback on stale rejections - redaction: attached -p for mysql/mariadb tools, line continuations, --proxy-user, key/value and YAML block credentials, NFKC + zero-width stripping - verification lint: broad targets restored, markdown stripped, clause-level negation for lists - a reflection with any stale-rejected delta keeps all its signals open - cap eviction never removes a bullet changed since the snapshot - replacement model calls run outside the lock - document the security posture (promote is the boundary; redaction and lint are guard rails) and the lock residual risk; promote prints a review notice Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): reflect with the session model; scope lint negation; restore password flags - reflection model: explicit -m > learn.model/ALTIMATE_LEARN_MODEL > the source session model > global default - verification lint: negation stops at a new clause - redact and lint mongosh/mongo -p, sshpass -p, redis-cli -a again - model test lets unrelated requests reach the real fetch Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(learn): union redaction, adjacency negation, frozen guard-rail corpus - each credential rule scans the text independently; matches are merged, so adding a rule cannot hide another - verification lint negates a bypass only when the negator directly precedes the verb, carried across or/nor only - guardrail-corpus.test.ts: 316 rows collected from five review rounds, asserted through redactor, lint, curate and promote validation Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * refactor(learn): flag verification-weakening lessons for human review instead of parsing negation Regex negation parsing could not be right in both directions across six review rounds. Lessons mentioning skipped or disabled verification are now staged with a warning; show and promote display it, and `promote --yes` refuses flagged lessons without `--allow-flagged`. Credentials and PII are still rejected. Redaction: linear scanning (100 KB < 10 ms), tool names only at command positions and outside quotes, docker login passwords, Bearer terminology left intact. Corpus: 352 rows. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): recognize tools mentioned in prose; tighten verification flags A real-CLI smoke test stored `sqlcmd ... -P hunter2` unredacted in signals.jsonl because tools were only recognized at command start. Tools now count at any word occurrence unless quoted or the value of a preceding flag; Bearer in Authorization headers always redacts; flags cover validation/verification and "no need to test", CI only as the bypass object; warning scan is bounded and linear; numeric IDs are no longer redacted. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs: record decision to store learn lessons as memory blocks Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs: backend requirements for storing learned rules in workspace memory Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs: learn v1 plan rev 2 (tiered retrieval, append-only additions, benchmark matrix) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): snapshot lesson store, migration from the playbook skill, 140-char lessons - approved/candidate/versions/retired JSON snapshots under .altimate-code/learn/<name>/, atomic, locked - curator 25-bullet cap replaced by learn.max_stored (default 1000), pinned lessons never evicted - idempotent, resumable migration from a learn-managed playbook skill; malformed files quarantined - learn-managed skills excluded from auto-load; publish exports on demand - new/edited lessons <=140 chars, longer ones grandfathered; no dbt defaults; learn search Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): harness-selected lessons per session, request and file - core tier (pinned, then net helpful) + local BM25 retrieval, frozen per session for cache safety - per-request additions appended to the user turn; file hook appends lessons for touched paths once - limits: learn.core_lessons, retrieved_lessons, budget_tokens, session_max_lessons (config + env) - shown.jsonl selection log; applied counters updated at flush, never in the prompt - projects without approved lessons are untouched Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): learn enable/disable/status and a recoverable reflection scheduler - enable/disable write learn.capture/auto_reflect to project config; status shows counts and limits - threshold (3 open signals), 10-min idle debounce cancelled by activity, flush-only exit - startup recovery after first idle, bounded by count/time with backoff - cross-process claims so the same signals are never reflected twice; run auto-reflect uses them Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): learn bootstrap seeds candidate lessons from past sessions - scope preview and confirmation (--yes for non-TTY, --dry-run sends nothing) - bounded, paged traversal of this project's root sessions; resumable, deduped extraction - batched signal writes; claimed, budgeted reflection; summary with tokens and estimated cost Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): learn import-reviews turns human PR review comments into lessons - GitHub/GHE via gh: access checks, GraphQL review threads (resolved) and review bodies of merged PRs - bots excluded (type, [bot] suffix, default list, config); short/LGTM-only dropped - paginated with rate-limit backoff and a resumable checkpoint; deduped ids; redacted - scope preview, --dry-run, --yes; claimed, budgeted reflection with a cost summary Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): import-reviews drops PR-author replies and duplicate comments A live dry run against AltimateAI/altimate-code kept the PR author's own thread replies as review signals (72 -> 17 after the fix). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): a quiet one-time nudge to try learning - TUI only, learning off, not run/JSON/CI/non-TTY; corrections counted in memory only - one toast after the second correction of a session; once per project, at most 3 overall - `learn nudge off` or `learn enable` stops it; only write is the global nudge state file Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): separate request/file limits, file-hook switch, reflection cost reporting learn.request_lessons (default 5), learn.file_hook / ALTIMATE_LEARN_FILE_HOOK, learn.file_lessons (default 5); learn reflect, scheduler history, bootstrap and import-reviews report tokens and estimated cost. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): learn pin/unpin, and git-ignore learn state except approved lessons Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): show lesson scope, rank file-hook lessons by path specificity Benchmark-driven: a staging-scoped cents lesson retrieved for an analysis request was over-applied; broad globs crowded the specific timestamp lesson out of the file-hook cap at 1,000 lessons. Lessons with trigger paths now render as [applies to: ...]; file-hook candidates rank by glob specificity, then request relevance; request-tier retrieval prefers path-compatible lessons. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): user guide with v1 benchmark results, and the benchmark harness - `docs/docs/configure/learn.md`: how learn works, delivery tiers (with scope rendering and file-hook path-specificity ranking), commands including `learn pin`/`unpin`, configuration, cost, privacy, benchmarks, limitations - `research/.../learn-v1-results.md`: all v1 arms (scale to 1,000 lessons, compression, topic switch, file hook, drift, bootstrap, review import, cost), before and after the retrieval fix - `experiments/rsi-workspace/harness/v1bench/`: harness, lesson pools, tasks, analysis script and result tables Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): scope specificity, truncated scopes, redacted session query; marker and guard fixes - `pathSpecificity`: ignore negated globs and count only leading literal segments, so `{a/b/c,models}/**` cannot outrank `models/staging/**` - `lessonLine`: append `(+N more)` when path scopes are truncated, so a lesson whose matching scope was cut does not look inapplicable - `Delivery.prepare`: persist the request text redacted in session state - wrap two learn additions in `config.ts` and `tui/runtime.ts` in `altimate_change` start/end blocks (Marker Guard) - widen the fork guard's `resolve-tools` → `sessionID` window to fit the marked `lessons` argument Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): correct guide claims against the code; drop a tracker-style id from a pool lesson - correct the guide where it overstated behaviour: what reflection sends to the model provider, retries after backoff, pin limits, behaviour with capture off, the default model for imports, publish conflicts, session state location, compaction and the cache, `--feedback` not being stored - benchmark pool: replace a tracker-style ticket key with `PROJ-123` in one distractor lesson (the tracker-leak check flags it; the benchmark ran with the old text, which no task needed) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): wrap the local config schema call in altimate_change markers Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore: ignore synthetic credentials in learn redaction tests for GitGuardian Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): address independent review findings - lock: recover a lock whose owner process on this host is dead; prompt-path delivery waits at most ~5 s for the lock, then skips lessons for that call - reflection whose proposals are all rejected consumes its signals and records a rejected outcome, so idle reflection, `bootstrap` and `import-reviews` no longer repeat the same model call - delivery: skip torn `shown.jsonl` lines and unparseable stores instead of disabling every store; usage counters move to git-ignored `usage.json`, so `approved.json` is no longer rewritten each session - delivery sanitizes every lesson (whitespace in paths and text, 140-char cap, curator lint) and budgets request and file notes - redaction: URL passwords containing `@`, typed and arrow assignments, escaped JSON, `*_PASS`/`*_TOKEN`-style env keys - `history.jsonl`: rejected text and reasons are redacted, capped, mode 0600 - `skill publish --replace`: adopt the remote id only after the PATCH succeeds - `run` exit waits only for its own session's reflection, at most 60 s - lint: more injection and verification-weakening phrasings - `import-reviews`: only owners, members and collaborators, and only comments made before merge (`--any-author` disables the association filter) - CLI: `learn enable` survives a failed nudge dismissal and reports a config that keeps capture off; `status` skips torn history lines; repeated `--bots`; a working retry command after a failed publish Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * chore: move the learn benchmark harness to its own PR (#1407) `experiments/rsi-workspace/` is research tooling, not product code; it lands unchanged in #1407 so this PR's review covers the feature only. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): address review comments on the research notes - backend requirements: project-scoped visibility follows open question 2; retired records carry the replacing id and reason (R7) - cache-cost arithmetic: write price replaces the input price, not added to it - design example: scoped `applyPaths`, no provenance in the published skill - findings: state that the headline excludes the defective fourth task - v1 plan: `learn graduate` is deferred; compaction rebuild is budgeted - SkillsBench numbers reconciled with the arXiv abstract Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): address bot review comments on #1405 - anchors: keep identifiers in grouped expressions; ignore SQL string literals and comparison operators - correction classifier: ignore reassurance, pasted diagnostics and quoted questions - capture: distinct retry episodes keep their part id; per-session caches are bounded - curator: freshness guard before automatic removal; rejected removals no longer authorize implicit supersession; removal reasons redacted - reflection prompts escape untrusted sections; identifiers must be backticked - digest bounds aggregate prompts so calls and the final message survive - run-end reports redact provider errors; post-commit status is best-effort - signals: CI precedence over review in mixed batches - store: validate grandfathered entries; migrate candidate-only legacy stores - `.gitignore`: append missing managed rules to an existing file - `import-reviews`: restart mutable searches and dedupe instead of resuming a stale cursor - `bootstrap --max-reflections 0` needs no model - `learn enable` honours nested config files; `learn` telemetry attribution - startup survives capture/scheduler import failures - compaction keeps the original request id for attachment-only replays - `skill publish --replace` adopts only skills in the linked workspace - tests isolate global state; guide updated for the new behaviour Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): count only runs whose agent turn completed The benchmark analysis also scored runs that ended without a completed turn (mostly an exhausted turn budget). Rescoring the saved runs with the same verifier and task project, excluding those 40 of 612 runs, moves held-out results by at most one run per arm (no lessons: 3/9 -> 2/9) and lowers several control results. Control failures are mostly incomplete runs, with and without lessons; real over-application is 1 of 6 runs in three retrieval arms. Nothing was rerun. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): record the verifier weaknesses kept unchanged for comparability Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): address the second round of review comments - lock: refuse a symlinked `.gitignore`; process liveness check tolerates zombies and reused pids - delivery: keep grandfathered lessons deliverable; a failed lock does not consume a compaction marker; one bad snapshot no longer drops the session ledger - scheduler: run exit no longer disables startup recovery for the process - reflection: deadline is checked before committing, not after; history writes are 0600 everywhere; replacement prompts require backticked identifiers; prompt sections escape delimiters without rewriting code - redaction: escaped whitespace in env secrets; credential keys matched at word boundaries (no more `MAX_TOKENS`) - lint: no token cap for bare domains, no false positives on file names, `--force-with-lease` flagged - `import-reviews`: model resolved only when a reflection can run; bounded imports keep progress past the PR limit - lesson ids: wider random space, no collision with retired ids - config writes are atomic; `learn status` collapses torn-history warnings - `skill publish`: carry the original scope; 403/404 retries keep replace - tests isolate global state and learn env overrides - docs: incomplete runs score as failures (not excluded); guide and results doc wording; R3 edit authorization Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): address the third round of review comments - lock: read an existing `.gitignore` without write access; stale-lock takeover is left to the heartbeat grace (prompt-path delivery still gives up after ~5 s) - `learn enable`/`disable`: write through a symlinked config to its target and keep the file mode - delivery: a failed grandfathering lookup skips the unsafe snapshot - history: removed lesson text is capped and redacted too - `import-reviews`: completed revisions are compacted; skill lookup rejects inconsistent page counts - anchors: `exists` and `values` are keywords - docs: delivery may write `shown.jsonl`/`usage.json`; `--any-author` noted; results headline notes the 8/9 arm Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): open .gitignore non-blocking; tighten two tests - a FIFO `.gitignore` can no longer block the learn lock - the bootstrap interruption fixture always releases its pending startup - the global-config test proves the global file is loaded Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * test(learn): assert delivery counts in usage.json, not approved.json The prompt integration test still expected the per-session counter in `approved.json`; delivery now records it in git-ignored `usage.json` and leaves the committed file unchanged (the CI TypeScript job failure). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * perf(learn): bound redaction work when digesting very large tool values The digest redacted each whole tool input, output and error before clipping it to a few hundred characters, so a 1 MB value cost ~100 ms locally and ~240 ms on CI (the TypeScript job's timing test). Redaction now scans a bounded window well past the shown prefix: a 1 MB input digests in ~1 ms. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): mask credentials left open by the bounded redaction window The previous change clipped a value before its closing delimiter could be seen, so `scheme://user:<long password>@host` or `"password": "<long>"` cut by the window left the start of the secret visible. Redaction now scans a 64 KB window and, when a value is cut, masks a URL credential or quoted secret that the cut leaves open (linear scans; no end-anchored regex). Top-level string inputs are bounded too, and leading whitespace is trimmed before the window is taken. Tests now fail when redaction or the masking is removed. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): remove internal identifiers from the research notes Drop the test tenant name, workspace id and GCP project number; the passages now say "a test tenant", "a test workspace" and "a GCP organization policy". Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * feat(learn): one kill switch for the whole feature `learn.enabled: false` or `ALTIMATE_LEARN=0` turns off lesson delivery, capture, scheduling, auto-reflection and the TUI nudge together; the env var overrides config both ways. Default on, so behaviour is unchanged. Explicit `learn` commands still run; `learn status` reports the switch and `learn enable` does not flip it. Capture/auto-reflect flag resolution is shared in one helper. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * docs(learn): say precisely what stays on when capture is off The guide introduction and the CLI help said learning is off by default; capture and reflection are, while approved-lesson delivery and the one-time reminder stay on unless the kill switch is off. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): address the Codex review - shared config schema accepts every `learn` key (including `enabled`), so the config API no longer drops them; SDK types and OpenAPI regenerated - file lessons are delivered for tools run inside the batch tool - a first message is never classified as a correction because of a later assistant reply - published playbooks load on teammates' checkouts; the exported skill is suppressed only where a same-name local approved store exists - a successful manual reflection clears the automatic backoff - export refuses unmanaged skill directories, unexpected files and symlinks - results doc: disclose two benchmark confounds found in review (topic-switch work directory name, compression-pair lesson order), tracked in #1408 Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): refuse symlinked learn paths; review round 8 - every learn write goes through one guard that rejects symlinks at the target and in any ancestor under the project root (no-follow opens); the prompt path skips delivery, CLI commands fail loudly - the kill switch also stops published learn skills from auto-loading - `learn disable` reports environment overrides that keep capture on - export recovers its own stale temps and detects edited managed exports; rollback refreshes the export - capture orders by stored message order, not id, and keeps text parts that arrive before their message; direct-feedback reflection clears backoff; shared-worktree reflections run in their originating instance Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): review round 9 - the kill switch also hides learn-managed skills from the skill list and the skill tool - path triggers are linted at promotion and delivery; unsafe lessons dropped - export keeps temp files it cannot prove it created; rollback restores approved.json even when the export was hand-edited, with a note - deferred startup recovery is retried when its directory opens - skill publish lookup dedupes ids repeated across pages - over-budget request notes are ignored on resume - symlink write test asserts with stat instead of fs.promises.exists Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): review round 10, including two regressions from round 9 - request-note cleanup trusts only server-recorded provenance, so client text is never dropped; notes are retired immutably, which keeps the plan-layer source-structure test passing - disabled learn skills are filtered before LLM skill selection and from cached selections - cancelled reflections with open signals stay retryable; deferred recovery lookups do not consume the time budget - a skill test removes an originally unset OPENCODE_TEST_HOME Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * fix(learn): review round 11 - a failed signal-store read after a cancelled startup reflection refunds the recovery attempt, so open signals stay retryable - the skill selector cache is keyed by the candidate skill set, so lesson skills return after learning is re-enabled in the same process Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent f8191ab commit cc22cda

136 files changed

Lines changed: 27372 additions & 62 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitguardian.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,11 @@ paths-ignore:
2020
# (PWD here is the unix env var for present working directory, not a
2121
# password). Test exercises symlink/cwd resolution.
2222
- packages/opencode/test/cli/tui/thread.test.ts
23+
# `learn` redaction and guard-rail tests: synthetic credentials (`hunter2`,
24+
# AWS's documented `AKIAIOSFODNN7EXAMPLE`, fake JWT/basic-auth strings) that
25+
# verify secrets are scrubbed before signals are stored or sent to a model.
26+
- packages/opencode/test/altimate/learn/guardrail-corpus.test.ts
27+
- packages/opencode/test/altimate/learn/review-redaction.test.ts
28+
- packages/opencode/test/altimate/learn/digest.test.ts
29+
- packages/opencode/test/altimate/learn/import-reviews.test.ts
30+
- packages/opencode/test/altimate/learn/sensitive-data.test.ts

‎docs/docs/configure/learn.md‎

Lines changed: 586 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ nav:
9696
- Memory Tools: data-engineering/tools/memory-tools.md
9797
- Custom Tools: configure/tools/custom.md
9898
- Skills: configure/skills.md
99+
- Learn: configure/learn.md
99100
- Commands: configure/commands.md
100101
- Validators: data-engineering/validators.md
101102
- Trace: configure/trace.md
@@ -177,6 +178,7 @@ nav:
177178
- Memory Tools: data-engineering/tools/memory-tools.md
178179
- Custom Tools: configure/tools/custom.md
179180
- Skills: configure/skills.md
181+
- Learn: configure/learn.md
180182
- Commands: configure/commands.md
181183
- Validators: data-engineering/validators.md
182184
- Trace: configure/trace.md

‎packages/core/src/v1/config/config.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,67 @@ export const Info = Schema.Struct({
244244
// altimate_change end
245245
}),
246246
),
247+
// altimate_change start — share every learning setting across config files and HTTP APIs
248+
learn: Schema.optional(
249+
Schema.Struct({
250+
enabled: Schema.optional(Schema.Boolean).annotate({
251+
description:
252+
"Enable lesson delivery, capture, automatic reflection and the TUI reminder (default: true). Env: ALTIMATE_LEARN overrides in both directions.",
253+
}),
254+
capture: Schema.optional(Schema.Boolean).annotate({
255+
description:
256+
"Record learning signals (user corrections, repeated tool failures) to .altimate-code/learn/signals.jsonl (default: false). Stored locally; reflection sends redacted text to the model provider. Env: ALTIMATE_LEARN_CAPTURE=1.",
257+
}),
258+
auto_reflect: Schema.optional(Schema.Boolean).annotate({
259+
description:
260+
"At the end of `run`, reflect on open signals into the playbook candidate (default: false; requires capture). Env: ALTIMATE_LEARN_AUTO=1.",
261+
}),
262+
recovery_max_reflections: Schema.optional(NonNegativeInt).annotate({
263+
description:
264+
"Maximum reflections on leftover signals at startup (default: 3). Env: ALTIMATE_LEARN_RECOVERY_MAX_REFLECTIONS.",
265+
}),
266+
recovery_max_seconds: Schema.optional(NonNegativeInt).annotate({
267+
description:
268+
"Time budget in seconds for startup recovery (default: 300). Env: ALTIMATE_LEARN_RECOVERY_MAX_SECONDS.",
269+
}),
270+
model: Schema.optional(Schema.String).annotate({
271+
description: "Model (provider/model) for auto-reflect. Env: ALTIMATE_LEARN_MODEL. Default: the default model.",
272+
}),
273+
review_bots: Schema.optional(Schema.mutable(Schema.Array(Schema.String))).annotate({
274+
description: "Extra bot logins excluded by import-reviews (default: []).",
275+
}),
276+
core_lessons: Schema.optional(NonNegativeInt).annotate({
277+
description: "Maximum core lessons at session start (default: 15). Env: ALTIMATE_LEARN_CORE_LESSONS.",
278+
}),
279+
retrieved_lessons: Schema.optional(NonNegativeInt).annotate({
280+
description: "Maximum retrieved lessons at session start (default: 15). Env: ALTIMATE_LEARN_RETRIEVED_LESSONS.",
281+
}),
282+
request_lessons: Schema.optional(NonNegativeInt).annotate({
283+
description:
284+
"Maximum lessons added per request after session start (default: 5; 0 disables). Env: ALTIMATE_LEARN_REQUEST_LESSONS.",
285+
}),
286+
file_hook: Schema.optional(Schema.Boolean).annotate({
287+
description:
288+
"Deliver lessons matching file paths or identifiers (default: true). Env: ALTIMATE_LEARN_FILE_HOOK=0 disables.",
289+
}),
290+
file_lessons: Schema.optional(NonNegativeInt).annotate({
291+
description: "Maximum lessons added per file event (default: 5; 0 disables). Env: ALTIMATE_LEARN_FILE_LESSONS.",
292+
}),
293+
budget_tokens: Schema.optional(NonNegativeInt).annotate({
294+
description:
295+
"Token budget for the frozen Team rules section (default: 1500). Env: ALTIMATE_LEARN_BUDGET_TOKENS.",
296+
}),
297+
session_max_lessons: Schema.optional(NonNegativeInt).annotate({
298+
description:
299+
"Maximum distinct lessons shown in one session (default: 40). Env: ALTIMATE_LEARN_SESSION_MAX_LESSONS.",
300+
}),
301+
max_stored: Schema.optional(PositiveInt).annotate({
302+
description:
303+
"Maximum stored lessons, enforced during curation; pinned lessons are retained (default: 1000). Env: ALTIMATE_LEARN_MAX_STORED.",
304+
}),
305+
}),
306+
),
307+
// altimate_change end
247308
// altimate_change start - tracing config (re-applied from main during the v1.17.9 reconciliation)
248309
tracing: Schema.optional(
249310
Schema.Struct({
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Learning reminder
2+
3+
`learn.enabled` is the master switch and defaults to `true`. The shared
4+
`learnEnabled` helper in `config.ts` resolves it; `ALTIMATE_LEARN=0`/`false`
5+
disables and `1`/`true` enables, case-insensitively, overriding config in both
6+
directions. When disabled, lesson delivery and its `.sessions/`, `shown.jsonl`
7+
and `usage.json` writes, capture, scheduling, automatic reflection, startup
8+
recovery and the TUI reminder do not start. Explicit `learn` commands still
9+
work. `learn status` reports the switch, and `learn enable` sets capture and
10+
automatic reflection without changing it, explaining if learning stays disabled.
11+
12+
When capture is off and the master switch is on, the interactive TUI can show one
13+
quiet reminder after two user corrections in a session, on the first idle event
14+
after the second correction.
15+
It uses the existing local correction classifier, loaded lazily in the TUI. It
16+
does not call a model, capture signals, or save message text, correction counts,
17+
or classifier results. Counts are per session and disappear when the TUI exits.
18+
Only messages following a completed assistant turn count.
19+
20+
The reminder never runs in `run`, JSON output, CI (including an empty `CI`
21+
environment variable), or with non-TTY input or output. It is suppressed when
22+
`learn.capture` is true or `ALTIMATE_LEARN_CAPTURE` enables capture.
23+
It is also suppressed in `attach`: a remote server's capture environment is
24+
not available to the TUI, and the suggested local enable command may not apply
25+
to that server. Locally launched TUIs share their worker's capture environment.
26+
27+
With capture off and the master switch on, already-approved lessons are still
28+
delivered (writing `.sessions/` and possibly `shown.jsonl` and `usage.json` when
29+
lessons are shown). The reminder can also write
30+
`learn-nudge.json` in the global state directory (`$XDG_STATE_HOME/altimate-code`,
31+
normally `~/.local/state/altimate-code`). It contains only `shownProjectHashes`,
32+
`totalCount`, and `dismissed`. The reminder is limited to once per project and
33+
three times across all projects. A project is identified by a SHA-256 hash of
34+
its existing project ID (shared by linked worktrees), with a canonical root
35+
path fallback for directories without a repository. Neither the ID nor the raw
36+
path is stored.
37+
38+
Updates use an exclusive sibling staging file and atomic rename, so concurrent
39+
TUIs share the same limits. No project learning files are created. If state
40+
cannot be safely read or updated, the reminder is skipped. An interrupted
41+
staging write is left in place rather than risking concurrent replacement.
42+
43+
The TUI's toast has no dismissal button. It includes the command
44+
`altimate-code learn nudge off` as its “Don't show again” action. That command
45+
and `altimate-code learn enable` permanently dismiss reminders in all projects,
46+
including after learning is subsequently disabled.
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
// altimate_change - new file
2+
3+
const STOP = new Set([
4+
"select", "from", "where", "as", "not", "and", "or", "is", "in", "null", "true", "false",
5+
"case", "when", "then", "else", "end", "on", "join", "by", "group", "order", "having", "qualify", "exists", "values",
6+
"distinct", "over", "partition", "with", "union", "all", "asc", "desc", "limit", "offset",
7+
"coalesce", "cast", "try_cast", "safe_cast", "sum", "count", "min", "max", "avg",
8+
"lower", "upper", "trim", "ltrim", "rtrim", "round", "abs", "ceil", "floor", "nullif", "ifnull", "nvl",
9+
"date_trunc", "date", "timestamp", "dateadd", "datediff", "extract", "current_date", "current_timestamp",
10+
"concat", "substring", "substr", "replace", "length", "greatest", "least", "ref", "source", "config", "var", "if",
11+
"col", "column", "table", "entity", "x", "sql",
12+
])
13+
14+
const GROUPING_KEYWORDS = new Set([
15+
"and", "or", "not", "in", "where", "on", "when", "then", "else", "having", "select",
16+
"from", "case", "by", "as", "is", "exists", "over", "partition", "join", "with", "values", "distinct", "qualify",
17+
])
18+
19+
/** Remove call arguments, including nested calls, while keeping the function identifier. */
20+
function withoutArguments(text: string): string {
21+
let depth = 0
22+
let quote = ""
23+
let result = ""
24+
for (const char of text) {
25+
if (depth && quote) {
26+
if (char === quote) quote = ""
27+
continue
28+
}
29+
if (depth && (char === "'" || char === '"')) {
30+
quote = char
31+
continue
32+
}
33+
if (char === "(") {
34+
const name = result.match(/[a-z_][a-z0-9_]*\s*$/)?.[0].trim()
35+
if (depth || (name && !GROUPING_KEYWORDS.has(name))) {
36+
if (!depth) result += " "
37+
depth++
38+
} else result += char
39+
} else if (char === ")") {
40+
if (depth) depth--
41+
else result += char
42+
} else if (!depth) result += char
43+
}
44+
return result
45+
}
46+
47+
/** Only backtick code identifiers establish overlap; prose-only contradictions rely on the reflector and replacement step. */
48+
export function anchors(text: string): Set<string> {
49+
const result = new Set<string>()
50+
for (const span of text.matchAll(/(`+)([\s\S]*?)\1/g)) {
51+
const code = withoutArguments(span[2].toLowerCase().replace(/'(?:''|\\[\s\S]|[^'\\])*(?:'|$)/g, " "))
52+
.replace(/<[a-z_][a-z0-9_]*>/g, "")
53+
for (const token of code.match(/[a-z_][a-z0-9_]*/g) ?? []) {
54+
if (!STOP.has(token) && /[a-z]/.test(token)) result.add(token)
55+
}
56+
}
57+
return result
58+
}
59+
60+
function matches(affix: string, token: string): boolean {
61+
return (affix.startsWith("_") && token.endsWith(affix)) || (affix.endsWith("_") && token.startsWith(affix))
62+
}
63+
64+
/** Shared identifiers, using the affix itself when it matches a concrete identifier. */
65+
export function sharedAnchors(a: string, b: string): string[] {
66+
const shared = new Set<string>()
67+
const left = anchors(a)
68+
const right = anchors(b)
69+
for (const x of left) {
70+
for (const y of right) {
71+
if (x === y || matches(x, y)) shared.add(x)
72+
else if (matches(y, x)) shared.add(y)
73+
}
74+
}
75+
return [...shared].sort()
76+
}
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
// altimate_change - new file
2+
//
3+
// Auto-reflect from the scheduler or the end of `altimate-code run`: open learning signals use the
4+
// same claimed reflect path in-process and report one line. Edits are only staged as the candidate; promotion
5+
// stays explicit (`learn promote`). Never throws: learning must not change a run's outcome.
6+
import * as Playbook from "./playbook"
7+
import { DEFAULT_MAX_STORED, summarize } from "./curator"
8+
import { autoReflectEnabled, captureEnabled, flushCapture } from "./capture"
9+
import { DEFAULT_TIMEOUT_MS, providerGenerate } from "./reflect"
10+
import { candidatePath, errText, reflectSessionSignals, sourceFromSession } from "./session-reflect"
11+
import { readScheduleState, recordReflection } from "./schedule-state"
12+
import type { InstanceContext } from "@/project/instance-context"
13+
import type { ReflectionOptions } from "./schedule"
14+
import { Effect } from "effect"
15+
import { InstanceRef } from "@/effect/instance-ref"
16+
import { createUsageTracker, type UsageSummary } from "./usage"
17+
import { redactSecrets } from "./digest"
18+
import { Log } from "@/util/log"
19+
20+
const log = Log.create({ service: "learn.auto" })
21+
export const RUN_EXIT_TIMEOUT_MS = 60_000
22+
23+
export interface AutoReflectOutcome {
24+
/** One line for the user. */
25+
line: string
26+
ok: boolean
27+
summary?: string
28+
signals?: number
29+
usage?: UsageSummary
30+
}
31+
32+
/** `learn.model` (config), overridden by ALTIMATE_LEARN_MODEL; otherwise use the source session's model. */
33+
export function learnModel(cfgModel: string | undefined, env: NodeJS.ProcessEnv = process.env): string | undefined {
34+
return env["ALTIMATE_LEARN_MODEL"]?.trim() || cfgModel?.trim() || undefined
35+
}
36+
37+
/** The local lesson cap is enforced during curation; pinned lessons are never evicted. */
38+
export function learnMaxStored(configured?: number, env: NodeJS.ProcessEnv = process.env): number {
39+
const override = env["ALTIMATE_LEARN_MAX_STORED"]?.trim()
40+
const value = override ? Number(override) : (configured ?? DEFAULT_MAX_STORED)
41+
if (!Number.isSafeInteger(value) || value < 1)
42+
throw new Error("learn.max_stored / ALTIMATE_LEARN_MAX_STORED must be a positive integer.")
43+
return value
44+
}
45+
46+
export function describeOutcome(summary: string, signals: number, candidate: string | undefined): string {
47+
return `learn: ${signals} signal${signals === 1 ? "" : "s"} -> ${summary}${candidate ? `; staged ${candidate}, review with \`altimate-code learn show\`` : ""}`
48+
}
49+
50+
/** Runs inside the instance context (after `run`'s session has completed). Returns undefined when disabled or idle. */
51+
export async function autoReflectSession(
52+
sessionID: string,
53+
options: Partial<ReflectionOptions> & { context?: InstanceContext; waitForScheduled?: boolean } = {},
54+
): Promise<AutoReflectOutcome | undefined> {
55+
if (!options.waitForScheduled) return runReflection(sessionID, options)
56+
const deadline = Math.min(options.deadline ?? Infinity, Date.now() + RUN_EXIT_TIMEOUT_MS)
57+
const abort = new AbortController()
58+
const abortSignal = options.abortSignal ? AbortSignal.any([options.abortSignal, abort.signal]) : abort.signal
59+
let timer: ReturnType<typeof setTimeout> | undefined
60+
const timeout = new Promise<AutoReflectOutcome>((resolve) => {
61+
timer = setTimeout(() => {
62+
abort.abort()
63+
// Publication may already be completing; the deadline cannot promise signals remain open.
64+
const line = "learn: auto-reflect deadline reached; check learn status before retrying"
65+
log.warn(line, { sessionID })
66+
resolve({ ok: false, line })
67+
}, Math.max(0, deadline - Date.now()))
68+
})
69+
try {
70+
return await Promise.race([
71+
runReflection(sessionID, { ...options, deadline, abortSignal, recoverPending: false }),
72+
timeout,
73+
])
74+
} finally {
75+
clearTimeout(timer)
76+
}
77+
}
78+
79+
async function runReflection(
80+
sessionID: string,
81+
options: Partial<ReflectionOptions> & { context?: InstanceContext; waitForScheduled?: boolean; recoverPending?: boolean },
82+
): Promise<AutoReflectOutcome | undefined> {
83+
let root: string | undefined
84+
const tracker = createUsageTracker()
85+
const ready = () => !options.abortSignal?.aborted && (options.shouldContinue?.() ?? true) &&
86+
(options.deadline === undefined || Date.now() < options.deadline)
87+
try {
88+
const { Config } = await import("@/config/config")
89+
const learn = options.context ? Config.peek(options.context)?.learn : (await Config.get()).learn
90+
if (captureEnabled(learn)) await flushCapture()
91+
if (!autoReflectEnabled(learn) || !ready()) return undefined
92+
const { Instance } = await import("@/project/instance")
93+
const context = options.context ?? Instance.current
94+
root = context.worktree !== "/" ? context.worktree : context.directory
95+
// `run` may reach its end while an idle-triggered reflection owns the batch. Finish that
96+
// session's work before disposal invalidates it. Startup recovery is not part of this wait.
97+
const scheduled = options.waitForScheduled
98+
? await import("./schedule").then((m) => m.drainScheduledReflections(root!, sessionID, options.abortSignal))
99+
: undefined
100+
if (!ready()) return scheduled
101+
if (((await readScheduleState(root)).recoveries[sessionID]?.retryAt ?? 0) > Date.now()) return scheduled
102+
const modelArg = learnModel(learn?.model)
103+
const modelLabel = modelArg ? `model ${modelArg}` : undefined
104+
const out = await reflectSessionSignals({
105+
root,
106+
name: Playbook.DEFAULT_NAME,
107+
sessionID,
108+
signalIDs: options.signalIDs,
109+
shouldContinue: ready,
110+
recoverPending: options.recoverPending,
111+
loadSource: (id) => sourceFromSession(id, context),
112+
maxStored: learnMaxStored(learn?.max_stored),
113+
modelLabel,
114+
getGenerate: async (source) => {
115+
const { Provider } = await import("@/provider/provider")
116+
const { AppRuntime } = await import("@/effect/app-runtime")
117+
const model = modelArg ? Provider.parseModel(modelArg) : source.model
118+
const deadlineSignal = options.deadline === undefined ? undefined : AbortSignal.timeout(Math.max(1, options.deadline - Date.now()))
119+
const abortSignal = options.abortSignal && deadlineSignal
120+
? AbortSignal.any([options.abortSignal, deadlineSignal])
121+
: options.abortSignal ?? deadlineSignal
122+
return AppRuntime.runPromise(providerGenerate(model, DEFAULT_TIMEOUT_MS, abortSignal, tracker.add).pipe(Effect.provideService(InstanceRef, context)))
123+
},
124+
})
125+
if (out.status === "none") return scheduled
126+
const { curated } = out.result
127+
// Candidate/history and signal consumption have committed; status must not change that outcome.
128+
await recordReflection(root, sessionID, "success", summarize(curated), undefined, undefined, out.result.usage)
129+
.catch((e) => log.warn("auto-reflect status update failed", { error: redactSecrets(errText(e)) }))
130+
return {
131+
ok: true,
132+
summary: summarize(curated),
133+
signals: out.signals.length,
134+
usage: out.result.usage,
135+
line: describeOutcome(
136+
summarize(curated),
137+
out.signals.length,
138+
curated.applied.length > 0 ? candidatePath(root, Playbook.DEFAULT_NAME) : undefined,
139+
),
140+
}
141+
} catch (e) {
142+
if (options.abortSignal?.aborted) return undefined
143+
const error = redactSecrets(errText(e))
144+
log.warn("auto-reflect skipped", { error })
145+
if (root) await recordReflection(root, sessionID, "failure", error, undefined, undefined, tracker.usage).catch(() => {})
146+
return { ok: false, line: `learn: auto-reflect skipped (${error}); signals stay open for \`learn reflect --session ${sessionID}\`` }
147+
}
148+
}

0 commit comments

Comments
 (0)