Repository navigation
fix(recall): judge native Stop replies from session recall evidence - #1374
Conversation
- 让 opt-in judge 接受 Codex、Muse 与 ZCode Stop 事件中的 last_assistant_message,不再强制要求 Claude 格式 transcript。 - 从同一会话已确认并关联的 recall 日志提取候选路径及投票键,继续保留作用域 eligibility、受信知识根校验与每会话投票账本去重。 - 新增三种工具的原生回复、重复 Stop、继承用户范围只读和伪造召回标记回归,更新中英说明及技能排查指南。 - 329 个评分与会话归属单测、构建、类型检查、lint 和隔离模型替身的真实 CLI recall/Stop/投票 E2E 通过;修复前三个原生回复用例失败。
|
No confident findings in the specified diff. The PR description satisfies the testing requirement: it records a representative built-CLI run covering recall → native Codex Stop → persisted vote, plus a repeated Stop verifying no double-credit. The deterministic model stand-in and untested providers/native sessions are explicitly disclosed. Review was read-only; I did not run, build, or install PR code. |
jeff-r2026
left a comment
There was a problem hiding this comment.
@nateEc Thanks, the native-reply path looks good otherwise (ledger dedupe, scope guard and repeated-Stop handling all check out). One behavior question before merging:
In src/hook-handlers.ts the evidence source is chosen as finalReply ? recall-log : parseTranscriptForVotes(transcriptPath). So whenever a Stop payload carries last_assistant_message, the recall log is used even if transcript_path is also present. If Claude Code (or another Claude-family tool) also sends last_assistant_message in Stop, Claude sessions would silently switch from transcript evidence to recall-log evidence — e.g. a doc that only appears as a recall region in the transcript, without a settled linked run in the log, would stop being judged. That contradicts the PR description ("Existing transcript-backed behavior ... remain intact") and the new docs line.
Suggested fix: prefer the transcript when one exists, and only fall back to the native reply when there is no transcript:
const voteData = transcriptPath
? await parseTranscriptForVotes(transcriptPath)
: { finalAssistantText: finalReply, /* recall-log docs */ };plus a test where both transcript_path and last_assistant_message are set, asserting the transcript path is used.
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
@nateEc Approving at 3672eb3. We ran the built CLI on a local git team with an isolated HOME, TEAMAI_UPVOTE_JUDGE=1 and a stub model CLI, doing pull, then a real recall that returns the seeded learning, then Stop payloads shaped like Codex's:
| Stop | merge-base f3cd042 (judge calls / votes) | this head | main 744d982 |
|---|---|---|---|
| final reply, no transcript | 0 / 0 | 1 / 1 | 0 / 0 |
| same payload again | 0 / 0 | 1 / 1 | 0 / 0 |
| neither transcript nor reply | 0 / 0 | 1 / 1, no change | 0 / 0 |
| empty reply | 0 / 0 | 1 / 1, no change | 0 / 0 |
The judge's argv on this head carries the reply text and the candidate excerpt. tsc and lint pass. The three changed unit files (329 tests) pass, and their new Codex, Muse and ZCode cases fail on the base. native-stop-judge.test.ts fails on the base (expected +0 to be 1) and passes here, also with TMPDIR behind a symlink. Thirteen other e2e files that run recall, Stop or the upvote judge pass (224 tests).
Non-blocking notes:
- [P3 nit] No test covers the settled and linked gate on the reply path. The code reuses
viewRoot, which already applies those rules, so this is coverage only. - [P3 nit] When a payload carries both
transcript_pathandlast_assistant_message, the reply branch wins. We found no tool that sends both, so this is only worth a comment in the code. - [P3 nit] The new paragraph in
skill-data/core/references/troubleshooting.mdis one 219-character line, while the rest of the file wraps near 80 columns.
Not run: a real Codex, Muse or ZCode session, and Windows.
The CI review reports no findings and accepts the testing record. We agree. All CI jobs pass; the GitHub-provider e2e job is skipped by the fork policy. This PR and #1335, #1339, #1340 and #1341 all edit troubleshooting.md, so whichever lands second needs a rebase there.
@jeff-r2026 Sorry, we merged this before your question was answered. You were right: Claude Code sends |
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
|
Thanks for catching the Claude payload carrying both fields. I checked the follow-up in #1396: it keeps actual Claude transcript recall evidence and falls back to the settled recall log for native Codex rollout/reply payloads, with regressions for both. That PR is already owned by SaulMoro, so I am leaving the fix there rather than pushing a competing change to this merged branch. |
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
… sends the reply (#1396) Claude Code sends last_assistant_message in Stop along with transcript_path. Since #1374 the reply took precedence, so a Claude session was judged from the recall log instead of its transcript, and a doc the transcript shows but no settled run of the session records was no longer judged. Use the transcript whenever it shows a recall. Fall back to the reply and the recall log only when it shows none: Codex sends its own rollout as transcript_path, which is not a Claude transcript.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
…tor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page.
* feat(tools): state each tool's run-time and AI support in the descriptor (#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of #1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page. * docs: generate the tool support page from the descriptor (#1221) docs/guide/tool-support.md and its Chinese page are rendered from TOOLS and diffed by a snapshot test, as the command reference is. Headers and the legend are translated; cell text stays English on both pages. AGENTS.md records that exception and the review rule for descriptor updates. * docs(readme): mark usage and sessions the descriptor records, and JoyCode's inlined rules (#1221) Usage: skill use is recorded for OpenCode, Pi, Oh My Pi and DeepSeek Harness since #1245 (`features.stats`). Sessions: `teamai session` reads OpenCode (`TOOLS.opencode.sessions`). Rules: JoyCode's user rules are inlined into rules.txt, so they stay on, as for every other tool marked ✓¹. * docs(readme): add culture, workflows and scorers to the agent matrix, and check it against the descriptor (#1221) Learnings, codebase and teamwiki were the same for every agent, so Team Context is one column. That leaves room for culture (team culture and shared instructions need an instruction-blocks channel, which Qoder, Qoder CN, Kiro, Trae and Trae CN do not have), workflows (every agent drives one from its session) and scorers (the agents whose sessions teamai can judge: a session adapter in `TOOLS[tool].sessions`). Team Execution is ordered skills, rules, culture, agents, hooks, mcp, workflows, env, models, and the layer bullets follow it. readme-matrix.test.ts keeps every README's cells equal and checks the cells the code owns: culture, models, usage, scorers and ✓¹. * docs: regenerate the tool support page for Kiro's hook file (#1221)
- 让 opt-in judge 接受 Codex、Muse 与 ZCode Stop 事件中的 last_assistant_message,不再强制要求 Claude 格式 transcript。 - 从同一会话已确认并关联的 recall 日志提取候选路径及投票键,继续保留作用域 eligibility、受信知识根校验与每会话投票账本去重。 - 新增三种工具的原生回复、重复 Stop、继承用户范围只读和伪造召回标记回归,更新中英说明及技能排查指南。 - 329 个评分与会话归属单测、构建、类型检查、lint 和隔离模型替身的真实 CLI recall/Stop/投票 E2E 通过;修复前三个原生回复用例失败。
… sends the reply (Tencent#1396) Claude Code sends last_assistant_message in Stop along with transcript_path. Since Tencent#1374 the reply took precedence, so a Claude session was judged from the recall log instead of its transcript, and a doc the transcript shows but no settled run of the session records was no longer judged. Use the transcript whenever it shows a recall. Fall back to the reply and the recall log only when it shows none: Codex sends its own rollout as transcript_path, which is not a Claude transcript.
… (Tencent#1406) * feat(tools): state each tool's run-time and AI support in the descriptor (Tencent#1221) The env, workflows, recall, stats, AI backend, votes judge and HTTP agent hook rows of Tencent#1221's matrix become a required `features` facet, one `SupportCell` per row: the mark and the evidence letter, as the matrix writes them. A cell a later fix changed (Tencent#1193, Tencent#1224, Tencent#1245, Tencent#1374) says so in its note and keeps teamai's-code-only evidence until it is rerun. Data only: nothing reads it but the support page. * docs: generate the tool support page from the descriptor (Tencent#1221) docs/guide/tool-support.md and its Chinese page are rendered from TOOLS and diffed by a snapshot test, as the command reference is. Headers and the legend are translated; cell text stays English on both pages. AGENTS.md records that exception and the review rule for descriptor updates. * docs(readme): mark usage and sessions the descriptor records, and JoyCode's inlined rules (Tencent#1221) Usage: skill use is recorded for OpenCode, Pi, Oh My Pi and DeepSeek Harness since Tencent#1245 (`features.stats`). Sessions: `teamai session` reads OpenCode (`TOOLS.opencode.sessions`). Rules: JoyCode's user rules are inlined into rules.txt, so they stay on, as for every other tool marked ✓¹. * docs(readme): add culture, workflows and scorers to the agent matrix, and check it against the descriptor (Tencent#1221) Learnings, codebase and teamwiki were the same for every agent, so Team Context is one column. That leaves room for culture (team culture and shared instructions need an instruction-blocks channel, which Qoder, Qoder CN, Kiro, Trae and Trae CN do not have), workflows (every agent drives one from its session) and scorers (the agents whose sessions teamai can judge: a session adapter in `TOOLS[tool].sessions`). Team Execution is ordered skills, rules, culture, agents, hooks, mcp, workflows, env, models, and the layer bullets follow it. readme-matrix.test.ts keeps every README's cells equal and checks the cells the code owns: culture, models, usage, scorers and ✓¹. * docs: regenerate the tool support page for Kiro's hook file (Tencent#1221)
The opt-in recall adoption judge returns immediately unless Stop supplies a Claude transcript path. Codex, Muse and ZCode provide
last_assistant_message, so their final replies never reach the judge even when this session has recorded recalls.Accept that final reply and derive candidates from the session's settled, linked recall runs, using their vote keys, paths and scope. Preserve the eligible-scope filter, trusted-root excerpt grounding, tool-use credits and per-session vote ledger. A reply cannot invent candidates through recall-looking markers, inherited user docs stay read-only in project scope, and repeated Stop events do not double-credit. Existing transcript-backed behavior and default opt-in policy remain intact.
Refs #1221 ("teamai's own AI" / votes judge).
Validation:
npx vitest run src/__tests__/recall-attribution.test.ts src/__tests__/hook-handlers.test.ts src/__tests__/votes-judge.test.ts— 329 passed, covering transcript behavior, native replies, repeated Stop, inherited scope and grounding defenses.npx tsc --noEmit,npm run lint,npm run build— passed.npx vitest run --config vitest.e2e.config.ts src/__tests__/e2e/native-stop-judge.test.ts— passed. An isolated local git team runs pull → real recall → Codex-shaped Stop without a transcript → persisted vote; a second Stop leaves the count at one and makes no second judge call. Only the model CLI boundary is a deterministic local stand-in; recall, dispatch, grounding, ledger and reports use the built CLI.Updated bilingual knowledge guidance and the troubleshooting skill. No hosted model, native tool session, GitLab or GitHub provider was exercised locally.