Skip to content

fix(recall): judge native Stop replies from session recall evidence - #1374

Merged
SaulMoro merged 1 commit into
Tencent:mainfrom
nateEc:fix/1221-native-stop-judge
Oct 25, 2026
Merged

SaulMoro merged 1 commit into
Tencent:mainfrom
nateEc:fix/1221-native-stop-judge

Conversation

@nateEc

@nateEc nateEc commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

The opt-in recall adoption judge returns immediately unless Stop supplies a Claude transcript path. Codex, Muse and ZCode provide last_assistant_message, so their final replies never reach the judge even when this session has recorded recalls.

Accept that final reply and derive candidates from the session's settled, linked recall runs, using their vote keys, paths and scope. Preserve the eligible-scope filter, trusted-root excerpt grounding, tool-use credits and per-session vote ledger. A reply cannot invent candidates through recall-looking markers, inherited user docs stay read-only in project scope, and repeated Stop events do not double-credit. Existing transcript-backed behavior and default opt-in policy remain intact.

Refs #1221 ("teamai's own AI" / votes judge).

Validation:

  • Before the fix: the three Codex/Muse/ZCode reply regressions failed; the no-candidate control passed.
  • npx vitest run src/__tests__/recall-attribution.test.ts src/__tests__/hook-handlers.test.ts src/__tests__/votes-judge.test.ts — 329 passed, covering transcript behavior, native replies, repeated Stop, inherited scope and grounding defenses.
  • npx tsc --noEmit, npm run lint, npm run build — passed.
  • Real built CLI: npx vitest run --config vitest.e2e.config.ts src/__tests__/e2e/native-stop-judge.test.ts — passed. An isolated local git team runs pull → real recall → Codex-shaped Stop without a transcript → persisted vote; a second Stop leaves the count at one and makes no second judge call. Only the model CLI boundary is a deterministic local stand-in; recall, dispatch, grounding, ledger and reports use the built CLI.

Updated bilingual knowledge guidance and the troubleshooting skill. No hosted model, native tool session, GitLab or GitHub provider was exercised locally.

- 让 opt-in judge 接受 Codex、Muse 与 ZCode Stop 事件中的 last_assistant_message,不再强制要求 Claude 格式 transcript。
- 从同一会话已确认并关联的 recall 日志提取候选路径及投票键,继续保留作用域 eligibility、受信知识根校验与每会话投票账本去重。
- 新增三种工具的原生回复、重复 Stop、继承用户范围只读和伪造召回标记回归,更新中英说明及技能排查指南。
- 329 个评分与会话归属单测、构建、类型检查、lint 和隔离模型替身的真实 CLI recall/Stop/投票 E2E 通过;修复前三个原生回复用例失败。
@github-actions

Copy link
Copy Markdown

No confident findings in the specified diff.

The PR description satisfies the testing requirement: it records a representative built-CLI run covering recall → native Codex Stop → persisted vote, plus a repeated Stop verifying no double-credit. The deterministic model stand-in and untested providers/native sessions are explicitly disclosed.

Review was read-only; I did not run, build, or install PR code.

@jeff-r2026 jeff-r2026 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nateEc Thanks, the native-reply path looks good otherwise (ledger dedupe, scope guard and repeated-Stop handling all check out). One behavior question before merging:

In src/hook-handlers.ts the evidence source is chosen as finalReply ? recall-log : parseTranscriptForVotes(transcriptPath). So whenever a Stop payload carries last_assistant_message, the recall log is used even if transcript_path is also present. If Claude Code (or another Claude-family tool) also sends last_assistant_message in Stop, Claude sessions would silently switch from transcript evidence to recall-log evidence — e.g. a doc that only appears as a recall region in the transcript, without a settled linked run in the log, would stop being judged. That contradicts the PR description ("Existing transcript-backed behavior ... remain intact") and the new docs line.

Suggested fix: prefer the transcript when one exists, and only fall back to the native reply when there is no transcript:

const voteData = transcriptPath
  ? await parseTranscriptForVotes(transcriptPath)
  : { finalAssistantText: finalReply, /* recall-log docs */ };

plus a test where both transcript_path and last_assistant_message are set, asserting the transcript path is used.

@SaulMoro SaulMoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

@nateEc Approving at 3672eb3. We ran the built CLI on a local git team with an isolated HOME, TEAMAI_UPVOTE_JUDGE=1 and a stub model CLI, doing pull, then a real recall that returns the seeded learning, then Stop payloads shaped like Codex's:

Stop merge-base f3cd042 (judge calls / votes) this head main 744d982
final reply, no transcript 0 / 0 1 / 1 0 / 0
same payload again 0 / 0 1 / 1 0 / 0
neither transcript nor reply 0 / 0 1 / 1, no change 0 / 0
empty reply 0 / 0 1 / 1, no change 0 / 0

The judge's argv on this head carries the reply text and the candidate excerpt. tsc and lint pass. The three changed unit files (329 tests) pass, and their new Codex, Muse and ZCode cases fail on the base. native-stop-judge.test.ts fails on the base (expected +0 to be 1) and passes here, also with TMPDIR behind a symlink. Thirteen other e2e files that run recall, Stop or the upvote judge pass (224 tests).

Non-blocking notes:

  • [P3 nit] No test covers the settled and linked gate on the reply path. The code reuses viewRoot, which already applies those rules, so this is coverage only.
  • [P3 nit] When a payload carries both transcript_path and last_assistant_message, the reply branch wins. We found no tool that sends both, so this is only worth a comment in the code.
  • [P3 nit] The new paragraph in skill-data/core/references/troubleshooting.md is one 219-character line, while the rest of the file wraps near 80 columns.

Not run: a real Codex, Muse or ZCode session, and Windows.

The CI review reports no findings and accepts the testing record. We agree. All CI jobs pass; the GitHub-provider e2e job is skipped by the fork policy. This PR and #1335, #1339, #1340 and #1341 all edit troubleshooting.md, so whichever lands second needs a rebase there.

@SaulMoro
SaulMoro merged commit 3752769 into Tencent:main Oct 25, 2026
18 checks passed
@SaulMoro

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

@jeff-r2026 Sorry, we merged this before your question was answered. You were right: Claude Code sends last_assistant_message in Stop as well, so on main a Claude session is now judged from the recall log instead of its transcript. #1396 keeps the transcript whenever it shows a recall. The fallback is not "only without a transcript", because Codex sends its own rollout as transcript_path, and that would turn this PR off for Codex. Both cases have a test there.

SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
@nateEc

nateEc commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for catching the Claude payload carrying both fields. I checked the follow-up in #1396: it keeps actual Claude transcript recall evidence and falls back to the settled recall log for native Codex rollout/reply payloads, with regressions for both. That PR is already owned by SaulMoro, so I am leaving the fix there rather than pushing a competing change to this merged branch.

SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 26, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 27, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
jeff-r2026 pushed a commit that referenced this pull request Oct 27, 2026
… sends the reply (#1396)

Claude Code sends last_assistant_message in Stop along with transcript_path.
Since #1374 the reply took precedence, so a Claude session was judged from the
recall log instead of its transcript, and a doc the transcript shows but no
settled run of the session records was no longer judged.

Use the transcript whenever it shows a recall. Fall back to the reply and the
recall log only when it shows none: Codex sends its own rollout as
transcript_path, which is not a Claude transcript.
jeff-r2026 pushed a commit that referenced this pull request Oct 27, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 27, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
SaulMoro added a commit that referenced this pull request Oct 27, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
jeff-r2026 pushed a commit that referenced this pull request Oct 27, 2026
…tor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.
jeff-r2026 pushed a commit that referenced this pull request Oct 27, 2026
* feat(tools): state each tool's run-time and AI support in the descriptor (#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of #1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (#1193, #1224, #1245, #1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.

* docs: generate the tool support page from the descriptor (#1221)

docs/guide/tool-support.md and its Chinese page are rendered from TOOLS
and diffed by a snapshot test, as the command reference is. Headers and
the legend are translated; cell text stays English on both pages.
AGENTS.md records that exception and the review rule for descriptor
updates.

* docs(readme): mark usage and sessions the descriptor records, and JoyCode's inlined rules (#1221)

Usage: skill use is recorded for OpenCode, Pi, Oh My Pi and DeepSeek Harness
since #1245 (`features.stats`). Sessions: `teamai session` reads OpenCode
(`TOOLS.opencode.sessions`). Rules: JoyCode's user rules are inlined into
rules.txt, so they stay on, as for every other tool marked ✓¹.

* docs(readme): add culture, workflows and scorers to the agent matrix, and check it against the descriptor (#1221)

Learnings, codebase and teamwiki were the same for every agent, so Team
Context is one column. That leaves room for culture (team culture and
shared instructions need an instruction-blocks channel, which Qoder,
Qoder CN, Kiro, Trae and Trae CN do not have), workflows (every agent
drives one from its session) and scorers (the agents whose sessions
teamai can judge: a session adapter in `TOOLS[tool].sessions`). Team
Execution is ordered skills, rules, culture, agents, hooks, mcp,
workflows, env, models, and the layer bullets follow it.

readme-matrix.test.ts keeps every README's cells equal and checks the
cells the code owns: culture, models, usage, scorers and ✓¹.

* docs: regenerate the tool support page for Kiro's hook file (#1221)
Walternater pushed a commit to Walternater/teamai-cli that referenced this pull request Nov 2, 2026
- 让 opt-in judge 接受 Codex、Muse 与 ZCode Stop 事件中的 last_assistant_message,不再强制要求 Claude 格式 transcript。
- 从同一会话已确认并关联的 recall 日志提取候选路径及投票键,继续保留作用域 eligibility、受信知识根校验与每会话投票账本去重。
- 新增三种工具的原生回复、重复 Stop、继承用户范围只读和伪造召回标记回归,更新中英说明及技能排查指南。
- 329 个评分与会话归属单测、构建、类型检查、lint 和隔离模型替身的真实 CLI recall/Stop/投票 E2E 通过;修复前三个原生回复用例失败。
Walternater pushed a commit to Walternater/teamai-cli that referenced this pull request Nov 2, 2026
… sends the reply (Tencent#1396)

Claude Code sends last_assistant_message in Stop along with transcript_path.
Since Tencent#1374 the reply took precedence, so a Claude session was judged from the
recall log instead of its transcript, and a doc the transcript shows but no
settled run of the session records was no longer judged.

Use the transcript whenever it shows a recall. Fall back to the reply and the
recall log only when it shows none: Codex sends its own rollout as
transcript_path, which is not a Claude transcript.
Walternater pushed a commit to Walternater/teamai-cli that referenced this pull request Nov 2, 2026
… (Tencent#1406)

* feat(tools): state each tool's run-time and AI support in the descriptor (Tencent#1221)

The env, workflows, recall, stats, AI backend, votes judge and HTTP agent
hook rows of Tencent#1221's matrix become a required `features` facet, one
`SupportCell` per row: the mark and the evidence letter, as the matrix
writes them. A cell a later fix changed (Tencent#1193, Tencent#1224, Tencent#1245, Tencent#1374) says
so in its note and keeps teamai's-code-only evidence until it is rerun.
Data only: nothing reads it but the support page.

* docs: generate the tool support page from the descriptor (Tencent#1221)

docs/guide/tool-support.md and its Chinese page are rendered from TOOLS
and diffed by a snapshot test, as the command reference is. Headers and
the legend are translated; cell text stays English on both pages.
AGENTS.md records that exception and the review rule for descriptor
updates.

* docs(readme): mark usage and sessions the descriptor records, and JoyCode's inlined rules (Tencent#1221)

Usage: skill use is recorded for OpenCode, Pi, Oh My Pi and DeepSeek Harness
since Tencent#1245 (`features.stats`). Sessions: `teamai session` reads OpenCode
(`TOOLS.opencode.sessions`). Rules: JoyCode's user rules are inlined into
rules.txt, so they stay on, as for every other tool marked ✓¹.

* docs(readme): add culture, workflows and scorers to the agent matrix, and check it against the descriptor (Tencent#1221)

Learnings, codebase and teamwiki were the same for every agent, so Team
Context is one column. That leaves room for culture (team culture and
shared instructions need an instruction-blocks channel, which Qoder,
Qoder CN, Kiro, Trae and Trae CN do not have), workflows (every agent
drives one from its session) and scorers (the agents whose sessions
teamai can judge: a session adapter in `TOOLS[tool].sessions`). Team
Execution is ordered skills, rules, culture, agents, hooks, mcp,
workflows, env, models, and the layer bullets follow it.

readme-matrix.test.ts keeps every README's cells equal and checks the
cells the code owns: culture, models, usage, scorers and ✓¹.

* docs: regenerate the tool support page for Kiro's hook file (Tencent#1221)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants