Skip to content

Commit b046c6a

Browse files
author
zhouzekun
committed
Anchor the opencode usage tests to a tool step under the v2 defect
opencode v2's CLI drops the final step_finish from stdout (canonical upstream issue anomalyco/opencode#53207, open): once session.wait() resolves, the forwarding loop only lets session.execution.* through — which carry no usage — so a text-only run reports nothing and every run undercounts its last step. The server event stream and the session DB still hold the data; only stdout loses it, and AgentFabric consumes stdout by design. The three real-integration usage assertions therefore failed on a defect that isn't ours to fix. Instead of relaxing them, their prompts now ask for a shell tool step first: an intermediate step_finish still reaches the Run, so the assertions stay strict (a broken usage pipeline still fails) and need no edit once upstream fixes the CLI. The reference moves from the downstream report (multica#8892) to the canonical upstream issue, the README paragraph describes the actual mechanism, and parseOpenCodeUsage carries a comment so nobody re-investigates "is there a renamed event" — there isn't.
1 parent dd552cd commit b046c6a

4 files changed

Lines changed: 27 additions & 5 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -821,7 +821,7 @@ bash scripts/build-harness-images.sh # 构建 pi + dsh(AGENTFABRIC_
821821
Local 与 Docker 共用同一个 Harness Parser(事件 / Session Ref / Usage / 错误),Execution Backend 只做传输:容器的 stdout/stderr 以原始行流交给对应 Harness Adapter,由同一套解析器处理。
822822

823823
* **Pi**(`pi --print --mode json`):`session` 头、`agent_start/end`、`turn_start/end`、`message_start/update/end`、`tool_execution_start/update/end` → `run.progress` / `agent.message` / `agent.thinking` / `tool.started` / `tool.progress` / `tool.completed` / `runtime.error`。
824-
* **OpenCode**(`opencode run --standalone --format json`,v2):`step_start` / `text` / `reasoning` / `tool_use` / `step_finish` / `error`(每行携带 `sessionID`)→ 同一套标准事件。v2 的 `tool_use` part 用 `id` 承载 call id(v1 是 `callID`)、错误用 `error.type` 承载类型(v1 是 `error.name`),Adapter 两者都读。**已知上游缺陷**:v2 的 `run --format json` 不再为最后一步输出 `step_finish`([multica#8892](https://github.com/multica-ai/multica/issues/8892),数据仍在 session DB 里,只是 stdout 流被提前拆掉),因此 v2 下 Run 的 Usage / Cost 会少算最后一步(纯文本回答的 Run 显示为 0)——这是 opencode v2 CLI 的行为,不是 Adapter 的解析问题。
824+
* **OpenCode**(`opencode run --standalone --format json`,v2):`step_start` / `text` / `reasoning` / `tool_use` / `step_finish` / `error`(每行携带 `sessionID`)→ 同一套标准事件。v2 的 `tool_use` part 用 `id` 承载 call id(v1 是 `callID`)、错误用 `error.type` 承载类型(v1 是 `error.name`),Adapter 两者都读。**已知上游缺陷**:v2 CLI 的 stdout 转发会丢掉最后一步的 `step_finish`([opencode#53207](https://github.com/anomalyco/opencode/issues/53207);下游报告 [multica#8892](https://github.com/multica-ai/multica/issues/8892))。实测(2.0.21/2.0.26)稳定复现:服务端事件流与 session DB 里最后一步的 `tokens`/`cost` 完整存在(`session.step.ended`),丢的是 CLI——`session.wait()` 落定后转发循环进入 finalizing、只放行不带 usage 的 `session.execution.*`,最后一步的 `step_finish` 因此到不了 stdout。后果:v2 下 Run 的 Usage / Cost 少算最后一步(纯文本回答的 Run 显示为 0),中间工具步的 `step_finish` 照常输出。这是 CLI 的行为,不是 Adapter 的解析问题。
825825
* **Codex**(`codex exec --json`):agent_message / reasoning / command_execution / file_change / mcp_tool_call / web_search / turn usage。
826826
* **Claude Code**(`claude -p --output-format stream-json --verbose`):assistant text / thinking / tool_use→Bash·Edit·Write·Read… / tool_result / result usage。
827827
* **DSH**(`dsh --profile headless --json [--session-id <id>] -- "<task>"`):`session` → `status`(turn/step,`step_end` 携带该次模型请求的 usage)→ `text` / `thinking`(已提交的助手消息投影,重试中的尝试不会外泄)→ `tool_call` / `tool_result`(按 callId 配对)→ `final`。退出码 0 = 完成、1 = 中止/出错。

‎packages/core/src/v3.real.test.ts‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -311,7 +311,7 @@ test("REAL: opencode local native resume continues Run #1's context (v3 §20/§2
311311
const runtime = await h.runtimes.create({ name: "oc-real", kind: "opencode", env: { XDG_DATA_HOME: xdg } });
312312

313313
const { task, run } = await h.runService.submit({
314-
prompt: "Remember the secret word MANGO-9. Reply with exactly: OK",
314+
prompt: "Remember the secret word MANGO-9. First run the shell command `echo READY`, then reply with exactly: OK",
315315
runtimeId: runtime.id,
316316
workspaceId: ws.id,
317317
timeoutMs: RUN_TIMEOUT_MS,
@@ -325,6 +325,10 @@ test("REAL: opencode local native resume continues Run #1's context (v3 §20/§2
325325
assert.ok(ref, "opencode persisted a native session reference");
326326
assert.ok(ref.nativeSessionRef.startsWith("ses_"));
327327
assert.ok(harnessStatePersisted(join(xdg, "opencode")), "opencode persisted its session state (opencode.db)");
328+
// opencode 2 never emits the final step_finish (anomalyco/opencode#53207),
329+
// so a single-step run reports no usage at all. The prompts ask for a
330+
// tool call on purpose: that step's step_finish still reaches the Run,
331+
// so a broken usage pipeline still fails here.
328332
assert.ok((first.usage?.inputTokens ?? 0) > 0, "step_finish tokens reached Run Usage");
329333
assert.ok((first.usage?.modelRequests ?? 0) >= 1);
330334

@@ -358,7 +362,7 @@ test("REAL: opencode docker native resume across destroyed containers (v3 §20/
358362
writeFileSync(join(state.path, "auth.json"), opencodeAuthJson()!);
359363

360364
const { task, run } = await h.runService.submit({
361-
prompt: "Remember the secret word NEBULA-5. Reply with exactly: OK",
365+
prompt: "Remember the secret word NEBULA-5. First run the shell command `echo READY`, then reply with exactly: OK",
362366
runtimeId: runtime.id,
363367
workspaceId: ws.id,
364368
timeoutMs: RUN_TIMEOUT_MS,
@@ -372,6 +376,8 @@ test("REAL: opencode docker native resume across destroyed containers (v3 §20/
372376
const ref = rs.list({ taskId: task.id })[0];
373377
assert.ok(ref, "session reference captured from the container stream");
374378
assert.ok(harnessStatePersisted(state.path), "session persisted in the opaque native state");
379+
// Same anchor as the local test: the tool step's step_finish is what
380+
// opencode 2 still emits (anomalyco/opencode#53207).
375381
assert.ok((first.usage?.inputTokens ?? 0) > 0, "usage survived the container boundary");
376382

377383
const cont = await h.runService.continueTask(task.id, { prompt: "What was the secret word I told you? Reply with just the word." });

‎packages/core/src/v4.real.test.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,7 @@ test(
253253
});
254254

255255
const { run } = await h.runService.submit({
256-
prompt: "Reply with exactly: PONG",
256+
prompt: "First run the shell command `echo READY`, then reply with exactly: PONG",
257257
runtimeId: runtime.id,
258258
modelId: model.id,
259259
workspaceId: ws.id,
@@ -266,6 +266,10 @@ test(
266266
const injected = await configInjectionEvent(h.runService, run.id);
267267
assert.ok(injected, "config-injected event present");
268268
assert.equal(injected!.slug, "af-real-oc");
269+
// The prompt asks for a shell call on purpose: opencode 2 never emits
270+
// the final step_finish (anomalyco/opencode#53207), so only a run with
271+
// a tool step still reports usage — and a broken usage pipeline still
272+
// fails here.
269273
assert.ok((finished.usage?.inputTokens ?? 0) > 0, "usage recorded through the generated config");
270274
}
271275
);
@@ -295,7 +299,7 @@ test(
295299
});
296300

297301
const { run } = await h.runService.submit({
298-
prompt: "Reply with exactly: DOCK-OK",
302+
prompt: "First run the shell command `echo READY`, then reply with exactly: DOCK-OK",
299303
runtimeId: runtime.id,
300304
modelId: model.id,
301305
workspaceId: ws.id,
@@ -306,6 +310,7 @@ test(
306310
assert.equal(finished.status, "completed", finished.error);
307311
assert.ok(finished.containerId, "ran inside a real container");
308312
assert.match(await agentMessages(h.runService, run.id), /DOCK-OK/, "containerized opencode used the injected provider");
313+
// Same tool-step anchor as the local §27 test (anomalyco/opencode#53207).
309314
assert.ok((finished.usage?.inputTokens ?? 0) > 0, "usage recorded across the container boundary");
310315
}
311316
);

‎packages/runtimes/src/opencode.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,17 @@ export function extractOpenCodeSessionRef(raw: string): string | undefined {
294294
* Extracts per-step usage from an opencode `step_finish` line (v3 §8):
295295
* tokens {input, output, reasoning, cache{read, write}} and a single
296296
* USD cost number. Each step_finish is one completed model request.
297+
*
298+
* Known upstream defect (anomalyco/opencode#53207): the v2 CLI drops the
299+
* FINAL step_finish from stdout — its forwarding loop goes into
300+
* finalizing once `session.wait()` resolves and only lets
301+
* `session.execution.*` through (which carry no usage), so a text-only
302+
* run reports nothing and every run undercounts its last step. The
303+
* server-side event stream and the session DB still hold the data; only
304+
* the CLI's stdout loses it, and AgentFabric consumes stdout by design
305+
* (native state stays opaque). Do not "fix" this here — there is no
306+
* renamed event to parse; intermediate tool steps still arrive and are
307+
* what the real tests anchor their usage assertions to.
297308
*/
298309
export function parseOpenCodeUsage(raw: string): Usage | undefined {
299310
let evt: OpenCodeEvent;

0 commit comments

Comments
 (0)