Skip to content

Commit bb4a9ce

Browse files
committed
feat(webui): redesign tool-call rendering with inline diffs, group summaries, and cheap-model purpose titles
Kind-aware tool rows (terminal-style commands with wrapper stripping and display highlighting, file edits with inline numbered diffs and diffstat, write previews, key-value args), aggregate group summaries with live run status, and a new batched chat.toolTitles gateway RPC that titles complex calls via the configured utilityModel or the OpenAI Luna default (gated to OpenAI-primary agents, cached in the per-agent SQLite cache_entries). Also fixes two transcript pairing bugs: result blocks now inherit call id/name/details at merge time, and results pair with any open call in the current tool run so parallel calls render as single rows. Fixes #103554
1 parent 2d54be3 commit bb4a9ce

35 files changed

Lines changed: 3351 additions & 124 deletions

docs/gateway/protocol.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -473,6 +473,7 @@ methods. Treat this as feature discovery, not a full enumeration of
473473
- `sessions.get` returns the full stored session row.
474474
- Chat execution still uses `chat.history`, `chat.send`, `chat.abort`, and `chat.inject`. `chat.history` is display-normalized for UI clients: inline directive tags are stripped from visible text, plain-text tool-call XML payloads (`<tool_call>...</tool_call>`, `<function_call>...</function_call>`, `<tool_calls>...</tool_calls>`, `<function_calls>...</function_calls>`, and truncated tool-call blocks) and leaked ASCII/full-width model control tokens are stripped, pure silent-token assistant rows (exact `NO_REPLY` / `no_reply`) are omitted, and oversized rows can be replaced with placeholders.
475475
- `chat.message.get` is the additive bounded full-message reader for a single visible transcript entry. Pass `sessionKey`, optional `agentId` when session selection is agent-scoped, and a transcript `messageId` previously surfaced through `chat.history`; the gateway returns the same display-normalized projection without the lightweight history truncation cap when the stored entry is still available and not oversized.
476+
- `chat.toolTitles` returns short purpose titles for tool calls rendered in the Control UI (batched, max 24 items with bounded inputs). Titles come from the agent's configured `utilityModel` when set, otherwise the low-cost `openai/gpt-5.6-luna` default — and that default applies only when the agent's primary model already routes to OpenAI, so tool arguments never reach a provider the session was not already using. When no eligible cheap model is usable the gateway returns no titles and clients keep deterministic labels. Results cache in the per-agent state database keyed by tool name + input, so repeated views never re-bill the same calls.
476477
- `chat.send` accepts one-turn `fastMode: "auto"` to use fast mode for model calls started before the auto cutoff, then start later retry, fallback, tool-result, or continuation calls without fast mode. The cutoff defaults to 60 seconds (`DEFAULT_FAST_MODE_AUTO_ON_SECONDS`) and can be configured per model with `agents.defaults.models["<provider>/<model>"].params.fastAutoOnSeconds`. A `chat.send` caller can pass one-turn `fastAutoOnSeconds` to override the cutoff for that request.
477478

478479
</Accordion>

docs/web/control-ui.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,8 @@ A **Search** field at the top of the sidebar opens the command palette (⌘K). T
170170
- Chat history refreshes request a bounded recent window with per-message text caps, so large sessions do not force the browser to render a full transcript payload before chat becomes usable.
171171
- Hovering or keyboard-focusing a public GitHub issue or pull request link shows its state, title, author, recent activity, comments, and change statistics. The connected Gateway fetches and caches public metadata without changing the link target, including when the UI uses a remote Gateway. The Gateway uses `GH_TOKEN` or `GITHUB_TOKEN` when available, after confirming the repository is public; otherwise it uses GitHub's anonymous API with a longer cache.
172172
- Talk through browser realtime sessions. OpenAI uses direct WebRTC, Google Live uses a constrained one-use browser token over WebSocket, and backend-only realtime voice plugins use the Gateway relay transport. Client-owned provider sessions start with `talk.client.create`; Gateway relay sessions start with `talk.session.create`. The relay keeps provider credentials on the Gateway while the browser streams microphone PCM through `talk.session.appendAudio`, forwards `openclaw_agent_consult` provider tool calls through `talk.client.toolCall` for Gateway policy and the larger configured OpenClaw model, and routes active-run voice steering through `talk.client.steer` or `talk.session.steer`.
173-
- Stream tool calls and live tool output cards in Chat (agent events).
173+
- Stream tool calls and live tool output cards in Chat (agent events). Tool activity renders as kind-aware rows: shell commands show the syntax-highlighted command with terminal-style output, file edits show an inline diff with line numbers and a `+added -removed` stat, file writes preview the created content, and consecutive calls collapse into an aggregate card such as "Ran 13 commands, read 6 files, edited 9 files". While a run is live, the newest running call names the group header.
174+
- Complex tool calls (long shell commands, argument-heavy plugin tools) get short AI-generated purpose titles through `chat.toolTitles`, powered by the agent's configured `utilityModel` or the low-cost `openai/gpt-5.6-luna` default (used only when the agent already runs on OpenAI, so tool arguments never reach a new provider). Titles are cached gateway-side per agent; when no cheap model is usable, rows keep their deterministic labels.
174175
- Start or dismiss ephemeral model-suggested follow-up tasks; accepted suggestions open a fresh managed-worktree session with the proposed prompt.
175176
- Activity tab with browser-local, redaction-first summaries of live tool activity from existing `session.tool` / tool event delivery.
176177

packages/gateway-protocol/src/index.ts

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,10 @@ import {
165165
type ChatInjectParams,
166166
ChatInjectParamsSchema,
167167
ChatSendParamsSchema,
168+
type ChatToolTitlesParams,
169+
type ChatToolTitlesResult,
170+
ChatToolTitlesParamsSchema,
171+
ChatToolTitlesResultSchema,
168172
type ConfigApplyParams,
169173
ConfigApplyParamsSchema,
170174
type ConfigGetParams,
@@ -1188,6 +1192,9 @@ export const validateTerminalEvent = lazyCompile<TerminalEvent>(TerminalEventSch
11881192
export const validateChatHistoryParams = lazyCompile(ChatHistoryParamsSchema);
11891193
export const validateChatMetadataParams = lazyCompile<ChatMetadataParams>(ChatMetadataParamsSchema);
11901194
export const validateChatMessageGetParams = lazyCompile(ChatMessageGetParamsSchema);
1195+
export const validateChatToolTitlesParams = lazyCompile<ChatToolTitlesParams>(
1196+
ChatToolTitlesParamsSchema,
1197+
);
11911198
export const validateChatSendParams = lazyCompile(ChatSendParamsSchema);
11921199
export const validateChatAbortParams = lazyCompile<ChatAbortParams>(ChatAbortParamsSchema);
11931200
export const validateChatInjectParams = lazyCompile<ChatInjectParams>(ChatInjectParamsSchema);
@@ -1549,6 +1556,8 @@ export {
15491556
ChatMetadataParamsSchema,
15501557
ChatSendParamsSchema,
15511558
ChatInjectParamsSchema,
1559+
ChatToolTitlesParamsSchema,
1560+
ChatToolTitlesResultSchema,
15521561
UpdateRunParamsSchema,
15531562
TickEventSchema,
15541563
ShutdownEventSchema,
@@ -1695,6 +1704,8 @@ export type {
16951704
AgentsListParams,
16961705
AgentsListResult,
16971706
ChatMetadataParams,
1707+
ChatToolTitlesParams,
1708+
ChatToolTitlesResult,
16981709
CommandsListParams,
16991710
CommandsListResult,
17001711
CommandEntry,

packages/gateway-protocol/src/schema/logs-chat.ts

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,36 @@ export const ChatMetadataParamsSchema = Type.Object(
4646
{ additionalProperties: false },
4747
);
4848

49+
/** Batched purpose-title request for tool calls rendered in the Control UI. */
50+
export const ChatToolTitlesParamsSchema = Type.Object(
51+
{
52+
sessionKey: NonEmptyString,
53+
agentId: Type.Optional(NonEmptyString),
54+
items: Type.Array(
55+
Type.Object(
56+
{
57+
id: NonEmptyString,
58+
name: Type.String({ minLength: 1, maxLength: 200 }),
59+
input: Type.String({ minLength: 1, maxLength: 4_000 }),
60+
},
61+
{ additionalProperties: false },
62+
),
63+
{ minItems: 1, maxItems: 24 },
64+
),
65+
},
66+
{ additionalProperties: false },
67+
);
68+
69+
/** Titles keyed by the caller-provided item id; missing ids mean no title. */
70+
export const ChatToolTitlesResultSchema = Type.Object(
71+
{
72+
titles: Type.Record(Type.String(), Type.String()),
73+
},
74+
{ additionalProperties: false },
75+
);
76+
/** Typed result shape for tool-title consumers. */
77+
export type ChatToolTitlesResult = Static<typeof ChatToolTitlesResultSchema>;
78+
4979
/** Fetches one stored chat message without forcing history callers to request huge payloads. */
5080
export const ChatMessageGetParamsSchema = Type.Object(
5181
{

packages/gateway-protocol/src/schema/protocol-schemas.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,8 @@ import {
234234
ChatMessageGetResultSchema,
235235
ChatInjectParamsSchema,
236236
ChatSendParamsSchema,
237+
ChatToolTitlesParamsSchema,
238+
ChatToolTitlesResultSchema,
237239
LogsTailParamsSchema,
238240
LogsTailResultSchema,
239241
} from "./logs-chat.js";
@@ -772,6 +774,8 @@ export const ProtocolSchemas = {
772774
ChatMetadataParams: ChatMetadataParamsSchema,
773775
ChatMessageGetParams: ChatMessageGetParamsSchema,
774776
ChatMessageGetResult: ChatMessageGetResultSchema,
777+
ChatToolTitlesParams: ChatToolTitlesParamsSchema,
778+
ChatToolTitlesResult: ChatToolTitlesResultSchema,
775779
ChatSendParams: ChatSendParamsSchema,
776780
ChatAbortParams: ChatAbortParamsSchema,
777781
ChatInjectParams: ChatInjectParamsSchema,

packages/gateway-protocol/src/schema/types.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,7 @@ export type ModelChoice = SchemaType<"ModelChoice">;
256256
export type ModelsListParams = SchemaType<"ModelsListParams">;
257257
export type ModelsListResult = SchemaType<"ModelsListResult">;
258258
export type ChatMetadataParams = SchemaType<"ChatMetadataParams">;
259+
export type ChatToolTitlesParams = SchemaType<"ChatToolTitlesParams">;
259260
export type CommandEntry = SchemaType<"CommandEntry">;
260261
export type CommandsListParams = SchemaType<"CommandsListParams">;
261262
export type CommandsListResult = SchemaType<"CommandsListResult">;
Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
// Gateway tests cover cheap-model tool-call title generation and its SQLite cache.
2+
import fs from "node:fs";
3+
import os from "node:os";
4+
import path from "node:path";
5+
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
6+
7+
const completeWithPreparedSimpleCompletionModel = vi.hoisted(() => vi.fn());
8+
const prepareSimpleCompletionModelForAgent = vi.hoisted(() => vi.fn());
9+
const resolveSimpleCompletionSelectionForAgent = vi.hoisted(() => vi.fn());
10+
11+
vi.mock("../agents/simple-completion-runtime.js", () => ({
12+
completeWithPreparedSimpleCompletionModel,
13+
prepareSimpleCompletionModelForAgent,
14+
resolveSimpleCompletionSelectionForAgent,
15+
}));
16+
17+
import type { OpenClawConfig } from "../config/types.openclaw.js";
18+
import { closeOpenClawAgentDatabases } from "../state/openclaw-agent-db.js";
19+
import { closeOpenClawStateDatabaseForTest } from "../state/openclaw-state-db.js";
20+
import { generateToolCallTitles } from "./chat-tool-titles.js";
21+
22+
const AGENT_ID = "main";
23+
24+
function mockPreparedModel(): void {
25+
prepareSimpleCompletionModelForAgent.mockResolvedValue({
26+
selection: { provider: "openai", modelId: "gpt-test", agentDir: "/tmp/openclaw-agent" },
27+
model: { provider: "openai", id: "gpt-test", maxTokens: 8192 },
28+
auth: { apiKey: "k", mode: "api-key" },
29+
});
30+
}
31+
32+
function mockCompletionTitles(titles: Record<string, string>): void {
33+
completeWithPreparedSimpleCompletionModel.mockResolvedValue({
34+
stopReason: "stop",
35+
content: [{ type: "text", text: JSON.stringify({ titles }) }],
36+
});
37+
}
38+
39+
describe("generateToolCallTitles", () => {
40+
let stateDir: string;
41+
let previousStateDir: string | undefined;
42+
43+
beforeEach(() => {
44+
completeWithPreparedSimpleCompletionModel.mockReset();
45+
prepareSimpleCompletionModelForAgent.mockReset();
46+
resolveSimpleCompletionSelectionForAgent.mockReset();
47+
// Default: the agent's primary model already routes to OpenAI, so the
48+
// Luna fallback is permitted by the egress gate.
49+
resolveSimpleCompletionSelectionForAgent.mockReturnValue({
50+
provider: "openai",
51+
modelId: "gpt-5.5",
52+
agentDir: "/tmp/openclaw-agent",
53+
});
54+
// realpath: macOS tmpdir is a /var -> /private/var symlink and DB paths resolve canonically.
55+
stateDir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-tool-titles-")));
56+
previousStateDir = process.env.OPENCLAW_STATE_DIR;
57+
process.env.OPENCLAW_STATE_DIR = stateDir;
58+
});
59+
60+
afterEach(() => {
61+
closeOpenClawAgentDatabases();
62+
closeOpenClawStateDatabaseForTest();
63+
if (previousStateDir === undefined) {
64+
delete process.env.OPENCLAW_STATE_DIR;
65+
} else {
66+
process.env.OPENCLAW_STATE_DIR = previousStateDir;
67+
}
68+
fs.rmSync(stateDir, { recursive: true, force: true });
69+
});
70+
71+
it("generates titles keyed by item id", async () => {
72+
mockPreparedModel();
73+
mockCompletionTitles({ "item-1": "Checked repo status", "item-2": "Listed source files" });
74+
75+
const result = await generateToolCallTitles({
76+
cfg: {} satisfies OpenClawConfig,
77+
agentId: AGENT_ID,
78+
items: [
79+
{ id: "item-1", name: "bash", input: "git status --short" },
80+
{ id: "item-2", name: "bash", input: "ls -la src" },
81+
],
82+
});
83+
84+
expect(result).toEqual({
85+
"item-1": "Checked repo status",
86+
"item-2": "Listed source files",
87+
});
88+
expect(completeWithPreparedSimpleCompletionModel).toHaveBeenCalledTimes(1);
89+
});
90+
91+
it("serves repeated items from the SQLite cache without a second completion", async () => {
92+
mockPreparedModel();
93+
mockCompletionTitles({ "item-1": "Checked repo status" });
94+
const params = {
95+
cfg: {} satisfies OpenClawConfig,
96+
agentId: AGENT_ID,
97+
items: [{ id: "item-1", name: "bash", input: "git status --short" }],
98+
};
99+
100+
const first = await generateToolCallTitles(params);
101+
const second = await generateToolCallTitles(params);
102+
103+
expect(first).toEqual({ "item-1": "Checked repo status" });
104+
expect(second).toEqual(first);
105+
expect(completeWithPreparedSimpleCompletionModel).toHaveBeenCalledTimes(1);
106+
});
107+
108+
it("fails closed to an empty result when model preparation errors", async () => {
109+
prepareSimpleCompletionModelForAgent.mockResolvedValue({
110+
error: 'No API key resolved for provider "openai".',
111+
});
112+
113+
await expect(
114+
generateToolCallTitles({
115+
cfg: {} satisfies OpenClawConfig,
116+
agentId: AGENT_ID,
117+
items: [{ id: "item-1", name: "bash", input: "git status --short" }],
118+
}),
119+
).resolves.toEqual({});
120+
expect(completeWithPreparedSimpleCompletionModel).not.toHaveBeenCalled();
121+
});
122+
123+
it("prepares the configured utility model when one is set", async () => {
124+
mockPreparedModel();
125+
mockCompletionTitles({ "item-1": "Checked repo status" });
126+
const cfg = { agents: { defaults: { utilityModel: "openai/gpt-test" } } };
127+
128+
await generateToolCallTitles({
129+
cfg,
130+
agentId: AGENT_ID,
131+
items: [{ id: "item-1", name: "bash", input: "git status --short" }],
132+
});
133+
134+
expect(prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith({
135+
cfg,
136+
agentId: AGENT_ID,
137+
useUtilityModel: true,
138+
useAsyncModelResolution: true,
139+
allowMissingApiKeyModes: ["aws-sdk"],
140+
});
141+
});
142+
143+
it("falls back to the Luna default model ref without a utility model", async () => {
144+
mockPreparedModel();
145+
mockCompletionTitles({ "item-1": "Checked repo status" });
146+
const cfg = {} satisfies OpenClawConfig;
147+
148+
await generateToolCallTitles({
149+
cfg,
150+
agentId: AGENT_ID,
151+
items: [{ id: "item-1", name: "bash", input: "git status --short" }],
152+
});
153+
154+
expect(prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith({
155+
cfg,
156+
agentId: AGENT_ID,
157+
modelRef: "openai/gpt-5.6-luna",
158+
useAsyncModelResolution: true,
159+
allowMissingApiKeyModes: ["aws-sdk"],
160+
});
161+
});
162+
163+
it("skips the Luna default when the agent's primary provider is not OpenAI", async () => {
164+
resolveSimpleCompletionSelectionForAgent.mockReturnValue({
165+
provider: "anthropic",
166+
modelId: "claude-test",
167+
agentDir: "/tmp/openclaw-agent",
168+
});
169+
170+
await expect(
171+
generateToolCallTitles({
172+
cfg: {} satisfies OpenClawConfig,
173+
agentId: AGENT_ID,
174+
items: [{ id: "item-1", name: "bash", input: "git status --short" }],
175+
}),
176+
).resolves.toEqual({});
177+
expect(prepareSimpleCompletionModelForAgent).not.toHaveBeenCalled();
178+
expect(completeWithPreparedSimpleCompletionModel).not.toHaveBeenCalled();
179+
});
180+
});

0 commit comments

Comments
 (0)