Skip to content

Commit e45c9ce

Browse files
feat(tui): add live tokens/sec display
Shows generation speed while an assistant response streams (refreshing every 500ms) and a final rate once the turn completes. Ported from upstream anomalyco#5374 (111 reactions): the footer placement/refresh approach and the ~4-chars/token live estimate come from PR anomalyco#46882 (closed by the 2-hour PR-template bot, same day, never reviewed); the "show a rate after completion too" idea comes from PR anomalyco#37052 (a bundled PR with an unrelated paste- expand feature -- only the tokens/sec piece used). anomalyco#37052's own final-rate formula was rewritten here because it counted input and cache tokens, badly inflating the number -- this counts only output and reasoning tokens. New: the measurement window starts at the first streamed token rather than message creation, and stops at the last finished text/reasoning part, so time-to-first-token and tool execution time don't distort the rate. New packages/tui/src/util/throughput.ts (~40 lines). Known gaps: the live number is a character-count estimate, not exact (providers only report real token counts at step-finish); each tool step gets its own footer so the rate shown is per-step, not averaged across a turn; an interrupted response shows no rate. New packages/tui/test/util/throughput.test.ts (9 tests). Full packages/tui suite: 208 pass. bun typecheck clean. Co-Authored-By: Claude Sonnet 5 <[email protected]>
1 parent 878fe22 commit e45c9ce

4 files changed

Lines changed: 169 additions & 0 deletions

File tree

‎FORK_CHANGES.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,16 @@ Each entry names the upstream gap it addresses, whether it originated as our own
66

77
## Implemented
88

9+
### Live tokens/sec display
10+
11+
The TUI now shows generation speed while an assistant response streams (`▣ Build · <model> · ~NN tok/s`, refreshing every 500ms) and a final rate once the turn completes.
12+
13+
- **Upstream gap:** [anomalyco/opencode#5374](https://github.com/anomalyco/opencode/issues/5374) — 111 reactions. Two small PRs existed, both closed by automated bots and never reviewed: [#46882](https://github.com/anomalyco/opencode/pull/46882) (closed by the 2-hour PR-template-compliance bot, same day) and [#42372](https://github.com/anomalyco/opencode/pull/42372) (closed by the 1-month/low-reaction bot; targets the web app, only computes a rate after the turn ends).
14+
- **Origin:** implemented by an Opus 5 agent. The footer placement and refresh approach come from #46882, which already targeted `packages/tui`. The ~4-characters-per-token live estimate and the "only count text/reasoning parts" rule are also from #46882. The "show a rate after completion too" idea is from [#37052](https://github.com/anomalyco/opencode/pull/37052) (a bundled PR also containing an unrelated paste-expand feature — only the tokens/sec piece was used) — but #37052's own final-rate formula was rewritten because it counted input and cache tokens, badly inflating the number; this fork's version counts only output and reasoning tokens. New: the measurement window starts at the first streamed token rather than message creation (so waiting for the first token doesn't deflate the rate) and stops at the last finished text/reasoning part (so time spent running tools doesn't either).
15+
- **Where:** new `packages/tui/src/util/throughput.ts` (~40 lines: `tokensPerSecond`, `liveTokensPerSecond`, `finalTokensPerSecond`), wired into the assistant-message footer in `packages/tui/src/routes/session/index.tsx`.
16+
- **Known gaps:** the live number is a character-count estimate (`~4 chars/token`), not exact — providers only report real output-token counts at step-finish, and getting an exact live rate would require Core to send usage mid-stream, which wasn't touched here. Each tool step gets its own assistant message with its own footer, so the shown rate is for the final step of a turn, not averaged across the whole turn. An interrupted response shows no rate (it never gets a completed timestamp). Not yet watched running live in the actual TUI, only unit-tested.
17+
- **Verified:** `bun typecheck` clean in `packages/tui`. New `packages/tui/test/util/throughput.test.ts` (9 tests) covers the rate math, the under-1-second cutoff, a growing stream sampled over time, text+reasoning summed together, tool time excluded, missing-timestamp fallback, and the final-rate formula ignoring input/cache tokens. Full `packages/tui` suite: 208 pass (re-verified against this fork's current `dev`, not just the agent's own stale worktree base).
18+
919
### Model fallback on exhausted retries
1020

1121
When a provider/model keeps failing after `RequestExecutor`'s transport retry budget is spent (429/5xx/529), the V2 session runner now fails over to the next model in the agent's configured `fallback` chain instead of erroring the turn out to the user.

‎packages/tui/src/routes/session/index.tsx‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ import type {
3838
} from "@opencode-ai/sdk/v2"
3939
import { useLocal } from "../../context/local"
4040
import { Locale } from "../../util/locale"
41+
import { finalTokensPerSecond, liveTokensPerSecond } from "../../util/throughput"
4142
import { webSearchProviderLabel } from "../../util/tool-display"
4243
import { Dynamic, useRenderer, useTerminalDimensions, type JSX } from "@opentui/solid"
4344
import { useSDK } from "../../context/sdk"
@@ -1486,6 +1487,16 @@ function AssistantMessage(props: { message: AssistantMessage; parts: Part[]; las
14861487
return props.message.time.completed - user.time.created
14871488
})
14881489

1490+
const streaming = createMemo(() => !props.message.time.completed && !props.message.error)
1491+
const [now, setNow] = createSignal(Date.now())
1492+
createEffect(() => {
1493+
if (!streaming()) return
1494+
const id = setInterval(() => setNow(Date.now()), 500)
1495+
onCleanup(() => clearInterval(id))
1496+
})
1497+
const liveTps = createMemo(() => (streaming() ? liveTokensPerSecond(props.message, props.parts, now()) : undefined))
1498+
const finalTps = createMemo(() => (final() ? finalTokensPerSecond(props.message, props.parts) : undefined))
1499+
14891500
const childShortcut = useCommandShortcut("session.child.first")
14901501
const backgroundShortcut = useCommandShortcut("session.background")
14911502

@@ -1561,9 +1572,15 @@ function AssistantMessage(props: { message: AssistantMessage; parts: Part[]; las
15611572
</span>{" "}
15621573
<span style={{ fg: theme.text }}>{Locale.titlecase(props.message.mode)}</span>
15631574
<span style={{ fg: theme.textMuted }}> · {model()}</span>
1575+
<Show when={liveTps()}>
1576+
<span style={{ fg: theme.textMuted }}> · ~{liveTps()} tok/s</span>
1577+
</Show>
15641578
<Show when={duration()}>
15651579
<span style={{ fg: theme.textMuted }}> · {Locale.duration(duration())}</span>
15661580
</Show>
1581+
<Show when={finalTps()}>
1582+
<span style={{ fg: theme.textMuted }}> · {finalTps()} tok/s</span>
1583+
</Show>
15671584
<Show when={props.message.error?.name === "MessageAbortedError"}>
15681585
<span style={{ fg: theme.textMuted }}> · interrupted</span>
15691586
</Show>
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
import type { AssistantMessage, Part, ReasoningPart, TextPart } from "@opencode-ai/sdk/v2"
2+
3+
// Providers only report output tokens at step-finish, so while a response is streaming
4+
// the token count is estimated from generated characters (~4 chars per token).
5+
const CHARS_PER_TOKEN = 4
6+
// Rates over very short windows are dominated by the first burst of deltas.
7+
const MIN_WINDOW_MS = 1000
8+
9+
export function tokensPerSecond(tokens: number, startMs: number, endMs: number) {
10+
const elapsed = endMs - startMs
11+
if (tokens <= 0 || elapsed < MIN_WINDOW_MS) return
12+
return Math.round(tokens / (elapsed / 1000))
13+
}
14+
15+
export function liveTokensPerSecond(message: AssistantMessage, parts: Part[], now: number) {
16+
const generated = parts.filter(isGenerated)
17+
const chars = generated.reduce((sum, part) => sum + part.text.length, 0)
18+
const window = generationWindow(message, generated, now)
19+
return tokensPerSecond(chars / CHARS_PER_TOKEN, window.start, window.end)
20+
}
21+
22+
export function finalTokensPerSecond(message: AssistantMessage, parts: Part[]) {
23+
if (!message.time.completed) return
24+
const window = generationWindow(message, parts.filter(isGenerated), message.time.completed)
25+
return tokensPerSecond(message.tokens.output + message.tokens.reasoning, window.start, window.end)
26+
}
27+
28+
function isGenerated(part: Part): part is TextPart | ReasoningPart {
29+
return part.type === "text" || part.type === "reasoning"
30+
}
31+
32+
// Measure from the first streamed token rather than message creation so time-to-first-token
33+
// latency does not deflate the rate, and stop at the last finished text/reasoning part so
34+
// tool execution time does not either. Any still-open part keeps the window running to `now`.
35+
function generationWindow(message: AssistantMessage, parts: (TextPart | ReasoningPart)[], now: number) {
36+
const timed = parts.flatMap((part) => (part.time ? [part.time] : []))
37+
if (timed.length === 0) return { start: message.time.created, end: now }
38+
return {
39+
start: Math.min(...timed.map((time) => time.start)),
40+
end: timed.some((time) => time.end === undefined) ? now : Math.max(...timed.map((time) => time.end ?? now)),
41+
}
42+
}
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
import { describe, expect, test } from "bun:test"
2+
import type { AssistantMessage, Part } from "@opencode-ai/sdk/v2"
3+
import { finalTokensPerSecond, liveTokensPerSecond, tokensPerSecond } from "../../src/util/throughput"
4+
5+
function message(input: { created: number; completed?: number; output?: number; reasoning?: number }) {
6+
return {
7+
id: "msg_1",
8+
sessionID: "ses_1",
9+
role: "assistant",
10+
parentID: "msg_0",
11+
modelID: "model",
12+
providerID: "provider",
13+
mode: "build",
14+
agent: "build",
15+
path: { cwd: "/", root: "/" },
16+
cost: 0,
17+
time: { created: input.created, completed: input.completed },
18+
tokens: { input: 500, output: input.output ?? 0, reasoning: input.reasoning ?? 0, cache: { read: 900, write: 0 } },
19+
} satisfies AssistantMessage
20+
}
21+
22+
function text(id: string, chars: number, time?: { start: number; end?: number }): Part {
23+
return { id, sessionID: "ses_1", messageID: "msg_1", type: "text", text: "x".repeat(chars), time }
24+
}
25+
26+
function reasoning(id: string, chars: number, time: { start: number; end?: number }): Part {
27+
return { id, sessionID: "ses_1", messageID: "msg_1", type: "reasoning", text: "x".repeat(chars), time }
28+
}
29+
30+
describe("tokensPerSecond", () => {
31+
test("divides tokens by elapsed seconds", () => {
32+
expect(tokensPerSecond(200, 1_000, 3_000)).toBe(100)
33+
expect(tokensPerSecond(125, 0, 1_500)).toBe(83)
34+
})
35+
36+
test("returns undefined for windows under a second or no tokens", () => {
37+
expect(tokensPerSecond(200, 1_000, 1_999)).toBeUndefined()
38+
expect(tokensPerSecond(200, 1_000, 1_000)).toBeUndefined()
39+
expect(tokensPerSecond(200, 2_000, 1_000)).toBeUndefined()
40+
expect(tokensPerSecond(0, 0, 5_000)).toBeUndefined()
41+
})
42+
})
43+
44+
describe("liveTokensPerSecond", () => {
45+
test("rate tracks a growing stream of timestamped deltas", () => {
46+
const msg = message({ created: 0 })
47+
// First token arrives at 2s (time-to-first-token), then 50 tok/s (200 chars/s) of output.
48+
const samples = [
49+
{ now: 2_500, chars: 100 },
50+
{ now: 3_000, chars: 200 },
51+
{ now: 4_000, chars: 400 },
52+
{ now: 6_000, chars: 800 },
53+
]
54+
const rates = samples.map((sample) =>
55+
liveTokensPerSecond(msg, [text("a", sample.chars, { start: 2_000 })], sample.now),
56+
)
57+
expect(rates).toEqual([undefined, 50, 50, 50])
58+
})
59+
60+
test("sums text and reasoning parts and measures from the earliest part start", () => {
61+
const parts = [reasoning("r", 400, { start: 1_000, end: 2_000 }), text("t", 400, { start: 2_000 })]
62+
// 800 chars = 200 tokens over 2s.
63+
expect(liveTokensPerSecond(message({ created: 0 }), parts, 3_000)).toBe(100)
64+
})
65+
66+
test("excludes tool execution after the last generated part finished", () => {
67+
const parts = [
68+
text("t", 800, { start: 1_000, end: 3_000 }),
69+
{
70+
id: "tool",
71+
sessionID: "ses_1",
72+
messageID: "msg_1",
73+
type: "tool",
74+
callID: "c",
75+
tool: "bash",
76+
state: { status: "pending", input: {}, raw: "" },
77+
} satisfies Part,
78+
]
79+
expect(liveTokensPerSecond(message({ created: 0 }), parts, 30_000)).toBe(100)
80+
})
81+
82+
test("falls back to message creation when parts have no timestamps", () => {
83+
expect(liveTokensPerSecond(message({ created: 1_000 }), [text("t", 400)], 3_000)).toBe(50)
84+
})
85+
86+
test("returns undefined before any output", () => {
87+
expect(liveTokensPerSecond(message({ created: 0 }), [], 5_000)).toBeUndefined()
88+
})
89+
})
90+
91+
describe("finalTokensPerSecond", () => {
92+
test("uses reported output and reasoning tokens, ignoring input and cache", () => {
93+
const msg = message({ created: 0, completed: 5_000, output: 300, reasoning: 100 })
94+
expect(finalTokensPerSecond(msg, [text("t", 10, { start: 1_000, end: 5_000 })])).toBe(100)
95+
})
96+
97+
test("returns undefined until the message completes", () => {
98+
expect(finalTokensPerSecond(message({ created: 0, output: 300 }), [])).toBeUndefined()
99+
})
100+
})

0 commit comments

Comments
 (0)