Skip to content

Commit 612fd85

Browse files
anandgupta42claude
andcommitted
refactor: build error_fingerprint detail from an allowlist template
- revert all `maskString` changes; it is byte-identical to main again, so other consumers and fingerprint hashes are unaffected - `error_message` is a template of the first line: only short plain-letter words and basic punctuation survive, everything else becomes `<x>` (200 char cap) - `stack` keeps only frames from the CLI's own source, relative to the bundle root - payload building lives in `error-detail.ts` and is tested through the helpers `prompt.ts` calls - docs describe the template and its residual risk Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 65df0bf commit 612fd85

5 files changed

Lines changed: 189 additions & 195 deletions

File tree

‎docs/docs/reference/telemetry.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ We collect the following categories of events:
4949
| `task_outcome_signal` | Behavioral quality signal at session end — accepted, error, abandoned, or cancelled. Includes tool count, step count, duration, and last tool category. No user content. |
5050
| `task_classified` | Intent classification of the first user message using keyword matching — category (e.g. `debug_dbt`, `write_sql`, `optimize_query`), confidence score, and detected warehouse type. No user text is sent — only the classified category. |
5151
| `tool_chain_outcome` | Aggregated tool execution sequence at session end — ordered tool names (capped at 50), error count, recovery count, final outcome, duration, and cost. No tool arguments or outputs. |
52-
| `error_fingerprint` | Hashed error pattern for anonymous grouping — SHA-256 hash of masked error message, error class, tool name, and whether recovery succeeded. `error_message` is the first line of the error only, masked and capped at 300 characters (a line that is a SQL statement is replaced by a placeholder); `stack` holds up to 5 `file:line:function` frames from source files with the home prefix removed and only the last 3 path segments kept. Full error output is never sent. |
52+
| `error_fingerprint` | Error pattern for grouping — SHA-256 hash of the masked error message, error class, tool name, whether recovery succeeded, plus `error_message` and `stack`. `error_message` is a template of the error's first line: only short plain-letter words and basic punctuation are kept, and every identifier, number, path, URL, quoted value or capitalized name is replaced with `<x>` (max 200 characters). `stack` holds up to 5 frames from Altimate Code's own source only, as `src/...:line:function`; frames from user scripts, dependencies or other paths are dropped. |
5353
| `sql_fingerprint` | SQL structural shape via AST parsing — statement types, table count, function count, subquery/aggregation/window function presence, and AST node count. No table names, column names, or SQL content. |
5454
| `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
5555
| `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
@@ -157,11 +157,11 @@ Two escape hatches exist for reporting from a test run deliberately:
157157
We take your privacy seriously. Altimate Code telemetry **never** collects:
158158

159159
- SQL queries or query results
160-
- Code content, file contents, or file paths
160+
- Code content, file contents, or file paths (stack frames are limited to Altimate Code's own source files)
161161
- Credentials, API keys, or tokens
162162
- Database connection strings or hostnames
163163
- Personally identifiable information (your email is SHA-256 hashed before sending and is used only for anonymous user correlation)
164-
- Tool arguments or outputs
164+
- Tool arguments or outputs (the `error_fingerprint` error template keeps only plain English words from the first line of an error; identifiers, numbers, paths and quoted values are replaced with `<x>`. Residual risk: a statement or message written entirely in plain letters, such as an unquoted SQL fragment of simple words, can survive the template)
165165
- AI prompt content or responses
166166

167167
Error messages are truncated to 500 characters and scrubbed of file paths before sending.
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
import { Telemetry } from "./index"
2+
3+
const MAX_MESSAGE = 200
4+
const MAX_FRAMES = 5
5+
const PLACEHOLDER = "<x>"
6+
const PEEL = /^([:,.()'"\-?!]*)(.*?)([:,.()'"\-?!]*)$/
7+
const WORD = /^[A-Za-z]{1,24}$/
8+
const FRAME = /^\s*at\s+(?:(.+?)\s+\()?(.+?):(\d+)(?::\d+)?\)?\s*$/
9+
const BUNDLE_ROOT = /(?:^|\/)(?:\$bunfs|~BUN)\/root\/(.+)$/
10+
const SOURCE_ROOT = /(?:^|\/)packages\/opencode\/(src\/.+)$/
11+
const SOURCE_FILE = /\.[cm]?[jt]sx?$/
12+
const IDENT = /^[A-Za-z_$][\w$.]{0,60}$/
13+
14+
/**
15+
* Allowlist template of an error's first line. Only short plain-letter words and
16+
* a little punctuation survive; every other token (digits, `_`, `/`, `.` inside,
17+
* `@`, `=`, quoted content, capitalized names) becomes `<x>`. English-like words
18+
* that happen to spell a SQL statement are not recognised and can survive.
19+
*/
20+
export function templateErrorMessage(raw: string): string {
21+
const first = raw.split("\n", 1)[0].replace(/'[^']*'|"[^"]*"/g, ` ${PLACEHOLDER} `)
22+
const out: string[] = []
23+
let position = 0
24+
for (const chunk of first.split(/\s+/)) {
25+
if (!chunk) continue
26+
if (chunk === PLACEHOLDER) {
27+
out.push(PLACEHOLDER)
28+
position++
29+
continue
30+
}
31+
const [, lead, core, trail] = PEEL.exec(chunk)!
32+
const plainWord = WORD.test(core) && !(core.length > 3 && core === core.toUpperCase())
33+
const name = position > 0 && /^[A-Z]/.test(core)
34+
out.push(core === "" || (plainWord && !name) ? chunk : lead + PLACEHOLDER + trail)
35+
position++
36+
}
37+
return out
38+
.join(" ")
39+
.replace(/<x>(?:[:,.()'"\-?!]*\s+<x>)+/g, PLACEHOLDER)
40+
.slice(0, MAX_MESSAGE)
41+
}
42+
43+
/** Frames from altimate-code's own source only, as `src/...:line:function`; everything else is dropped. */
44+
export function ownSourceFrames(raw: string): string {
45+
const frames: string[] = []
46+
for (const line of raw.slice(0, 8192).split("\n")) {
47+
const m = FRAME.exec(line)
48+
if (!m) continue
49+
const file = m[2].replace(/^file:\/\//, "").replace(/\\/g, "/")
50+
if (file.includes("@") || file.includes("://") || file.includes("node_modules")) continue
51+
const rel = BUNDLE_ROOT.exec(file)?.[1] ?? SOURCE_ROOT.exec(file)?.[1]
52+
if (!rel || !SOURCE_FILE.test(rel)) continue
53+
const fn = (m[1] ?? "").replace(/^(?:async|new)\s+/, "")
54+
frames.push(`${rel}:${m[3]}:${IDENT.test(fn) ? fn : "<anonymous>"}`)
55+
if (frames.length >= MAX_FRAMES) break
56+
}
57+
return frames.join("\n")
58+
}
59+
60+
/** Hash input is unchanged from before the detail fields existed, so fingerprints stay stable. */
61+
export function describeToolError(raw: string) {
62+
return {
63+
errorHash: Telemetry.hashError(Telemetry.maskString(raw).slice(0, 500)),
64+
errorMessage: templateErrorMessage(raw),
65+
stack: ownSourceFrames(raw),
66+
}
67+
}
68+
69+
export function buildErrorFingerprintEvent(
70+
sessionID: string,
71+
err: {
72+
errorHash: string
73+
errorClass: string
74+
toolName: string
75+
toolCategory: string
76+
errorMessage: string
77+
stack: string
78+
recovered: boolean
79+
recoveryTool: string
80+
},
81+
): Telemetry.Event {
82+
return {
83+
type: "error_fingerprint",
84+
timestamp: Date.now(),
85+
session_id: sessionID,
86+
error_hash: err.errorHash,
87+
error_class: err.errorClass,
88+
tool_name: err.toolName,
89+
tool_category: err.toolCategory,
90+
error_message: err.errorMessage,
91+
...(err.stack ? { stack: err.stack } : {}),
92+
recovery_successful: err.recovered,
93+
recovery_tool: err.recoveryTool,
94+
}
95+
}

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 2 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -942,9 +942,9 @@ export namespace Telemetry {
942942
tool_name: string
943943
/** Tool category */
944944
tool_category: string
945-
/** Masked first line of the error, capped at 300 chars; a SQL statement becomes a placeholder */
945+
/** Allowlist template of the error's first line (identifiers and numbers replaced by <x>), max 200 chars */
946946
error_message?: string
947-
/** Top stack frames as file:line:function from source files, masked; omitted when none */
947+
/** Frames from altimate-code's own source as src-relative file:line:function; omitted when none */
948948
stack?: string
949949
/** Whether a subsequent tool call succeeded (error was recovered) */
950950
recovery_successful: boolean
@@ -1309,57 +1309,6 @@ export namespace Telemetry {
13091309
return createHash("sha256").update(maskedMessage).digest("hex").slice(0, 16)
13101310
}
13111311

1312-
// altimate_change start — actionable error detail for error_fingerprint
1313-
const MAX_ERROR_MESSAGE = 300
1314-
const MAX_STACK_FRAMES = 5
1315-
const STACK_FRAME = /^\s*at\s+(?:(.+?)\s+\()?(.+?):(\d+)(?::\d+)?\)?\s*$/
1316-
const SOURCE_FILE = /(?:\.(?:[cm]?[jt]sx?|py|rs|go)|^(?:node|bun):.+)$/
1317-
// SQL statement shapes, not bare keywords: "cannot delete file" or "read from cache" stay readable
1318-
const SQL_WORDS =
1319-
/\bselect\b[\s\S]*\bfrom\b|\binsert\s+into\b|\bupdate\s+\S+\s+set\b|\bdelete\s+from\b|\bmerge\s+into\b|\bwhere\b[\s\S]*(?:=|\blike\b|\bin\s*\()/i
1320-
1321-
/** Drops URL query strings and fragments, which commonly carry tokens or identifiers. */
1322-
function stripUrlQuery(s: string): string {
1323-
return s.replace(/(https?:\/\/[^\s?#'")]*)[?#][^\s'")]*/g, "$1?<redacted>")
1324-
}
1325-
1326-
/** SQL text cannot be reliably masked token by token, so a SQL-looking line is replaced outright. */
1327-
function redactSql(line: string): string {
1328-
return SQL_WORDS.test(line) ? "<sql redacted>" : line
1329-
}
1330-
1331-
/**
1332-
* Conservative message plus the top stack frames of an error's text. The
1333-
* message is the masked first line only, with SQL-looking lines replaced by
1334-
* a placeholder. Frames are `file:line:function` from source files only,
1335-
* home prefix removed, last three path segments kept. The message derives
1336-
* from the same masked text as the fingerprint input but is not identical
1337-
* to it; the hash itself is computed by the caller and is unaffected.
1338-
*/
1339-
export function errorDetail(raw: string): { message: string; stack: string } {
1340-
const first = raw.split("\n", 1)[0]
1341-
const message = redactSql(stripUrlQuery(maskString(first))).slice(0, MAX_ERROR_MESSAGE)
1342-
const frames: string[] = []
1343-
for (const line of raw.slice(0, 8192).split("\n")) {
1344-
const m = STACK_FRAME.exec(line)
1345-
if (!m) continue
1346-
const target = m[2]
1347-
if (target.includes("@") || (target.includes("://") && !target.startsWith("file://"))) continue
1348-
if (!SOURCE_FILE.test(target)) continue
1349-
const file = target
1350-
.replace(/^file:\/\//, "")
1351-
.replace(/^(?:[A-Za-z]:)?[\\/](?:Users|home)[\\/][^\\/]+[\\/]?/i, "")
1352-
.split(/[\\/]/)
1353-
.slice(-3)
1354-
.join("/")
1355-
frames.push(stripUrlQuery(maskString(`${file}:${m[3]}:${m[1] ?? "<anonymous>"}`)))
1356-
if (frames.length >= MAX_STACK_FRAMES) break
1357-
}
1358-
return { message, stack: frames.join("\n") }
1359-
}
1360-
// altimate_change end
1361-
1362-
13631312
/** Classify user intent from the first message text.
13641313
* Pure regex/keyword matcher — zero LLM cost, <1ms. */
13651314
export function classifyTaskIntent(text: string): { intent: string; confidence: number } {
@@ -1751,23 +1700,6 @@ export namespace Telemetry {
17511700
.replace(/\x1b(?:\[[0-?]*[ -\/]*[@-~]|\][^\x07\x1b]*(?:\x07|\x1b\\))/g, "")
17521701
.replace(/sk-(?:ant-)?[A-Za-z0-9_-]{20,}/g, "sk-***")
17531702
.replace(/Bearer\s+[A-Za-z0-9._-]{20,}/gi, "Bearer ***")
1754-
// altimate_change start — credential shapes beyond sk-/Bearer: URL userinfo, cloud/VCS/chat
1755-
// tokens, JWTs, key=value secrets, long opaque runs and warehouse/cloud hostnames
1756-
.replace(/(?<![a-z0-9+.-])([a-z][a-z0-9+.-]*:\/\/)[^\s]*@/gi, "$1***@")
1757-
.replace(/\beyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]*/g, "<jwt>")
1758-
.replace(/\b(?:AKIA|ASIA)[A-Z0-9]{12,}\b/g, "<aws-key>")
1759-
.replace(/\bgh[pousr]_[A-Za-z0-9]{20,}/g, "<github-token>")
1760-
.replace(/\bxox[a-z]-[A-Za-z0-9-]{10,}/g, "<slack-token>")
1761-
.replace(
1762-
/\b([A-Za-z0-9_]*(?:password|passwd|pwd|secret|token|key|credential)[A-Za-z0-9_]*)\s*=\s*(?:"[^"]*"|'[^']*'|[^\s,;&]+)/gi,
1763-
"$1=***",
1764-
)
1765-
.replace(/\b(?=[A-Za-z0-9+_-]*\d)[A-Za-z0-9+_-]{32,}/g, "<opaque>")
1766-
.replace(
1767-
/(?<![A-Za-z0-9.-])[A-Za-z0-9.-]*(?:snowflakecomputing\.com|amazonaws\.com|database\.windows\.net|azure\.com|azure-api\.net|bigquery[A-Za-z0-9.-]*|databricks[A-Za-z0-9.-]*)/gi,
1768-
"<host>",
1769-
)
1770-
// altimate_change end
17711703
.replace(/'(?:[^'\\]|\\.)*'/g, "?")
17721704
.replace(/"(?:[^"\\]|\\.)*"/g, "?")
17731705
// Fast path: a string with no separator cannot contain a path — skip the

‎packages/opencode/src/session/prompt.ts‎

Lines changed: 3 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,7 @@ import { stampRegistryToolSource, describeMcpTool } from "../altimate/tool-sourc
113113
// altimate_change end
114114
// altimate_change end
115115
import { Telemetry } from "@/telemetry" // altimate_change — session telemetry
116+
import { describeToolError, buildErrorFingerprintEvent } from "@/altimate/telemetry/error-detail" // altimate_change — error_fingerprint payload
116117
import { TraceContext } from "@/altimate/observability/trace-context" // altimate_change — client trace per turn
117118
import * as OnboardingTelemetry from "@/altimate/telemetry/onboarding" // altimate_change — onboarding funnel
118119
// altimate_change start — keyless public Zen predicate (flat module)
@@ -2178,15 +2179,11 @@ export namespace SessionPrompt {
21782179
lastToolWasError = true
21792180
const errorMsg =
21802181
part.state.status === "error" && typeof part.state.error === "string" ? part.state.error : "unknown"
2181-
const masked = Telemetry.maskString(errorMsg).slice(0, 500)
2182-
const detail = Telemetry.errorDetail(errorMsg)
21832182
pendingError = {
21842183
toolName: part.tool,
21852184
toolCategory,
21862185
errorClass: Telemetry.classifyError(errorMsg),
2187-
errorHash: Telemetry.hashError(masked),
2188-
errorMessage: detail.message,
2189-
stack: detail.stack,
2186+
...describeToolError(errorMsg),
21902187
}
21912188
} else {
21922189
if (lastToolWasError && pendingError) {
@@ -2538,19 +2535,7 @@ export namespace SessionPrompt {
25382535
}
25392536
// Error fingerprints — one event per unique error (capped at 20)
25402537
for (const err of errorRecords.slice(0, 20)) {
2541-
Telemetry.track({
2542-
type: "error_fingerprint",
2543-
timestamp: Date.now(),
2544-
session_id: sessionID,
2545-
error_hash: err.errorHash,
2546-
error_class: err.errorClass,
2547-
tool_name: err.toolName,
2548-
tool_category: err.toolCategory,
2549-
error_message: err.errorMessage,
2550-
...(err.stack ? { stack: err.stack } : {}),
2551-
recovery_successful: err.recovered,
2552-
recovery_tool: err.recoveryTool,
2553-
})
2538+
Telemetry.track(buildErrorFingerprintEvent(sessionID, err))
25542539
}
25552540
// altimate_change end — emit quality signal, tool chain, and error fingerprint events
25562541
// altimate_change start — populate agent_outcome diagnostic fields

0 commit comments

Comments
 (0)