Skip to content

Commit 5ea02f9

Browse files
anandgupta42claude
andcommitted
feat(telemetry): add a masked error message template to error_fingerprint
`error_fingerprint` carried only a hash of the error. It now also carries `error_message`: an allowlist template of the error's first line in which identifiers, numbers, quoted text, names and credential values become `<x>`. `maskString` and the fingerprint hash input are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 3ed4561 commit 5ea02f9

5 files changed

Lines changed: 259 additions & 16 deletions

File tree

‎docs/docs/reference/telemetry.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ We collect the following categories of events:
4949
| `task_outcome_signal` | Behavioral quality signal at session end — accepted, error, abandoned, or cancelled. Includes tool count, step count, duration, and last tool category. No user content. |
5050
| `task_classified` | Intent classification of the first user message using keyword matching — category (e.g. `debug_dbt`, `write_sql`, `optimize_query`), confidence score, and detected warehouse type. No user text is sent — only the classified category. |
5151
| `tool_chain_outcome` | Aggregated tool execution sequence at session end — ordered tool names (capped at 50), error count, recovery count, final outcome, duration, and cost. No tool arguments or outputs. |
52-
| `error_fingerprint` | Hashed error pattern for anonymous grouping — SHA-256 hash of masked error message, error class, tool name, and whether recovery succeeded. Raw error content is never sent. |
52+
| `error_fingerprint` | Error pattern for grouping — SHA-256 hash of the masked error message (`error_hash`), plus separate fields for error class, tool name, whether recovery succeeded, and `error_message`. `error_message` is a template of the error's first line: only short plain-letter words and basic punctuation are kept, and identifiers containing digits, `_`, `.`, `/` or `@`, numbers, paths, URLs, quoted values, capitalized names and values after credential keywords are replaced with `<x>` (max 200 characters). Plain lowercase words such as an unquoted table name can survive. |
5353
| `sql_fingerprint` | SQL structural shape via AST parsing — statement types, table count, function count, subquery/aggregation/window function presence, and AST node count. No table names, column names, or SQL content. |
5454
| `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
5555
| `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
@@ -161,7 +161,7 @@ We take your privacy seriously. Altimate Code telemetry **never** collects:
161161
- Credentials, API keys, or tokens
162162
- Database connection strings or hostnames
163163
- Personally identifiable information (your email is SHA-256 hashed before sending and is used only for anonymous user correlation)
164-
- Tool arguments or outputs
164+
- Tool arguments or outputs, except the first line of tool errors in `error_fingerprint` (the `error_fingerprint` error template keeps only plain English words from the first line of an error; identifiers, numbers, paths and quoted values are replaced with `<x>`. Residual risk: a statement or message written entirely in plain letters, such as an unquoted SQL fragment of simple words, can survive the template)
165165
- AI prompt content or responses
166166

167167
Error messages are truncated to 500 characters and scrubbed of file paths before sending.
@@ -210,7 +210,7 @@ Event type names use **snake_case** with a `domain_action` pattern:
210210
- `error_recovered` for error recovery events
211211
- `task_outcome_signal`, `task_classified` for session quality signals
212212
- `tool_chain_outcome` for tool execution chain aggregation
213-
- `error_fingerprint` for anonymous error pattern grouping
213+
- `error_fingerprint` for error pattern grouping, with a masked one-line error message
214214
- `sql_fingerprint` for SQL structural analysis
215215
- `schema_complexity` for warehouse schema metrics
216216

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
import { Telemetry } from "./index"
2+
3+
const MAX_MESSAGE = 200
4+
const PLACEHOLDER = "<x>"
5+
const PEEL = /^([:,.()'"\-?!]*)(.*?)([:,.()'"\-?!]*)$/
6+
const WORD = /^(?:[a-z]{1,24}|[A-Z][a-z]{0,23}|[A-Z]{1,3})$/
7+
const SENTENCE_STARTERS = new Set([
8+
"error",
9+
"cannot",
10+
"can",
11+
"could",
12+
"failed",
13+
"failure",
14+
"unable",
15+
"invalid",
16+
"unknown",
17+
"missing",
18+
"no",
19+
"not",
20+
"the",
21+
"a",
22+
"an",
23+
"permission",
24+
"access",
25+
"connection",
26+
"timeout",
27+
"request",
28+
"query",
29+
"command",
30+
"file",
31+
"table",
32+
"column",
33+
"unexpected",
34+
"expected",
35+
"syntax",
36+
"authentication",
37+
"operation",
38+
"http",
39+
"tool",
40+
"bad",
41+
"login",
42+
"user",
43+
"this",
44+
"it",
45+
"there",
46+
"too",
47+
"failed",
48+
])
49+
const CREDENTIAL_WORDS = new Set([
50+
"password",
51+
"passwd",
52+
"pwd",
53+
"secret",
54+
"token",
55+
"key",
56+
"auth",
57+
"authorization",
58+
"credential",
59+
"credentials",
60+
"apikey",
61+
"bearer",
62+
])
63+
const CREDENTIAL_SPAN = 2
64+
65+
/**
66+
* Allowlist template of an error's first line. Only short plain-letter words and
67+
* a little punctuation survive; every other token (digits, `_`, `/`, `.` inside,
68+
* `@`, `=`, quoted content, capitalized names) becomes `<x>`. English-like words
69+
* that happen to spell a SQL statement are not recognised and can survive.
70+
*/
71+
export function templateErrorMessage(raw: string): string {
72+
const first = raw
73+
.split(/[\r\n]/, 1)[0]
74+
.replace(/'[^']*'|"[^"]*"/g, ` ${PLACEHOLDER} `)
75+
.replace(/['"].*$/, ` ${PLACEHOLDER}`)
76+
const out: string[] = []
77+
let position = 0
78+
let secretSpan = 0
79+
let secretToEol = false
80+
for (const chunk of first.split(/\s+/)) {
81+
if (!chunk) continue
82+
if (chunk === PLACEHOLDER) {
83+
out.push(PLACEHOLDER)
84+
position++
85+
continue
86+
}
87+
const [, lead, core, trail] = PEEL.exec(chunk)!
88+
// the tokens after a credential keyword are values, however plain they look
89+
if (secretSpan > 0 && core !== "") {
90+
out.push(lead + PLACEHOLDER + trail)
91+
position++
92+
secretSpan = secretToEol ? 1 : /[,.)!?]/.test(trail) ? 0 : secretSpan - 1
93+
continue
94+
}
95+
if (
96+
core
97+
.toLowerCase()
98+
.split(/[_\-.]/)
99+
.some((part) => CREDENTIAL_WORDS.has(part))
100+
) {
101+
secretSpan = CREDENTIAL_SPAN
102+
// `label: value` / `label=value`: the value may be a multi-word passphrase, so mask to end of line
103+
secretToEol = /[:=]/.test(trail) || /[:=]/.test(lead)
104+
}
105+
const plainWord = WORD.test(core) && !(core.length > 3 && core === core.toUpperCase())
106+
const lower = core.toLowerCase()
107+
const name = /^[A-Z]/.test(core) && (position > 0 || !(SENTENCE_STARTERS.has(lower) || CREDENTIAL_WORDS.has(lower)))
108+
out.push(core === "" || (plainWord && !name) ? chunk : lead + PLACEHOLDER + trail)
109+
position++
110+
}
111+
return out
112+
.join(" ")
113+
.replace(/<x>(?:[:,.()'"\-?!]*\s+<x>)+/g, PLACEHOLDER)
114+
.slice(0, MAX_MESSAGE)
115+
}
116+
117+
/** Hash input is unchanged from before the detail fields existed, so fingerprints stay stable. */
118+
export function describeToolError(raw: string) {
119+
return {
120+
errorHash: Telemetry.hashError(Telemetry.maskString(raw).slice(0, 500)),
121+
errorMessage: templateErrorMessage(raw),
122+
}
123+
}
124+
125+
export function buildErrorFingerprintEvent(
126+
sessionID: string,
127+
err: {
128+
errorHash: string
129+
errorClass: string
130+
toolName: string
131+
toolCategory: string
132+
errorMessage: string
133+
recovered: boolean
134+
recoveryTool: string
135+
},
136+
): Telemetry.Event {
137+
return {
138+
type: "error_fingerprint",
139+
timestamp: Date.now(),
140+
session_id: sessionID,
141+
error_hash: err.errorHash,
142+
error_class: err.errorClass,
143+
tool_name: err.toolName,
144+
tool_category: err.toolCategory,
145+
error_message: err.errorMessage,
146+
recovery_successful: err.recovered,
147+
recovery_tool: err.recoveryTool,
148+
}
149+
}
150+
151+
export * as ErrorDetail from "./error-detail"

‎packages/opencode/src/altimate/telemetry/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -942,6 +942,8 @@ export namespace Telemetry {
942942
tool_name: string
943943
/** Tool category */
944944
tool_category: string
945+
/** Allowlist template of the error first line, max 200 chars: plain lowercase words are kept, identifiers, numbers, quoted values, capitalized names and credential values become <x> */
946+
error_message?: string
945947
/** Whether a subsequent tool call succeeded (error was recovered) */
946948
recovery_successful: boolean
947949
/** Tool that succeeded after the error (if recovered) */

‎packages/opencode/src/session/prompt.ts‎

Lines changed: 4 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,7 @@ import { stampRegistryToolSource, describeMcpTool } from "../altimate/tool-sourc
113113
// altimate_change end
114114
// altimate_change end
115115
import { Telemetry } from "@/telemetry" // altimate_change — session telemetry
116+
import { ErrorDetail } from "@/altimate/telemetry/error-detail" // altimate_change — error_fingerprint payload
116117
import { TraceContext } from "@/altimate/observability/trace-context" // altimate_change — client trace per turn
117118
import * as OnboardingTelemetry from "@/altimate/telemetry/onboarding" // altimate_change — onboarding funnel
118119
// altimate_change start — keyless public Zen predicate (flat module)
@@ -801,6 +802,7 @@ export namespace SessionPrompt {
801802
toolCategory: string
802803
errorClass: string
803804
errorHash: string
805+
errorMessage: string
804806
recovered: boolean
805807
recoveryTool: string
806808
}
@@ -2176,12 +2178,11 @@ export namespace SessionPrompt {
21762178
lastToolWasError = true
21772179
const errorMsg =
21782180
part.state.status === "error" && typeof part.state.error === "string" ? part.state.error : "unknown"
2179-
const masked = Telemetry.maskString(errorMsg).slice(0, 500)
21802181
pendingError = {
21812182
toolName: part.tool,
21822183
toolCategory,
21832184
errorClass: Telemetry.classifyError(errorMsg),
2184-
errorHash: Telemetry.hashError(masked),
2185+
...ErrorDetail.describeToolError(errorMsg),
21852186
}
21862187
} else {
21872188
if (lastToolWasError && pendingError) {
@@ -2533,17 +2534,7 @@ export namespace SessionPrompt {
25332534
}
25342535
// Error fingerprints — one event per unique error (capped at 20)
25352536
for (const err of errorRecords.slice(0, 20)) {
2536-
Telemetry.track({
2537-
type: "error_fingerprint",
2538-
timestamp: Date.now(),
2539-
session_id: sessionID,
2540-
error_hash: err.errorHash,
2541-
error_class: err.errorClass,
2542-
tool_name: err.toolName,
2543-
tool_category: err.toolCategory,
2544-
recovery_successful: err.recovered,
2545-
recovery_tool: err.recoveryTool,
2546-
})
2537+
Telemetry.track(ErrorDetail.buildErrorFingerprintEvent(sessionID, err))
25472538
}
25482539
// altimate_change end — emit quality signal, tool chain, and error fingerprint events
25492540
// altimate_change start — populate agent_outcome diagnostic fields
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
import { describe, expect, test } from "bun:test"
2+
import { Telemetry } from "../../src/altimate/telemetry"
3+
import { ErrorDetail } from "../../src/altimate/telemetry/error-detail"
4+
5+
const { buildErrorFingerprintEvent, describeToolError, templateErrorMessage } = ErrorDetail
6+
// built at runtime so no credential-shaped literal is committed
7+
const basicAuth = Buffer.from(["adm", "pw"].join(":")).toString("base64")
8+
const pgUri = ["postgres:", "//", "u", ":", "p", "@db:5432/x"].join("")
9+
10+
describe("templateErrorMessage", () => {
11+
test.each([
12+
["column ssn_number does not exist in table hr.salaries", "column <x> does not exist in table <x>"],
13+
["password: hunter2", "password: <x>"],
14+
["Cannot delete file: permission denied", "Cannot delete file: permission denied"],
15+
["failed (code 12)!", "failed (code <x>)!"],
16+
])("%s", (raw, expected) => {
17+
expect(templateErrorMessage(raw)).toBe(expected)
18+
})
19+
20+
test("Basic auth header keeps no secret", () => {
21+
const out = templateErrorMessage(`Authorization: Basic ${basicAuth}`)
22+
expect(out).toBe("Authorization: <x>")
23+
})
24+
25+
test("unquoted SQL keeps no identifiers, numbers or names", () => {
26+
const out = templateErrorMessage("SELECT ssn FROM prod.hr.salaries WHERE id = 4417 AND name = Alice")
27+
for (const leak of ["prod", "hr.salaries", "4417", "Alice"]) expect(out).not.toContain(leak)
28+
})
29+
30+
test.each([
31+
"/Users/bob/proj/a.ts",
32+
"C:\\Users\\bob\\a.ts",
33+
34+
"https://h.example.com/p?token=zzz",
35+
pgUri,
36+
"AKIAIOSFODNN7EXAMPLE",
37+
"ghp_abcdefghijklmnopqrstuvwxyz0123456789",
38+
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0In0.sig",
39+
"deadbeefdeadbeefdeadbeefdeadbeef0123",
40+
"'hunter2' \"s3cret\"",
41+
"stg_salesforce_history_2024.sql",
42+
])("replaces identifier-like token in %s", (raw) => {
43+
const out = templateErrorMessage(`bad ${raw} here`)
44+
expect(out).toBe("bad <x> here")
45+
})
46+
47+
test.each([
48+
["login failed password: supersecret", "login failed password: <x>"],
49+
["Authorization: Bearer secrettoken", "Authorization: <x>"],
50+
["bad api key hunter, retry", "bad api key <x>, retry"],
51+
["client_secret: supersecret", "<x>"],
52+
["access_token: secrettoken", "<x>"],
53+
["api-key: secretvalue", "<x>"],
54+
["password: correct horse battery staple", "password: <x>"],
55+
["Alice does not have access", "<x> does not have access"],
56+
["failed\raccount alice unavailable", "failed"],
57+
["Error: value 'private customer phrase is invalid", "Error: value <x>"],
58+
["xKqPzLmNaBcD failed", "<x> failed"],
59+
["DROP TABLE prod.customer_records", "<x>"],
60+
])("masks alphabetic credential values in %s", (raw, expected) => {
61+
expect(templateErrorMessage(raw)).toBe(expected)
62+
})
63+
64+
test("only the first line, capped at 200 chars", () => {
65+
expect(templateErrorMessage("first line\nsecond line")).toBe("first line")
66+
expect(templateErrorMessage("word ".repeat(200)).length).toBeLessThanOrEqual(200)
67+
})
68+
})
69+
70+
describe("error_fingerprint wiring", () => {
71+
const raw = "column ssn_number does not exist in table hr.salaries\n at run (/w/packages/opencode/src/a.ts:1:1)"
72+
73+
test("error_hash is identical to the pre-existing computation", () => {
74+
expect(describeToolError(raw).errorHash).toBe("3a0bc6389647e452")
75+
})
76+
77+
test("event carries the template", () => {
78+
const d = describeToolError(raw)
79+
const base = {
80+
...d,
81+
errorClass: "unknown",
82+
toolName: "bash",
83+
toolCategory: "shell",
84+
recovered: false,
85+
recoveryTool: "",
86+
}
87+
const ev: any = buildErrorFingerprintEvent("s1", base)
88+
expect(ev.type).toBe("error_fingerprint")
89+
expect(ev.error_message).toBe("column <x> does not exist in table <x>")
90+
expect(ev.error_hash).toBe(d.errorHash)
91+
expect("stack" in ev).toBe(false)
92+
})
93+
94+
test("prompt.ts emits through these helpers", async () => {
95+
const src = await Bun.file(new URL("../../src/session/prompt.ts", import.meta.url)).text()
96+
expect(src).toContain("...ErrorDetail.describeToolError(errorMsg)")
97+
expect(src).toContain("Telemetry.track(ErrorDetail.buildErrorFingerprintEvent(sessionID, err))")
98+
})
99+
})

0 commit comments

Comments
 (0)