Skip to content

Commit 8feb417

Browse files
anandgupta42claude
andcommitted
refactor: build error_message from a fixed vocabulary of error words
Split the first line into letter runs and keep only words in `ERROR_VOCABULARY`; every other run becomes `<x>`. Drops the punctuation/quote/credential-stem heuristics, which could not be made complete. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 232a2c8 commit 8feb417

3 files changed

Lines changed: 434 additions & 149 deletions

File tree

‎docs/docs/reference/telemetry.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ We collect the following categories of events:
4949
| `task_outcome_signal` | Behavioral quality signal at session end — accepted, error, abandoned, or cancelled. Includes tool count, step count, duration, and last tool category. No user content. |
5050
| `task_classified` | Intent classification of the first user message using keyword matching — category (e.g. `debug_dbt`, `write_sql`, `optimize_query`), confidence score, and detected warehouse type. No user text is sent — only the classified category. |
5151
| `tool_chain_outcome` | Aggregated tool execution sequence at session end — ordered tool names (capped at 50), error count, recovery count, final outcome, duration, and cost. No tool arguments or outputs. |
52-
| `error_fingerprint` | Error pattern for grouping — SHA-256 hash of the masked error message (`error_hash`), plus separate fields for error class, tool name, whether recovery succeeded, and `error_message`. `error_message` is a template of the error's first line: only short plain-letter words and basic punctuation are kept, and identifiers containing digits, `_`, `.`, `/` or `@`, numbers, paths, URLs, quoted values, capitalized names and values after credential keywords are replaced with `<x>` (max 200 characters). Plain lowercase words such as an unquoted table name can survive. |
52+
| `error_fingerprint` | Error pattern for grouping — SHA-256 hash of the masked error message, error class, tool name, whether recovery succeeded, plus `error_message`. `error_message` contains only words from a fixed list of common error terms taken from the first line of the error; every other word is replaced with `<x>`, so no user text, identifiers, paths, values or SQL can appear (max 200 characters). |
5353
| `sql_fingerprint` | SQL structural shape via AST parsing — statement types, table count, function count, subquery/aggregation/window function presence, and AST node count. No table names, column names, or SQL content. |
5454
| `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
5555
| `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
@@ -156,12 +156,12 @@ Two escape hatches exist for reporting from a test run deliberately:
156156

157157
We take your privacy seriously. Altimate Code telemetry **never** collects:
158158

159-
- SQL queries or query results (except that the `error_fingerprint` template can keep a fragment made only of plain words, see below)
159+
- SQL queries or query results
160160
- Code content, file contents, or file paths
161161
- Credentials, API keys, or tokens
162162
- Database connection strings or hostnames
163163
- Personally identifiable information (your email is SHA-256 hashed before sending and is used only for anonymous user correlation)
164-
- Tool arguments or outputs, except the first line of tool errors in `error_fingerprint` (the `error_fingerprint` error template keeps only plain English words from the first line of an error; identifiers, numbers, paths and quoted values are replaced with `<x>`. Residual risk: a statement or message written entirely in plain letters, such as an unquoted SQL fragment of simple words, can survive the template)
164+
- Tool arguments or outputs
165165
- AI prompt content or responses
166166

167167
Error messages are truncated to 500 characters and scrubbed of file paths before sending.
@@ -210,7 +210,7 @@ Event type names use **snake_case** with a `domain_action` pattern:
210210
- `error_recovered` for error recovery events
211211
- `task_outcome_signal`, `task_classified` for session quality signals
212212
- `tool_chain_outcome` for tool execution chain aggregation
213-
- `error_fingerprint` for error pattern grouping, with a masked one-line error message
213+
- `error_fingerprint` for error pattern grouping, with a one-line error message built from a fixed vocabulary of common error words
214214
- `sql_fingerprint` for SQL structural analysis
215215
- `schema_complexity` for warehouse schema metrics
216216

0 commit comments

Comments
 (0)