Skip to content

Commit 080698e

Browse files
anandgupta42claude
andcommitted
fix: cap model labeling at 500 rows without user confirmation
The sample path in the `data-question` skill and `analyst.txt` had no size limit. Labeling by any path (own reading, subagents, warehouse AI functions) is now capped at 500 rows unless the user confirms more after seeing the row count; larger populations get a documented sample of up to 500 rows reported as an estimate. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 7788d7f commit 080698e

2 files changed

Lines changed: 2 additions & 2 deletions

File tree

‎.opencode/skills/data-question/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ Dialect references: [DuckDB query_table](https://www.duckdb.org/docs/current/gui
139139

140140
Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141141

142-
Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling every row with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself; do not loop model calls or subagents over a large population. Warehouse AI functions (for example classification functions in Snowflake, Databricks or BigQuery) are billed per row too: never run them over a large population without telling the user the row count and getting their confirmation. By default, for an aggregate question over a large population, label a documented random sample and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
142+
Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself. Without the user's confirmation, label at most 500 rows in total by any path: your own reading, subagents, or warehouse AI functions (for example classification functions in Snowflake, Databricks or BigQuery), which are billed per row too. To label more, first tell the user the row count and how the labels would be produced, and wait for their confirmation. When the population is larger than 500 rows, by default label a documented random sample of up to 500 rows (roughly ±4.5 percentage points at 95% confidence for a proportion) and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
143143

144144
To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145145

‎packages/opencode/src/altimate/prompts/analyst.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Inspect connections, schemas and actual rows before joining or parsing; key form
44

55
Derive large intermediate sets in-engine. For a cross-engine join, move the smaller side: a few hundred keys go into an IN list or VALUES against the big table; materialize into `scratch` only the side that is small after filtering, never a multi-million-row table when a filter or aggregate could shrink it first. Preserve requested weighting: group means weighted by contributing counts can be exact; unweighted averages change the weighting. For many similar tables, prefer catalog-driven SQL (information_schema, DuckDB query_table/UNION ALL BY NAME) after confirming dialect support.
66

7-
Treat inferred text labels as data: fixed rubric, stable IDs, coverage and duplicate checks, then deterministic counting, never by eye. Count the population first. Labeling grows in cost with the row count, so do not fan labeling out across many model calls or `data-explorer` subagents. For a large population, label a documented random sample and report an estimate with its sample size, unless the user confirms a full run after seeing the row count. When available, `task` with read-only `data-explorer` can verify reasoning.
7+
Treat inferred text labels as data: fixed rubric, stable IDs, coverage and duplicate checks, then deterministic counting, never by eye. Count the population first. Labeling grows in cost with the row count: without the user's confirmation, label at most 500 rows in total by any path (your own reading, `data-explorer` subagents or warehouse AI functions). For a larger population, label a documented random sample of up to 500 rows and report an estimate with its sample size; label more only after telling the user the row count and getting their confirmation. When available, `task` with read-only `data-explorer` can verify reasoning.
88

99
## Cost-conscious exploration
1010
Use selected columns, partition filters and LIMIT/sampling to inspect cheaply; estimates are not exact final computations. Read JSONL spill paths with `read`; spills contain returned rows, not necessarily all matches. Aggregate in SQL for exact answers. On timeout, revise the query. Report costs only from evidence.

0 commit comments

Comments
 (0)