You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a660824
Browse filesBrowse the repository at this point in the historyBrowse files
fix: remove the text_label bulk labeling tool and add a cost side-effects rule
- Remove `text_label` (tool, registry entry, `data-explorer` permission and
its tests). It made one session-model call per ~50-row batch, so its cost
grew with table size, and its calls were missing from traces and session
cost.
- `tool-schema-anthropic.test.ts`: assert no agent registers `text_label`;
keep the generic empty-sentinel schema check.
- `data-question` skill: count the population first; prefer
warehouse-native AI classification where permitted, otherwise label a
documented random sample and report an estimate.
- `REVIEW.md` focus area 12, `packages/opencode/AGENTS.md` and the PR
template require checking and flagging cost side effects that grow with
user data.
Closes#1547
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: .opencode/skills/data-question/SKILL.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ Use the available read-only tools to answer the user's question. Source database
11
11
12
12
1.**Inspect.** Use `warehouse_list`, `schema_inspect` and small `sql_execute` queries as needed. Inspect actual keys and text before choosing joins, casts or parsing rules: schemas alone hide whitespace, case, prefixes and mixed formats. Preserve meaningful leading zeros; check that normalization does not merge distinct identities.
13
13
2.**Plan.** State the population, row grain, denominator, units and material assumptions. Decide which joins and aggregation levels preserve them. Keep this brief; a trivial lookup does not need a checklist or several validation scans.
14
-
3.**Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate. For exact final answers, aggregate the full intended population in SQL.
14
+
3.**Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate, or an exact answer would need per-row model labeling of a large population (see "Batch semantic labels, then count"); then say the answer is an estimate. For exact final answers, aggregate the full intended population in SQL.
15
15
4.**Verify relevant invariants.** Check key uniqueness, before/after join counts, missing matches and null share when they affect the answer. Use `sql_check` for these invariants and extraction coverage; for cross-warehouse comparisons, first materialize both sides into `scratch` with `sql_materialize`. Reconcile components with totals and preserve weighting. `SUM(group_mean * contributing_count) / SUM(contributing_count)` recovers the overall mean for disjoint, complete groups using unrounded means and counts of the same non-null measure; `AVG(group_mean)` gives each group equal weight. `sql_analyze` is optional. If available, a read-only `data-explorer` via `task` can independently re-derive a difficult result.
16
16
5.**Answer.** Explain the basis in one or two sentences (population, grain, caveats), retaining useful tables or explanation before it. Finish with `**Answer:** <requested value(s), with units>`. If evidence or available tools are insufficient, finish with `**Answer:** unable to determine — <why>`. Never guess a missing result.
Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141
141
142
-
Inspect text, then define the complete population of stable IDs. When `text_label` is available, materialize that population in `scratch` with `sql_materialize`, then call `text_label` with SQL over scratch, the stable ID and text column names, a fixed label set with a description for each label, and any rubric/tie-breaking instructions. It classifies every row through the session's LLM in bounded batches and persists the results in scratch; do not substitute regex or keyword guesses for semantic labels. `text_label` accepts at most 30,000 rows per call. For a larger population, split it into disjoint, deterministic id ranges (one call per range, each into a distinct result table), combine the persisted result tables in SQL, and verify that combined coverage equals the population count before claiming results. Check `rows_in`, `labeled`, `failed` and `unparseable`, and retrieve failed IDs before claiming complete coverage. Optional spot-check agreement measures repeatability, not correctness. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
142
+
Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling every row with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself; do not loop model calls over a large population. If the warehouse has its own AI classification functions and the user's configuration permits them, prefer those, since the warehouse runs and bills them under the user's controls. Otherwise, for an aggregate question over a large population, label a documented random sample and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
143
143
144
-
Use the persisted `text_label` output for SQL counts. It retains the source ID column name, so use `message_id` as the ID column for this example. To reuse the checks below, select `message_id`, `label` and your declared rubric version as `rubric_version` in a SELECT/CTE named `labels`; `population` must cover the same intended IDs. If `text_label` is unavailable, small audited annotation batches can use an available permitted ingestion path to form `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
144
+
To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145
145
146
146
```sql
147
147
WITH annotations(message_id, label, rubric_version) AS (
@@ -152,9 +152,9 @@ WITH annotations(message_id, label, rubric_version) AS (
152
152
SELECT*FROM annotations;
153
153
```
154
154
155
-
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. Use `text_label` for large labeling jobs; if it is unavailable and there is no permitted way to ingest complete returned annotation records, report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
155
+
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. When the full population cannot be labeled within these limits, label a sample or report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
156
156
157
-
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
157
+
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs (the sampled IDs for a sample) and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
158
158
159
159
```sql
160
160
-- Must return no rows: includes duplicates across batches.
@@ -179,7 +179,7 @@ SELECT label, COUNT(*) AS messages
179
179
FROM labels GROUP BY label ORDER BY label;
180
180
```
181
181
182
-
Verify the label counts sum to the full population count. Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries.
182
+
Verify the label counts sum to the `population` count (the sample size when you labeled a sample). Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries. For a sample, report proportions with the sample size and the size of the full population, and do not present scaled-up counts as exact.
Copy file name to clipboardExpand all lines: REVIEW.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -49,6 +49,8 @@ Before flagging: trace the actual caller and the actual shipped surface. This re
49
49
50
50
11.**Test isolation leaks under parallel CI** (Warning). `mock.module`/dispatcher `reset()` leaked state across test files (52 CI failures fixed via #460). **Flag** new tests using global mocks/dispatchers without confirmed teardown under parallel `bun test`.
51
51
52
+
12. **Unbounded cost side effects** (Critical; always check, every PR). A feature whose spend grows with user data rather than with the conversation. `text_label` shipped on by default and labeled every row of a table with one session-model call per ~50-row batch: one evaluation day billed ~$680 for it, about 7x the agent sessions themselves, none of it visible in traces or session cost. On a million-row table that is ~20,000 model calls per question. **Flag when** a change can, without an explicit user decision: make model calls per row, batch, file or item (fan-out inside a tool, loops over query results, per-item summarization or classification); run unbounded or user-data-sized warehouse queries, scans or exports; start cloud resources; or add hidden model calls (sub-calls inside tools, background or automatic calls). **Require** all of: (a) off by default or bounded by a hard cap that does not scale with data size; (b) above the cap, an estimate shown and the user's confirmation required before the work runs; (c) the cheapest adequate model by default, with no cache breakpoint on one-off content; (d) its usage counted in session cost and traces. A PR that adds such a path without these is not mergeable; say so even when the feature itself is correct.
53
+
52
54
## Repo invariants & landmines
53
55
54
56
- Effect (`effect` v4 beta) is used pervasively; `Effect.promise`/`Effect.sync`-returning functions look like normal async functions but must be run via `AppRuntime.runPromise` (or equivalent) — a bare `await` silently no-ops the effect.
Copy file name to clipboardExpand all lines: packages/opencode/AGENTS.md
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,19 @@
12
12
- Capture the current TUI output with: `tmux capture-pane -pt opencode-dev`.
13
13
- Stop the session explicitly when done: `tmux kill-session -t opencode-dev`.
14
14
15
+
# Cost side effects
16
+
17
+
Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR body in a "Cost side effects" section (the PR template's checklist asks for it), even when the answer is "none". Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
18
+
19
+
Such a path must be:
20
+
21
+
- Off by default, or bounded by a hard cap that does not scale with data size.
22
+
- Above the cap, shown as a cost estimate and run only after the user confirms.
23
+
- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
24
+
- Counted in session cost and traces, so its spend is visible.
25
+
26
+
Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
27
+
15
28
# Module shape
16
29
17
30
Do not use `export namespace Foo { ... }` for module organization. It is not
0 commit comments