Skip to content

Commit a660824

Browse files
anandgupta42claude
andcommitted
fix: remove the text_label bulk labeling tool and add a cost side-effects rule
- Remove `text_label` (tool, registry entry, `data-explorer` permission and its tests). It made one session-model call per ~50-row batch, so its cost grew with table size, and its calls were missing from traces and session cost. - `tool-schema-anthropic.test.ts`: assert no agent registers `text_label`; keep the generic empty-sentinel schema check. - `data-question` skill: count the population first; prefer warehouse-native AI classification where permitted, otherwise label a documented random sample and report an estimate. - `REVIEW.md` focus area 12, `packages/opencode/AGENTS.md` and the PR template require checking and flagging cost side effects that grow with user data. Closes #1547 Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent de3d05a commit a660824

10 files changed

Lines changed: 29 additions & 1247 deletions

File tree

‎.github/pull_request_template.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,5 +25,8 @@ _If this is a UI change, please include a screenshot or recording._
2525

2626
- [ ] I have tested my changes locally
2727
- [ ] I have not included unrelated changes in this PR
28+
<!-- altimate_change start — cost side-effects check (see packages/opencode/AGENTS.md) -->
29+
- [ ] I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
30+
<!-- altimate_change end -->
2831

2932
_If you do not follow this template your PR will be automatically rejected._

‎.opencode/skills/data-question/SKILL.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Use the available read-only tools to answer the user's question. Source database
1111

1212
1. **Inspect.** Use `warehouse_list`, `schema_inspect` and small `sql_execute` queries as needed. Inspect actual keys and text before choosing joins, casts or parsing rules: schemas alone hide whitespace, case, prefixes and mixed formats. Preserve meaningful leading zeros; check that normalization does not merge distinct identities.
1313
2. **Plan.** State the population, row grain, denominator, units and material assumptions. Decide which joins and aggregation levels preserve them. Keep this brief; a trivial lookup does not need a checklist or several validation scans.
14-
3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate. For exact final answers, aggregate the full intended population in SQL.
14+
3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate, or an exact answer would need per-row model labeling of a large population (see "Batch semantic labels, then count"); then say the answer is an estimate. For exact final answers, aggregate the full intended population in SQL.
1515
4. **Verify relevant invariants.** Check key uniqueness, before/after join counts, missing matches and null share when they affect the answer. Use `sql_check` for these invariants and extraction coverage; for cross-warehouse comparisons, first materialize both sides into `scratch` with `sql_materialize`. Reconcile components with totals and preserve weighting. `SUM(group_mean * contributing_count) / SUM(contributing_count)` recovers the overall mean for disjoint, complete groups using unrounded means and counts of the same non-null measure; `AVG(group_mean)` gives each group equal weight. `sql_analyze` is optional. If available, a read-only `data-explorer` via `task` can independently re-derive a difficult result.
1616
5. **Answer.** Explain the basis in one or two sentences (population, grain, caveats), retaining useful tables or explanation before it. Finish with `**Answer:** <requested value(s), with units>`. If evidence or available tools are insufficient, finish with `**Answer:** unable to determine — <why>`. Never guess a missing result.
1717

@@ -139,9 +139,9 @@ Dialect references: [DuckDB query_table](https://www.duckdb.org/docs/current/gui
139139

140140
Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141141

142-
Inspect text, then define the complete population of stable IDs. When `text_label` is available, materialize that population in `scratch` with `sql_materialize`, then call `text_label` with SQL over scratch, the stable ID and text column names, a fixed label set with a description for each label, and any rubric/tie-breaking instructions. It classifies every row through the session's LLM in bounded batches and persists the results in scratch; do not substitute regex or keyword guesses for semantic labels. `text_label` accepts at most 30,000 rows per call. For a larger population, split it into disjoint, deterministic id ranges (one call per range, each into a distinct result table), combine the persisted result tables in SQL, and verify that combined coverage equals the population count before claiming results. Check `rows_in`, `labeled`, `failed` and `unparseable`, and retrieve failed IDs before claiming complete coverage. Optional spot-check agreement measures repeatability, not correctness. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
142+
Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling every row with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself; do not loop model calls over a large population. If the warehouse has its own AI classification functions and the user's configuration permits them, prefer those, since the warehouse runs and bills them under the user's controls. Otherwise, for an aggregate question over a large population, label a documented random sample and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
143143

144-
Use the persisted `text_label` output for SQL counts. It retains the source ID column name, so use `message_id` as the ID column for this example. To reuse the checks below, select `message_id`, `label` and your declared rubric version as `rubric_version` in a SELECT/CTE named `labels`; `population` must cover the same intended IDs. If `text_label` is unavailable, small audited annotation batches can use an available permitted ingestion path to form `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
144+
To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145145

146146
```sql
147147
WITH annotations(message_id, label, rubric_version) AS (
@@ -152,9 +152,9 @@ WITH annotations(message_id, label, rubric_version) AS (
152152
SELECT * FROM annotations;
153153
```
154154

155-
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. Use `text_label` for large labeling jobs; if it is unavailable and there is no permitted way to ingest complete returned annotation records, report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
155+
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. When the full population cannot be labeled within these limits, label a sample or report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
156156

157-
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
157+
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs (the sampled IDs for a sample) and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
158158

159159
```sql
160160
-- Must return no rows: includes duplicates across batches.
@@ -179,7 +179,7 @@ SELECT label, COUNT(*) AS messages
179179
FROM labels GROUP BY label ORDER BY label;
180180
```
181181

182-
Verify the label counts sum to the full population count. Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries.
182+
Verify the label counts sum to the `population` count (the sample size when you labeled a sample). Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries. For a sample, report proportions with the sample size and the size of the full population, and do not present scaled-up counts as exact.
183183

184184
## Reading spilled results
185185

‎REVIEW.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ Before flagging: trace the actual caller and the actual shipped surface. This re
4949

5050
11. **Test isolation leaks under parallel CI** (Warning). `mock.module`/dispatcher `reset()` leaked state across test files (52 CI failures fixed via #460). **Flag** new tests using global mocks/dispatchers without confirmed teardown under parallel `bun test`.
5151

52+
12. **Unbounded cost side effects** (Critical; always check, every PR). A feature whose spend grows with user data rather than with the conversation. `text_label` shipped on by default and labeled every row of a table with one session-model call per ~50-row batch: one evaluation day billed ~$680 for it, about 7x the agent sessions themselves, none of it visible in traces or session cost. On a million-row table that is ~20,000 model calls per question. **Flag when** a change can, without an explicit user decision: make model calls per row, batch, file or item (fan-out inside a tool, loops over query results, per-item summarization or classification); run unbounded or user-data-sized warehouse queries, scans or exports; start cloud resources; or add hidden model calls (sub-calls inside tools, background or automatic calls). **Require** all of: (a) off by default or bounded by a hard cap that does not scale with data size; (b) above the cap, an estimate shown and the user's confirmation required before the work runs; (c) the cheapest adequate model by default, with no cache breakpoint on one-off content; (d) its usage counted in session cost and traces. A PR that adds such a path without these is not mergeable; say so even when the feature itself is correct.
53+
5254
## Repo invariants & landmines
5355

5456
- Effect (`effect` v4 beta) is used pervasively; `Effect.promise`/`Effect.sync`-returning functions look like normal async functions but must be run via `AppRuntime.runPromise` (or equivalent) — a bare `await` silently no-ops the effect.

‎packages/opencode/AGENTS.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,19 @@
1212
- Capture the current TUI output with: `tmux capture-pane -pt opencode-dev`.
1313
- Stop the session explicitly when done: `tmux kill-session -t opencode-dev`.
1414

15+
# Cost side effects
16+
17+
Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR body in a "Cost side effects" section (the PR template's checklist asks for it), even when the answer is "none". Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
18+
19+
Such a path must be:
20+
21+
- Off by default, or bounded by a hard cap that does not scale with data size.
22+
- Above the cap, shown as a cost estimate and run only after the user confirms.
23+
- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
24+
- Counted in session cost and traces, so its spend is visible.
25+
26+
Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
27+
1528
# Module shape
1629

1730
Do not use `export namespace Foo { ... }` for module organization. It is not

‎packages/opencode/src/agent/agent.ts‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -281,9 +281,6 @@ export const layer = Layer.effect(
281281
tool_lookup: "allow",
282282
// Materialized scratch tables for multi-step analysis (tool added separately; key is harmless if absent)
283283
sql_materialize: "allow",
284-
// altimate_change start — model labels are persisted only in session-local scratch tables.
285-
text_label: "allow",
286-
// altimate_change end
287284
bash: "deny",
288285
} as const
289286
// Cap every configured ruleset at the subagent's read-only allow-list.

0 commit comments

Comments
 (0)