Skip to content

Commit 7b5dfcd

Browse files
anandgupta42claude
andauthored
fix: remove the text_label bulk labeling tool and add a cost side-effects rule (AltimateAI#1548)
* fix: remove the `text_label` bulk labeling tool and add a cost side-effects rule - Remove `text_label` (tool, registry entry, `data-explorer` permission and its tests). It made one session-model call per ~50-row batch, so its cost grew with table size, and its calls were missing from traces and session cost. - `tool-schema-anthropic.test.ts`: assert no agent registers `text_label`; keep the generic empty-sentinel schema check. - `data-question` skill: count the population first; prefer warehouse-native AI classification where permitted, otherwise label a documented random sample and report an estimate. - `REVIEW.md` focus area 12, `packages/opencode/AGENTS.md` and the PR template require checking and flagging cost side effects that grow with user data. Closes AltimateAI#1547 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: close cost gaps in labeling guidance and enforce the PR cost section - `analyst.txt`: count the population first, do not fan labeling out across model calls or `data-explorer` subagents, and sample with an estimate unless the user confirms a full run. - `data-question` skill: warehouse AI functions bill per row too; require the row count and the user's confirmation before running them over a large population. - PR template: required `### Cost side effects` section; `pr-standards.yml` flags it when missing or empty; `AGENTS.md` matches. - `tool-schema-anthropic.test.ts`: prettier formatting. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: cap model labeling at 500 rows without user confirmation The sample path in the `data-question` skill and `analyst.txt` had no size limit. Labeling by any path (own reading, subagents, warehouse AI functions) is now capped at 500 rows unless the user confirms more after seeing the row count; larger populations get a documented sample of up to 500 rows reported as an estimate. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: apply the cost rule repo-wide and require every PR checklist box - Move the cost side-effects rule from `packages/opencode/AGENTS.md` to a new root `AGENTS.md` so agents editing skills or other packages get it. - `pr-standards.yml`: require every checklist box (the template now has three); anchor the cost-section match to real headings so nested headings and prose mentions are handled. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: give the data-explorer subagent the labeling cap and size the job above it - `data-explorer.txt`: label at most 500 rows by any path; it cannot ask the user, so above that it samples and reports an estimate. - `analyst.txt` and `data-question` skill: above the cap, tell the user the job size (about one model call per 50 rows, or a per-row warehouse charge) along with the row count before asking to confirm. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: require all three PR checklist boxes A body that deleted the cost checkbox still passed with two checked boxes. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix: keep the cost rule out of the forbidden root AGENTS.md The upstream guard tests forbid a root `AGENTS.md`. The rule now lives in `packages/opencode/AGENTS.md` and, repo-wide, in `CONTRIBUTING.md`; `REVIEW.md` keeps the review bots checking every PR. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent de3d05a commit 7b5dfcd

14 files changed

Lines changed: 64 additions & 1249 deletions

File tree

‎.github/pull_request_template.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@ Please provide a description of the issue, the changes you made to fix it, and w
1717

1818
### How did you verify your code works?
1919

20+
<!-- altimate_change start — required cost side-effects section (see CONTRIBUTING.md) -->
21+
### Cost side effects
22+
23+
_Does this change add model calls, warehouse queries or cloud resources whose cost grows with user data (rows, files, items)? Describe each one and how it is capped, estimated and confirmed, or write "None"._
24+
25+
<!-- altimate_change end -->
2026
### Screenshots / recordings
2127

2228
_If this is a UI change, please include a screenshot or recording._
@@ -25,5 +31,8 @@ _If this is a UI change, please include a screenshot or recording._
2531

2632
- [ ] I have tested my changes locally
2733
- [ ] I have not included unrelated changes in this PR
34+
<!-- altimate_change start — cost side-effects check (see CONTRIBUTING.md) -->
35+
- [ ] I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
36+
<!-- altimate_change end -->
2837

2938
_If you do not follow this template your PR will be automatically rejected._

‎.github/workflows/pr-standards.yml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -252,14 +252,26 @@ jobs:
252252
}
253253
}
254254
255+
// altimate_change start — every PR states its cost side effects, even when there are none
256+
const costMatch = body.match(/^### Cost side effects[ \t]*\r?\n([\s\S]*?)(?=^#{1,3}[ \t]|(?![\s\S]))/m);
257+
const costContent = costMatch
258+
? costMatch[1].replace(/<!--[\s\S]*?-->/g, '').replace(/^_Does this change add[\s\S]*?_\s*$/m, '').trim()
259+
: '';
260+
if (!costContent) {
261+
issues.push('"Cost side effects" section is missing or empty. Describe any model calls, queries or cloud resources whose cost grows with user data, or write "None".');
262+
}
263+
// altimate_change end
264+
255265
// Check: checklist boxes are checked
256266
if (hasChecklistSection) {
257267
const checklistMatch = body.match(/### Checklist\s*\n([\s\S]*?)(?=###|$)/);
258268
const checklistContent = checklistMatch ? checklistMatch[1] : '';
259269
const unchecked = (checklistContent.match(/- \[ \]/g) || []).length;
260270
const checked = (checklistContent.match(/- \[x\]/gi) || []).length;
261-
if (checked < 2) {
262-
issues.push('Not all checklist items are checked. Please confirm you have tested locally and have not included unrelated changes.');
271+
// altimate_change start — require all three checklist boxes; a deleted line must not pass
272+
if (unchecked > 0 || checked < 3) {
273+
issues.push('Not all checklist items are checked. Please confirm you have tested locally, have not included unrelated changes and have checked for cost side effects.');
274+
// altimate_change end
263275
}
264276
}
265277

‎.opencode/skills/data-question/SKILL.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Use the available read-only tools to answer the user's question. Source database
1111

1212
1. **Inspect.** Use `warehouse_list`, `schema_inspect` and small `sql_execute` queries as needed. Inspect actual keys and text before choosing joins, casts or parsing rules: schemas alone hide whitespace, case, prefixes and mixed formats. Preserve meaningful leading zeros; check that normalization does not merge distinct identities.
1313
2. **Plan.** State the population, row grain, denominator, units and material assumptions. Decide which joins and aggregation levels preserve them. Keep this brief; a trivial lookup does not need a checklist or several validation scans.
14-
3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate. For exact final answers, aggregate the full intended population in SQL.
14+
3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate, or an exact answer would need per-row model labeling of a large population (see "Batch semantic labels, then count"); then say the answer is an estimate. For exact final answers, aggregate the full intended population in SQL.
1515
4. **Verify relevant invariants.** Check key uniqueness, before/after join counts, missing matches and null share when they affect the answer. Use `sql_check` for these invariants and extraction coverage; for cross-warehouse comparisons, first materialize both sides into `scratch` with `sql_materialize`. Reconcile components with totals and preserve weighting. `SUM(group_mean * contributing_count) / SUM(contributing_count)` recovers the overall mean for disjoint, complete groups using unrounded means and counts of the same non-null measure; `AVG(group_mean)` gives each group equal weight. `sql_analyze` is optional. If available, a read-only `data-explorer` via `task` can independently re-derive a difficult result.
1616
5. **Answer.** Explain the basis in one or two sentences (population, grain, caveats), retaining useful tables or explanation before it. Finish with `**Answer:** <requested value(s), with units>`. If evidence or available tools are insufficient, finish with `**Answer:** unable to determine — <why>`. Never guess a missing result.
1717

@@ -139,9 +139,9 @@ Dialect references: [DuckDB query_table](https://www.duckdb.org/docs/current/gui
139139

140140
Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141141

142-
Inspect text, then define the complete population of stable IDs. When `text_label` is available, materialize that population in `scratch` with `sql_materialize`, then call `text_label` with SQL over scratch, the stable ID and text column names, a fixed label set with a description for each label, and any rubric/tie-breaking instructions. It classifies every row through the session's LLM in bounded batches and persists the results in scratch; do not substitute regex or keyword guesses for semantic labels. `text_label` accepts at most 30,000 rows per call. For a larger population, split it into disjoint, deterministic id ranges (one call per range, each into a distinct result table), combine the persisted result tables in SQL, and verify that combined coverage equals the population count before claiming results. Check `rows_in`, `labeled`, `failed` and `unparseable`, and retrieve failed IDs before claiming complete coverage. Optional spot-check agreement measures repeatability, not correctness. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
142+
Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself. Without the user's confirmation, label at most 500 rows in total by any path: your own reading, subagents, or warehouse AI functions (for example classification functions in Snowflake, Databricks or BigQuery), which are billed per row too. To label more, first tell the user the row count, how the labels would be produced and the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions), and wait for their confirmation. When the population is larger than 500 rows, by default label a documented random sample of up to 500 rows (roughly ±4.5 percentage points at 95% confidence for a proportion) and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
143143

144-
Use the persisted `text_label` output for SQL counts. It retains the source ID column name, so use `message_id` as the ID column for this example. To reuse the checks below, select `message_id`, `label` and your declared rubric version as `rubric_version` in a SELECT/CTE named `labels`; `population` must cover the same intended IDs. If `text_label` is unavailable, small audited annotation batches can use an available permitted ingestion path to form `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
144+
To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145145

146146
```sql
147147
WITH annotations(message_id, label, rubric_version) AS (
@@ -152,9 +152,9 @@ WITH annotations(message_id, label, rubric_version) AS (
152152
SELECT * FROM annotations;
153153
```
154154

155-
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. Use `text_label` for large labeling jobs; if it is unavailable and there is no permitted way to ingest complete returned annotation records, report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
155+
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. When the full population cannot be labeled within these limits, label a sample or report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
156156

157-
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
157+
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs (the sampled IDs for a sample) and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
158158

159159
```sql
160160
-- Must return no rows: includes duplicates across batches.
@@ -179,7 +179,7 @@ SELECT label, COUNT(*) AS messages
179179
FROM labels GROUP BY label ORDER BY label;
180180
```
181181

182-
Verify the label counts sum to the full population count. Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries.
182+
Verify the label counts sum to the `population` count (the sample size when you labeled a sample). Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries. For a sample, report proportions with the sample size and the size of the full population, and do not present scaled-up counts as exact.
183183

184184
## Reading spilled results
185185

‎CONTRIBUTING.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,19 @@ Some debug methods that can be problematic:
204204

205205
With that said, you may want to try these methods, as they might work for you.
206206

207+
## Cost side effects
208+
209+
Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR template's "Cost side effects" section, even when the answer is "None". The PR-standards check flags a missing or empty section on contributor PRs (it skips team members), and the review bots check every PR against `REVIEW.md` focus area 12. Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
210+
211+
Such a path must be:
212+
213+
- Off by default, or bounded by a hard cap that does not scale with data size.
214+
- Above the cap, shown as a cost estimate and run only after the user confirms.
215+
- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
216+
- Counted in session cost and traces, so its spend is visible.
217+
218+
Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
219+
207220
## AI Contributions Policy
208221

209222
We welcome contributions from developers who use AI tools as part of their workflow. However, we require that all contributions reflect genuine understanding and effort from the contributor.

‎REVIEW.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ Before flagging: trace the actual caller and the actual shipped surface. This re
4949

5050
11. **Test isolation leaks under parallel CI** (Warning). `mock.module`/dispatcher `reset()` leaked state across test files (52 CI failures fixed via #460). **Flag** new tests using global mocks/dispatchers without confirmed teardown under parallel `bun test`.
5151

52+
12. **Unbounded cost side effects** (Critical; always check, every PR). A feature whose spend grows with user data rather than with the conversation. `text_label` shipped on by default and labeled every row of a table with one session-model call per ~50-row batch: one evaluation day billed ~$680 for it, about 7x the agent sessions themselves, none of it visible in traces or session cost. On a million-row table that is ~20,000 model calls per question. **Flag when** a change can, without an explicit user decision: make model calls per row, batch, file or item (fan-out inside a tool, loops over query results, per-item summarization or classification); run unbounded or user-data-sized warehouse queries, scans or exports; start cloud resources; or add hidden model calls (sub-calls inside tools, background or automatic calls). **Require** all of: (a) off by default or bounded by a hard cap that does not scale with data size; (b) above the cap, an estimate shown and the user's confirmation required before the work runs; (c) the cheapest adequate model by default, with no cache breakpoint on one-off content; (d) its usage counted in session cost and traces. A PR that adds such a path without these is not mergeable; say so even when the feature itself is correct.
53+
5254
## Repo invariants & landmines
5355

5456
- Effect (`effect` v4 beta) is used pervasively; `Effect.promise`/`Effect.sync`-returning functions look like normal async functions but must be run via `AppRuntime.runPromise` (or equivalent) — a bare `await` silently no-ops the effect.

‎packages/opencode/AGENTS.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,19 @@
1212
- Capture the current TUI output with: `tmux capture-pane -pt opencode-dev`.
1313
- Stop the session explicitly when done: `tmux kill-session -t opencode-dev`.
1414

15+
# Cost side effects
16+
17+
Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR template's "Cost side effects" section, even when the answer is "None". The PR-standards check flags a missing or empty section on contributor PRs (it skips team members), and the review bots check every PR against `REVIEW.md` focus area 12. Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
18+
19+
Such a path must be:
20+
21+
- Off by default, or bounded by a hard cap that does not scale with data size.
22+
- Above the cap, shown as a cost estimate and run only after the user confirms.
23+
- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
24+
- Counted in session cost and traces, so its spend is visible.
25+
26+
Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
27+
1528
# Module shape
1629

1730
Do not use `export namespace Foo { ... }` for module organization. It is not

‎packages/opencode/src/agent/agent.ts‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -281,9 +281,6 @@ export const layer = Layer.effect(
281281
tool_lookup: "allow",
282282
// Materialized scratch tables for multi-step analysis (tool added separately; key is harmless if absent)
283283
sql_materialize: "allow",
284-
// altimate_change start — model labels are persisted only in session-local scratch tables.
285-
text_label: "allow",
286-
// altimate_change end
287284
bash: "deny",
288285
} as const
289286
// Cap every configured ruleset at the subagent's read-only allow-list.

‎packages/opencode/src/altimate/prompts/analyst.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Inspect connections, schemas and actual rows before joining or parsing; key form
44

55
Derive large intermediate sets in-engine. For a cross-engine join, move the smaller side: a few hundred keys go into an IN list or VALUES against the big table; materialize into `scratch` only the side that is small after filtering, never a multi-million-row table when a filter or aggregate could shrink it first. Preserve requested weighting: group means weighted by contributing counts can be exact; unweighted averages change the weighting. For many similar tables, prefer catalog-driven SQL (information_schema, DuckDB query_table/UNION ALL BY NAME) after confirming dialect support.
66

7-
Treat inferred text labels as data: fixed rubric, stable IDs, full coverage and duplicate checks, then deterministic counting, never by eye. When available, `task` with read-only `data-explorer` can batch work or verify reasoning.
7+
Treat inferred text labels as data: fixed rubric, stable IDs, coverage and duplicate checks, then deterministic counting, never by eye. Count the population first. Labeling grows in cost with the row count: without the user's confirmation, label at most 500 rows in total by any path (your own reading, `data-explorer` subagents or warehouse AI functions). For a larger population, label a documented random sample of up to 500 rows and report an estimate with its sample size; label more only after telling the user the row count, the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions) and getting their confirmation. When available, `task` with read-only `data-explorer` can verify reasoning.
88

99
## Cost-conscious exploration
1010
Use selected columns, partition filters and LIMIT/sampling to inspect cheaply; estimates are not exact final computations. Read JSONL spill paths with `read`; spills contain returned rows, not necessarily all matches. Aggregate in SQL for exact answers. On timeout, revise the query. Report costs only from evidence.

0 commit comments

Comments
 (0)