You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7b5dfcd
Browse filesBrowse the repository at this point in the historyBrowse files
fix: remove the text_label bulk labeling tool and add a cost side-effects rule (AltimateAI#1548)
* fix: remove the `text_label` bulk labeling tool and add a cost side-effects rule
- Remove `text_label` (tool, registry entry, `data-explorer` permission and
its tests). It made one session-model call per ~50-row batch, so its cost
grew with table size, and its calls were missing from traces and session
cost.
- `tool-schema-anthropic.test.ts`: assert no agent registers `text_label`;
keep the generic empty-sentinel schema check.
- `data-question` skill: count the population first; prefer
warehouse-native AI classification where permitted, otherwise label a
documented random sample and report an estimate.
- `REVIEW.md` focus area 12, `packages/opencode/AGENTS.md` and the PR
template require checking and flagging cost side effects that grow with
user data.
ClosesAltimateAI#1547
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: close cost gaps in labeling guidance and enforce the PR cost section
- `analyst.txt`: count the population first, do not fan labeling out across
model calls or `data-explorer` subagents, and sample with an estimate
unless the user confirms a full run.
- `data-question` skill: warehouse AI functions bill per row too; require
the row count and the user's confirmation before running them over a
large population.
- PR template: required `### Cost side effects` section; `pr-standards.yml`
flags it when missing or empty; `AGENTS.md` matches.
- `tool-schema-anthropic.test.ts`: prettier formatting.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: cap model labeling at 500 rows without user confirmation
The sample path in the `data-question` skill and `analyst.txt` had no size
limit. Labeling by any path (own reading, subagents, warehouse AI
functions) is now capped at 500 rows unless the user confirms more after
seeing the row count; larger populations get a documented sample of up to
500 rows reported as an estimate.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: apply the cost rule repo-wide and require every PR checklist box
- Move the cost side-effects rule from `packages/opencode/AGENTS.md` to a
new root `AGENTS.md` so agents editing skills or other packages get it.
- `pr-standards.yml`: require every checklist box (the template now has
three); anchor the cost-section match to real headings so nested
headings and prose mentions are handled.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: give the data-explorer subagent the labeling cap and size the job above it
- `data-explorer.txt`: label at most 500 rows by any path; it cannot ask
the user, so above that it samples and reports an estimate.
- `analyst.txt` and `data-question` skill: above the cap, tell the user the
job size (about one model call per 50 rows, or a per-row warehouse
charge) along with the row count before asking to confirm.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: require all three PR checklist boxes
A body that deleted the cost checkbox still passed with two checked boxes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix: keep the cost rule out of the forbidden root AGENTS.md
The upstream guard tests forbid a root `AGENTS.md`. The rule now lives in
`packages/opencode/AGENTS.md` and, repo-wide, in `CONTRIBUTING.md`;
`REVIEW.md` keeps the review bots checking every PR.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
_Does this change add model calls, warehouse queries or cloud resources whose cost grows with user data (rows, files, items)? Describe each one and how it is capped, estimated and confirmed, or write "None"._
24
+
25
+
<!-- altimate_change end -->
20
26
### Screenshots / recordings
21
27
22
28
_If this is a UI change, please include a screenshot or recording._
@@ -25,5 +31,8 @@ _If this is a UI change, please include a screenshot or recording._
25
31
26
32
-[ ] I have tested my changes locally
27
33
-[ ] I have not included unrelated changes in this PR
34
+
<!-- altimate_change start — cost side-effects check (see CONTRIBUTING.md) -->
35
+
-[ ] I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
36
+
<!-- altimate_change end -->
28
37
29
38
_If you do not follow this template your PR will be automatically rejected._
Copy file name to clipboardExpand all lines: .github/workflows/pr-standards.yml
+14-2Lines changed: 14 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -252,14 +252,26 @@ jobs:
252
252
}
253
253
}
254
254
255
+
// altimate_change start — every PR states its cost side effects, even when there are none
256
+
const costMatch = body.match(/^### Cost side effects[ \t]*\r?\n([\s\S]*?)(?=^#{1,3}[ \t]|(?![\s\S]))/m);
257
+
const costContent = costMatch
258
+
? costMatch[1].replace(/<!--[\s\S]*?-->/g, '').replace(/^_Does this change add[\s\S]*?_\s*$/m, '').trim()
259
+
: '';
260
+
if (!costContent) {
261
+
issues.push('"Cost side effects" section is missing or empty. Describe any model calls, queries or cloud resources whose cost grows with user data, or write "None".');
issues.push('Not all checklist items are checked. Please confirm you have tested locally and have not included unrelated changes.');
271
+
// altimate_change start — require all three checklist boxes; a deleted line must not pass
272
+
if (unchecked > 0 || checked < 3) {
273
+
issues.push('Not all checklist items are checked. Please confirm you have tested locally, have not included unrelated changes and have checked for cost side effects.');
Copy file name to clipboardExpand all lines: .opencode/skills/data-question/SKILL.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ Use the available read-only tools to answer the user's question. Source database
11
11
12
12
1.**Inspect.** Use `warehouse_list`, `schema_inspect` and small `sql_execute` queries as needed. Inspect actual keys and text before choosing joins, casts or parsing rules: schemas alone hide whitespace, case, prefixes and mixed formats. Preserve meaningful leading zeros; check that normalization does not merge distinct identities.
13
13
2.**Plan.** State the population, row grain, denominator, units and material assumptions. Decide which joins and aggregation levels preserve them. Keep this brief; a trivial lookup does not need a checklist or several validation scans.
14
-
3.**Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate. For exact final answers, aggregate the full intended population in SQL.
14
+
3.**Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate, or an exact answer would need per-row model labeling of a large population (see "Batch semantic labels, then count"); then say the answer is an estimate. For exact final answers, aggregate the full intended population in SQL.
15
15
4.**Verify relevant invariants.** Check key uniqueness, before/after join counts, missing matches and null share when they affect the answer. Use `sql_check` for these invariants and extraction coverage; for cross-warehouse comparisons, first materialize both sides into `scratch` with `sql_materialize`. Reconcile components with totals and preserve weighting. `SUM(group_mean * contributing_count) / SUM(contributing_count)` recovers the overall mean for disjoint, complete groups using unrounded means and counts of the same non-null measure; `AVG(group_mean)` gives each group equal weight. `sql_analyze` is optional. If available, a read-only `data-explorer` via `task` can independently re-derive a difficult result.
16
16
5.**Answer.** Explain the basis in one or two sentences (population, grain, caveats), retaining useful tables or explanation before it. Finish with `**Answer:** <requested value(s), with units>`. If evidence or available tools are insufficient, finish with `**Answer:** unable to determine — <why>`. Never guess a missing result.
Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141
141
142
-
Inspect text, then define the complete population of stable IDs. When `text_label` is available, materialize that population in `scratch` with `sql_materialize`, then call `text_label` with SQL over scratch, the stable ID and text column names, a fixed label set with a description for each label, and any rubric/tie-breaking instructions. It classifies every row through the session's LLM in bounded batches and persists the results in scratch; do not substitute regex or keyword guesses for semantic labels. `text_label` accepts at most 30,000 rows per call. For a larger population, split it into disjoint, deterministic id ranges (one call per range, each into a distinct result table), combine the persisted result tables in SQL, and verify that combined coverage equals the population count before claiming results. Check `rows_in`, `labeled`, `failed` and `unparseable`, and retrieve failed IDs before claiming complete coverage. Optional spot-check agreement measures repeatability, not correctness. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
142
+
Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself. Without the user's confirmation, label at most 500 rows in total by any path: your own reading, subagents, or warehouse AI functions (for example classification functions in Snowflake, Databricks or BigQuery), which are billed per row too. To label more, first tell the user the row count, how the labels would be produced and the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions), and wait for their confirmation. When the population is larger than 500 rows, by default label a documented random sample of up to 500 rows (roughly ±4.5 percentage points at 95% confidence for a proportion) and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
143
143
144
-
Use the persisted `text_label` output for SQL counts. It retains the source ID column name, so use `message_id` as the ID column for this example. To reuse the checks below, select `message_id`, `label` and your declared rubric version as `rubric_version` in a SELECT/CTE named `labels`; `population` must cover the same intended IDs. If `text_label` is unavailable, small audited annotation batches can use an available permitted ingestion path to form `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
144
+
To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145
145
146
146
```sql
147
147
WITH annotations(message_id, label, rubric_version) AS (
@@ -152,9 +152,9 @@ WITH annotations(message_id, label, rubric_version) AS (
152
152
SELECT*FROM annotations;
153
153
```
154
154
155
-
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. Use `text_label` for large labeling jobs; if it is unavailable and there is no permitted way to ingest complete returned annotation records, report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
155
+
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. When the full population cannot be labeled within these limits, label a sample or report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
156
156
157
-
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
157
+
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs (the sampled IDs for a sample) and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
158
158
159
159
```sql
160
160
-- Must return no rows: includes duplicates across batches.
@@ -179,7 +179,7 @@ SELECT label, COUNT(*) AS messages
179
179
FROM labels GROUP BY label ORDER BY label;
180
180
```
181
181
182
-
Verify the label counts sum to the full population count. Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries.
182
+
Verify the label counts sum to the `population` count (the sample size when you labeled a sample). Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries. For a sample, report proportions with the sample size and the size of the full population, and do not present scaled-up counts as exact.
Copy file name to clipboardExpand all lines: CONTRIBUTING.md
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -204,6 +204,19 @@ Some debug methods that can be problematic:
204
204
205
205
With that said, you may want to try these methods, as they might work for you.
206
206
207
+
## Cost side effects
208
+
209
+
Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR template's "Cost side effects" section, even when the answer is "None". The PR-standards check flags a missing or empty section on contributor PRs (it skips team members), and the review bots check every PR against `REVIEW.md` focus area 12. Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
210
+
211
+
Such a path must be:
212
+
213
+
- Off by default, or bounded by a hard cap that does not scale with data size.
214
+
- Above the cap, shown as a cost estimate and run only after the user confirms.
215
+
- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
216
+
- Counted in session cost and traces, so its spend is visible.
217
+
218
+
Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
219
+
207
220
## AI Contributions Policy
208
221
209
222
We welcome contributions from developers who use AI tools as part of their workflow. However, we require that all contributions reflect genuine understanding and effort from the contributor.
Copy file name to clipboardExpand all lines: REVIEW.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -49,6 +49,8 @@ Before flagging: trace the actual caller and the actual shipped surface. This re
49
49
50
50
11.**Test isolation leaks under parallel CI** (Warning). `mock.module`/dispatcher `reset()` leaked state across test files (52 CI failures fixed via #460). **Flag** new tests using global mocks/dispatchers without confirmed teardown under parallel `bun test`.
51
51
52
+
12. **Unbounded cost side effects** (Critical; always check, every PR). A feature whose spend grows with user data rather than with the conversation. `text_label` shipped on by default and labeled every row of a table with one session-model call per ~50-row batch: one evaluation day billed ~$680 for it, about 7x the agent sessions themselves, none of it visible in traces or session cost. On a million-row table that is ~20,000 model calls per question. **Flag when** a change can, without an explicit user decision: make model calls per row, batch, file or item (fan-out inside a tool, loops over query results, per-item summarization or classification); run unbounded or user-data-sized warehouse queries, scans or exports; start cloud resources; or add hidden model calls (sub-calls inside tools, background or automatic calls). **Require** all of: (a) off by default or bounded by a hard cap that does not scale with data size; (b) above the cap, an estimate shown and the user's confirmation required before the work runs; (c) the cheapest adequate model by default, with no cache breakpoint on one-off content; (d) its usage counted in session cost and traces. A PR that adds such a path without these is not mergeable; say so even when the feature itself is correct.
53
+
52
54
## Repo invariants & landmines
53
55
54
56
- Effect (`effect` v4 beta) is used pervasively; `Effect.promise`/`Effect.sync`-returning functions look like normal async functions but must be run via `AppRuntime.runPromise` (or equivalent) — a bare `await` silently no-ops the effect.
Copy file name to clipboardExpand all lines: packages/opencode/AGENTS.md
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,19 @@
12
12
- Capture the current TUI output with: `tmux capture-pane -pt opencode-dev`.
13
13
- Stop the session explicitly when done: `tmux kill-session -t opencode-dev`.
14
14
15
+
# Cost side effects
16
+
17
+
Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR template's "Cost side effects" section, even when the answer is "None". The PR-standards check flags a missing or empty section on contributor PRs (it skips team members), and the review bots check every PR against `REVIEW.md` focus area 12. Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
18
+
19
+
Such a path must be:
20
+
21
+
- Off by default, or bounded by a hard cap that does not scale with data size.
22
+
- Above the cap, shown as a cost estimate and run only after the user confirms.
23
+
- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
24
+
- Counted in session cost and traces, so its spend is visible.
25
+
26
+
Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
27
+
15
28
# Module shape
16
29
17
30
Do not use `export namespace Foo { ... }` for module organization. It is not
Copy file name to clipboardExpand all lines: packages/opencode/src/altimate/prompts/analyst.txt
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ Inspect connections, schemas and actual rows before joining or parsing; key form
4
4
5
5
Derive large intermediate sets in-engine. For a cross-engine join, move the smaller side: a few hundred keys go into an IN list or VALUES against the big table; materialize into `scratch` only the side that is small after filtering, never a multi-million-row table when a filter or aggregate could shrink it first. Preserve requested weighting: group means weighted by contributing counts can be exact; unweighted averages change the weighting. For many similar tables, prefer catalog-driven SQL (information_schema, DuckDB query_table/UNION ALL BY NAME) after confirming dialect support.
6
6
7
-
Treat inferred text labels as data: fixed rubric, stable IDs, full coverage and duplicate checks, then deterministic counting, never by eye. When available, `task` with read-only `data-explorer` can batch work or verify reasoning.
7
+
Treat inferred text labels as data: fixed rubric, stable IDs, coverage and duplicate checks, then deterministic counting, never by eye. Count the population first. Labeling grows in cost with the row count: without the user's confirmation, label at most 500 rows in total by any path (your own reading, `data-explorer` subagents or warehouse AI functions). For a larger population, label a documented random sample of up to 500 rows and report an estimate with its sample size; label more only after telling the user the row count, the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions) and getting their confirmation. When available, `task` with read-only `data-explorer` can verify reasoning.
8
8
9
9
## Cost-conscious exploration
10
10
Use selected columns, partition filters and LIMIT/sampling to inspect cheaply; estimates are not exact final computations. Read JSONL spill paths with `read`; spills contain returned rows, not necessarily all matches. Aggregate in SQL for exact answers. On timeout, revise the query. Report costs only from evidence.
0 commit comments