Skip to content

Commit 979ff37

Browse files
anandgupta42claude
andcommitted
feat: add text_label tool for bulk LLM labeling of scratch rows
Labels every row of a scratch query against a fixed label set in bounded, concurrent batches outside the model context, writes results to a scratch table, and reports coverage (rows_in / labeled / failed / unparseable) with failed ids retrievable. Optional agreement spot-check. Allowed for the `analyst`; `ALTIMATE_TEXT_LABEL=0` disables it. The `data-question` skill now points bulk labeling at `text_label`. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 6da61ce commit 979ff37

7 files changed

Lines changed: 1186 additions & 6 deletions

File tree

‎.opencode/skills/data-question/SKILL.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -139,9 +139,9 @@ Dialect references: [DuckDB query_table](https://www.duckdb.org/docs/current/gui
139139

140140
Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141141

142-
Inspect text, then define the complete population of stable IDs. If `task` and read-only `data-explorer` are available, assign disjoint ID ranges or catalog-defined batches with the same rubric/version. Each child returns exactly one `{message_id, label, rubric_version}` record per assigned ID, including `uncertain`; request records, not a guessed count. Otherwise label manageable batches directly using the same contract. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
142+
Inspect text, then define the complete population of stable IDs. When `text_label` is available, materialize that population in `scratch` with `sql_materialize`, then call `text_label` with SQL over scratch, the stable ID and text column names, a fixed label set with a description for each label, and any rubric/tie-breaking instructions. It classifies every row through the session's LLM in bounded batches and persists the results in scratch; do not substitute regex or keyword guesses for semantic labels. Check `rows_in`, `labeled`, `failed` and `unparseable`, and retrieve failed IDs before claiming complete coverage. Optional spot-check agreement measures repeatability, not correctness. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
143143

144-
Use an available permitted ingestion path to turn these records into a relation `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, small audited annotation batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
144+
Use the persisted `text_label` output for SQL counts. It retains the source ID column name, so use `message_id` as the ID column for this example. To reuse the checks below, select `message_id`, `label` and your declared rubric version as `rubric_version` in a SELECT/CTE named `labels`; `population` must cover the same intended IDs. If `text_label` is unavailable, small audited annotation batches can use an available permitted ingestion path to form `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145145

146146
```sql
147147
WITH annotations(message_id, label, rubric_version) AS (
@@ -152,7 +152,7 @@ WITH annotations(message_id, label, rubric_version) AS (
152152
SELECT * FROM annotations;
153153
```
154154

155-
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. For large jobs require a permitted way to ingest complete returned annotation records; `sql_materialize` alone does not imply an arbitrary file writer. If unavailable, report the coverage limitation. With no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
155+
These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. Use `text_label` for large labeling jobs; if it is unavailable and there is no permitted way to ingest complete returned annotation records, report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
156156

157157
**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
158158

‎packages/opencode/src/agent/agent.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,9 @@ export const layer = Layer.effect(
281281
tool_lookup: "allow",
282282
// Materialized scratch tables for multi-step analysis (tool added separately; key is harmless if absent)
283283
sql_materialize: "allow",
284+
// altimate_change start — model labels are persisted only in session-local scratch tables.
285+
text_label: "allow",
286+
// altimate_change end
284287
bash: "deny",
285288
} as const
286289
// Cap every configured ruleset at the subagent's read-only allow-list.

0 commit comments

Comments
 (0)