Skip to content

Commit f076c06

Browse files
lloydclaude
andauthored
buzz-acp: the task extractor's core, and an instrument that can fail (#85)
* buzz-acp: the task extractor's core, and an instrument that can fail `scripts/task-extractor/` (#74) shipped the gate and the corpus. Nothing consumed them. This is the extraction itself: the prompt, the model call, the parse, the validation, and a replay that scores it offline. Not wired in yet. Publication and the `lib.rs` hook are the next commit; this is the half that has to clear the gate before any of it is worth running against a live relay. What the corpus changed, measured rather than assumed: **The harness could not tell a failure from a verdict.** `extract()` fails open, so a timeout, a truncation and a content filter all resolve to an empty list — which is exactly what "there is no work here" looks like. The first corpus replay scored 5/11 and four of those `none`s were calls the model never answered. `Extraction::error` makes the two readable apart, and the replay refuses to exit 0 when any row errored. Every finding below was invisible until that existed. **A thinking model spends `max_tokens` before it writes.** At 1600, 5 of 11 replies came back truncated mid-object. `parse_tasks` could only call that malformed, which sends the reader after a model bug instead of a budget; the call site now reads `finish_reason` and names the truncation. Default budget 8000, and a retry doubles it. **Retries are not politeness.** Three consecutive replays failed on `6bfd1e01`, then `17ced753`, then `46c1401d` — a different utterance every time, at one or two calls in eleven. A single-shot extractor drops about one ask in eight to transients that correlate with nothing. And a retry must take a *different* path: Gemini's `RECITATION` filter is a property of the sampled continuation, so three identical greedy calls are blocked three identical times. The first ask is greedy; a retry is not. **`reasoning_effort: low` is a trap.** It cuts thinking ~5x and stops the truncations, which looks like a pure win. Decomposition *is* the thinking: at `low` the model returned one task for a message asking four, and mistook two board-status questions for new work. Unset, same prompt, same retries: four tasks, and the attach cases land. Truncation is the retry's job, not the budget's. **The prompt's order mattered more than its content.** The first version opened with the board-flooding argument and five vivid examples of the `none` case. Same model, same corpus: 5/11 with four false `none`s, including a message that plainly asks for three things — which the same model, given two lines, got right. The gate fails a false `none` and forgives an overcount, so the asymmetry now leads. **An enumeration the model has to write is one it has to do.** Telling it to list the asks first did nothing: `6bfd1e01` came back 1, 1, none, 3, none across five replays. Making `asks` a required output field took it to 3, 3, 3. The harness never publishes that field. **`attach` is undecidable without a board**, and the corpus ships labels, not one. "where are we with assessing spark 1.3?" attaches and "where are we at in all our initiatives?" does not; nothing in the text separates them. So the replay walks the window in order and grows the board as it goes — the board at utterance N is what 1..N-1 created, which is what the live path does. It also means a miss cascades: the window's first message is what puts the Spark task on the board for a question nineteen hours later. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp(task-extract): the decompose pass, measured and switched off rock asked for a decompose-only first pass to fix the one utterance that will not hold still (`6bfd1e01`, three asks in three clauses, 4 of 7 real judgements where every other row sits at 7 or 8 of 8). Built it, measured it, and it is worse. Keeping it behind `two_pass`, default off, because the measurement is worth more than the deletion and a different model may not behave this way. Pass one is genuinely good. Asked only to enumerate, the model returns `9aee0484`'s four asks verbatim and identically on repeat trials, and five for `6bfd1e01` — an over-split the gate forgives. Pass two is what breaks. Handed a list and told to emit one entry per ask, it stops deciding: * `2279229f` says outright "I know we already are running a comparison. Can we just add in Luna to that?" — 8/8 `attach` in one call, 2/5 split, because three enumerated asks read as three things to create. * `63e0c989`, a status sweep that is 8/8 `none`, attached to a board item. * The utterance this existed to fix did not move: 0 of 4. Eight replays against six, transport failures excluded from both denominators: 81/87 (93%) in one call, 49/63 (78%) split. The lesson is not that two passes are bad. It is that an enumeration handed over as input becomes an instruction to emit that many tasks, and that is the wrong contract for a classifier whose three answers include "attach to something that already exists" and "nothing". Also here, and worth more than the experiment: I read those numbers wrong the first time. My comparison table scored a `none` row as a model verdict when the replay had already reported it as a failed call — the exact collapse `Extraction::error` was added to prevent, committed by the analysis instead of the harness. Three of the split's rows were network errors. Excluding them from the denominator is what turned "the split is catastrophic" into "the split is worse", and only one of those two statements is true. The HTTP post is factored out of `ask` so both passes name their failures identically, which is why the errors were visible at all. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp(task-extract): the anti-merge rule, measured and reverted rock's option 1: tell the classifier that splitting two asks is cheap and merging them loses work, without forcing an exact count. The premise was sound and I had the evidence for it — across seven non-errored replays the model enumerated `6bfd1e01`'s three asks CORRECTLY every time and then emitted one or two tasks. The enumeration was never the defect. The ask-to-task merge was. So the rule was pushed as hard as it goes: count `asks`, count `tasks`, THESE ARE THE SAME NUMBER, never combine two entries from `asks` — not the same project, not the same file, not the same person, not one sounding like a step of the other. It did exactly what it was written to do on the target, and broke the utterance that was working. 6bfd1e01 create/3 4/8 -> reached 3 or 4 tasks in 4 of 7 9aee0484 create/4 7/8 -> 0/7, ONE task every single run `9aee0484` collapsed deterministically: four asks enumerated correctly, one task emitted, eight runs out of eight. `17ced753` followed it from 7/8 to 0/7, but that is a cascade rather than a second failure — with `9aee0484` producing one task instead of four, the Spark task is not on the board for a question nineteen hours later to attach to. 60/82 (73%) against 81/87 (93%), and the gate from 3 of 8 to 0 of 8. Reverted; the measurement is in the test that binds the softer clause, so the next person to reach for the harder one finds the number first. A method note, because it nearly cost me the conclusion. Midway through I probed the v6 prompt by hand against `9aee0484` and got a clean four-task answer with the blocked-by link correct — flatly contradicting the replay. One probe is not a measurement: the sweep says that reply was a lucky draw and the collapse is real at 0 of 7. I have now made the same mistake twice in one session in the same shape, once with a 0/3 I called deterministic and once here. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp(task-extract): bind the count in the schema, per wren's review wren read the saved replays and the premise I had been building on was wrong. Under the anti-merge prompt the model did not merge four asks into one task: it emitted the FIRST ask alone, `doneWhen` absent, and closed the array — seven of seven runs, `asks` correct at 4 every time, `dropped` empty every time. Early termination, not merging. Verified before implementing: `doneWhen` absent went from 30% of `create` records to 85% under that prompt, and two rows repeated a subject verbatim, which a verdict counting array entries reads as decomposition. Prose asking the model to re-check its own output has nowhere to run under `response_format`, so the check happens in thinking and the visible JSON gets terser. So the rule moves into the schema. One array of per-ask records, each carrying its own disposition, with `oneOf` per disposition pinning what that disposition must supply. There is no second array whose `]` is a fresh decision, and `none` survives as a per-ask verdict rather than being deleted by an instruction to emit one task per ask. It works, and it costs something. Four sweeps of eight replays: design accuracy attach gate two arrays + prose count rule 93% 23/24 3/8 per-ask records 74% 3/24 8/8 + attach-first prose 77% 6/24 7/8 + a required "why not attach" 76% 3/24 6/8 Decomposition is solved. The two utterances that held the gate at 3 of 8 — `9aee0484` (four asks) and `6bfd1e01` (three) — went to 7-8 of 8 and stayed. `attach` broke, and three attempts to talk it back recovered at most a quarter. A create where an attach belonged is a duplicate, which is the board-flooding the design note warns about, so this is a real trade and it is the open question on the module. The untried lever is schema-level like everything else that worked here: `attachTo` as an enum of the actual open-task ids. Shipping the middle row. Confirmed at this head over four more replays: 3 of 4 gate, 8-9 of 11 correct, zero transport failures. Instrument fixes from §4 of the review, all of which earned themselves: * Raw reply and `finish_reason` saved per row. Two faults were invisible without them and obvious with them: a repetition loop in an unbounded `why` ("properly cleanly fast well nicely reliably" for hundreds of tokens) that ate the budget so no `doneWhen` was ever written, and `attach` emitted with no `attachTo` at all on all three attach rows. * The raw reply survives the ERROR path too. The first version lost it to `?`, and the one row where the bytes name the bug instantly is the row that failed to parse — which is exactly what happened: `no_open_task_covers` camel-cases to `noneOpenTaskCovers`, one character off the schema's `noOpenTaskCovers`, and every call on the corpus failed. * `deny_unknown_fields` and `additionalProperties: false`. Work nested under a key the harness does not know is now a retry, not a quiet undercount — and it is what caught the field-name bug above. * Bounded free text, so a decoder that starts repeating cannot spend the whole budget doing it. * `create` counted by distinct normalised subject, with duplicates reported separately. * A create with no observable `doneWhen` is a defect row the eval prints. `--strict-done-when` makes it fail the gate; off by default, so the gate's rule does not move under a measurement in flight. * Errored rows are out of the accuracy denominator and named. They still fail the gate: in production a lost extraction is lost work. * A dropped record is retried, like a transport error, because a dropped record is a lost ask. * `--board-snapshot` pins the board across runs, and `--temperature` reaches the wire. A sweep at 0.0 is one draw plus endpoint noise, not N samples; comparing two prompts needs both. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp(task-extract): attachTo is an enum of the board, and that is not the fix rock's go on the schema lever for `attach`. Built, swept eight replays, and it does not recover it: 4 of 24 against 6 of 24 for prose alone, with `2279229f` at 0 of 8 even though that message says outright "I know we already are running a comparison. Can we just add in Luna to that?". Four interventions on the same corpus now: design accuracy attach decomposition gate two arrays + prose count rule 93% 23/24 11/15 3/8 per-ask records 74% 3/24 14/16 8/8 + attach-first prose 77% 6/24 14/16 7/8 + attachTo enum (this) 75% 4/24 14/16 8/8 The constraint was never the barrier. Asked about one isolated ask, the model does not recognise "where are we with spark 1.3?" as being about a board row someone else titled "Assess Spark 1.3 against Flash", and making the right answer mechanically free does not make it visible. Shipping it anyway, on gate and on correctness rather than on accuracy. The gate is 8 of 8, the best of the four and the contract rock set; decomposition holds at 14/16; and a dangling `attachTo` becomes structurally impossible instead of rejected after the fact, which is worth having whatever the score does. An empty board offers no attach shape at all — an enum with no members is not a legal schema, and an attach the harness would have to drop is worse than one the model could never proposeo. What is left to try is a second question rather than a better constraint: keep this schema for decomposition, then ask once per proposed `create` whether any open task already covers that subject, with only the subject and the board in front of it. That is the question the per-ask framing cannot hold, and it is one extra call per create. Not built; rock's call. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp(task-extract): ask the board question on its own, and it answers rock's option 1. The per-ask schema decomposes well and matches the board badly, and four attempts to fix the second inside the first all failed — a board-first rule, worked examples, a required field justifying the create, and finally making `attachTo` an enum so that attaching was mechanically free. `attach` never left the floor. So it is asked separately: one candidate subject, the board, and nothing else. No asymmetry paragraph, no create/attach/none judgement, no doneWhen — a two-item comparison instead of a judgement smuggled into a classification. One call per proposed `create`, run concurrently, and only when the board is non-empty. Five designs on the same corpus, eight replays each: design accuracy attach decomp gate two arrays + prose count rule 93% 23/24 11/15 3/8 per-ask records 74% 3/24 14/16 8/8 + attach-first prose 77% 6/24 14/16 7/8 + attachTo enum 75% 4/24 14/16 8/8 + board-match pass 94% 20/24 15/16 7/8 Best on every axis. `17ced753` 2/8 -> 8/8. `2279229f` — the message that says outright "I know we already are running a comparison, can we just add in Luna to that?" — 0/8 -> 7/8. Accuracy past the two-array design it replaced, which never got the gate above 3 of 8. The gate reads 7 of 8 against that design's 8 of 8, and the one failure is `6bfd1e01` emitting one task for three: the known decomposition residual, not the match pass over-reaching. At n=8 one run is not a difference I would defend; 94% against 75% is. Fails soft in the safe direction. A match call that errors leaves its create alone, because a duplicate is closeable and a lost task is the failure this module exists to remove. A match naming an id that is not on the board is ignored. Two creates matching the same board item collapse to one link. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp: wire the task extractor behind BUZZ_ACP_TASK_EXTRACT_AUTHORS The hook sits **after the author gate and before the subscription rules**, which is the one placement the corpus forces: zero of the owner's eleven readable utterances carry a `p` tag — he addresses people in prose and posts top level — so a seat that is not woken by a message still has to capture the work in it. Downstream of the rules would extract only from messages that already reached the model. Spawned, not awaited. The extraction's only output is relay writes and nothing in the turn depends on it, so blocking would put a model call in front of every matching message's time-to-first-response for no gain. A detached task survives the failure §5 actually cares about — the seat's own model turn dying on budget exhaustion — because it is not downstream of that turn. Off unless BOTH an author list and an endpoint are configured. Either alone is a misconfiguration that would otherwise announce itself once per message or not at all, so it is one warn line at startup instead. A malformed pubkey is refused at startup rather than extracting nothing quietly: "the extractor is silent" and "nobody asked for anything" look identical from outside. `task_board_source` reads the board the extractor decides against. Deliberately a reduced read — `buzz tasks board` resolves assignment, blocked-by links, thread activity and the Blocked/In Progress/Up Next ordering at one extra query per linked thread, and the extractor wants one thing from all of it: which work is open and what it is called. Two queries, no per-issue round trips. Two reductions worth their tests: an issue with no status event is OPEN, because `buzz issues create` publishes none and treating "no status" as "not open" would hide every task nobody has touched — which is most of them, and exactly the ones a duplicate gets created for. And the newest status wins by (created_at, id), so a reopened task is open and two status events in one second do not make the board depend on relay ordering. A board that cannot be read **skips the message** rather than extracting against an empty one: an empty board turns every `attach` into a `create`, which is the duplicate the whole second pass exists to prevent. Absent is honest; empty is a lie with a cost. `--task-extract-publish` is off, and that is the rollout rather than an omission. The extractor makes judgements, and the cheapest way to learn whether it makes them well on real traffic is to let it decide in the open — one log line per create and attach, with the ask, the subject and the done-when — without writing anything anyone has to close. Publication is the next commit; setting the flag today says so once per message rather than dropping the work silently. 1101 + 9 pass, clippy clean. Corpus unchanged at this head: 9/11, gate passed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * buzz-acp(task-extract): rustfmt, and take the key the seat already has Two fixes on the hook. CI caught the first: I had been hand-writing this module and never ran `cargo fmt`. The Rust Lint job is a format check before it is a clippy run, and it was right. The second is a design correction I should have made before asking rock to stage anything. `deploy/buzz-agent-run` already fetches `gemini-live-api-key` from Secret Manager at seat start and exports it as `GEMINI_API_KEY` for any seat whose adapter is `rebrand-acp --provider gemini` — which is the seat this runs on first. Its own comment says secrets are "fetched at start rather than stored in an EnvironmentFile". Asking for `BUZZ_ACP_TASK_EXTRACT_API_KEY` would have meant writing a second copy of the same key for the same model into `rock.env`, to avoid touching a launcher this crate does not own. So the extractor falls back to `GEMINI_API_KEY`, and the staged config needs no secret at all. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent bb545ba commit f076c06

5 files changed

Lines changed: 3259 additions & 0 deletions

File tree

Lines changed: 364 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,364 @@
1+
//! Replay the labelled corpus through the real extractor and write a
2+
//! predictions file `scripts/task-extractor/task-extractor-eval.py` can score.
3+
//!
4+
//! ```text
5+
//! scripts/task-extractor/task-extractor-eval.py --fetch > utterances.json
6+
//!
7+
//! # Track 1 — "will production pass?": greedy, board built as it goes.
8+
//! cargo run -p buzz-acp --example task-extract-replay -- \
9+
//! --utterances utterances.json --out predictions.json --detail detail.json
10+
//!
11+
//! # Track 2 — "is prompt B better than A?": sampled, and the board held still.
12+
//! cargo run -p buzz-acp --example task-extract-replay -- \
13+
//! --utterances utterances.json --write-board-snapshot board.json --out /dev/null
14+
//! cargo run -p buzz-acp --example task-extract-replay -- \
15+
//! --utterances utterances.json --board-snapshot board.json \
16+
//! --temperature 0.7 --out predictions.json
17+
//! ```
18+
//!
19+
//! # Two tracks, because they answer different questions
20+
//!
21+
//! **Greedy is one sample.** Eight replays at `temperature: 0` against a
22+
//! byte-identical prompt are one draw plus endpoint noise, not eight
23+
//! observations — they tell you the mode and nothing about the spread around
24+
//! it. That is the right instrument for "will the fleet pass the gate", and the
25+
//! wrong one for "is this prompt better". A prompt comparison needs
26+
//! `--temperature 0.7` and k ≥ 8.
27+
//!
28+
//! **And the board is an input.** The corpus ships labels, not a board, and
29+
//! `attach` is undecidable without one: "where are we with assessing spark
30+
//! 1.3?" attaches, "where are we at in all our initiatives?" does not, and
31+
//! nothing in the text separates them. So the default replay walks the window
32+
//! in order and grows the board as it goes — the board at utterance N is what
33+
//! 1..N-1 created, which is what the live path does.
34+
//!
35+
//! That fidelity has a cost for comparison: a miss **cascades**, so the board a
36+
//! later utterance sees varies run to run, and some of what looks like model
37+
//! variance is input variance. `--board-snapshot` pins it. Produce one with
38+
//! `--write-board-snapshot` from a sequential run, then hold it fixed across
39+
//! both arms.
40+
//!
41+
//! The synthetic issue ids are derived from the source event id so a prediction
42+
//! can be traced back to the utterance that created its attach target.
43+
44+
use std::collections::HashMap;
45+
use std::collections::HashSet;
46+
use std::path::PathBuf;
47+
48+
use buzz_acp::task_extract::{
49+
BoardTask, ExtractInput, SourceMessage, TaskAction, TaskExtractConfig, TaskExtractor, Verdict,
50+
};
51+
52+
#[derive(serde::Deserialize)]
53+
struct Utterance {
54+
id: String,
55+
#[serde(default)]
56+
channel: String,
57+
#[serde(default)]
58+
text: String,
59+
#[serde(default, rename = "eventId")]
60+
event_id: String,
61+
}
62+
63+
struct Args {
64+
utterances: PathBuf,
65+
out: PathBuf,
66+
detail: Option<PathBuf>,
67+
board_snapshot: Option<PathBuf>,
68+
write_board_snapshot: Option<PathBuf>,
69+
endpoint: String,
70+
model: String,
71+
reasoning_effort: Option<String>,
72+
temperature: f64,
73+
no_board_match: bool,
74+
}
75+
76+
fn parse_args() -> Args {
77+
let mut utterances = None;
78+
let mut out = None;
79+
let mut detail = None;
80+
let mut board_snapshot = None;
81+
let mut write_board_snapshot = None;
82+
let mut endpoint = "https://generativelanguage.googleapis.com/v1beta/openai".to_string();
83+
let mut model = "gemini-3.8-flash".to_string();
84+
let mut reasoning_effort: Option<String> = None;
85+
let mut temperature = 0.0;
86+
let mut no_board_match = false;
87+
let mut it = std::env::args().skip(1);
88+
while let Some(flag) = it.next() {
89+
match flag.as_str() {
90+
"--utterances" => utterances = it.next().map(PathBuf::from),
91+
"--out" => out = it.next().map(PathBuf::from),
92+
"--detail" => detail = it.next().map(PathBuf::from),
93+
"--board-snapshot" => board_snapshot = it.next().map(PathBuf::from),
94+
"--write-board-snapshot" => write_board_snapshot = it.next().map(PathBuf::from),
95+
"--endpoint" => endpoint = it.next().unwrap_or(endpoint),
96+
"--model" => model = it.next().unwrap_or(model),
97+
"--temperature" => {
98+
temperature = it
99+
.next()
100+
.and_then(|v| v.parse().ok())
101+
.unwrap_or(temperature)
102+
}
103+
// `none` sends no field at all, so the endpoint's own default
104+
// thinking budget applies.
105+
"--no-board-match" => no_board_match = true,
106+
"--reasoning-effort" => reasoning_effort = it.next().filter(|v| v != "none"),
107+
other => {
108+
eprintln!("unknown flag {other}");
109+
std::process::exit(2);
110+
}
111+
}
112+
}
113+
Args {
114+
utterances: utterances.unwrap_or_else(|| {
115+
eprintln!("--utterances is required");
116+
std::process::exit(2);
117+
}),
118+
out: out.unwrap_or_else(|| PathBuf::from("predictions.json")),
119+
detail,
120+
board_snapshot,
121+
write_board_snapshot,
122+
endpoint,
123+
model,
124+
reasoning_effort,
125+
temperature,
126+
no_board_match,
127+
}
128+
}
129+
130+
/// A board id for the `n`th task created by `event_id`.
131+
///
132+
/// Traceable on sight: the first 62 hex of the source event, then the index.
133+
fn synthetic_id(event_id: &str, n: usize) -> String {
134+
let mut base: String = event_id
135+
.chars()
136+
.filter(|c| c.is_ascii_hexdigit())
137+
.take(62)
138+
.collect();
139+
while base.len() < 62 {
140+
base.push('0');
141+
}
142+
format!("{base}{:02x}", n & 0xff)
143+
}
144+
145+
#[tokio::main]
146+
async fn main() -> Result<(), Box<dyn std::error::Error>> {
147+
let args = parse_args();
148+
let api_key = std::env::var("TASK_EXTRACT_API_KEY")
149+
.ok()
150+
.filter(|k| !k.is_empty());
151+
if api_key.is_none() {
152+
eprintln!(
153+
"TASK_EXTRACT_API_KEY is empty. A replay with no key scores the endpoint's \
154+
401 page, which looks exactly like a model that answered `none` to everything."
155+
);
156+
std::process::exit(2);
157+
}
158+
159+
let utterances: Vec<Utterance> = serde_json::from_slice(&std::fs::read(&args.utterances)?)?;
160+
let pinned: Option<HashMap<String, Vec<BoardTask>>> = match &args.board_snapshot {
161+
Some(p) => Some(serde_json::from_slice(&std::fs::read(p)?)?),
162+
None => None,
163+
};
164+
eprintln!(
165+
"replaying {} utterances through {} at {}",
166+
utterances.len(),
167+
args.model,
168+
args.endpoint
169+
);
170+
eprintln!(
171+
" temperature={} reasoning_effort={} board={}",
172+
args.temperature,
173+
args.reasoning_effort
174+
.as_deref()
175+
.unwrap_or("<endpoint default>"),
176+
if pinned.is_some() {
177+
"pinned snapshot"
178+
} else {
179+
"grown sequentially"
180+
}
181+
);
182+
eprintln!(
183+
" board-match second pass: {}",
184+
if args.no_board_match { "off" } else { "on" }
185+
);
186+
187+
let cfg = TaskExtractConfig {
188+
endpoint: args.endpoint,
189+
model: args.model,
190+
timeout_ms: 60_000,
191+
max_tokens: 8_000,
192+
reasoning_effort: args.reasoning_effort.clone(),
193+
attempts: 3,
194+
temperature: args.temperature,
195+
board_match: !args.no_board_match,
196+
};
197+
// No roster: the corpus has no pubkeys, and an empty known-set means the
198+
// extractor accepts any well-formed 64-hex assignee rather than rejecting
199+
// every one. Scoring is on action and count, not assignment.
200+
let extractor = TaskExtractor::new(api_key, HashSet::new());
201+
202+
let mut board: Vec<BoardTask> = Vec::new();
203+
let mut snapshot: HashMap<String, Vec<BoardTask>> = HashMap::new();
204+
let mut predictions = serde_json::Map::new();
205+
let mut detail = Vec::new();
206+
let mut errors = 0usize;
207+
let mut thin_total = 0usize;
208+
let mut rematched_total = 0usize;
209+
let mut dup_total = 0usize;
210+
211+
for u in &utterances {
212+
let board_for_row = match &pinned {
213+
Some(p) => p.get(&u.id).cloned().unwrap_or_default(),
214+
None => board.clone(),
215+
};
216+
snapshot.insert(u.id.clone(), board_for_row.clone());
217+
218+
let input = ExtractInput {
219+
message: Some(SourceMessage {
220+
id: u.event_id.clone(),
221+
channel: u.channel.clone(),
222+
author: String::new(),
223+
text: u.text.clone(),
224+
thread_root: None,
225+
}),
226+
thread_context: vec![],
227+
board: board_for_row,
228+
};
229+
let board_before = input.board.len();
230+
let extraction = extractor.extract(&cfg, &input).await;
231+
if let Some(err) = &extraction.error {
232+
// A failed call and a `none` verdict both publish nothing. Scoring
233+
// them the same is how the first run of this corpus reported four
234+
// model verdicts that were really a prompt the model never
235+
// answered — and how a later comparison table called a regression
236+
// catastrophic when it was merely worse.
237+
eprintln!(" !! {} EXTRACTION FAILED: {err}", u.id);
238+
errors += 1;
239+
}
240+
let duplicates = extraction.duplicate_subjects();
241+
thin_total += extraction.thin.len();
242+
rematched_total += extraction.rematched;
243+
dup_total += duplicates;
244+
245+
let (action, count) = match extraction.verdict() {
246+
Verdict::Create { count } => ("create", count),
247+
Verdict::Attach { count } => ("attach", count),
248+
Verdict::None { count } => ("none", count),
249+
};
250+
println!(
251+
" {:<10} {:<7} count={} asks={} board={} rematched={} dup={} thin={} dropped={}",
252+
u.id,
253+
action,
254+
count,
255+
extraction.asks.len(),
256+
board_before,
257+
extraction.rematched,
258+
duplicates,
259+
extraction.thin.len(),
260+
extraction.dropped.len()
261+
);
262+
// `thin` and `duplicates` ride with the prediction so the scorer can
263+
// count a create nobody can close as the defect it is, rather than as
264+
// work extracted.
265+
predictions.insert(
266+
u.id.clone(),
267+
serde_json::json!({
268+
"action": action,
269+
"count": count,
270+
"thin": extraction.thin.len(),
271+
"duplicates": duplicates,
272+
"asks": extraction.asks.len(),
273+
"rematched": extraction.rematched,
274+
"error": extraction.error.is_some(),
275+
}),
276+
);
277+
278+
let mut emitted = Vec::new();
279+
for task in extraction.tasks.iter() {
280+
match task {
281+
TaskAction::Create {
282+
ask,
283+
subject,
284+
done_when,
285+
blocked_by,
286+
..
287+
} => {
288+
let id = synthetic_id(&u.event_id, board.len());
289+
emitted.push(serde_json::json!({
290+
"ask": ask, "id": id, "subject": subject,
291+
"doneWhen": done_when, "blockedBy": blocked_by
292+
}));
293+
if pinned.is_none() {
294+
board.push(BoardTask {
295+
id,
296+
subject: subject.clone(),
297+
state: "open".into(),
298+
assignee: None,
299+
});
300+
}
301+
}
302+
TaskAction::Attach {
303+
ask,
304+
attach_to,
305+
note,
306+
} => {
307+
emitted.push(serde_json::json!({
308+
"ask": ask, "attachTo": attach_to, "note": note
309+
}));
310+
}
311+
}
312+
}
313+
detail.push(serde_json::json!({
314+
"id": u.id,
315+
"action": action,
316+
"count": count,
317+
"boardBefore": board_before,
318+
"asks": extraction.asks,
319+
"tasks": emitted,
320+
"thin": extraction.thin,
321+
"duplicates": duplicates,
322+
"dropped": extraction
323+
.dropped
324+
.iter()
325+
.map(|d| serde_json::json!({ "index": d.index, "reason": d.reason }))
326+
.collect::<Vec<_>>(),
327+
"error": extraction.error,
328+
// The evidence that was missing when the one-task collapse had to
329+
// be explained: key order shows what the model wrote first, and an
330+
// unknown key shows work nested where the parser cannot see it.
331+
"finishReason": extraction.finish_reason,
332+
"rawReply": extraction.raw_reply,
333+
}));
334+
}
335+
336+
std::fs::write(
337+
&args.out,
338+
serde_json::to_string_pretty(&predictions)? + "\n",
339+
)?;
340+
eprintln!("wrote {}", args.out.display());
341+
if let Some(path) = &args.detail {
342+
std::fs::write(path, serde_json::to_string_pretty(&detail)? + "\n")?;
343+
eprintln!("wrote {}", path.display());
344+
}
345+
if let Some(path) = &args.write_board_snapshot {
346+
std::fs::write(path, serde_json::to_string_pretty(&snapshot)? + "\n")?;
347+
eprintln!("wrote {} (pin it with --board-snapshot)", path.display());
348+
}
349+
eprintln!(
350+
"defects: {thin_total} creates with no observable doneWhen, {dup_total} duplicate subjects; \
351+
board-match converted {rematched_total} create(s) into attaches"
352+
);
353+
if errors > 0 {
354+
eprintln!(
355+
"\n{errors} of {} extractions never got an answer. The predictions file is \
356+
written, but it is NOT a model result: an errored row scores as `none` and \
357+
will read as a deliberate verdict. Exclude those rows from any denominator, \
358+
and say how many there were.",
359+
utterances.len()
360+
);
361+
std::process::exit(1);
362+
}
363+
Ok(())
364+
}

0 commit comments

Comments
 (0)