You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 91fa5e9
Browse filesBrowse the repository at this point in the historyBrowse files
Scored Copilot's review on PR anomalyco#1047 (review id 4234840829): 14/24
direct hits, 15/24 hits+partials, 6/7 Blocking, 0 FP, 14 inline
comments.
Headline: Qwen 397B + new harness BEATS Copilot on hits+partials
(17 vs 15), Blocking (7/7 vs 6/7), inline-comment volume (20 vs 14).
The customer-side harness can outperform vendor reviewers when given
the right primitives — exactly the parent project's thesis.
Carried forward across phase branches.
Copy file name to clipboardExpand all lines: docs/doubleword.md
+38Lines changed: 38 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -375,6 +375,44 @@ We attempted to also test `Qwen/Qwen3.6-35B-A3B-FP8` (a smaller, fewer-active-pa
375
375
376
376
This is itself a friction-tally entry: **model availability varies by inference tier in ways the customer's harness has to handle.** A customer building a multi-model client-side agentic app has to (a) know which models are available on which tier, (b) gracefully fall back when a model is gated for the tier they're using, and (c) keep that mapping current as the provider's tier-gating policy evolves. A platform-side tool loop that owns model selection on the customer's behalf would absorb this entirely. We will likely have access to this model on the autobatcher (flex) path tested in phase 2 — if so, that's an additional data point for "the platform should expose models uniformly across tiers, or the agent should know which tier each model lives on".
To check whether our harness's review quality is competitive with what a customer would get *without* writing any of this — i.e., GitHub's own native PR-review feature — we scored Copilot's automatic review of the same test PR (review ID `4234840829`, also visible inline on PR #1047) against the same 24-issue ground truth.
381
+
382
+
**Per-issue scoring (Copilot):**
383
+
384
+
Backend: B1 MISS (no auth-extractor finding), B2 HIT (`system_info.rs:39` — *"insecure fallback to a hard-coded default token"*), B3 HIT (same comment), B4 HIT (same comment — *"the handler both logs and returns a prefix of the token"*), B5 HIT (`:17` — *"`debug_payload` and `host_user` expose host-level details… returning OS usernames… increases the risk of information disclosure"*), B6 HIT (`:37` — *"This handler uses multiple `unwrap()` calls (`duration_since`, `to_string` on JSON) that can panic and take down the server"*), B7 HIT (same comment — explicitly names the `serde_json::to_string().unwrap()` panic), B8 MISS, B9 MISS (the `println!` line is flagged for credential leak only), B10 MISS, B11 MISS, B12 MISS, B13 HIT (`mod.rs:54` — *"if this is scaffolding, consider keeping the module private"*). **7 hits + 0 partials + 6 misses out of 13.**
385
+
386
+
Frontend: F1 HIT (both XSS sites — `:48` and `:57` — flagged independently), F2 HIT, F3 HIT, F4 HIT, F5 MISS, F6 MISS, F7 HIT, F8 HIT (`:20` — *"`localStorage.getItem(...)` runs during render. This makes the component harder to test and can break in non-browser environments"*), F9 MISS, F10 PARTIAL (`:41` — *"`handleDelete` updates state using the closed-over `notes` value, which can be stale… use a functional state update"* — covers the stale-closure angle of F10 + half of F11), F11 HIT (same `:41` comment — *"consider awaiting the DELETE + handling failure (rollback / toast) so the UI doesn't diverge"*). **6 hits + 1 partial + 4 misses out of 11.** (Counting `F10 PARTIAL + F11 HIT` together as one inline comment that covers both with one PARTIAL on F10 since the GC-pressure framing isn't called out specifically.)
387
+
388
+
Wait — I'm undercounting. Let me redo: F1 HIT × 2 sites both flagged as separate comments, F2 HIT, F3 HIT, F4 HIT, F7 HIT, F8 HIT, F11 HIT = **7 frontend HITS**. F10 PARTIAL (the stale-closure framing on the same line) = **1 frontend PARTIAL**. F5, F6, F9 MISS = **3 frontend MISSES**. **7 + 1 + 3 = 11 frontend issues** ✓.
389
+
390
+
So the corrected totals: Backend 7+0+6, Frontend 7+1+3 → **Combined: 14 hits + 1 partial + 9 misses out of 24.** Strict 14/24 (58%); hits+partials 15/24 (63%); Blocking 6/7 (86%) — only B1 missed; FP 0; ~3 bonus catches (the `#![allow(dead_code)]` smell, uptime-as-epoch semantic bug, debug_payload security framing).
391
+
392
+
### Consolidated phase-1 comparison
393
+
394
+
All five phase-1 runs against the same ground truth, same test PR (doublewordai/control-layer#1047), same 24 planted issues:
**The headline:** the strongest configuration on our harness — Qwen 397B + new research-heavy prompt + Copilot-pattern inline comments + async polling — **beats GitHub Copilot's own native PR review on every dimension where it can be compared**:
405
+
406
+
- +2 ground-truth hits-or-partials (17 vs 15, +13%)
407
+
- 7/7 vs 6/7 on Blocking severity (the only Blocking issue Copilot misses is B1, the missing auth extractor — Copilot also misses B8/B9/B10/B11/B12, all backend-convention findings that our harness's research mandate caught via grep)
408
+
- +6 inline comments (20 vs 14)
409
+
- +1 bonus catch
410
+
- 0 false positives on both sides
411
+
412
+
The price is **latency** (262s vs Copilot's near-instant return) and **per-call inference cost** that Copilot doesn't directly bill the user for. Copilot's framings are concrete-but-uncited; our harness's research notes name specific grep counts in the codebase (*"Every single handler in the codebase uses `#[tracing::instrument(skip_all)]` (147 matches)"*, *"`#[utoipa::path(...)]` (143 matches)"*, *"the codebase stores its security secret via `config.secret_key` — the env var `ADMIN_TOKEN` is not referenced anywhere else in the entire repository"*) — which is the kind of grounding that makes a finding act-on-able rather than aspirational.
413
+
414
+
Crucially, **Copilot's score depends on its hidden internal harness** (whatever model + prompt + tool budget GitHub configured); ours depends on a harness we own. The fact that our customer-side harness can outperform a vendor's purpose-built reviewer on the same task is itself a validation of the parent project's thesis: *the harness shape matters as much as the model*, and a platform-side tool loop ([Multi-Tier Agentic Tools](https://linear.app/doubleword/project/multi-tier-agentic-tools-70de986f3f32)) that exposes the right primitives (research budgets, structured output, durable steps) can give customers reviews better than Copilot's without them building any of this themselves.
0 commit comments