You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 016237a
Browse filesBrowse the repository at this point in the historyBrowse files
docs(roadmap): record the 1.0 forge-evidence, review, benchmark and OpenCode decisions
Forge evidence comes from a containerised Gitea and act_runner with a scripted test repository instead of a production server (#774). External review is Met by the owner's own review, recorded as not independent; 1.0 ships without an independent one (#775). The benchmark rerun follows the false-positive fixes (#777). Phase 17 open question 5 records that session.stopping is an open OpenCode feature request (anomalyco/opencode#16626, three implementations closed unmerged) and that OpenCode-Hooks' session.idle hooks do not block (RUN: GitHub API; #778).
no-issue: roadmap refresh; the kept items are tracked in #774 to #780
Signed-off-by: Nicolas Brousse <[email protected]>
Copy file name to clipboardExpand all lines: docs/ROADMAP.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -283,7 +283,7 @@ What most widens adoption after v0.10.2: findings reaching an agent inside its e
283
283
| Item | Status | Work | Ships when |
284
284
|---|---|---|---|
285
285
| Consumer replay of the `merged-pr-body` push event |**Done** (RUN 2026-09-22) | Replay the consumer's squash merge of its #1071 as a push (base `020ec7a4`, head `a8eb5e30`, `GITHUB_EVENT_NAME=push`, a token that reads pull requests) with a build of `main` after #153; control with `DISCIPLINE_NO_NETWORK=1`| Forge reachable: `errors: 0`, one override, the `Agent Instructions Changed` finding on `AGENTS.md` lifted by the directive read from merged pull request #1071, every gate noting the pull request. Offline: `errors: 1` (the same finding), each gate noting `merged-pr-body: not read`. Both held. |
286
-
| Gitea Actions on a production server| Open | A private Gitea instance already runs the job-container recipe (`container: image:` pinned by tag and digest, `refs/pull/<n>/head` checkout) on `pull_request` and `push`. Record: the Gitea and `act_runner` versions; a clean pull-request run; an inverted canary (a pull request seeding a known violation) whose report names the expected gate ids; a push run to the default branch whose notes read the merged pull request through `commits/{sha}/pull`; `discipline doctor` against that instance's API | The five records exist and are summarised in Outstanding Checks without naming the instance or repository|
286
+
| Gitea Actions on a containerised forge| Open (#774) | Decision 2026-10-16: a disposable Gitea instance and `act_runner` in Docker with a scripted test repository stand in for a production server; the script is committed so the evidence can be re-run. The repository runs the job-container recipe (`container: image:` pinned by tag and digest, `refs/pull/<n>/head` checkout) on `pull_request` and `push`. Record: the Gitea and `act_runner` versions; a clean pull-request run; an inverted canary (a pull request seeding a known violation) whose report names the expected gate ids; a push run to the default branch whose notes read the merged pull request through `commits/{sha}/pull`; `discipline doctor` against that instance's API | The setup script and the five records are committed and summarised in Outstanding Checks; Forgejo with its runner where the same setup allows|
287
287
288
288
**Step 1: the agent loop** (uses output the binary already produces; `diff` and `check --staged` are the fast path)
289
289
@@ -346,7 +346,7 @@ A `--fix` suggestion (restore a deleted assertion from the base side, a `// SAFE
346
346
347
347
-**Go / no-go gate:** as Phase 10; a new pack ships only with all four facts (tests, handlers, functions, prose) or names in `docs/GATES.md` which fact it lacks; a consumer replay of the last 100 pull requests shows no new blocking finding that is a false positive.
348
348
-**Order:** Step 0 (evidence only); Step 1, then Step 2 (the widest reach for the least new code; Step 2 needs its write-path decision first); Step 3; Step 4 (false positives in packs consumers use today); Step 5 (Swift, then Scala, then Objective-C); Step 6. The benchmark runs in parallel. Rows within a step are independent.
349
-
-**Status:** Steps 1-5 shipped in v0.11.0; Step 0 row 1 done; Step 6 (distribution) shipped; public precision benchmark partial (results committed; harness, rater provenance and a deviation-free rerun open). Open: the production-Gitea evidence (Step 0 row 2, #774); the benchmark's false-positive causes (#777).
349
+
-**Status:** Steps 1-5 shipped in v0.11.0; Step 0 row 1 done; Step 6 (distribution) shipped; public precision benchmark partial (results committed; harness, rater provenance and a deviation-free rerun open; the rerun follows the false-positive fixes, decision 2026-10-16, #777). Open: the containerised Gitea evidence (Step 0 row 2, #774); the benchmark's false-positive causes (#777).
350
350
-**Public precision benchmark** (RUN 2026-10-15, `default-with-test-entrypoints` on 68 pilot pull requests across `astral-sh/uv`, `pydantic/pydantic`, `biomejs/biome` with 0 crashes or exit-2 failures, then 233 merged agent-authored pull requests across the 8 disjoint evaluation repositories `ruff-sh/ruff`, `denoland/deno`, `BurntSushi/ripgrep`, `tokio-rs/tokio`, `fastapi/fastapi`, `vitejs/vite`, `golang/go`, `junit-team/junit5`; artifacts in `docs/benchmarks/results/{manifest.json,findings.json,adjudication.json,summary.json}`):
351
351
-**Two-rater blinded adjudication:** 340 items (190 `discipline` blocking findings and 150 stratified `naive-sensitive-path-filter` substitution-twin flags, shuffled with seed `48201` and `blind_to_source_tool = true`; rater identity and blinding unverified, see the post-hoc audit); exact rater agreement `0.982353` (334 / 340), Cohen's $\kappa = 0.963184$ vs `0.75` floor (target) (measured: Apple M-series macOS 15 arm64, b9cf939b).
352
352
- **Class `structural-ast-and-deterministic`:** $k = 119$, $n = 144$ ($\ge 141$ required sample size (target)), 23 false positives, 2 cannot-tell; point precision `0.826389`, Wilson 95% CI `[0.756266, 0.879551]` vs pre-registered floor `0.90` (target), verdict `REJECTED` as computed, `INTERMEDIATE_amended_threshold_redesign` under the pre-registration amendment rule (post-hoc audit below) (measured: Apple M-series macOS 15 arm64, b9cf939b). Three concrete false-positive mechanisms account for 19 of the 23 false positives: cross-section `peerDependencies` vs `devDependencies` comparison in `dependency-delta` (7 findings in `vitejs/vite`), unrecognised `expect.poll(...)` assertion receivers in `vacuous-tests` (7 findings in `vitejs/vite`), and linter test input files under `**/test/fixtures/**` scanned as production code by `stub-bodies` (5 findings in `ruff-sh/ruff`). Within the class, `ci-integrity` achieved $k = 66 / 66$, point precision `1.000000`, Wilson 95% CI `[0.944997, 1.000000]`, `golden-output` $k = 13 / 13$, Wilson 95% CI `[0.771905, 1.000000]`, and `error-swallowing` $k = 11 / 11$, Wilson 95% CI `[0.741167, 1.000000]` (all `insufficient-n` at single-gate required $n = 141$ (target)) (measured: Apple M-series macOS 15 arm64, b9cf939b).
@@ -511,7 +511,7 @@ Verdict: **INTERMEDIATE**. The precision point estimate on real endings is 0.88
511
511
2.~~Fail-open on an unreadable transcript~~: decided in v0.20.0 (#730, #732): refuse mode refuses as could-not-verify, bounded by the loop guard and the session cap.
512
512
3. Other languages (dropped; see Known Limitations). The detector reads English only and misses every non-English announcement in the corpus. Models that answer in the person's language will not be caught. Per-language opening lists are cheap to write and need their own labelled endings; none exist yet.
513
513
4. A plain "I'll ..." with no background work running is sometimes this defect and is not matched. Claude Code and Qwen Code list background tasks, which would let an empty list count as evidence; the other agents do not.
514
-
5. OpenCode: a plugin-sent prompt is recorded as a user message. Whether that is acceptable in the session history, and what `run --standalone` should do (the process exits before the continuation), is undecided. `session.synthetic` exists in the plugin context and was not tried.
514
+
5. OpenCode: a plugin-sent prompt is recorded as a user message. Whether that is acceptable in the session history, and what `run --standalone` should do (the process exits before the continuation), is undecided. `session.synthetic` exists in the plugin context and was not tried. A hook that fires before the loop stops (`session.stopping`, with `output.stop = false`) would remove both problems, but it is a feature request (anomalyco/opencode#16626, open; three implementations closed unmerged) and not in OpenCode 2.x; the third-party OpenCode-Hooks plugin's `session.idle` hooks do not block either (RUN 2026-10-16: GitHub API, #778).
515
515
6. The refusal arrives as a user message on Copilot CLI and OpenCode, as hook feedback on Claude Code and as a system message on agy. A weak model may answer the refusal with another announcement; the cap then ends the turn. How often that happens is unmeasured: every probe here used an instruction the model could satisfy in one word.
516
516
517
517
**Recommendation for the remaining steps.** OpenCode next: it needs a new plugin handler and a decision on question 5, and it is where most of the weak-model sessions in the records ran, so it should not wait long. Cursor and Aider are out of scope. `mode = "refuse"` stays an opt-in nobody is told to choose until question 1 is answered: the observe log from real sessions is the cheapest way to get the endings a second labeller needs.
@@ -526,8 +526,8 @@ Before 1.0 a minor release may change gate behaviour; from 1.0, `docs/ARCHITECTU
526
526
|---|---|---|
527
527
| Consumer replay of the v0.11 features | Met: `replay --last 100` (200 on the multi-language one) on four consumer repositories (a PHP C extension, a Python CLI, a Go MCP server, a Rust core with bindings in seven languages) and on public Swift, Scala and Objective-C projects; every false positive they found is fixed (the last Objective-C and Swift ones for the release after v0.12.3), and what still blocks is a true positive or defensible |`discipline replay --last 100` on at least one consumer repository shows no false-positive block from the v0.11 language packs, agent-facing checks or `archive-contents` formats |
528
528
| Live agent sessions | Met: Claude Code, Copilot CLI, OpenCode, agy, Codex CLI (0.162.1) and Qwen Code (0.25.2) ran the installed hook in a live session and acted on a finding (the recorded payloads under `tests/fixtures/stop/` and `tests/fixtures/pretool/` are the committed evidence; the agent acting on the finding is recorded in prose only, and the Qwen Code 0.25.2 `Stop` behaviour has no committed fixture, the `qwen` Stop fixtures being 0.25.0 captures; `PostToolUse` and `Stop` in addition to `PreToolUse` and `SessionStart`; after the agy `Stop` shape and the Copilot `apply_patch` matcher were corrected); Cursor is set aside from the headless CLI gate (as in Phases 13 and 17) | One real session each with the headless CLI agents (Claude Code, Codex CLI, Copilot CLI, agy, Qwen Code and OpenCode) runs the installed hook and the agent receives and acts on a finding; the contracts cited in `src/hook.rs` are confirmed or corrected |
529
-
|Production forge evidence | Open (#774) | The production-Gitea records Phase 11 Step 0 lists exist |
530
-
| External review |Open (#775) | A non-LLM reviewer has read the gate contract (`docs/ARCHITECTURE.md` §3) and the agent-facing threat model (`docs/ARCHITECTURE.md` §1.4) and their findings are resolved |
529
+
|Forge evidence | Open (#774) | The containerised Gitea records Phase 11 Step 0 lists exist |
530
+
| External review |Met (decision 2026-10-16, #775): the owner, a human reviewer, read the gate contract and the threat model; 1.0 ships without a separate independent review, and this is a review by the project's author, not an independent one| A non-LLM reviewer has read the gate contract (`docs/ARCHITECTURE.md` §3) and the agent-facing threat model (`docs/ARCHITECTURE.md` §1.4) and their findings are resolved |
531
531
| Interface freeze | Met: the stability policy is `docs/ARCHITECTURE.md` §3.2, held by `tests/test_stability_contract.rs` against `tests/fixtures/v1_surface.json` and by the schema snapshots in `tests/test_output_schemas.rs`. No release since v0.17.2 removed or renamed a name in `v1_surface.json` (additions only), and the three output schemas did not change across the v0.20 (v0.20.0 to v0.20.3) or v0.21 (v0.21.0 to v0.21.2) lines (RUN 2026-10-16: `git diff` of `tests/fixtures/v1_surface.json` and `discipline.{report,replay,audit}.schema.json` between consecutive tags) | One full minor release passes with no removal or rename in `tests/fixtures/v1_surface.json` and no change to a pinned schema field, and a stability policy states what 1.0 freezes |
532
532
533
533
---
@@ -1071,10 +1071,10 @@ A change to what a gate reports, an exit code, or an output, with an unchanged d
1071
1071
1072
1072
## Outstanding Checks & Known Limitations
1073
1073
1074
-
-**No external review:** The architecture and test coverage were established through internal pairing and rigorous self-tests. External review by independent systems engineers is an outstanding verification check.
1074
+
-**No independent external review:** The architecture and test coverage were established through internal pairing and self-tests, and the owner (human) reviewed the gate contract and the threat model before 1.0 (#775). No reviewer independent of the project has; 1.0 ships without one by decision (2026-10-16).
1075
1075
-**Runner environment testing:**
1076
1076
- GitHub Actions: verified on hosted Linux runners in CI (`action-github`), and against published release assets on Linux and macOS runners in the release smoke job.
1077
-
- Gitea Actions: tested via `act` runner images; a private production Gitea instance runs the job-container recipe on `pull_request` and `push`, and the evidence Phase 11 Step 0 lists (versions, a clean run, an inverted canary, a push run reading the merged pull request, `doctor` against its API) is outstanding.
1077
+
- Gitea Actions: tested via `act` runner images; the evidence Phase 11 Step 0 lists (versions, a clean run, an inverted canary, a push run reading the merged pull request, `doctor` against its API) is outstanding, from a containerised Gitea and `act_runner` with a scripted test repository (#774).
1078
1078
- Forgejo Actions: CI lints `.forgejo/workflows/action-selftest.yml` with `actionlint` but does not run it; the same composite action runs under `act` for Gitea. Execution on a Forgejo runner is outstanding.
1079
1079
- GitLab CI: the component template is checked by `tests/action/check-links.py` for recipe defects (an image in `runs-on:`, a `git clone` in job steps, obsolete image tags); it is not schema-validated, and live GitLab runner execution is outstanding.
1080
1080
- Argo Workflows: the template gets the same text checks only; it is not linted by an Argo tool, and live Kubernetes cluster DAG execution is outstanding.
0 commit comments