Skip to content

Commit 8d27b65

Browse files
anandgupta42claude
andcommitted
test: [#1406] restore the published benchmark configuration; third review round
Model, arm, lesson and pool defaults match the commit that produced the published runs (780168a); comparison checkouts are required env vars. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent fe1862e commit 8d27b65

50 files changed

Lines changed: 607 additions & 140 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎experiments/rsi-workspace/README.md‎

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,12 @@ For example, from the repository root:
1414
python3.11 -m venv /tmp/rsi-dbt
1515
/tmp/rsi-dbt/bin/pip install dbt-core==1.12.5 dbt-duckdb==1.11.0
1616
export DBT_BIN=/tmp/rsi-dbt/bin/dbt
17-
export DBT_PY=/tmp/rsi-dbt/bin/python
17+
export DBT_PYTHON=/tmp/rsi-dbt/bin/python
1818
```
1919

2020
Pin package versions and record the CLI commit and model IDs alongside any reported results.
21-
The default CLI is this repository's `packages/opencode/src/index.ts`; an override is a
21+
The shared default CLI is this repository's `packages/opencode/src/index.ts`; baseline and fix
22+
comparison wrappers retain their historical `rsi-base` and `rsi` worktrees. An override is a
2223
shell-quoted command string, e.g. `ALTIMATE_CMD="bun run --conditions=browser '/path with spaces/src/index.ts'"`.
2324
Model-driven runs need provider authentication usable from the isolated homes. Account/org policies
2425
may block Claude, including Claude on Vertex; choose models your account can use. The harness checks
@@ -48,7 +49,7 @@ bash budget/run_budget2.sh # scope/conflict arms
4849
bash budget/run_drift.sh my-drift "${REFLECTOR_MODEL:-google-vertex/gemini-3.5-flash}"
4950
FIX_SRC_ROOT=/path/to/fix-checkout bash budget/run_drift_matrix.sh
5051
FIX_SRC_ROOT=/path/to/fix-checkout bash budget/run_drift_matrix2.sh
51-
bash v1bench/run_baselines.sh # none / real4 / long / compressed baselines
52+
bash v1bench/run_baselines.sh # n50-short / all-50 / all-300 / all-1000
5253
bash v1bench/run_all_v1.sh # retrieval, vague task, topic, bootstrap, drift suites
5354
bash v1bench/run_fix_v1.sh # focused fix comparison (ALTIMATE_CMD selects checkout)
5455
```
@@ -66,25 +67,25 @@ v1 resumes only outputs containing the exact unique expected task/run keys.
6667
| Variable | Default / meaning |
6768
|---|---|
6869
| `ALTIMATE_CMD` | Bun CLI in this repository; honors quoted paths |
69-
| `DBT_BIN`, `DBT_PY` | `dbt` on PATH; Python beside dbt, or explicitly selected interpreter |
70-
| `AGENT_MODEL` | `google-vertex/gemini-3.5-flash`; choose an available model |
71-
| `REFLECTOR_MODEL`, `REVIEWER_MODEL` | `AGENT_MODEL`; independently overridable |
72-
| `BACKEND` | `fake`, bound to `127.0.0.1` with isolated test users |
70+
| `DBT_BIN`, `DBT_PYTHON` | `dbt` on PATH; Python beside dbt, or explicitly selected interpreter |
71+
| `AGENT_MODEL` | Shared Python drivers: `google-vertex-anthropic/claude-haiku-4-5@20251001`; v1 and budget drift shell drivers: `google-vertex/gemini-3.5-flash` |
72+
| `REFLECTOR_MODEL`, `REVIEWER_MODEL` | Shared default: `google-vertex-anthropic/claude-sonnet-4-6@default`; drift shell reviewer: `google-vertex/gemini-3.1-pro-preview`; independently overridable |
73+
| `BACKEND` | Historical default `saas`, requiring explicit opt-in; `fake` uses loopback and isolated test users |
7374
| `ALLOW_REAL_SAAS` | Unset; must equal `1` to permit real workspace access |
7475
| `WORKSPACE_ID` / `--workspace-id` | Required positive ID for SaaS; binding must resolve to it |
7576
| `SAAS_CREDS_DIR` | No default; explicit directory containing `home-a/.altimate/altimate.json` and `home-b/...` for SaaS |
7677
| `RUNS`, `PARALLEL`, `K`, `RUNS_VAL` | Wrapper sample count, concurrency, iterations and validation repetitions |
7778
| `AGENT_TIMEOUT`, `STAGGER_SECONDS` | Agent deadline (600s), startup spacing (3s) |
78-
| `BASELINE_FROM` | Optional prior baseline JSONL; must match task/model/verifier; retain its original logs |
79-
| `FIX_SRC_ROOT`, `STRONG_MODEL`, `WEAK_MODEL` | Drift matrix comparison checkout (required), reflector choices |
80-
| `BUDGET_REAL_FILE` | Optional real-lesson source; defaults to committed `budget/arms/pull.md` |
79+
| `BASELINE_FROM` | Corrections driver defaults to `runs/saas-v2/eval/none.jsonl`; set empty to evaluate anew; retain original logs when reusing |
80+
| `FIX_SRC_ROOT`, `STRONG_MODEL`, `WEAK_MODEL` | Drift matrix checkout (historical `rsi-fix` worktree by default), reflector choices |
81+
| `BUDGET_REAL_FILE` | Optional real-lesson source; defaults to `runs/corr-main/playbooks/final.md` |
8182
| `FAKE_DEBUG_TOKEN` | Standalone fake-server admin token; unset disables debug routes; harness generates its own |
8283

8384
SaaS publishing can update an existing **owned** `team-playbook`. Only run it against an intended test
8485
workspace, with `BACKEND=saas ALLOW_REAL_SAAS=1 WORKSPACE_ID=... SAAS_CREDS_DIR=...` (Python entrypoints use
8586
`--backend saas --workspace-id ...`). No real SaaS or paid benchmark is needed for self-tests.
8687

8788
Lesson pools/playbooks are deterministic: regenerate with `python3 harness/v1bench/pool.py` then
88-
`python3 harness/v1bench/make_playbooks.py` from this directory. Committed outputs track those generators.
89+
`python3 harness/v1bench/make_playbooks.py` from this directory. Committed pools, playbooks, lesson IDs/order, and benchmark treatment defaults match `780168ab18`.
8990
Historical result Markdown predates these verifier and delivery fixes; it is retained as historical evidence,
9091
not rescored or claimed as current benchmark results.

‎experiments/rsi-workspace/fake-backend/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ Notes:
7575
- `altimate-code link` needs a TTY (`src/cli/cmd/link.ts:~164`), so the demo seeds the binding
7676
server-side (`FAKE_SEED_REMOTE`) and the CLI finds it with `GET /datamate-project-bindings/by-remote`.
7777
- Model auth for `run` is independent of the fake and model requests can incur charges.
78-
The demo defaults to `google-vertex/gemini-3.5-flash`; override `AGENT_MODEL` or `MODEL`
78+
The demo defaults to `google-vertex-anthropic/claude-haiku-4-5@20251001`; override `AGENT_MODEL` or `MODEL`
7979
with an available configured model. Claude on Vertex may be blocked by organization policy.
8080
Provider auth stored only in the usual home is unavailable in an isolated home.
8181
- `skill list` does NOT trigger a sync. The bind-time sync runs on the first prompt of a session

‎experiments/rsi-workspace/fake-backend/demo.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ SCRATCH="${1:-$HERE/.demo}"
88
mkdir -p -- "$SCRATCH"
99
S="$(mktemp -d "$(cd "$SCRATCH" && pwd)/demo.XXXXXX")"
1010
REMOTE=https://example.test/demo/rsi-demo.git # must be a URL: bare local paths are dropped by detect.ts
11-
MODEL="${MODEL:-${AGENT_MODEL:-google-vertex/gemini-3.5-flash}}"
11+
MODEL="${MODEL:-${AGENT_MODEL:-google-vertex-anthropic/claude-haiku-4-5@20251001}}"
1212
PORT="${PORT:-18787}"
1313

1414
echo "Demo artifacts: $S"

‎experiments/rsi-workspace/fake-backend/selftest.py‎

Lines changed: 59 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,12 @@
33
import json
44
import os
55
from pathlib import Path
6+
import select
67
import shlex
78
import subprocess
89
import sys
910
import tempfile
11+
import time
1012
from urllib.parse import urlencode
1113

1214

@@ -30,6 +32,48 @@
3032
'''
3133

3234

35+
def receive_result(proc, pending, timeout=10):
36+
deadline = time.monotonic() + timeout
37+
while True:
38+
remaining = deadline - time.monotonic()
39+
if remaining <= 0:
40+
proc.kill()
41+
proc.wait(timeout=5)
42+
raise AssertionError("Backend test worker response timed out")
43+
if b"\n" in pending:
44+
line, _, rest = pending.partition(b"\n")
45+
pending[:] = rest
46+
if line.startswith(b"RESULT "):
47+
return json.loads(line[len(b"RESULT "):])
48+
continue
49+
if not select.select([proc.stdout], [], [], remaining)[0]:
50+
continue
51+
chunk = os.read(proc.stdout.fileno(), 65536)
52+
if not chunk:
53+
raise AssertionError("Backend test worker exited unexpectedly")
54+
pending.extend(chunk)
55+
56+
57+
def response_deadline():
58+
for prefix in ("", "RESULT "):
59+
proc = subprocess.Popen(
60+
[sys.executable, "-u", "-c",
61+
f"import sys, time; sys.stdout.write({prefix!r}); sys.stdout.flush(); time.sleep(60)"],
62+
stdout=subprocess.PIPE)
63+
try:
64+
try:
65+
receive_result(proc, bytearray(), timeout=0.1)
66+
raise AssertionError("Stalled worker unexpectedly returned a response")
67+
except AssertionError as exc:
68+
assert str(exc) == "Backend test worker response timed out", exc
69+
assert proc.poll() is not None
70+
finally:
71+
if proc.poll() is None:
72+
proc.kill()
73+
proc.wait(timeout=5)
74+
proc.stdout.close()
75+
76+
3377
def demo_isolation():
3478
"""Exercise run_as without starting the demo server or calling a model."""
3579
script = Path(__file__).with_name("demo.sh").read_text()
@@ -62,14 +106,10 @@ def exercise(debug):
62106
env["FAKE_DEBUG_TOKEN"] = "selftest-admin-only"
63107
proc = subprocess.Popen(["bun", "-e", WORKER], env=env, stdin=subprocess.PIPE,
64108
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
109+
pending = bytearray()
65110

66111
def receive():
67-
while True:
68-
line = proc.stdout.readline()
69-
if not line:
70-
raise AssertionError("Backend test worker exited unexpectedly")
71-
if line.startswith("RESULT "):
72-
return json.loads(line[len("RESULT "):])
112+
return receive_result(proc, pending)
73113

74114
def call(method, path, payload=None, token="token-user-a", tenant="demo", status=200):
75115
headers = {"x-tenant": tenant}
@@ -102,6 +142,16 @@ def workspace(name, owner="token-user-a", privacy="private"):
102142

103143
target_a = workspace("target-a")
104144
target_b = workspace("target-b", "token-user-b")
145+
before_invalid = call("GET", "/__debug/state", token="selftest-admin-only")
146+
for endpoint in ("/datamate-project-bindings", "/datamate-project-bindings/bind"):
147+
for invalid in ("", " ", 0, False, {}, []):
148+
for key, other, valid in (("project_path", "repo_remote", "https://example.test/valid.git"),
149+
("repo_remote", "project_path", "/example/valid")):
150+
call("POST", endpoint,
151+
{"name": "invalid-identifiers", "datamate_id": target_a, key: invalid, other: valid},
152+
status=400)
153+
call("POST", endpoint, {"name": "missing-identifiers", "datamate_id": target_a}, status=400)
154+
assert call("GET", "/__debug/state", token="selftest-admin-only") == before_invalid
105155
for privacy in ("private", "shared"):
106156
for kind in ("remote", "path"):
107157
ws_id = workspace(f"{privacy}-{kind}", privacy=privacy)
@@ -157,7 +207,8 @@ def workspace(name, owner="token-user-a", privacy="private"):
157207
assert len(state["memories"]) == 1
158208
assert all(ws["name"] != "must-not-create" for ws in state["workspaces"])
159209
call("POST", "/__debug/reset", token="selftest-admin-only")
160-
assert call("GET", "/__debug/state", token="selftest-admin-only")["workspaces"] == []
210+
reset_state = call("GET", "/__debug/state", token="selftest-admin-only")
211+
assert all(reset_state[key] == [] for key in ("workspaces", "bindings", "skills", "memories"))
161212
finally:
162213
proc.stdin.close()
163214
proc.terminate()
@@ -170,6 +221,7 @@ def workspace(name, owner="token-user-a", privacy="private"):
170221

171222

172223
if __name__ == "__main__":
224+
response_deadline()
173225
demo_isolation()
174226
exercise(False)
175227
exercise(True)

‎experiments/rsi-workspace/fake-backend/server.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,11 @@ function requireIdentifier(value: unknown): asserts value is string {
6767
if (typeof value !== "string" || !value.trim()) throw new HttpError(400, "A nonempty project identifier is required")
6868
}
6969

70+
function requireProjectIdentifiers(remote: unknown, path: unknown) {
71+
requireIdentifier(remote ?? path)
72+
if (remote != null && path != null) requireIdentifier(path)
73+
}
74+
7075
function requireBindingOwner(b: Binding, uid: number) {
7176
const ws = wsOf(b.datamate_id)
7277
if (!ws || !visible(ws, uid)) throw new HttpError(404, "No binding for this project")
@@ -84,7 +89,7 @@ function bindingConflict(clash: Binding, uid: number, remote: string | null, pat
8489
}
8590

8691
function bindProject(uid: number, datamate_id: number, remote: string | null, path: string | null) {
87-
requireIdentifier(remote || path)
92+
requireProjectIdentifiers(remote, path)
8893
const ws = wsOf(datamate_id)
8994
if (!ws || !visible(ws, uid)) throw new HttpError(404, "Workspace not found")
9095
if (ws.user_id !== uid) throw new HttpError(403, "Only the workspace owner can bind projects to it")
@@ -146,7 +151,7 @@ async function route(req: Request, url: URL): Promise<Response> {
146151
if (p === "/datamate-project-bindings") {
147152
if (m === "POST") {
148153
const j = await body()
149-
requireIdentifier(j.repo_remote || j.project_path)
154+
requireProjectIdentifiers(j.repo_remote, j.project_path)
150155
const clash = state.bindings.find((x) => (j.repo_remote && x.repo_remote === j.repo_remote) || (j.project_path && x.project_path === j.project_path))
151156
if (clash) bindingConflict(clash, uid, j.repo_remote ?? null, j.project_path ?? null)
152157
const ws: Workspace = { id: nextId(), name: j.name, user_id: uid, privacy: "private", memory_enabled: true, description: j.description ?? null }

‎experiments/rsi-workspace/harness/budget/arms/applicable40.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ applyPaths: ["dbt_project.yml"]
1818
- [L-4a50] Always spell out `as` when aliasing a column, for example `id as refund_id`, never a bare alias. <!-- h:0 x:0 -->
1919
- [L-c83c] Name the two CTEs `source` and `renamed`, in that order, with nothing between them. <!-- h:0 x:0 -->
2020
- [L-52d0] Do not qualify column names with the CTE name (`source.id`); the `renamed` CTE reads from a single relation. <!-- h:0 x:0 -->
21-
- [L-2584] Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row surviving required filters out. <!-- h:0 x:0 -->
21+
- [L-3b6a] Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row out. <!-- h:0 x:0 -->
2222
- [L-8536] If a source table has a soft-delete flag (e.g. `_is_deleted`), filter it out with `where not _is_deleted` in the renamed CTE and exclude that column from the select list; staging models must not expose soft-delete flags as output columns. <!-- h:2 x:0 -->
2323
- [L-82e4] Staging models contain no `order by` and no `limit`. <!-- h:0 x:0 -->
2424
- [L-a997] Reference raw data only through `{{ source('<source>', '<entity>') }}`; never hardcode schema or table names such as `raw.raw_refunds`. <!-- h:0 x:0 -->

‎experiments/rsi-workspace/harness/budget/arms/arms2-review.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ existing models or analyses.
2424
| 11 | L-4a50 | Always spell out `as` when aliasing a column, for example `id as refund_id`, never a bare alias. | Alias style. | C2 regex needs `as <pk>`; this reinforces it. |
2525
| 12 | L-c83c | Name the two CTEs `source` and `renamed`, in that order, with nothing between them. | Shape convention from gold item 2. | Matches gold; no check on names. |
2626
| 13 | L-52d0 | Do not qualify column names with the CTE name (`source.id`); the `renamed` CTE reads from a single relation. | Style. | cents/to_utc regexes match the call, not the column prefix; C3/C4 take the bare name inside the call (a qualified name would break them, so this avoids that). |
27-
| 14 | L-2584 | Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row surviving required filters out. | Staging layer contract. | Gold has none. |
27+
| 14 | L-3b6a | Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row out. | Staging layer contract. | Gold has none. |
2828
| 15 | L-82e4 | Staging models contain no `order by` and no `limit`. | Ordering and limiting belong downstream. | Gold has none; row counts are checked and `limit` would break them, so this protects. |
2929
| 16 | L-a997 | Reference raw data only through `{{ source('<source>', '<entity>') }}`; never hardcode schema or table names such as `raw.raw_refunds`. | Lineage convention (gold item 1). | Gold uses source(). |
3030
| 17 | L-cb34 | Staging models do not `ref()` other models; they sit directly on sources. | Layering rule. | Gold has no refs. |

‎experiments/rsi-workspace/harness/budget/make_arms.py‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515

1616
HERE = os.path.dirname(os.path.abspath(__file__))
1717
EXP = os.path.abspath(os.path.join(HERE, "..", ".."))
18-
REAL_FILE = os.environ.get("BUDGET_REAL_FILE", os.path.join(HERE, "arms", "pull.md"))
18+
REAL_FILE = os.environ.get("BUDGET_REAL_FILE", os.path.join(HERE, "..", "runs", "corr-main", "playbooks", "final.md"))
1919
ARMS = os.path.join(HERE, "arms")
2020
TIERED_CAP = 8
2121
PREP_TASK = "heldout-disputes"
@@ -25,9 +25,6 @@ def parse_final(path):
2525
text = open(path).read()
2626
m = re.match(r"(---\n.*?\n---\n)(.*)", text, re.S)
2727
front, body = m.group(1), m.group(2)
28-
# The committed source is the pull arm; generated budget arms are auto-loaded.
29-
if "applyPaths:" not in front:
30-
front = front[:-4] + 'applyPaths: ["dbt_project.yml"]\n---\n'
3128
lines = body.splitlines()
3229
header = [l for l in lines if l.startswith("<!--")][0]
3330
bullets = [l for l in lines if l.startswith("- [L-")]

‎experiments/rsi-workspace/harness/budget/make_arms2.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@
3838
"Shape convention from gold item 2.", "Matches gold; no check on names."),
3939
("Do not qualify column names with the CTE name (`source.id`); the `renamed` CTE reads from a single relation.",
4040
"Style.", "cents/to_utc regexes match the call, not the column prefix; C3/C4 take the bare name inside the call (a qualified name would break them, so this avoids that)."),
41-
("Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row surviving required filters out.",
41+
("Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row out.",
4242
"Staging layer contract.", "Gold has none."),
4343
("Staging models contain no `order by` and no `limit`.",
4444
"Ordering and limiting belong downstream.", "Gold has none; row counts are checked and `limit` would break them, so this protects."),

0 commit comments

Comments
 (0)