You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8d27b65
Browse filesBrowse the repository at this point in the historyBrowse files
test: [#1406] restore the published benchmark configuration; third review round
Model, arm, lesson and pool defaults match the commit that produced the
published runs (780168a); comparison checkouts are required env vars.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|`BUDGET_REAL_FILE`| Optional real-lesson source; defaults to `runs/corr-main/playbooks/final.md`|
81
82
|`FAKE_DEBUG_TOKEN`| Standalone fake-server admin token; unset disables debug routes; harness generates its own |
82
83
83
84
SaaS publishing can update an existing **owned**`team-playbook`. Only run it against an intended test
84
85
workspace, with `BACKEND=saas ALLOW_REAL_SAAS=1 WORKSPACE_ID=... SAAS_CREDS_DIR=...` (Python entrypoints use
85
86
`--backend saas --workspace-id ...`). No real SaaS or paid benchmark is needed for self-tests.
86
87
87
88
Lesson pools/playbooks are deterministic: regenerate with `python3 harness/v1bench/pool.py` then
88
-
`python3 harness/v1bench/make_playbooks.py` from this directory. Committed outputs track those generators.
89
+
`python3 harness/v1bench/make_playbooks.py` from this directory. Committed pools, playbooks, lesson IDs/order, and benchmark treatment defaults match `780168ab18`.
89
90
Historical result Markdown predates these verifier and delivery fixes; it is retained as historical evidence,
90
91
not rescored or claimed as current benchmark results.
REMOTE=https://example.test/demo/rsi-demo.git # must be a URL: bare local paths are dropped by detect.ts
Copy file name to clipboardExpand all lines: experiments/rsi-workspace/harness/budget/arms/applicable40.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@ applyPaths: ["dbt_project.yml"]
18
18
-[L-4a50] Always spell out `as` when aliasing a column, for example `id as refund_id`, never a bare alias. <!-- h:0 x:0 -->
19
19
-[L-c83c] Name the two CTEs `source` and `renamed`, in that order, with nothing between them. <!-- h:0 x:0 -->
20
20
-[L-52d0] Do not qualify column names with the CTE name (`source.id`); the `renamed` CTE reads from a single relation. <!-- h:0 x:0 -->
21
-
-[L-2584] Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row surviving required filters out. <!-- h:0 x:0 -->
21
+
-[L-3b6a] Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row out. <!-- h:0 x:0 -->
22
22
-[L-8536] If a source table has a soft-delete flag (e.g. `_is_deleted`), filter it out with `where not _is_deleted` in the renamed CTE and exclude that column from the select list; staging models must not expose soft-delete flags as output columns. <!-- h:2 x:0 -->
23
23
-[L-82e4] Staging models contain no `order by` and no `limit`. <!-- h:0 x:0 -->
24
24
-[L-a997] Reference raw data only through `{{ source('<source>', '<entity>') }}`; never hardcode schema or table names such as `raw.raw_refunds`. <!-- h:0 x:0 -->
Copy file name to clipboardExpand all lines: experiments/rsi-workspace/harness/budget/arms/arms2-review.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,7 +24,7 @@ existing models or analyses.
24
24
| 11 | L-4a50 | Always spell out `as` when aliasing a column, for example `id as refund_id`, never a bare alias. | Alias style. | C2 regex needs `as <pk>`; this reinforces it. |
25
25
| 12 | L-c83c | Name the two CTEs `source` and `renamed`, in that order, with nothing between them. | Shape convention from gold item 2. | Matches gold; no check on names. |
26
26
| 13 | L-52d0 | Do not qualify column names with the CTE name (`source.id`); the `renamed` CTE reads from a single relation. | Style. | cents/to_utc regexes match the call, not the column prefix; C3/C4 take the bare name inside the call (a qualified name would break them, so this avoids that). |
27
-
| 14 | L-2584| Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row surviving required filters out. | Staging layer contract. | Gold has none. |
27
+
| 14 | L-3b6a| Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row out. | Staging layer contract. | Gold has none. |
28
28
| 15 | L-82e4 | Staging models contain no `order by` and no `limit`. | Ordering and limiting belong downstream. | Gold has none; row counts are checked and `limit` would break them, so this protects. |
29
29
| 16 | L-a997 | Reference raw data only through `{{ source('<source>', '<entity>') }}`; never hardcode schema or table names such as `raw.raw_refunds`. | Lineage convention (gold item 1). | Gold uses source(). |
30
30
| 17 | L-cb34 | Staging models do not `ref()` other models; they sit directly on sources. | Layering rule. | Gold has no refs. |
Copy file name to clipboardExpand all lines: experiments/rsi-workspace/harness/budget/make_arms2.py
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,7 @@
38
38
"Shape convention from gold item 2.", "Matches gold; no check on names."),
39
39
("Do not qualify column names with the CTE name (`source.id`); the `renamed` CTE reads from a single relation.",
40
40
"Style.", "cents/to_utc regexes match the call, not the column prefix; C3/C4 take the bare name inside the call (a qualified name would break them, so this avoids that)."),
41
-
("Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row surviving required filters out.",
41
+
("Staging models do no joins, aggregations or `distinct`; one source table in, one row per source row out.",
42
42
"Staging layer contract.", "Gold has none."),
43
43
("Staging models contain no `order by` and no `limit`.",
44
44
"Ordering and limiting belong downstream.", "Gold has none; row counts are checked and `limit` would break them, so this protects."),
0 commit comments