Skip to content

Commit c451717

Browse files
tae898claude
andcommitted
benchmarks: method docs match the instrument; MongoDB dense arm kept by the October freeze
Code (two lines, neither runs on the bench host): - make_paper_tables.MP_ARMS_OPTIONAL gains mongo: MongoDB's dense rows would have been dropped from the overlay without a word at qOH's landing (BUGS F138). - version_pin_check reads Dockerfile.composed too (0 disagreements). Docs, each claim checked against code (30 audit findings): the composed arm is the Qdrant server beside Neo4j in one container over REST (#117, F133), with the page-cache reserve disclosed; both ArcadeDB e2 arms pre-filter; per-engine TXN_SCOPE (#118); six gates, not four or five, and the tableId coverage refusal; the re-run mark and STALE_UNTIL_RERUN; the SurrealDB-server HNSW settle wait as a harness-defined settle row (F134); server_oom_killed and the 99%-of-cap rule; durability_class in the dedupe key and the current durability rule; ingest/index split engines; F1 to F14; land_stage's --only-lanes and its refusals; October image names and Neo4j 2026.08.1; tracked-artifact table rejoined. Comment-only edits in runner, e2_hybrid, export_web, fairness_check and refresh_web_page (AST-identical code). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01M8NdUMbUoCLwNJEWPir4YL
1 parent 34ded71 commit c451717

15 files changed

Lines changed: 95 additions & 70 deletions

‎benchmarks/README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
# ArcadeDB benchmarks: harnesses and results
22

3-
Two suites. Both reach ArcadeDB through the Python package, run every
4-
backend in Docker under the same CPU and memory caps, report five repetitions
5-
per cell, and publish the rows their numbers are computed from. They differ
3+
Two suites. Both reach embedded ArcadeDB through the Python package (the
4+
live harness's served arms reach it over HTTP instead, on a server built from
5+
the same upstream commit as the wheel), run every backend in Docker under the
6+
same CPU and memory caps, report five repetitions per cell, and publish the
7+
rows their numbers are computed from. They differ
68
in scope and in where they publish, not in the kind of thing they measure.
79

810
- `experiments/` is the **live harness**: ArcadeDB, embedded and served,

‎benchmarks/experiments/COMPARATORS.md‎

Lines changed: 7 additions & 5 deletions
Large diffs are not rendered by default.

‎benchmarks/experiments/FAIRNESS.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Every number on the page compares systems. A comparison is only worth printing if both sides were given the same thing. This file says what "the same thing" means, what is allowed to differ, and what is checked mechanically rather than remembered.
44

5-
`fairness_check.py` is one of the FIVE gates `refresh_web_page.py` runs (`equivalence_check`, `provenance_check`, `fairness_check`, `page_check`, `version_consistency_check`). It fails loudly rather than warning quietly.
5+
`fairness_check.py` is one of the SIX gates `refresh_web_page.py` runs (`equivalence_check`, `provenance_check`, `fairness_check`, `page_check`, `version_consistency_check`, `version_pin_check`). It fails loudly rather than warning quietly.
66

77
The failure mode this contract exists to close is **a correct number measured under conditions the row beside it did not get**. `claims_check` and `provenance_check` cannot see it: both verify a number against its own artifact, and such a number is correct about its own run.
88

@@ -130,9 +130,11 @@ does at commit was not established, and the row says
130130
cannot spread silently to another engine.
131131

132132
Every row records what it ran as `durability`; `fairness_check.check_durability`
133-
refuses a 2026-10 row with none, a strict string on an engine that has the knob,
134-
an unverified string on an engine not named above, or a PostgreSQL row whose
135-
server answered anything but `off`. The same check refuses two `instrument`
133+
refuses a 2026-10 row with none, a row whose engine reports a class other than
134+
the one its cell asked for in `durability_class` (DECISIONS #90), a row that
135+
declares no setting or carries an unverified string on an engine not named
136+
above, or a PostgreSQL row whose server answered anything but its class's
137+
setting (`off` relaxed, `on` strict). The same check refuses two `instrument`
136138
values in one table (rows before 2026-10 carry none and are the September
137139
instrument), and it refuses a time-series table whose engines disagree on the
138140
row counts of the two data-dependent queries (`q_groupby_rows`, `q_high_rows`),
@@ -271,7 +273,7 @@ Three things follow, and they are what make this checkable rather than a good in
271273

272274
* **A new arm must declare its index decision.** `fairness_check` fails a lane backend that declares nothing, the way the capability table refuses a kind its legend cannot define. Silence is the state this invariant exists to remove.
273275
* **And the declaration is checked against the rows, not trusted (F14b).** A map saying an arm builds an index is a claim; `index_s` on that arm's rows is the evidence. The gate fails an arm whose declaration cites a measured with/without ratio and whose every row records `index_s=0`, and fails one that declares `NONE` and records a build -- an index removed for costing that engine coming back. Only those two directions are asserted: a declaration like "record id carries pid" or "native TIMESERIES type" describes an index with no build step to time, and those arms are reported as not asserted rather than pattern-matched into an expectation. The count of arms checked, arms not asserted, and arms whose rows predate the ingest/index split is printed every run, so the coverage is visible rather than implied.
274-
* **And where one arm times the two phases, every arm does (F14c).** The ingest/index split turns "this engine took 40 s to build the corpus" into "12 s writing, 28 s indexing", and it arrives per adapter -- so it is exactly the kind of thing that stops arriving when someone adds the next engine to a lane and writes a `build()` without knowing its neighbours time two phases inside theirs. The cost is not a missing number but an unfair TABLE: the column exists because the other arms fill it, so the new arm prints a blank where everyone else prints a figure, and a blank in a benchmark reads as a result. Three states pass -- the arm splits, or it declares `index_before_load` (the index is defined before the first row lands and its work is spread through the load, as SurrealDB does on the cross-model lane), or no arm on that lane splits at all because every engine there builds its index as it ingests (sparse) or creates it in the schema before loading (graph). The finding is the fourth: a lane where some arms split, one does not, and it says nothing. On the dense lane the two phases are independent timers inside `build_s` rather than a division of it, so those rows also carry `setup_s` for the remainder, and the three add up (BUGS F101).
276+
* **And where one arm times the two phases, every arm does (F14c).** The ingest/index split turns "this engine took 40 s to build the corpus" into "12 s writing, 28 s indexing", and it arrives per adapter -- so it is exactly the kind of thing that stops arriving when someone adds the next engine to a lane and writes a `build()` without knowing its neighbours time two phases inside theirs. The cost is not a missing number but an unfair TABLE: the column exists because the other arms fill it, so the new arm prints a blank where everyone else prints a figure, and a blank in a benchmark reads as a result. Three states pass -- the arm splits, or it declares `index_before_load` (the index is defined before the first row lands and its work is spread through the load, as SurrealDB does on the document and time-series lanes), or no arm on that lane splits at all because every engine there builds its index as it ingests (sparse) or creates it in the schema before loading (graph). The finding is the fourth: a lane where some arms split, one does not, and it says nothing. On the dense lane the two phases are independent timers inside `build_s` rather than a division of it, so those rows also carry `setup_s` for the remainder, and the three add up (BUGS F101).
275277
* **The decision names its evidence.** "No index" is a finding when it is measured (ArangoDB, DuckDB on the document lane) and a defect when it is an omission (PostgreSQL on the document lane, DuckDB on the time-series lane, both until 2026-09-22).
276278
* **Index build time is its own column** wherever the engine has a boundary to time. The dense vector table has always separated `ingest s` from `index s`; the rest folded index build into ingest, which hides both the cost and the asymmetry -- SurrealDB's document arm must build its index BEFORE the load, so about 5.3 s of index work sits inside an ingest number every other engine pays without any.
277279

@@ -302,7 +304,7 @@ Two consequences:
302304

303305
## Allowed to differ (must be DISCLOSED, per PROTOCOL.md section 7)
304306

305-
- **Vendor settle steps** that have no equivalent elsewhere (Elasticsearch forcemerge, Milvus flush+load, Qdrant green-wait, ArcadeDB `COMPACT INDEX`). Each engine gets *its own*; none goes unmatched by the others having theirs.
307+
- **Vendor settle steps** that have no equivalent elsewhere (Elasticsearch forcemerge, Milvus flush+load, Qdrant green-wait, ArcadeDB `COMPACT INDEX`). Each engine gets *its own*; none goes unmatched by the others having theirs. The one settle step this harness defines rather than a vendor is the SurrealDB server's HNSW wait on the dense and cross-model lanes, a latency probe recorded as `settle_s` (BUGS F134, DECISIONS #117), and PROTOCOL.md section 7 lists it as such.
306308
- **Operating points deliberately not matched**, such as the dense fp32 arms with the build cache pinned to the corpus against INT8 at the engine default (DECISIONS #56), stated in the l3d condition.
307309
- **Quality and precision differences** (int8 against fp32 postings, ES pruning). Report recall next to latency, always.
308310
- **Intra-query parallelism at each engine's default.** ArcadeDB's SQL scans a type's buckets in parallel (`arcadedb.queryParallelScan`, on by default) only when the type has at least two buckets, and `arcadedb.typeDefaultBuckets` is 1. Every ArcadeDB type in this benchmark is created with the default, so its full scans run on one thread, as SQLite's and MongoDB's do, while DuckDB uses the whole cpuset. Not tuned: it is a knob that moves only ArcadeDB, so the default stands until a campaign decides otherwise (HANDOFF, 2026-09-23). The per-record cost that makes that one thread slow is engine code, filed as ArcadeData/arcadedb#8260.

0 commit comments

Comments
 (0)