SOTA refresh evidence wave: pin qualification, observability comparison, hardware profiles - #87

…markitdown, mteb

Resolves each target version live via gh api releases/latest, cites the
retained current-pin receipt (or its absence) found by rg over evidence/,
blueprints/, catalogs/, and re-runs the same scoped checks isolated under
~/.cache/sota-refresh-20260923/pins-mem/ without touching the live
ai-memory service, PATH binaries or paid APIs.

- repomix: already at the newest release (1.18.1); reused the retained
  148/148 upstream test result, verdict qualified.
- markitdown 0.1.8: isolated venv install reproduces the retained
  version + conversion functional checks, verdict qualified.
- ai-memory 2.4.0: isolated binary/data-dir/loopback-server checks pass
  for write/search/read/backup; restore is blocked by a host-wide
  running-process safety guard while the live service is up, and
  `status --json` was found to ignore --data-dir and query the live
  server's hardcoded default port instead (discovered, not exploited;
  live store independently verified unchanged). Verdict not_comparable.
- pageindex 0.2.19 and mteb 2.21.6: no retained functional receipt exists
  to reproduce (source-review only per docs/grand-catalog-handbook.md and
  catalog citations); isolated install+import verified as new evidence.
  Verdict not_comparable.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Adds scripts/hardware_profile.py (stdlib only), a declarative
adoption/hardware-profiles.json for workflow-concurrency, local-generation,
embedding/RAG and ecosystem-bounded-run tiers, synthetic unit tests, this
host's measured evidence, and a macos-15/ubuntu-24.04 hardware-profile-smoke
workflow (with a uv hash-locked mlx-lm smoke test pinned to a Hugging Face
revision SHA). Also fixes the handbook's Codex CLI/SDK version conflation and
documents the live-go gate's user_decision exception to the flip-condition
wording.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

… (obs-compare)

Ran the coordinator's five-part discriminating test natively on pinned
otelcol-contrib 0.161.0, Prometheus 3.14.0, Loki 3.7.8, restic 0.19.1 and
sandbox-runtime 0.0.77 in an isolated scratch area with loopback-only ports.
otelcol-contrib, Prometheus, Loki and restic all qualify: ingest, storage,
query-back, kill-9 retention and byte-identical restic backup/restore with
query parity all pass. sandbox-runtime regresses: its default bwrap network
isolation makes the sandboxed collector's OTLP receiver unreachable from
outside the sandbox, breaking ingest entirely. Does not change the
contested row; receipt and README report per-candidate findings for the
coordinator to re-judge.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…h receipts

- langgraph: store smoke_test.py and full raw stdout in the repo (not just
  the cache prefix), extend the test with an InMemorySaver interrupt/resume
  checkpoint cycle on a thread_id, and cite the catalog's checkpointer entries
  in acceptance_check_ref.
- opensandbox: actually run the catalog's own offline `--help`/SDK-import
  surface at release-1.1.0, fix the preregistration to check Docker absence
  before concluding it, and correctly attribute the release-notes vs README
  quotes with an exact commit/line citation. Verdict stays blocked for the
  unverified functional core.
- mcporter: correct the "byte-identical to retained receipt" overclaim to
  describe the actual same-session comparison, and add a real --help command
  with exit codes instead of an uncited prose summary.
- agentsview / openresearch: wrap --help checks in pipefail with recorded
  exit codes, add missing --version commands, replace the openresearch
  discover --help narrative with a hashed same-session diff, and disclose
  agentsview's new 0.44.0 subcommand surface.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…26-09-23, pins-runtime)

dagu 2.16.6 -> v2.17.0: checksum-verified the pre-staged binary against the
release tarball, reran the retained graceful stop/retry job-recovery
acceptance (pass), and reran PR #81's inflight SIGKILL-resume fixture with a
120s hold past both documented lock/heartbeat thresholds -- still refused by
start/retry past the hold, settling the previously inconclusive ~1-2s result
without claiming a regression from 2.16.6 (never tested at 120s). Qualified.

ClickHouse v26.8.7.19-lts -> v26.9.2.8-stable: no prior native acceptance
receipt exists for ClickHouse at any version (only an unopened-README
source_review disposition; DuckDB is the actual storage-compute incumbent),
so this ran a first-time checksum-verified storage-compute smoke test
(create/insert/aggregate/Parquet round-trip) instead of a comparable rerun.
Verdict recorded as not_comparable.

skfolio 1.2.9 -> v1.3.0: reran the retained WalkForward MeanRisk/HRP
optimizer acceptance from PR #81 against a fresh 1.3.0 venv over the same
frozen SPY/QQQ/IWM inputs; development and reserved-segment Sharpe
approximations and weights reproduced the 1.2.9 receipt exactly. Qualified.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…the ai-memory isolation failure, add the missing markitdown fixture and handoff check

Resolves the Opus review's blocker and major findings for the 2026-09-23
pins-mem wave: the first-pass ai-memory 2.4.0 check never scoped its CLI
client to the isolated server (AI_MEMORY_SERVER_URL defaulted to the live
service's port), so its write/search/status results and a stray canary
page actually hit the live production store. Re-ran the whole check with
the client explicitly env-scoped, AI_MEMORY_EMBEDDING_PROVIDER=none, and
distinct --workspace/--project, confirmed by direct filesystem inspection
that the live store gained nothing this time, and disclosed (without
deleting/rotating) the still-unresolved stray live page and an incidental
token_pepper view for the coordinator/user to decide. Also broadened the
ai-memory acceptance_check_ref to the retained lifecycle-probe receipt and
added a handoffs leg; re-ran markitdown against the retained upstream
fixtures/greeting.html fixture instead of only a non-identical README;
re-ran mteb's install through ecosystem-bounded-run; and relabelled the
pageindex/repomix preregistration notes as late rather than prior.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…math, ecosystem-bounded-run vars

Resolve the independent Opus review of unit hw-profiles' first submission
(1 blocker, 5 major, 8 minor findings; full disposition in
evidence/artifacts/sota-refresh-20260923/hw-profiles/review-findings-response.json):

- hardware-profile-smoke.yml: stop activating the venv / calling
  `python3 -m uv` (uv is not importable inside an activated venv without
  system-site-packages); call a standalone, SHA-256-verified uv binary
  directly with --python targeting the venv interpreter. Pin the mlx-lm
  lock compile with --exclude-newer so the CI redundant-compile diff check
  stays deterministic as new transitive releases publish. Add
  tools/mlx-smoke/run_smoke.py to the pull_request path filter.
- hardware_profile.py: prefer measured /proc/meminfo over the .wslconfig
  configured ceiling for RAM-tier arithmetic (source_review vs.
  native_proven); read the CPU model from /proc/cpuinfo instead of
  asserting it; scale macOS unified memory by a labelled GPU working-set
  fraction before comparing it to VRAM tiers, and flag concurrent
  generation+RAG use of the same memory pool; replace the single
  unnamed "memory ceiling" with named ECOSYSTEM_JOB_MEMORY_HIGH/MAX
  overrides sized against ecosystem-bounded-run's actual defaults and
  containment rationale; report every .wslconfig candidate found instead
  of silently picking one.
- adoption/hardware-profiles.json: mark the untested GitHub macOS runner
  entry `pending` instead of `native_proven`; update sizing arithmetic
  for the new fraction/variable-name rules.
- run_smoke.py: report tokens/s and peak memory from mlx-lm's own
  GenerationResponse instead of a wall-clock estimate; confirm the
  pinned HF revision and mlx-lm's load(revision=) support from source.
- tests/test_hardware_profile.py: add mocked-subprocess coverage for
  detect_nvidia_gpu and measure_macos, plus direct tests for the new
  RAM-source and multi-.wslconfig helpers (33 tests, up from 16).
- Re-run scripts/hardware_profile.py on this host and re-verify uv's
  aarch64-apple-darwin checksum, PyPI mlx-lm source, and the pinned HF
  revision; record results in the revised hardware-profile-tool.json and
  mlx-lm-hash-lock.json receipts.
- docs/grand-catalog-handbook.md: reword the Hardware profiles section
  for the fixes above, outside the generated verdicts block. The
  handbook's Codex CLI/SDK conflation inside that block is confirmed but
  left unfixed (protected path); recorded as a residual.

python3 -m unittest (2146 tests), scripts/validate.py and
scripts/build_ecosystem.py --check all pass.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…ing receipt

Re-scope c3 sandbox-runtime from "regression" to "not_comparable": the
original test wrapped the always-on collector, a role Codex never selected
srt for. Re-ran the discriminating egress-denial test for srt's actual
selected role (confining a research worker's outbound network access) from
the owned worktree, fixing a cwd bug that had the previous run's deny-path
enumeration touching the coordinator's checkout. Default settings deny
egress; an explicit allowlist controls it.

Correct c8 restic's overclaimed "live" backup: the original test stopped
both backends before backing them up (a cold backup). Ran a genuine hot
backup with both backends left running and an active concurrent-write
emitter loop confirmed alive at backup time; restic check/restore passed
with no corruption and correct partial-snapshot parity.

Disclose the original preregistration's written_at as an unverifiable
self-report and add a genuinely pre-run preregistration file for this fix
round. Correct the "fully cleaned up" scratch-directory claim and the
inaccurate "pid 386/386-successor" wording (verified the live production
stack ran continuously and was never touched). Regenerate manifests and
explorer via finalize_evidence.sh; rtk proxy python3 -m unittest,
scripts/validate.py and scripts/build_ecosystem.py --check all pass.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Retract the dagu inflight-resume-after-kill receipt's overclaim that the
120s hold "conclusively settled" self-healing: no dagu scheduler was ever
started, so scheduler-driven zombie detection (the only documented
self-heal path, needing ~90s heartbeat plus 3x45s zombie checks) was never
exercised; only a manual `retry` refusal without a scheduler was observed,
and the `start --run-id` refusal is a duplicate-ID error, not a staleness
signal. Rescope the result and limits to inconclusive-without-a-scheduler.

Disclose the earlier job-recovery rerun attempt that produced no
result.json instead of omitting it. Commit the three patched harness
copies (dagu job-recovery run.py, dagu inflight-kill fixture v2, skfolio
run_optimizer.py) plus their diffs against the tracked/PR #81 originals
under patched-harnesses/, with sha256 hashes referenced from the receipts,
so they are reproducible from the repository instead of only an ephemeral
session scratchpad. Record sha256 hashes of the raw cached result files.

Fix the clickhouse receipt's recorded command text to match what actually
ran (versioned binary path, --queries-file against the retained
work/smoke.sql with absolute paths) instead of the mismatched
--query/relative-path text from the earlier draft; the recorded output was
already correct.

Reword the skfolio README/receipt "reproduced exactly" claims to "to
reported precision", matching the 1.2.9 receipt's 3-decimal rounding
rather than implying a bit-for-bit match against the full-precision 1.3.0
run. Sanitize personal home paths and session-scratchpad identifiers out
of the newly committed patched-harness files per scripts/validate.py.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Corrects claims that survived the earlier fix round but were still
inaccurate against the underlying evidence: README step 4/Key finding
still described the original cold restic backup as off-host/live and
"exactly the gap" the sealed lanes flagged; the Key finding still
headlined sandbox-runtime step 5 as a failure without the fix round's
role correction; the JSON's fix_round_preregistration.note claimed the
committed preregistration was verbatim (it is redacted) and omitted
that the non-listed-host egress check was added after the
preregistration file was written; and fix_round_summary claimed a
"diff-free restore" for the hot restic backup when no diff was run.
No new checks were required; timestamps are tied to file mtimes rather
than asserted. Verified with rtk proxy python3 -m unittest (2112
tests, OK, skipped=324), scripts/validate.py (passed) and
scripts/build_ecosystem.py --check (passed).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…receipts

Corrects preregistration/checked_at timestamps to actual file-mtime-derived
values (the fix round's timestamps were later than the commit that recorded
them and are now labelled as same-round/late instead of implying prior
measurement), fixes catalog line-number citations for langgraph
(agents-operations.json:253-254, foundation.json:850), drops a false
same-session byte-for-byte claim in mcporter.json, updates the langgraph
smoke_test.py docstring to reflect its interrupt()/resume coverage, and
moves the mcporter/agentsview/openresearch raw help/discover captures from
/tmp into the unit's owned cache prefix (disclosing the new-version captures
that were not recoverable). Fetches and stores the opensandbox README at the
cited commit to verify its Docker-requirement line citation.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

… disclosure wording, coordinator canary-page deletion

Addresses all five minor findings from the pins-mem rereview: adds the
missing fixtures/greeting.html command to markitdown.json's commands
array and aligns README's Key findings bullet to name it as the primary
reproduction leg (not the non-identical README conversion); fixes the
stale port (58217 -> 58317) and false "no credential read" claim in the
Isolation and containment section; corrects the "git log unchanged since
9f9a01c" claim to state the six subsequent live-service commits precisely;
and discloses in ai-memory.json's limits/README that the fix-round rerun
covered only write/search/read/handoffs/status/backup, not the retained
probe's full update/delete/restart lifecycle.

Also records that the coordinator deleted the stray first-pass canary
page (agent-lab/agent-lab/canary/test.md) via memory_delete_page on
2026-09-23 (wiki checkpoints ea02e38 -> e2fadf4), noting per the #81
finding that this is a wiki-level delete, not byte-level erasure, and
that the live token_pepper value seen during diagnosis has been reported
to the user for a rotation decision (value itself never reproduced here).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Corrects four minor findings from the second independent re-review:
- dagu.json and skfolio.json wrongly claimed the committed
  patched-harness copies were "verbatim"; they are path-sanitized
  derivatives of the files that actually ran. Both files now record
  the executed (pre-sanitization) file's sha256 alongside the
  committed copy's sha256 and describe exactly what differs.
- dagu.json understated the job-recovery harness diff (it also
  rewrites the frozen repository_inputs digest-list paths and the
  result dagu_version literal, not just the version guard and ROOT)
  and wrongly attributed an EVAL_DIR constant to the job-recovery
  harness, which belongs only to the skfolio harness.
- skfolio.json had a leftover "reproduced exactly regardless" phrase
  in limits inconsistent with the already-corrected "to reported
  precision" wording used elsewhere in the same receipt.
- clickhouse.json's corrected command text contradicted its own note
  by implying the queries-file hang was "resolved"; reworded to match
  the note and limits, which correctly say the hang was never
  root-caused or reproduced.

No new native commands were run; checked_at is unchanged, and each
file's correction_note documents this second, documentation-only
correction with its actual (not backdated) timestamp.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…unt and lock-header doc errors

The macOS job's setup-python step stayed at 3.13 while the mlx-lm lock is
resolved and diff-checked for 3.12, and `uv venv` passed no --python, so the
smoke-test venv could have been built against the wrong interpreter. Align
setup-python to 3.12 and pass --python 3.12 to uv venv explicitly; quote $UV
consistently with the rest of the file.

Also fix two documentation inaccuracies the rereview found: the review had
14 findings (1 blocker, 5 major, 8 minor), not 12, in
review-findings-response.json and the evidence README; and
tools/mlx-smoke/requirements.in pointed at a requirements.lock.txt header
comment that does not exist because the lock is compiled with --no-header.

Adds review-findings-response-round2.json recording this round's disposition
with honest (post-fix) timestamps, and reruns the offline
YAML-validity/actionlint/zizmor checks plus the full test/validate/
build_ecosystem suite before committing.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…hash and timestamp claims

- Copy the ten raw help/list/discover captures (agentsview, mcporter,
  openresearch) that were falsely described as "not preserved / not
  recoverable / moved" into evidence/artifacts/sota-refresh-20260923/pins-tools/raw/
  via cp -p (mtime preserved), recording sha256 and mtime in the receipts.
  All ten files were in fact still present in /tmp the whole time; the
  false provenance claims are corrected to describe what actually happened.
  One line in the agentsview --help captures (the tool's own built-in
  sync_include_cwd_prefixes example path) is a placeholder baked into the
  tool's output, not a host path; it is rewritten to this repository's
  "example" placeholder convention in the committed raw/ copies only
  (sha256 recomputed after rewrite, mtime preserved via touch -r), with the
  unmodified original bytes still verifiable via the sha256 recorded for
  the unit's own $HOME/.cache/ copy.
- Fix langgraph.json/README.md citing a stale smoke_test.py sha256 from
  before the prior round's docstring fix; both now cite the current hash
  and disclose that only the docstring differs from the script that
  produced the recorded outputs.
- Set written_at values that did not match any measured file mtime
  (mcporter.json, langgraph.json, opensandbox.json) to the commit time
  that actually finalized the text, keeping the LATE label rather than an
  unmeasured value; remove opensandbox.json's "before running anything"
  phrasing in favor of naming the actual per-check exit-code evidence.
- Set agentsview.json/openresearch.json checked_at from the true last
  check (the newer-version capture, now that both sides are stored)
  instead of only the older capture; remove langgraph.json's unsupported
  claim that a date -u command ran after each smoke test.
- Correct README.md's mcporter bullet (no byte-for-byte same-session
  claim actually exists) and langgraph bullet (out_1.2.11/1.2.12.txt hold
  stdout only, not exit codes); disclose that those two committed files
  are renamed copies of the working directory's out_old.txt/out_new.txt
  (content verified identical).

Verification: python3 -m unittest (2112 tests, OK, skipped=324),
scripts/validate.py (passed, 1936 hashed files), finalize_evidence.sh
(explorer fixed point), gitleaks protect --staged (no leaks). Not pushed.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…imestamp wording, c8 preregistration deviation)

- results[6] hot-backup narration now matches the bounded "smaller, correct
  subset, not a proven exact durable-write count" wording already used in
  fix_round_summary, instead of the stronger "not corrupted or missing"
  claim that had no durable-write count to support it.
- Preregistration timing text no longer claims a `date -u` timestamp inside
  the fix-round preregistration file; it now says explicitly the file's own
  mtime (vs. the later command-log mtimes) is the evidence for the write
  order, matching fix_round_preregistration.note. The fix-round commands
  window is restated as starting at the preregistration write (00:40:24Z,
  first command log mtime 00:40:29Z) instead of 00:39Z, which preceded that
  write.
- Added an explicit preregistration-deviation note to c8's verdict summary:
  the hot-backup test did not fully meet fixround-preregistration.md's c8
  pass criteria (verbatim command output retained; comparison against a
  during-window ingested count), since the backup command's raw output was
  not kept and no durable-write count was captured at backup time. The
  "qualified" verdict rests on check --read-data/restore --verify passing,
  not on full compliance with the preregistered comparison method.
- Rehashed manifests/evidence.json for the changed receipt file via
  finalize_evidence.sh (evidence explorer rebuilt to its fixed point,
  build_ecosystem.py --check passed on the first iteration).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

…ations in pins-runtime receipts

The verify:pins-runtime rereview found two new minor defects introduced by
the prior correction pass: skfolio.json described the EVAL_DIR placeholder
substitution backwards (it named the placeholder string as if it were the
real path being rewritten), and both dagu.json and skfolio.json labeled the
executed pre-sanitization harness copies as "retained under" a
/tmp/session-scratchpad/... path, which is itself a sanitized placeholder
for an ephemeral Claude session scratchpad, not a real, inspectable
location. Both receipts now correctly state the substitution direction and
label the executed-file location as an ephemeral, non-retained scratchpad
placeholder. Rehashed manifests/evidence.json to match.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…0260923

# Conflicts:
#	manifests/evidence.json

…-20260923

# Conflicts:
#	manifests/evidence.json

…sh-20260923

# Conflicts:
#	manifests/evidence.json

…h-20260923

# Conflicts:
#	manifests/evidence.json

…h-20260923

# Conflicts:
#	docs/ecosystem/index.html
#	manifests/evidence.json

…0260923

# Conflicts:
#	docs/ecosystem/index.html
#	manifests/evidence.json