Skip to content

Local-model qualification results on the RTX 4090: C2 (Qwen3.8-27B fully on GPU + MTP) 5x decode at equal F1, now serving - #342

Merged
seathatflowsinourveins merged 5 commits into
mainfrom
claude/li26-results-20260926
Sep 26, 2026
Merged

Local-model qualification results on the RTX 4090: C2 (Qwen3.8-27B fully on GPU + MTP) 5x decode at equal F1, now serving#342
seathatflowsinourveins merged 5 commits into
mainfrom
claude/li26-results-20260926

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

These are the results of the preregistered local-model qualification on the workstation RTX 4090 (local-inference-latest-20260926: the plan was frozen in #323 and cleared to run by #326). The user approved GPU windows on 2026-09-26, using only the latest models: the Qwen3.8 family and releases from the past few weeks. The task is 8-K item extraction on the frozen 2020-03-02 daily-index cohort: 360 filings, with header items as labels.

Arm micro-F1 macro-F1 Median decode tok/s Median TTFT Peak device MiB Outcome
C0: production Qwen3.8-27B UD-Q4_K_M, 36 layers on GPU 0.9909 0.9867 5.25 11.4 s 16,345 control
C1: fully on GPU 0.9909 0.9867 8.40 4.5 s 20,083 passes
C2: C1 + MTP draft (n_max 3) 0.9909 0.9867 26.28 4.4 s 20,314 selected (change_serving_profile, final)
M: MiMo-V2.6-Distill-Qwen-9B Q8_0 0.9670 0.9512 80.16 0.43 s 15,621 retain_control (bootstrap lower bound −0.056 < −0.02)
  • C1 and C2 predicted exactly C0's item sets on all 360 filings, so their bootstrap bounds are 0. C2's MTP drafting accepted 4,735 of 4,737 drafted tokens.
  • B (Ternary-Bonsai) and X (Xing4.0) were not run, as plan.json specifies.
  • The results, decision.json (byte-identical when re-derived by the unchanged analyze.py), the window records and the acquisition summary are under results/. No filing text is published; the private raw outputs are only counted and hashed.
  • Recorded, not hidden:
    • the acquisition's first refusal (state_dir_must_be_owner_only, before any SEC request);
    • C2's admission refusal in window 2 (17,366 MiB free against 17,535 required; production was idle-sleeping, so stopping it freed only about 438 MiB);
    • the Windows applications closed for each window, including closures beyond README step 3's list (user-approved "all can be closed").

Production switch. At 11:05:40Z a unit drop-in switched nativestack-generation to the C2 profile. Only 4 flags changed (--gpu-layers 99, --n-cpu-ffn 20, --spec-type draft-mtp, --spec-draft-n-max 3); the other 23 are unchanged, including --sleep-idle-seconds 600. After the switch, /health returned 200 and the structured query answered {"answer":42}. Rollback is one command: remove the drop-in, reload and restart. Caveat: about 14.5 GB of the device is held while the model is loaded, against about 9.5 GB before. The device had 3.9 GB free with the Windows desktop apps closed, and #308's planned consolidation LLM will not fit alongside it while it is loaded.

Receipts:

  • llama-cpp install, first generation: independent review needs_changes (no -B), kept as is.
  • llama-cpp install -2: supersedes it, re-recorded from clean main 3822178 with -B; independent review agree.
  • llama-cpp use: qualified models are Qwen3.8-27B 4ca7207 under C2 (pass) and MiMo 81baddc (measured, not selected); independent review agree.
  • local-inference-c2-serving-switch-20260926: the switch record.
  • None of them rebinds the catalog's b11057 llama-cpp pin.

manifests/stack.json now names the artifact production actually serves: the stale ggml-org efbb3b1 entry becomes unsloth/Qwen3.8-27B-GGUF 4ca7207 UD-Q4_K_M, with its file sha256, runtime b11146 and the li26-c2 serving profile. experiment.json is recorded as observed, with the decision adopt_within_scope.

Evidence classes

  • Native execution on this host: one frozen-protocol run per arm. Speeds are this host's, under the load at the time.
  • The receipt reviews are independent headless sessions (independent_session).
  • Workflow review: Claude Opus found 5 low issues. GPT-6-Astra found 1 medium (a receipt command wrote a temp directory inside the checkout), which was fixed and re-recorded, with a regression test that keeps receipt commands read-only.

Tests: tests.test_local_inference_latest_20260926, test_stack_lifecycle and test_host_receipts ran 231, OK. validate.py, validate_convergence.py --all-recorded and host_receipts.py validate pass.

SOTA sources

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 26, 2026 13:04
Scout and others added 5 commits September 26, 2026 09:46
Scout @claude
li26 results: C2 serving profile selected and recorded, llama-cpp hos…
6d0157a 
…t receipts, served model pin

Record the preregistered local-inference-latest-20260926 comparison as run on
nativestack-5975wx-20260925 (native execution, one run per arm, frozen
protocol; speed and memory are this host's under its load at the time):

- results/: the frozen analyze.py decision (final, C2 selected,
  change_serving_profile; a rerun reproduces it byte for byte), the
  acquisition summary and every window summary and per-arm window record,
  all copied unchanged, plus results.md with the per-arm table, bootstrap
  bounds, why M failed criterion (1), B and X not run, the retained failed
  attempts and the dated operations record.
- experiment.json is now observed: adopt_within_scope for the C2 profile,
  qualified by li26-c2-w3; the acquisition's first refusal
  (state_dir_must_be_owner_only), C2's admission refusal in window 2 and
  M's failed criterion stay in failures_and_skips.
- README: a dated results section below the unchanged frozen text.
- llama-cpp install and use host receipts (catalog_revision 2e77144) with
  qualified models Qwen3.8-27B UD-Q4_K_M 4ca7207 (pass, C2 profile) and
  MiMo-V2.6-Distill-Qwen-9B Q8_0 81baddc (measured, fail); the b11057
  catalog pin is not rebound.
- evidence/receipts/local-inference-c2-serving-switch-20260926.json: the
  production switch, its post-switch checks and the rollback.
- manifests/stack.json: the generation model entry now names the served
  unsloth/Qwen3.8-27B-GGUF UD-Q4_K_M file (revision 4ca7207, sha256
  322e194f...) and the C2 serving profile flags.
- The component evidence matrix is regenerated (llama.cpp alternative:
  receipts_recorded); the li26 test now checks the recorded state and the
  results' no-document-text boundary.

manifests/evidence.json is not changed here: the coordinator registers the
19 files and the switch receipt.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scout @claude
li26 repair: re-record the llama-cpp use receipt with read-only commands
170ac39 
Review (gpt-6-astra, medium) found that the use receipt's sixth command made
a directory with mktemp inside the checkout, wrote decision.json there
through analyze.py --out and deleted it with rm -rf, contradicting the
receipt's statement that its commands changed no file. Confirmed against the
receipt and scripts/host_receipts.py record, which runs every --cmd in the
checkout.

- The use receipt is re-recorded at catalog_revision 2e77144 (published on
  origin/main). Its first five commands are unchanged. The sixth loads the
  frozen analyze.py under python3 -B, calls its analyze() on the retained
  private state and hashes the decision serialized as main() writes it, all
  in memory. It printed the same two lines as the first recording: final,
  C2 selected, change_serving_profile, and SHA-256 672be297..., the
  committed results/decision.json. A before/after mtime scan of the
  checkout, the private state directory, the systemd user configuration and
  the model and runtime directories showed no change beyond the receipt
  file and the manifests/evidence.json registration that host_receipts.py
  record itself writes (reverted; the coordinator registers). The receipt,
  results.md and the switch receipt say that it replaces the unpublished
  first recording (11:26:21Z).
- Dependent hashes: the switch receipt's post-switch readback now names both
  receipts with their observation times (install 11:26:02Z, use 12:41:06Z)
  and the new nvidia-smi reading, and its provenance carries the new use
  receipt and results.md hashes. experiment.json's results.md artifact hash
  follows results.md.
- test_host_receipt_commands_only_read fails on the first recording
  (command 6: mktemp) and passes on this one. It also ties the sixth
  command's printed hash to the committed decision.json.

The install receipt keeps its recording: neither of its commands creates,
writes or removes a path, and the new test checks both receipts.

Checks, with the registration applied temporarily and then reverted:
validate.py passed; validate_convergence.py --all-recorded 23 valid;
host_receipts.py validate 165 passed; component_matrix.py --check checked;
uv run --no-project --with jsonschema --with pyyaml python -m unittest over
10 modules, 652 tests OK.

manifests/evidence.json is not changed. Re-register experiment.json,
results/results.md, the use receipt, the switch receipt (its receipts[]
claim and limitations too) and the li26 test file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scout @claude
li26 receipts: independent reviews; install receipt re-recorded with …
2b48f01 
…python3 -B

A separate headless Claude session reviewed both llama-cpp host receipts:
- use receipt: independent_session agree.
- install receipt (first generation): needs_changes. verify-arms ran without -B (a
  possible bytecode write, against its read-only limitation) and limitations[1]
  overstated the recorded source of the MiMo pin. Kept unchanged with that review.
- install receipt -2 (supersedes it): re-recorded from a clean checkout at published
  main 3822178 with python3 -B and the corrected limitations; independent re-review:
  agree. The bytecode caches present predate the recording.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scout @claude
Register the li26 results, receipts and stack correction (hot-file co…
8d36702 
…mmit)

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scout @claude
Re-register after rebasing onto #329 (hot-file commit)
036ffbb 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement label Sep 26, 2026
@seathatflowsinourveins

seathatflowsinourveins commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator Author

Trading-lane ack (lane:shared), 2026-09-26, at 036ffbb.

The manifests/stack.json change replaces the local generation entry ggml-org/Qwen3.8-27B-GGUF@efbb3b1 Q4_K_M with the artifact production actually serves: unsloth/Qwen3.8-27B-GGUF@4ca7207 UD-Q4_K_M.

  • The entry records the file sha256, the llama.cpp b11146 runtime and the li26-c2 serving profile.
  • It names the rollback (removing the drop-in restores C0) and its selected_by decision file.
  • Its evidence id local-inference-c2-serving-switch-20260926 is registered in manifests/evidence.json.
  • No manifests/stack.json component changes, and the PR touches no catalogs/us-equities or blueprints/us-equities path.

Follow-up in the trading lane after merge: the Qwen/Qwen3.8-27B row in catalogs/us-equities/models.{json,md} still says "Financial extraction accuracy is unmeasured" and describes the historical Ollama quant. I'll update it from results/decision.json and this PR's receipts, keeping the caveat that the labels are the filings' own header ITEMS.

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 26, 2026
Open
2 tasks done
@seathatflowsinourveins
seathatflowsinourveins merged commit 771f25f into main Sep 26, 2026
25 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/li26-results-20260926 branch September 26, 2026 15:47
This was referenced Sep 26, 2026
Merged
Merged
Merged
seathatflowsinourveins added a commit that referenced this pull request Sep 27, 2026
@seathatflowsinourveins Scout @claude
Record the li26 native measurement on the Qwen3.8-27B model row (#354)
5af288d 
#342 merged the preregistered li26 comparison and its native_model_e2e receipt.
The Qwen/Qwen3.8-27B row moves from source_review to native_proven for that one
host and task: micro-F1 0.9909 against filer-declared header items on the
360-filing 2020-03-02 8-K cohort, and the C2 serving profile now in production
(26.28 tokens/s median decode). Limitations keep the label source, protocol,
memory and rollback boundaries, and the MiMo and B/X outcomes.

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

lane:shared Touches files owned by both lanes; needs both lanes' acknowledgement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seathatflowsinourveins