Repository navigation
fix: keep the previous artifact when reward synth or adapter fuse fails - #1454
Conversation
… lands (MakazhanAlpamys#1450) merge_adapter_to_dense (shared by `soup shrink` and `soup draft distill`) deleted out_dir before swapping the newly merged model in, so a failure between rmtree and os.replace (a lock on the staging dir, disk full) left neither the old nor the new model. `soup reward synth -o reward.py --force` had the same shape: it overwrote the existing verifier with the candidate before calibrating it, then deleted the file outright on refusal or a calibration error, destroying a previously working verifier. Both now rename the previous artifact aside instead of deleting it, and restore it if the replacement does not make it all the way to accepted: merge_adapter_to_dense renames out_dir to a sibling .old- path and puts it back if the final os.replace(staging, out_dir) itself fails; reward.py's synth moves the existing --force target to a temp file and restores it on both the "calibration failed" and "refused" exit paths.
…oo (MakazhanAlpamys#1450) Two gaps remained in the previous MakazhanAlpamys#1450 fix: - reward synth: atomic_write_text(result.source, output) ran outside any try/except. With --force, the previous verifier had already been moved to a .soup.reward-prev.*.tmp backup by then, so a failing candidate write (disk full, a permission error) exited 1 with the previous verifier gone and the backup orphaned under its .tmp name instead of restored. - adapter_fuse: when the final os.replace(staging, out_dir) failed, the handler tried to restore the renamed-aside backup but did not distinguish a successful restore from a failed one. If the restore itself also failed, the outer except still ran shutil.rmtree(staging, ignore_errors=True), destroying the only complete copy of the newly merged model even though neither the old nor the new model had made it back to out_dir. Both are now guarded: the candidate write restores (or removes) the backup on failure, and a failed restore keeps both the backup and the staging dir on disk and names them in the raised error instead of deleting either. Tests: a failing candidate write leaves the previous reward.py untouched and no orphaned backup; a regression guard that poisons shutil.rmtree(out_dir) to fail after deleting one file (the original bug's exact failure shape) and asserts merge_adapter_to_dense never calls it; a Windows-only test for an open handle on out_dir/model.safetensors. Stale present-tense docstrings/ comments in the existing tests are also corrected.
… restore fail Adds the test the issue's acceptance criteria call out explicitly: when os.replace(staging, out_dir) AND the rename-back of the backup both raise, neither copy may be deleted and the error must name both paths. Verified by mutation: reverting 'if not keep_staging:' to 'if True:' in adapter_fuse.py makes this new test fail while every other test in the file still passes.
… acceptance (MakazhanAlpamys#1450) reward synth --force moved the previous verifier aside and wrote the candidate in place at -o, then restored the backup on refusal or a calibration error. A crash between the write and that restore left the candidate sitting at -o with the real previous verifier hidden under its backup name. Switch to the issue's own design: write the candidate to a .soup.reward-candidate.*.py temp sibling, load and calibrate that file, and os.replace it onto -o only once calibration accepts it. -o is never moved or overwritten before that final replace, so a refusal, a calibration error, or a crash at any point before it leaves the previous verifier exactly as it was.
A lock held on -o or a disk-full error at the very last step -- swapping the accepted candidate onto -o -- used to propagate as a raw OSError traceback instead of the command's usual clean failure. Wrap it like every other step here: on OSError, clean up the now-useless candidate and fail with a message naming the path and the error. Round 3 review also flagged the changelog fragment's reward half, which still described the old move-aside-and-restore design after the temp- sibling rewrite -- reworded to match the code. The adapter_fuse half was already accurate.
| backup = os.path.join( | ||
| parent, f".{os.path.basename(os.path.normpath(out_dir))}.old-{os.getpid()}" | ||
| ) | ||
| os.replace(out_dir, backup) |
There was a problem hiding this comment.
I ran this against 2fb1212 in a clean container, stubbing transformers/peft the same way this PR's own test fixtures do.
adapter_fuse.py:126, the first os.replace(out_dir, backup) (renaming the previous model aside) sits outside the try/except that guards the swap below it. When it raises, control falls straight to the outer except BaseException at line 154, keep_staging is still False, and shutil.rmtree(staging, ...) deletes the newly merged model even though out_dir was never touched and the old model is intact.
exception raised: PermissionError(5, 'Access is denied (indexer/AV holding out_dir)')
out_dir contents: ['config.json', 'model.safetensors', 'tokenizer.json']
out_dir/model.safetensors: b'OLD'
staging dirs remaining: []
Same lock class this PR already guards against on the swap itself (an indexer or AV holding a path under out_dir), just one step earlier, on the rename that's supposed to protect the old copy. Wrapping the line-126 replace in its own try and setting keep_staging = True on failure (out_dir was never touched, so staging is still the only place the merge result exists) would keep it recoverable the same way the double-failure branch below already does.
There was a problem hiding this comment.
Fixed in 67af97b: a failed rename of the old model now keeps the staging dir and names it in the error, with a test for that case.
There was a problem hiding this comment.
That covers it, and the error message pointing at the staging dir is a nice touch for anyone who hits this manually.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you. This is a real fix for both sites in #1450, and #1450 is yours (you claimed it on 2026-09-28, and that claim is confirmed on the issue).
What is good: soup reward synth --force now never touches -o until the candidate has been loaded back and calibrated. The temp sibling plus os.replace is exactly the right shape, and you also guarded the final replace and the candidate write, which the issue did not ask for. In merge_adapter_to_dense you went beyond "rename aside and restore": when the restore fails as well, you keep both copies and name both paths instead of guessing which one to delete. The tests are strong: I ran your file on this head and it is 10 passed plus the one failure below, and on main it is 9 failed / 2 passed. I also mutated the fix 10 ways (rmtree instead of rename, no restore, if not keep_staging to if True, no keep_staging on the rename failure, aside copy never removed, swap onto -o before calibrating, unguarded final replace, and three "candidate left behind" variants), and all 10 were killed. Related files (test_v07133, test_v07140, test_v07201, test_cli_startup_is_light) pass, 290 passed, and ruff is clean. The gap @AmirF194 found in review on 2fb1212e (the first os.replace(out_dir, backup) outside the try) is fixed properly in 67af97b2.
One blocker: the two failing required checks are caused by this PR, and it is deterministic.
test (windows-latest, 3.10) and test (windows-latest, 3.12) (and the informational 3.11 cell) each fail on exactly one test, everything else passes ("1 failed, 30867 passed" / "1 failed, 30868 passed"). These runs are for the current head; they started 9 seconds after your push of 67af97b2. The test is your Windows-only one:
tests/test_issue1450_atomic_replace.py::TestMergeAdapterToDenseSurvivesAFailedSwap::
test_out_dir_survives_an_open_handle_on_model_safetensors
adapter_fuse.py:127: os.replace(out_dir, backup)
PermissionError: [WinError 5] Access is denied: 'out' -> '...\.out.old-2896'
adapter_fuse.py:134: RuntimeError: failed to move the previous model at 'out' aside (...); it is
unchanged, and the newly merged model is intact at '...\.fuse_n6trztvn' - move it into place manually
Why: Windows cannot rename a directory that contains an open file, so with a handle open on out/model.safetensors the rename-aside always fails. Since 67af97b2 that failure is re-raised as RuntimeError, but the test still says except OSError: pass, so the RuntimeError escapes. Because the test is Windows-only it is skipped on Linux and macOS, so it could not fail on your machine. It is not flaky: all three Windows cells fail the same way, the previous head 2fb1212e (before the new commit) was green on all three, and I reproduced it on a Windows box (10 passed, 1 failed, same message; also on the merge with current main).
The code is right; what was wrong is the test, which is also too weak. In this scenario the guarantee is that the old model is byte-identical, and the merged model survives at the path the error names. Replace the try/except OSError and the old_intact or new_intact assertion with exactly that:
handle = open(out / "model.safetensors", "rb")
try:
# Windows refuses to rename a directory that holds an open file, so
# the rename-aside fails: a RuntimeError that names the staged model.
with pytest.raises(RuntimeError, match="move the previous model") as excinfo:
adapter_fuse.merge_adapter_to_dense(
base_model="base", adapter_dir=str(adapter), out_dir="out"
)
finally:
handle.close()
# The old model is complete and untouched ...
for name in ("config.json", "model.safetensors", "tokenizer.json"):
assert (out / name).read_bytes() == b"OLD"
# ... and the merged model survives at the path the error names.
staging_dirs = list(tmp_path.glob(".fuse_*"))
assert len(staging_dirs) == 1, staging_dirs
assert (staging_dirs[0] / "model.safetensors").read_bytes() == b"NEW"
assert staging_dirs[0].name in str(excinfo.value)
assert not list(tmp_path.glob(".out.old-*"))I ran this replacement: your file goes to 11 passed on Windows, and the 10 mutations above are still all killed with it in place. (Changing only except OSError to except (OSError, RuntimeError) would also turn CI green, but it keeps the weak assertion.)
Smaller items, worth doing in the same push:
adapter_fuse.py, theexcept BaseException as backup_exc:around the rename-aside (andas restore_exc): a Ctrl-C during the rename becomes aRuntimeErrorsaying "it is unchanged". I ran it: withos.replaceraisingKeyboardInterrupton the aside rename, the caller seesRuntimeError, notKeyboardInterrupt. Useexcept OSErrorin both places, so an interrupt keeps propagating as one. Your tests injectPermissionError, so they still pass.- Stale text this change made wrong:
src/soup_cli/commands/draft.py:289(the_guard_output_targetdocstring saysmerge_adapter_to_dense"doesrmtree(out_dir)thenos.replace") and the comment atsrc/soup_cli/utils/adapter_fuse.py:108("otherwisermtree(out_dir)also hits error 1224"; it is a rename now). Themerge_adapter_to_densedocstring also does not yet mention the third outcome you added (rename-aside fails:out_dirunchanged, staging kept and named), and the issue asks for the docstring to say what the swap guarantees. reward.py, the finalos.replace(candidate_path, output):outputis validated once at the top, before load and calibration, whereas the oldatomic_write_textre-validated immediately before writing.os.replaceover a symlink replaces the link itself, so I did not find an escape, butmerge_adapter_to_densealready re-validates right before its swap for exactly this reason. Oneenforce_under_cwd_and_no_symlink(output, "output path")inside the existingtry, just before the replace, matches that pattern.
The fragment and the credit line are fine. Once the Windows test is fixed and the required checks are green, this is ready.
Thanks also to @AmirF194, whose earlier review of the rename-aside failure on 2fb1212e is what the change in 67af97b2 answers; that exchange is worth reading for anyone touching this code.
… docs, re-guard reward output Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
|
Pushed ad338ea with the Windows test replaced by your snippet, both fuse handlers narrowed to OSError, the stale docstring and comment text fixed with the third outcome documented, and the output re-check added before the final os.replace in reward |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, review of ad338ead. Everything from the last review is in, and this is merge-ready: every required check is green on this head, including the three Windows cells that were red.
The blocker is closed, checked by running it. The open-handle test now asserts exactly what the scenario guarantees: a RuntimeError that names the staged model, the old model byte-identical in all three files, the merged model at the path the error names, and no .out.old-* left behind. I ran your file on a Windows machine: 11 passed, including that test. CI shows test (windows-latest, 3.10), 3.11 and 3.12 all green on this head.
The three smaller items:
except OSErrorin both fuse handlers (adapter_fuse.py:134and:155). Withos.replaceraisingKeyboardInterrupton the aside rename, and again inside the restore, the caller now seesKeyboardInterrupt.- Stale text. The
draft.py:289docstring, theadapter_fuse.py:114comment and themerge_adapter_to_densedocstring (now with the third outcome and the guarantee) are all correct. - The re-check before the final replace (
reward.py:246). I planted a symlink atreward.pywhile calibration was running: the command exits 1, and the link and its target are untouched.
What I ran (CPU, your head merged with current main, a clean merge; ruff check src/soup_cli/ tests/ scripts/ benchmarks/ is clean):
- Your file plus
test_v07133,test_v07140,test_v07201, the startup-lightness test, the fragment validator and the #748 consumer guard: 367 passed, 0 failed. Four more draft and shrink neighbour files: 145 passed. - 14 mutations of the fix against your file. These 11 fail a test:
rmtree(out_dir)instead of the rename-aside (5 failed), no restore after a failed swap (2),if not keep_stagingturned intoif True(3), nokeep_stagingon the aside-rename failure (2), the aside copy never removed (1), the refusal path deleting-o(1), the calibration-error path deleting-o(2), the final-replace guard unguarded (1), a refusal leaving the candidate behind (1), the candidate written straight to-o(4), and a failed candidate write leaving the candidate behind (1). - The 3 that pass are the three items above:
except OSErrorback toexcept BaseExceptionat either handler, and the re-check before the final replace removed. They are hardening you added on request, and the tests do not pin them yet. - On
mainyour file gives 9 failed / 2 passed. The 2 that pass are the accepted-path test and the candidate-write-failure test.
Optional follow-up, not blocking. If you want those three pinned, these tests do it. I ran all three: they pass on your head, and each of the three mutations fails one of them. They use the helpers already in your file (_fake_train_stack, _fresh_adapter_and_out, _write_jsonl):
def test_an_interrupt_during_the_rename_aside_keeps_propagating(self, tmp_path, monkeypatch):
from soup_cli.utils import adapter_fuse
monkeypatch.chdir(tmp_path)
_fake_train_stack(monkeypatch)
adapter, out = _fresh_adapter_and_out(tmp_path)
real_replace = adapter_fuse.os.replace
def _interrupt(src, dst, *a, **kw):
if ".old-" in os.path.basename(dst):
raise KeyboardInterrupt
return real_replace(src, dst, *a, **kw)
monkeypatch.setattr(adapter_fuse.os, "replace", _interrupt)
with pytest.raises(KeyboardInterrupt):
adapter_fuse.merge_adapter_to_dense(
base_model="base", adapter_dir=str(adapter), out_dir="out"
)
assert (out / "model.safetensors").read_bytes() == b"OLD"
def test_an_interrupt_during_the_restore_keeps_propagating(self, tmp_path, monkeypatch):
from soup_cli.utils import adapter_fuse
monkeypatch.chdir(tmp_path)
_fake_train_stack(monkeypatch)
adapter, out = _fresh_adapter_and_out(tmp_path)
real_replace = adapter_fuse.os.replace
def _swap_fails_restore_interrupted(src, dst, *a, **kw):
name = os.path.basename(src)
if name.startswith(".fuse_"):
raise PermissionError(5, "Access is denied")
if ".old-" in name:
raise KeyboardInterrupt
return real_replace(src, dst, *a, **kw)
monkeypatch.setattr(adapter_fuse.os, "replace", _swap_fails_restore_interrupted)
with pytest.raises(KeyboardInterrupt):
adapter_fuse.merge_adapter_to_dense(
base_model="base", adapter_dir=str(adapter), out_dir="out"
)and, in TestRewardSynthForceKeepsPreviousOnFailure:
def test_output_swapped_for_a_symlink_during_calibration_is_refused(
self, tmp_path, monkeypatch
):
from soup_cli.commands.reward import app
from soup_cli.utils import reward_synth as rs
work = tmp_path / "work"
work.mkdir()
outside = tmp_path / "outside.py"
outside.write_text("# outside\n", encoding="utf-8")
monkeypatch.chdir(work)
_write_jsonl(Path("refs.jsonl"), [{"answer": "4"}, {"answer": "6"}])
real_calibrate = rs.calibrate
def _plant_link_then_calibrate(*a, **kw):
try:
os.symlink(outside, "reward.py")
except (OSError, NotImplementedError):
pytest.skip("cannot create symlinks here")
return real_calibrate(*a, **kw)
monkeypatch.setattr(rs, "calibrate", _plant_link_then_calibrate)
res = self._runner().invoke(app, ["synth", "refs.jsonl", "-o", "reward.py"])
assert res.exit_code == 1, (res.output, repr(res.exception))
assert outside.read_text(encoding="utf-8") == "# outside\n"
assert os.path.islink("reward.py"), "the planted link was replaced by the candidate"
assert not list(work.glob(".soup.reward-candidate.*.py"))Two smaller observations, neither blocking:
draft.py:604still says "The merge REPLACES -o wholesale (rmtree + os.replace)". It is the same stale wording you fixed at:289.- If Ctrl-C lands inside the restore after a failed swap, the
KeyboardInterruptnow propagates, as intended. I ran it: the old model is left at.out.old-<pid>with noout/, and the staged merge is removed, becausekeep_stagingis still false at that point. Nothing is lost for good, but the new docstring's "at a path the caller is told" does not literally hold for that one window. Treatingbackup is not Noneaskeep_stagingin the outer handler would close it.
Thanks again to @AmirF194, whose review of the first rename-aside is what 67af97b2 answered.
|
Merged as |
commit 4e3ab348994dba22bff836fe255936b5212a14a2 Author: Alpamys Makazhan <[email protected]> Date: Fri Oct 2 22:40:37 2026 +0500 fix: soup init and the MCP plan snapshot refuse links at their write target (#1553) soup init wrote the starter config with Path.write_text, which follows a symbolic link or junction at the target, and a dangling link skipped the overwrite prompt. It now checks the target before prompting, creates a new file exclusively (O_CREAT | O_EXCL with O_NOFOLLOW on POSIX; a staging file renamed into place on Windows, where an exclusive create still follows a dangling link), opens an existing file through open_no_follow, and truncates it only after the open-time check has passed. A refusal names the path and exits 1; ordinary paths write the same bytes and mode as before. The MCP plan snapshot (.soup/mcp-runs/<run_id>/config.yaml) also refuses a link at the run directory itself, re-checks the parents after creating them, creates the run directory with exist_ok=False and config.yaml with O_EXCL through open_no_follow(check_parent=True), and requires the run id to be a single path component. Refusals stay path-free ExecutionErrors. Co-Authored-By: Claude Opus 5.5 <[email protected]> commit 12807838df1604f8873949ab2ca7340afaefd358 Author: Mohd Uwaish <[email protected]> Date: Fri Oct 2 18:24:07 2026 +0200 fix(config): refuse LoRA+ with model-only optimizers and GaLore at load (#1210) (#1255) `training.loraplus_lr_ratio` combined with `optimizer: apollo_adamw`, `lomo` or `adalomo`, or with `use_galore`, passed config load and failed only after the base model had been downloaded and loaded. LoRA+ builds its optimizer through `build_loraplus_optimizer`, which asks `Trainer.get_optimizer_cls_and_kwargs(args)` for it without a model, and transformers can only build those three optimizers (and GaLore) from one; the GaLore combination was refused only at runtime. A new `TrainingConfig._validate_loraplus_compat`, next to `_validate_lorafa_compat`, refuses both combinations at load with a message that names both fields, and suggests an optimizer transformers can build without the model. The runtime GaLore check in `peft_wiring.py` stays as a backstop. The refusal does not depend on the task, and LoRA+ with any other optimizer (`adamw_torch`, `adamw_8bit`, `adafactor`, ...) and each of the refused settings on its own still load. Users of LoRA+ with one of those optimizers now see the error from `soup train` in about a second instead of after the model download. `docs/peft-and-efficiency.md` gains an `Optimizers` bullet in the LoRA+ section, under the existing `Compatibility` bullet. Verified: `tests/test_issue1210_loraplus_model_only_optimizers.py` (36 tests: three refusals, the case-insensitive refusal, GaLore and seven controls, each under `sft`, `dpo` and `ppo`) passes, and 15 of them fail on `main` without the change; ten single-point mutations, including moving the refusal into an `sft`-only validator (10 failures), all fail the tests; 1333 related tests pass on the tree merged with `main`; `ruff` is clean. Closes #1210. commit ef44c0c0af75e9aefe3b109a17fe15ec843ec878 Author: Pernav Jain <[email protected]> Date: Fri Oct 2 21:38:54 2026 +0530 fix(notebook): make proof-4gb reproducible (#1545) `notebooks/proof-4gb.ipynb`, the "run it yourself" proof of 4 GB layer streaming, did not run on a fresh hosted multi-GPU runtime (Kaggle 2x T4) with a current `huggingface-hub`, for two reasons in its install cell. (1) `huggingface_hub` 1.32 keeps blobs in a cache-wide shared store that soup-cli 0.75.0's snapshot materialization treats as outside the cache and refuses (#1509, fix open in #1510); the install now caps `huggingface-hub<1.32.0`, which resolves to 1.31.0 and satisfies transformers' `>=1.5.0,<2.0`, with a comment saying why and when to drop it. (2) With two visible CUDA devices, layer streaming refuses to start because it cannot run under `nn.DataParallel` (`refuse_if_data_parallel` names `CUDA_VISIBLE_DEVICES=0`); the cell now sets `os.environ.setdefault("CUDA_VISIBLE_DEVICES", "0")` before torch is imported, so a caller who pre-selected a card keeps it. Only cell 3's source changes; the outputs, metadata and execution counts of all 19 cells are identical. `tests/test_issue844_notebook_pin_guard.py` gains two tests: the GPU is selected in a cell at or before the first cell that imports torch, and the install cell caps the hub below 1.32. Verified: the new tests are 2 failed / 7 passed on `main` and 9 passed with the PR; mutations deleting or relocating the device block, deleting the cap, or loosening it each fail a test; cell 3 was run in isolation with the `%pip` lines recorded and left the variable at `'0'` when unset and unchanged when preset. The two-GPU run itself was done by the author and not reproduced here. Closes #1543. commit c6271bffe35455c1e675800344dfd9f3c6215e82 Author: qwist1233-cpu <[email protected]> Date: Fri Oct 2 19:07:18 2026 +0300 fix(advise): print the ROI table when --probe-model implies --probe (#1448 slice) (#1521) `soup advise run <data> --probe-model X` (without `--probe`) paid for the live probe and then never printed it. `--probe-model` implies `--probe`, and the probe ran under the derived `run_probe = probe or probe_model is not None`, but the `ROI deltas (probe)` table was gated on the raw `--probe` option, so the measured baselines, LoRA estimate and base-model proximity were computed and thrown away. The table is now gated on `run_probe`, the same flag that runs the probe, at `src/soup_cli/commands/advise.py:333`. Users notice: `soup advise run data.jsonl --probe-model org/model` now prints the ROI table, as `docs/evaluation.md` already said it implies `--probe`; plain `--probe` still prints the table and a run with no probe flag still prints none. Verified: the new `tests/test_issue1448_advise_probe_model_roi_table.py` (4 tests, probe helpers monkeypatched) has 1 failed / 3 passed on `main`; with the PR, 228 passed / 1 skipped across it, `tests/test_v0717.py`, `tests/test_v0540.py` and the changelog-fragment tests. Four mutations of the gate each fail a test (`if probe:`, `if probe_model is not None:`, `if True:`, `if False:`). Part of #1448. commit 255a5ecce26204367d5ce72a96e19029c256c0fd Author: Jillur Rahman <[email protected]> Date: Fri Oct 2 22:05:04 2026 +0600 fix(distill): leave unaligned student positions out of aligned ULD (#1426) (#1541) Aligned ULD (`training.uld_strategy: wasserstein_aligned`, the strategy `soup draft distill` picks for every cross-tokenizer pair) mean-pools teacher logits onto student positions through a character alignment. A student position that aligns to no teacher token (a byte-fallback piece decoding to U+FFFD, an empty decoded piece, a student-only special token) got an all-zero teacher row, which softmaxes to a uniform distribution, and stayed in the loss mask. Against a confident student each such position scored about V/2 (7999.4 for the reproduction at a 32k vocabulary) while aligned positions scored near 0, so the loss was dominated by positions with nothing to distill and its gradient pushed the student toward maximum entropy there. `uld_aligned_loss` now counts a student position only when its alignment names at least one existing teacher position (`aligned_position_mask`, the same in-range rule `aggregate_aligned_logits` uses). That mask multiplies the label mask or the attention mask and, with neither, is the whole mask, so the position leaves both the sum and the denominator; a sequence with no aligned supervised position gives a finite 0.0. The distill trainer decodes the student's per-token strings with `skip_special_tokens=True`, as the teacher text already is, so a student-only special token decodes to "" and aligns to nothing. Users notice: distillation loss across different tokenizers no longer carries a large constant per byte-fallback piece, empty piece or special token, and the student is no longer pulled toward a uniform distribution at those positions. With no `labels` and no `attention_mask` the aligned loss is now the mean over aligned positions instead of over every position; with every position aligned the value is bit-identical (800 old-vs-new comparisons across four mask modes). Verified: the issue reproduction gives 7999.4 on main and 0.0 on the branch under labels, attention mask and no mask; a batch of rows with different alignments and an all-unaligned row stays finite with finite gradients; 15 new tests (11 fail on main, 4 are controls) including a trainer-level test through the real `_DistillTrainer.compute_loss`; 331 passed across the new file and six neighbouring ULD/distill files, 565 across nine more; ten mutations of the fix, each caught. Closes #1426. commit ed50391a1711f6b5fc9903c785d9b4b3ecb9bb81 Author: Urooj Fatima Mukhtar <[email protected]> Date: Fri Oct 2 14:40:32 2026 +0500 fix(eval): make contains scoring linear again (#1528) (#1540) The custom-eval `contains` scorer stopped being linear when #1490 made it match `expected` only as a whole token. Its pattern began with a lookbehind and ran under `re.IGNORECASE`, which keeps `re` off its literal-prefix search, so the search restarted at every position and compared up to `len(expected)` characters at each word start: time proportional to `len(output) * len(expected)`. On repetitive token-separated text a 200,000-character output against a 100,000-character `expected` took 27 s on `main` (68 s at 400,000 against 200,000, as measured in the #1490 review), where v0.75.1's substring test took 0 s. `score_contains` now lowercases both sides once and puts the escaped literal first, so `re` finds candidates with its literal-prefix search. The left boundary is a fixed-width lookbehind after the literal and the right boundary is the existing lookahead, so each candidate costs O(1). The token bound from #1490 is unchanged and nothing is truncated: a gold token at character 60,000 of a 60,016-character output still scores `True`. What users notice: scores are unchanged for ASCII text, and the same 200,000-character case takes about 0.13 s. Case folding is `str.lower()` on both sides, as in v0.75.1 (the regex had not shipped in a release), so `istanbul` no longer matches `İstanbul` and `s` no longer matches `ſ`; the `contains` row in docs/evaluation.md says so. Verified: the 8 new tests fail 4/8 on `main` and pass on the branch; test_issue1342 (30) unchanged; 620 tests across the 11 files that touch contains-scoring pass; 120,000 ASCII cases agree with #1490's regex and 320,000 cases (Unicode included) agree with an explicit overlapping scan; 16 mutations of the function all fail the new tests; growth is linear on 8 input shapes up to 400,000 characters. Closes #1528. commit 6879f323f7c8b893c3679abd53a34d61f3616fc1 Author: Srinivasan R <[email protected]> Date: Fri Oct 2 14:57:53 2026 +0530 fix(expect): refuse top-level suite keys other than expectations (#1483) (#1499) `soup expect` read only the `expectations` key of a suite and dropped every other top-level key, so a suite with `fail_fast: true` (or any label or typo) beside `expectations:` loaded, ran and exited 0 with no message: the gate that ran was not the gate the user wrote. #1461 had already made `parse_suite_spec` refuse unknown keys inside an entry and unknown argument names; this closes the same gap one level up. `parse_suite_spec` now refuses every top-level key other than `expectations` before it reads anything else, naming each key through #1461's `repr()`-based `_key_repr` so a hostile key cannot reach the terminal as a control sequence, and `soup expect` exits 3 before the data file is opened. Users with a suite that carried a label such as `name:` beside `expectations:` now get exit 3 with the key named instead of a silent pass (documented in `docs/compliance.md`); one existing fixture in `test_issue1231_expect_all_formats.py` was such a suite and loses that line. Verified on the PR merged with main: 270 passed across the expect, gate-exit-code and ANSI-safety files, the new tests give 8 failed / 2 passed on main, and mutations removing the check, the `repr()`, the all-keys naming, or widening the allowlist to `name`, `description`, `version` or an `expectation*` prefix are each caught. Real CLI: `fail_fast:` and `name:` beside `expectations:` go from exit 0 to exit 3. Closes #1483. commit 8fcd8b90e320cf99d2abd3816b334611b1460707 Author: Alpamys Makazhan <[email protected]> Date: Thu Oct 1 23:33:22 2026 +0500 ci: run the full matrix at approval (ci:full), a quick subset on other PR pushes (#1381) A push to a pull request now runs a quick subset of `ci.yml` (`lint` and `test (ubuntu-latest, 3.12)`); the full matrix runs on pushes to `main` and `release/**` and on pull requests that carry the `ci:full` label, which a maintainer adds at approval. Why: the Actions account runs at its free-plan cap (20 concurrent jobs, 5 on macOS). Measured over 1,906 runs since 2026-09-19, one full `ci.yml` run costs 358 job-minutes, saturated service is 61-71 runs a day while weekday demand is 87-99, and most of that demand was review iterations: every push to a pull request paid for the full matrix even while the head waited on its author. Running the full matrix once per pull request, at approval, is worth about x1.36 of capacity on the binding limit; dropping Windows and macOS on 3.11 from the full run (they stay in the release matrix) takes a full run from 4 macOS jobs to 3. How: `pull_request.types` adds `labeled` (still a plain `pull_request`, never `pull_request_target`); a new `plan` job declares the RELEASE, FULL and QUICK matrices once and `test` takes `fromJSON(needs.plan.outputs.matrix)`, so a quick run never creates the other required `test (...)` contexts and branch protection keeps them Expected: nothing can be merged on a quick run. The concurrency key gives a `labeled` event for any label other than `ci:full` its own group, so labelling cannot cancel a running full matrix. `CONTRIBUTING.md` tells contributors what they will see. Verification: `tests/test_ci_full_matrix_on_approval.py` pins the decision table, the matrices, the gated jobs, the concurrency keys and the decide step's shell (the matrix literals are parsed by `tests/test_requires_python_bound.py`); its own full matrix ran green on this head (11 of 11 required checks). commit 4142fdd89889989e08f38de81394392a242af74a Author: UmranPros <[email protected]> Date: Thu Oct 1 23:49:08 2026 +0530 fix(train): a blend that cannot be computed refuses instead of scaling one loss (#1425) (#1513) `training.preference_loss_weights` promised a runtime blend of 2-5 preference losses, but no trl 0.29 trainer puts per-sequence log-probs on the batch, so the term loop never ran and the combine wrapper silently fell back to `primary_loss * primary_weight`. A config such as `{dpo: 0.7, simpo: 0.3}` printed `Multi-objective preference loss: 0.70·dpo + 0.30·simpo` at setup and then trained DPO alone scaled by 0.7 (measured on a real trl 0.29.1 trainer: exit 0, `Loss: 0.4852` = 0.7 x 0.693147), with nothing in the output saying so. The fallback is deleted (with the now-dead `_pick_primary` and the `primary_loss` unpack): when a requested term cannot be computed the wrapper raises, naming the terms it could not compute, the terms it did compute, and what was requested. What users notice: every `preference_loss_weights` config now stops at the first step instead of training one loss under a blend's name; to train one loss, remove the weights and set `training.preference_loss`. The weighted-sum path itself is unchanged, so a trl that does carry log-probs on the batch would still blend. `docs/training.md` no longer claims a true weighted sum. Verified on the PR merged with main: 183 passed across the five preference/combine files, a real-trainer test (tiny random-init Llama, CPU, two blends) that passes on the PR and fails on main, and mutations restoring the fallback, naming every term as missing, emptying the computed list, accepting a partial blend and changing the exception type each caught. This is partial progress on #1425: the real blend (deriving log-probs from the trainer's own forward pass) and the "changing only the secondary loss changes the loss" check are not in this change. Refs #1425. commit 909d86c48a8de062ebc1e48704fc0cb174980700 Author: Alpamys <[email protected]> Date: Thu Oct 1 23:02:17 2026 +0500 docs(contributors): #1496, #1481, #1482, #1514 [skip ci] Display names from `gh api users/<login>`. commit 92e6a1fd00405c10f60c2080bf6e04b2b43526bd Author: UmranPros <[email protected]> Date: Thu Oct 1 23:30:50 2026 +0530 fix(train): refuse the RL-signal flags PPO cannot feed (#1441) (#1514) fix(train): `task: ppo` refuses the reward-hacking and echo-trap flags it cannot feed (#1441) The trl PPO trainer this build supports (`trl.experimental.ppo.PPOTrainer`) takes no `reward_funcs`: its reward comes from the reward model it calls internally. Soup still built the callable reward wrappers and a shared signal buffer at setup and then dropped them, so the buffer was never written, and because the buffer object existed the detector's `on_log` fallback returned early too. A run with `reward_hack_detector`, `reward_hack_mitigation` or `echo_trap_enabled` was announced, loaded, and then never halted, mitigated or trapped anything. `PPOTrainerWrapper.setup` now refuses these three flags (the exact set that makes `rl_callbacks_need_buffer` create the buffer) on the experimental trl API, before `datasets`, the model, the reward model or the tokenizer is loaded. The message names every flag that was set and points at `task: grpo`, where the buffer is fed. `reward_hack_mitigation: off`, the other `reward_hack_*` / `echo_trap_*` tunables (already refused at config load unless one of the three is set) and the transitional trl API are unaffected, and none of the 200 shipped configs, templates and recipes is refused. `docs/training.md` and `docs/commands.md` no longer say PPO is supported for these flags. Feeding the buffer from the PPO rollout and the meaning of `reward_fn` on PPO are separate follow-ups. Verified: 21 tests in `tests/test_issue1441_ppo_refuses_unfed_rl_flags.py`, including setup-level tests that stub the loaders and assert nothing loaded, and a drift test pinning `_unsupported_rl_flags` to `rl_callbacks_need_buffer`; 11 mutations of the refusal (call removed, condition inverted, each flag dropped, refusal moved after each load, never raising, always raising, no flag names, drift) all fail a test; the neighbouring PPO suites (222 tests with the changelog and backend-support guards) pass on the merge with current `main`; and a real `soup train --config soup.yaml --yes` with all three flags stops at "Setting up model + trainer..." with the new message, while the same config without them is not refused. Closes #1441. commit 5c335b93c3e4993ead7e3f6ef552d4c3ad4f870e Author: Alpamys Makazhan <[email protected]> Date: Thu Oct 1 23:00:43 2026 +0500 fix(net): harden outbound endpoint validation (#1527) Outbound endpoint checks for the vLLM provider, `soup data generate --api-base`, judge URLs (flags, eval-gate suites, `soup ship --judge-model`, `training.online_dpo_judge`) and the Web UI chat proxy share one helper, `net_guard.refuse_private_ip_literal`, which refuses a private, link-local or reserved IP literal on every scheme. Loopback stays allowed; hostnames are not resolved. `0.0.0.0` is no longer loopback for `soup data generate` and the chat proxy. The shared predicate also counts 100.64.0.0/10 and fec0::/10 as non-public, and the shared parser classifies a non-ASCII host in the ASCII forms an HTTP client connects to (stdlib IDNA 2003 and the idna package), so webhooks, the OTLP endpoint, telemetry, hub endpoints and `soup ingest --pull` inherit both. Also: `soup ship` checks `--judge-model` before it builds the base and tuned models; the chat proxy answers 400, not 500, for an endpoint that does not parse; `online_dpo_judge` names itself on an unparsable URL. Docs say which addresses each setting accepts and name the two ranges. Tests: `tests/test_outbound_endpoint_ip_literals.py` (every entry point through its real surface, a differential test against `httpx.URL(url).host`, no network I/O) and the extended #616 ratchet. No changelog fragment: the change is recorded in the 0.75.2 section of CHANGELOG.md, which main gains at its next release. Co-Authored-By: Claude Opus 5.5 <[email protected]> commit 1d0e7384dcb5bfa9e900d6632e6efb0685c5b447 Author: Areen Agrawal <[email protected]> Date: Thu Oct 1 23:30:16 2026 +0530 fix(lora): map mistral3 target_modules and add modality to mistral-medium-3-5-sft (#1395) (#1482) `mistral-medium-3-5-sft` could not attach LoRA: its base, `mistralai/Mistral-Medium-3.5-128B`, is `Mistral3ForConditionalGeneration` (a Pixtral vision tower plus a `ministral3` text tower), but the recipe had no `modality: vision`, so SFT loaded it with `AutoModelForCausalLM`, which does not know `Mistral3Config` (`soup recipes verify` failed at stage `build`), and `training.lora.target_modules: auto` had no row for `mistral3` (stage `resolve`: no mapping for model_type ['ministral3']), so the architecture was refused even with the right loader. The recipe now sets `modality: vision` (SFT loads the image-text class), and `MOE_TEXT_LORA_TARGETS` gains a `mistral3` row: `.*language_model\..*\.self_attn\.(q_proj|k_proj|v_proj|o_proj)`, scoped to the language tower so the Pixtral vision tower, the multimodal projector and the MLP projections stay unadapted (a plain `q_proj/k_proj/v_proj/o_proj` suffix list would have adapted 16 modules, 8 of them in the vision tower). Measured on a 2-layer meta-device skeleton of the real config: 8 modules adapted, 0 in the vision tower, 0 experts. The recipe's `EXCEPTIONS` pin in `scripts/check_recipe_attach.py` is removed (the ratchet reports "0 pinned that now attach" without it, and flags the pin if it is put back), the recipe snapshot gains the `modality` line, and docs/peft-and-efficiency.md lists `mistral3` among the architectures `target_modules: auto` maps. The recipe count does not change. Verified: `soup recipes verify --config` on the real recipe moves from "0 attach, 1 cannot" (exit 2) to "1 attach, 0 cannot" (exit 0); 30 new tests (20 fail on the previous tree; the other 10 are documentation tests that build their own regex), 731 passed across the related files on the tree merged with main, `scripts/check_recipe_attach.py` exits 0 (and exits 2 when the pin is put back), and seven mutations of the fix are all caught (delete the row, drop the leading `.*`, plain suffix tuple, unscoped regex, drop `o_proj`, drop `modality: vision`, drop the snapshot line). Closes #1395. commit be0049993e2fa93ab26205cf6721d983aeb08547 Author: fukalous <[email protected]> Date: Fri Oct 2 01:59:43 2026 +0800 fix(diagnose): detect memorization in long rows (#1481) `soup diagnose` could not see memorization on long training rows. The live probe asks the adapter to continue the first quarter of a row and caps the generation at 64 tokens, but the memorization score compared that capped completion with the whole held-out suffix by set-Jaccard, so the longer the row the more the untouched suffix diluted the overlap: a verbatim 48-word echo read MAJOR on a 60 or 100-word row and OK from 150 words up, and a model that had memorized its long rows passed with a clean 1.00. The overlap is now clipped completion-bigram precision against the suffix: the share of the completion's adjacent token pairs that occur, counted at most as often as they occur, in the suffix. The default path pairs adjacent normalized words (stopwords and short ASCII words are still ignored, as before); `--tokenizer` pairs adjacent sub-word tokens. Bigrams, not single words, are what keep the score honest: a single-word precision made unrelated text flag as MAJOR more often the longer the row (59 percent of unrelated completions at 4000 words of real prose), flagged one-word completions, and treated a repetition loop of a common word as a verbatim echo. What users notice: a verbatim echo now reads MAJOR at every row length (60 to 4000 words, default and `--tokenizer`, Latin, Cyrillic and CJK text), and an unrelated continuation, a short completion or a repetition loop stays OK. The 0.5 threshold now means bigram precision rather than set-Jaccard, so a partial echo can read MAJOR slightly earlier than before (on a 60-word row a completion copying 55 percent of its words in order flips, where set-Jaccard needed 67 percent), and memorization results saved by earlier releases are not directly comparable; the changelog and `docs/evaluation.md` say so. Verified on the PR head merged with main (CPU): 34 new tests (13 of them fail on clean main) and 406 passed with the neighbouring diagnose and changelog-fragment tests; 7 of 8 mutations of the scoring killed (the survivor is the pre-existing `>=` versus `>` boundary at exactly the threshold); real-prose runs (docs/*.md, SmolLM2 tokenizer, 120 rows per length) show verbatim 1.00 and unrelated 0.00 to 0.03 from 60 to 4000 words on both paths. Closes #1437. commit b9ad2fb4f7d42b9a76ff417d000f8336abe6bd54 Author: Alpamys Makazhan <[email protected]> Date: Thu Oct 1 20:33:45 2026 +0500 test(serve): pin that --device is refused on every backend, not only transformers (#1478) (#1523) The review of #1479 (merged as `0f7ab958`) ran 24 mutations of the `soup serve --device` fix and one survived: validating `--device` only when `backend == "transformers"` passed every test, because the refusal tests all ran on the default backend. That mutation would be a real regression (an unknown device reaching vLLM/SGLang unchecked). This adds one parametrised case to `tests/test_issue1478_serve_device_map.py`: `--device tpu` must exit 2 on `--backend vllm` and `--backend sglang`, before either server is built or `uvicorn.run` is called. Mutation check: guarding the validation block in `serve.py` with `and backend == "transformers"` makes the two new cases fail (2 failed, 34 passed); unmutated, 36 pass. Test-only, no source change. commit 4d14b3ddc246f5fd2f3f5255c872d5ecfaa8f3dc Author: Chaitanya Reddy <[email protected]> Date: Thu Oct 1 19:47:24 2026 +0530 fix(data): detect format from a bounded prefix, not only the first row (closes #1424) (#1496) `format: auto` chose a dataset's format from the first row alone, so a first row that happened to lack an optional key picked the poorer converter for the whole file and silently dropped that key from every row that had it: a LLaVA file whose first row was text-only loaded as `sharegpt` and lost every image, an OpenAI-shape tools file whose first row had no `tools` lost the tool-schema turn, and audio and ASR files lost their audio path. Detection now reads a bounded prefix (`MAX_DETECT_ROWS = 100` rows). Each row resolves to the richest format its keys fit; a uniform file detects exactly as before (a golden comparison over all 22 `.jsonl` files in the tree is identical to main apart from the wording "Cannot detect format at row 0"). A mixed file either upgrades to the richer format of the same family (today: `chatml` rows mixed with `tool-calling` rows load as `tool-calling`), but only when the richer converter accepts every poorer row and converts it to exactly the messages the poorer converter would (so a chatml row carrying message-level `name` or `weight` is refused rather than silently stripped), or it is refused with a message naming both shapes and the row where each was first seen and telling the user to set `data.format`. Rows that match no signature, and non-object rows, are skipped as the converters skip them instead of crashing on `.keys()`. What users notice: a mixed file that used to load with a key stripped, or with rows dropped under a warning, now stops at load; `soup data validate` exits 3 on it with the refusal. The Web UI data explorer detects from the same window and reports the refusal in a new `format_error` field instead of answering 500, and `docs/data.md` documents the 100-row window, the upgrade rule and the refusals. Verified on the PR head merged with main (CPU): 34 new tests (19 fail on main with only the constant added so the module imports), 815 passed across the 22 existing modules that call `detect_format` or the explorer, 15 of 18 mutations killed (the three survivors are ancillary lines; their tests are optional), ruff clean. Closes #1424. commit 8d8abc933a50056bb8ce850f1fcdf7620dfbf3ec Author: t <t@t> Date: Thu Oct 1 17:20:25 2026 +0500 docs(contributors): #1398, #1455, #1457, #1419, #1516, #1517, #1469, #1498, #1306, #1454 Display names from `gh api users/<login>`. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> commit 1a67520c07669acdcd3ed76afbacf75cef2e0dc8 Author: Younes Beriane <[email protected]> Date: Thu Oct 1 14:18:54 2026 +0200 fix: keep the previous artifact when reward synth or adapter fuse fails (#1454) Two commands replaced an existing artifact in an order that removed the old copy before the new one was known to be valid and in place, so a failed replace could leave no good copy at all. `merge_adapter_to_dense` (used by `soup shrink` to fuse the heal adapter into the pruned model, and by `soup draft distill` to write the dense draft) deleted `out_dir` and then moved the staged model in, and its error handler also deleted the staged model, so a failure between the two steps (an antivirus or indexer lock, disk full) left neither copy. It now renames the previous model aside, swaps the merged model in, restores the previous model if the swap fails, and removes the aside copy only after the swap succeeded. If the restore also fails, both copies are kept and both paths are named in the error; if the rename-aside itself fails, `out_dir` is left unchanged and the staged model is kept and named. Interrupts (Ctrl-C) are no longer turned into a "the old model is unchanged" error. The docstring says what the swap guarantees. `soup reward synth -o reward.py --force` overwrote the existing verifier with the candidate before calibrating it, then deleted the file when calibration refused or failed, which destroyed a previously working verifier. It now writes the candidate to a `.soup.reward-candidate.*.py` sibling, loads and calibrates that file, re-validates `-o`, and replaces `-o` only once calibration accepts. A refusal (exit 2), a calibration error (exit 1), a failed candidate write or a failed final replace leaves the previous verifier byte-identical and removes the candidate. Verified: 11 new tests (the Windows open-handle case runs on Windows); 367 tests across the related files pass on the PR merged with `main`; the 11 mutations of the fix all fail a test; the new file gives 9 failed / 2 passed on `main`. Closes #1450. commit 2bcdbc9a7c6f50d7fe40199d6e86753b4320484e Author: Nurkhan Esenbek <[email protected]> Date: Thu Oct 1 17:18:42 2026 +0500 fix(data): wire classifier, reranker, and cross_encoder data loading (#1219) (#1306) `soup train` could not train the sequence-classification family. For `task: classifier` and `task: reranker` the loader kept source columns only for `grpo`, so `label` was dropped during format normalisation and the run died after the model loaded with `TypeError: label must be int / str / list, got NoneType`. For `task: cross_encoder`, `{text_a, text_b}` and `{question, answer}` pair rows could not be loaded at all. `--dry-run` said `Ready to train!` for all of them. Source columns are now kept for the whole family (`task_preserves_source_columns`, shared by `soup train` and `soup sweep`). `data.format: cross_encoder` exists, and `format: auto` resolves to it only when `task: cross_encoder` (`data_config_for_task`); `detect_format` and `FORMAT_SIGNATURES` are untouched, so GSM8K-shaped `{question, answer}` rows are still refused for `grpo`, `sft` and `pretrain` exactly as before. `data.format: cross_encoder` on any other task is refused at config load. Every classifier-family row (train and val) is checked on `--dry-run` and before the model loads, and an error names the row (`train row 3: missing required 'label' field for task 'cross_encoder'`). When `training.label_names` is set, `id2label` / `label2id` are written to the saved `config.json`. `docs/data.md` and `docs/training.md` document the row shapes per task. Verified: 791 tests across 16 files pass on the PR merged with `main`; 13 of 13 mutations of the fix fail a test; the new pin file gives 6 failed / 4 passed on `main`. Closes #1219. commit 387ccb3c63d9d0998bec239890ffe0cfcc4f3ab9 Author: UmranPros <[email protected]> Date: Thu Oct 1 17:48:02 2026 +0530 fix(data): from-traces reads the signals Soup's own producers write (#1440) (#1498) fix(data): from-traces reads the signals Soup's own producers write (#1440) `soup ingest` writes a top-level `signal` (`thumbs_up` / `thumbs_down` / `none`) into the trace stream that `soup data from-traces` and the loop harvest stage are documented to consume, but `parse_soup_serve` read the signal only from a nested `feedback.rating` that no Soup producer writes, and no parser ever set `edited_output`. So every ingest row read as `none`, `--signal thumbs_up` and the loop harvest built no pair, `--signal user_edit` could never match any log, and the command ended with a green `Wrote 0 preference pair(s)` and exit 0. `parse_soup_serve` now reads the top-level `signal` when it names a value of the closed vocabulary (`thumbs_up`, `thumbs_down`, `regenerated`, `user_edit`; anything else, including non-string values, reads as `none`) and keeps `feedback.rating` as the fallback for older logs; a top-level `signal` wins over a conflicting `feedback.rating`. `edited_output` is read from `edited_output`, `edited_response` or a nested `feedback.edited_output` (non-string and whitespace-only values ignored). When `from-traces` reads traces but builds no pair it now prints how many it read, which `--signal` it wanted and which signals were present (or that no trace carried one); an empty directory stays silent, and the `soup ingest` record-shape hint is printed only for `--format soup-serve`, the one parser that reads a top-level `signal`. `docs/data.md` is corrected: `--format soup-serve` and `--signal regenerations` are the accepted spellings, `--logs` for the soup-serve parser is a directory, and the record shape is documented with who writes which signal (`soup ingest` writes `thumbs_up` / `thumbs_down` / `none` under `trace_id`; `user_edit` and `regenerated` rows come from your own pipeline; `soup serve --trace-log` records carry no signal yet). Users notice that the round trip `soup ingest --source langfuse` then `soup data from-traces --format soup-serve --signal thumbs_up` now writes the pair instead of an empty file, and that a log with no usable signal says so instead of reporting a normal write of 0. Not in this change: `soup serve --trace-log` records still carry no signal and the harvest does not read the `--record-thumbs` database, so the loop still cannot learn from Soup's own serving; that producer half stays open on the issue. A single-file `--logs traces.jsonl` with `--format soup-serve` still ends with a green `Wrote 0` (the parser reads a directory); documented, not changed. Verified: the new test file fails 11 of 16 on the parent commit and passes 16 of 16 on the change; 414 tests across 9 related files pass on the merged tree; ruff clean; the round trip, the three `user_edit` shapes, a serve-trace-log shaped directory and hostile `signal` / `edited_output` / `feedback` / `id` values were executed through the real CLI; 19 mutations of the parsers and the message were applied one at a time and 15 are killed by the tests (the survivors are the openai/langchain hint gating, `regenerated` clamp membership and the non-dict `feedback` guard, each checked by hand). Refs #1440. commit d73e281187fdc6cd59dc64c4fb6f6767f21a6120 Author: Kratos <[email protected]> Date: Thu Oct 1 20:17:48 2026 +0800 Fix #1445: validate compile / GGUF-export / unlearn destinations before the expensive step (#1469) Three commands validated their destination only after the expensive step and then threw the finished work away. `soup compile -o <path outside cwd, or a symlink>` ran the whole optimizer and its LLM calls, then refused the path with an unhandled `ValueError` (exit 1). A quantized GGUF export into a directory that did not exist yet crashed in `tempfile.mkdtemp` after the adapter had been merged, and the cleanup deleted the merged model. `task: unlearn` loaded the policy and a frozen reference copy and ran every NPO / SimNPO / RMU step before refusing an `output` outside the working directory or a symlinked one. `build_compile_plan` now runs the containment and symlink check at plan time, so the optimizer is never called and the command exits 2 (the write-time check stays as the TOCTOU guard, and its `ValueError` is caught and exits 2 as well); `soup compile --plan-only` now refuses such a destination too instead of printing the plan and exiting 0. The export creates the destination's parent directory before the merge. `UnlearnTrainerWrapper.setup()` calls `_validated_output_dir` first, before torch, peft or any model is loaded; the check after the run stays as the TOCTOU guard. Verified: the new test file (9 tests, each with an in-cwd control) fails 6 of 9 on `main`; 4 of 4 mutations of the fix are killed (export `mkdir`, plan-time check, write-time `except`, unlearn `setup()` call); 1362 related tests pass; the real CLI run before and after for all three commands, including a Windows junction pointing outside the working directory. Closes #1445. commit 1bc20670c3c0245fb4a09f34b6e57340061e6960 Author: Mohammad Hijjawi <[email protected]> Date: Thu Oct 1 13:02:14 2026 +0100 fix(local-rl): render --min-pairs and -o into the nightly-train job (#1448) (#1517) Without `--once`, `soup local-rl train` renders a systemd `.service` / `.timer` and a launchd `.plist` that run `soup local-rl train --once` every night, but the rendered command carried `--db`, `--model` and `--train-method` only. The nightly run therefore trained on as few as 10 pairs into `local_rl_adapter/` whatever `--min-pairs` and `-o` the user had passed, and nothing said so. `build_train_argv`, `render_systemd_service`, `render_launchd_plist` and `write_scheduler_files` now take `min_pairs` and `output_dir` and always render `--min-pairs N --output DIR`, and the CLI passes both through, so the job does what `--once` with the same flags does. Both values are checked with the runner's own validators (`validate_min_pairs` / `validate_output_dir`, moved out of `run_nightly_train`), so a value the nightly run would refuse, such as `--min-pairs 0`, exits 2 before any scheduler directory or unit file is written, and a newline in the output path is refused like `--model` and `--db`. `docs/commands.md` lists the flags in the scheduled-mode line. `--backend` (the other half of this bullet in #1448) is unchanged. Verified: 14 new tests (12 fail on `main`), 309 passed across the neighbouring local-rl and docs-ratchet files, the issue's repro through the real CLI renders `--min-pairs 200 --output adapters/nightly` into both the `.service` and the `.plist`, and 15 of 16 mutations fail a test. Part of #1448. commit 65265a8b86abae367a3e5a0ad430749fb9f463d5 Author: Mohammad Hijjawi <[email protected]> Date: Thu Oct 1 13:01:56 2026 +0100 test(peft): count the MLX LoRA builder in the LoRA+ wiring scan (#1324 follow-up) (#1516) The #724 wiring scan counted only `get_peft_model(` and a `peft_config=` handed to trl as "builds a LoRA adapter", so `mlx_sft.py`, which builds its adapter with mlx-lm's `linear_to_lora_layers`, was neither wired nor exempt. That is how `backend: mlx` ignored `training.loraplus_lr_ratio` without a test noticing (#1324). The scan now counts `linear_to_lora_layers`, `mlx_sft.py` is exempt with its reason (the config-load refusal from #1383), and a new test fails when an exemption reason names a test file that does not exist. `tests/test_issue1324_mlx_loraplus_refusal.py` gains the cases #1383's file lacked: every ratio value, direct `SoupConfig(...)` construction, an explicit `null` ratio, the `transformers` / `unsloth` controls on `pretrain` and `dpo`, plain `soup train` stopping at config load, and `soup doctor --config` exiting 2. `docs/peft-and-efficiency.md` states that `backend: unsloth` gets the same LoRA+ optimizer as `transformers`. No source change and no behaviour change. Verified: 135 passed on the tree merged with `main`; six mutations (scan regex, exemption removed, exemption pointing at a renamed file, docs sentence, #1383 refusal deleted, refusal widened to `unsloth`) each fail 1 to 9 tests. Follow-up to #1324 / #1383. commit f47d6668ba0e08f790a7594ebd2f0ba8ff58c14c Author: Abdulwaarith Zakariyya <[email protected]> Date: Thu Oct 1 14:59:03 2026 +0300 fix(ab): decide soup ab with a variance-robust NIG Bayes factor, no burn-in (#1265) (#1419) `soup ab` estimated the row-to-row variance from the rows it was testing, so the #1227 two-sided mSPRT needed a burn-in of 30 or 40 rows per arm, was calibrated only up to 1000 rows and down to `--alpha 0.01`, and still sat near 0.011 at 0.01. It now decides with the normal-inverse-gamma mixture Bayes factor (right-Haar limit), which depends on the rows only through the two-sample t statistic, so its false-positive rate stays at most `--alpha` however often `soup ab` is re-run: the worst simulated rate is 0.033 at `--alpha 0.05` and 0.0066 at 0.01, against 0.051 and 0.011 before (22 ratios, 100,000 runs each; record and harness in `benchmarks/gate-1265-ab-nig.md`). The first 5 rows per arm set the scale for `--effect-size` and show as `prior_scale`; the burn-in table, the 1000-row limit and both calibration warnings are gone. The cost is stated in the docs and the fragment: with no difference, `accept_h0` takes several times as many rows (281 pairs against 43 at 0.5 standard deviations), and the "under 0.4%" wrong-accept figure is for simulated Gaussian rows. When the held-out rows barely vary (a warm cache, a judge that gives 1.0 early), `accept_h0` is held at `continue` while the tested rows' standard deviation is more than 3 times the held-out one (`ACCEPT_HOLD_SPREAD_RATIO`); a `reject_h0` is never held back, so the false-positive bound is unchanged. In a 2000-run simulation that start had ended 99.0% of runs in a wrong `accept_h0` at about 7 pairs, and with the hold it ends none that way; on Gaussian rows 15 of 6000 first verdicts change and no rejection does. The `--alpha + --beta < 1` refusal from #1339 stays with its reason restated, and #1339's and #1384's refusals of values a float cannot carry cover the new statistic, which is never compared with the boundaries unless finite; an `--effect-size` past about 1.9e151 standard deviations of the held-out rows, or held-out rows whose variance underflows to 0, is refused at every row count. Verified on the head merged with `main`: 208 passed across the four `soup ab` files and three neighbours, 16 of 17 mutations of the hold and the prior bound killed (the survivor, `>=` for `>`, differs only at exact float equality), the harness self-check matches the shipped code, and the new tests fail on `main` (50 failed / 12 passed in `tests/test_issue1265_ab_nig.py`). A faster accept rule is #1418. Closes #1265. commit 0ae08c7fbbf40f4744d90bdb7129b08147b4c780 Author: Kratos <[email protected]> Date: Thu Oct 1 19:56:14 2026 +0800 Fix #1434: soup expect fails closed on BOM, malformed and non-object JSONL rows (#1457) `soup expect` could pass a dataset it never checked. It opened the file as plain `utf-8`, so a UTF-8 BOM broke the first line's JSON and that row was silently skipped (the issue's repro passed with `Rows 1` where the same file without the BOM failed on the PII in row 1). Malformed JSON lines were counted into a yellow "skipped" note, valid JSON that was not an object (`[1, 2]`, `"text"`, `42`) was dropped without a word, and a file with no checkable row at all passed with `Rows 0`. As a CI gate, that is a run that checked nothing and exited 0. The loader now reads `utf-8-sig`, like the training loader, so a BOM is stripped. Any malformed line or non-object line makes the run refuse with exit 3 (input error, distinct from exit 2, a real suite failure), naming the count and the first ten line numbers with `(+N more)`. A file that yields no checkable rows is refused the same way. `docs/compliance.md` and `docs/data.md` now state exit 3, the BOM strip and the zero-row refusal. Reusing the training loader so `.json`/`.csv`/`.parquet` are read too stays out of scope: a `.json` array file is refused (fail-closed) rather than read. Verified on the PR merged with current `main` (which includes #1461): the 9 new tests (all fail on `main`) plus the existing expect suites, 247 passed; `ruff` clean; twelve mutations of `commands/expect.py` (encoding, zero-row check, uncheckable check, both recording branches, count, line number, ten-line cap, `(+N more)`, exit code) all killed; by hand against `main`, a BOM file goes from exit 0 to exit 2, array-per-line, empty and malformed files from exit 0 to exit 3, and a clean file still exits 0. Merged cleanly together with #1499 and #1505, which touch the same module. Closes #1434. commit 8bfeaa2e82887d533a1400cdb289a9da2cd4eb88 Author: Kratos <[email protected]> Date: Thu Oct 1 19:56:02 2026 +0800 Fix #1429: `tokenizer train` saves a complete byte-level BPE (decoder, byte alphabet, no hard-coded unk) (#1455) `soup tokenizer train` saved a byte-level BPE that could not round-trip text. The trainer attached no `ByteLevel` decoder, so `decode(encode(x))` returned `Ġ` for every space and `h ll o Ġ Ġ` for `héllo 😀 世界`. It also did not seed the 256-byte initial alphabet, so characters unseen in the corpus (accents, emoji, CJK) encoded to `<unk>`. And it hard-coded `unk_token="<unk>"` into the model, so passing `--special-token` without `<unk>` saved a tokenizer that names an unk token missing from its own vocabulary, and encoding then raised `Unk token <unk> not found in the vocabulary`. The trainer now attaches `decoders.ByteLevel()`, seeds `ByteLevel.alphabet()` as the initial alphabet, and builds the model with no `unk_token`. `--vocab-size` is checked after the special tokens are known: it must be at least `256 + len(special tokens) + 1`, which is the true boundary (measured with `tokenizers` alone: any size up to 256 + specials yields a vocabulary with zero merges and no error, so `main` accepted `-v 256` and saved a merge-less tokenizer). Below that the command exits 2 and names the minimum. Users notice that ASCII, accented, emoji and CJK text round-trips through `Tokenizer`, `PreTrainedTokenizerFast` and `AutoTokenizer`, and that a custom `--special-token` run encodes and saves `model.unk_token = null`. Verified on the PR merged with current `main`: the 9 new tests (7 fail on `main`, the other 2 are the sizes `main` also accepts) plus `tests/test_v0539.py` and the changelog-fragment tests, 87 passed; `ruff` clean; eight mutations of `tokenizer.py` (decoder removed, initial alphabet removed, hard-coded unk restored, floor disabled, `+ 1` dropped, floor ignoring specials, wrong message number), each killed by the new tests; a manual `soup tokenizer train` at `-v 260` (exit 2) and `-v 261` (exit 0, exact round trip). `tokenizer_config.json` is deliberately left out, as the issue asks. Closes #1429. commit 4b5f4a3cda36d482278bd4400da51a204cca2b39 Author: Srinivasan R <[email protected]> Date: Thu Oct 1 17:05:42 2026 +0530 fix(recipes): drop the recipes that cannot load; MiniMax-M3 SFT loads through the vision path (#1145) (#1398) Four shipped recipes named a repo that exists but a model that cannot load as written (#1145). `mistral-large-3-sft` and `mistral-large-3-dpo` are removed: `mistralai/Mistral-Large-3-675B-Instruct-2512` publishes Mistral's native `params.json` + `consolidated-*.safetensors` layout with no `config.json`, which `AutoConfig` cannot read; `docs/models.md` now says so and names the Hugging Face-format conversion on the Hub as a fact, not a recommendation. `minimax-m3-dpo` is removed, because the DPO trainer builds `AutoModelForCausalLM` for every modality and cannot build the `minimax_m3_vl` wrapper (#1393). `minimax-m3-sft` now sets `modality: vision`, so SFT builds the image-text wrapper and `target_modules: auto` keeps LoRA on the language tower; it drops `moe_lora` (refused on SFT vision by #1179) and `moe_aux_loss_coeff` (read only on the text path), and because that path loads `AutoProcessor`, which is custom code for this base, its description says to run `soup train --trust-remote-code` (verified: the flag reaches the vision processor). The recipe YAML also notes that the vision collator supervises every token, so `train_on_responses_only` is not applied on this path (#1156). The catalog loses three recipes (173 -> 170, 112 bases); every count site, both recipe fixtures, the MoE tables in `docs/performance-and-quantization.md` (29 recipes set `moe_lora`: 14 SFT, 8 GRPO, 7 DPO; MiniMax 0, kimi 7) and the two attach-ratchet pins for MiniMax are updated. A new test pins the `docs/models.md` paragraph. Users notice: `soup recipes list` shows three fewer recipes, and `minimax-m3-sft` attaches LoRA (16 modules on the meta device) instead of failing at load. Verified on CPU: 2446 tests over 20 recipe, count, snapshot, ratchet and docs-guard files, 0 failed; `scripts/check_recipe_attach.py` exits 0 (118 verified, 18 pinned, 0 new, 0 now attach); 9 of 9 mutations fail a test; the PR's tests fail on unmodified `main` (14 failed, 642 passed). Closes #1145. commit aae7508e88f6f7261d2e001df86e987bc2d5e870 Author: Alpamys <[email protected]> Date: Thu Oct 1 16:26:16 2026 +0500 docs(contributors): fix Bakdaulet's entry, the first-contribution text for #1489 was missing The entry written with the batch credit repeated the name line instead of the description. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> commit d70398d5113b106eb2405b096dcae8a2b97cbfe6 Author: Alpamys <[email protected]> Date: Thu Oct 1 16:25:33 2026 +0500 docs(contributors): #1308, #1460, #1461, #1168, #1366, #1462, #1463, #1364, #1492, #1493, #1489, #1416, #1484, #1490, #1479, #1485, #1495 Display names from `gh api users/<login>`. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> commit 7e681f07a72e529ce26276af4d8350011a19c0b2 Author: Ashutosh Panda <[email protected]> Date: Thu Oct 1 16:54:00 2026 +0530 fix(tests): pin #1408 helper with regression tests (#1495) When the suite ran from inside a virtualenv whose base interpreter has none of the dependencies (the CONTRIBUTING setup on, for example, Homebrew Python), `test_worker_verification_survives_a_real_venv_launcher` failed with an empty `RuntimeError: MLX teacher worker exited with 1:`. The child venv is created with `system_site_packages=True`, which from inside a venv exposes only the base interpreter's site-packages, not the outer venv's, so the worker died importing pydantic before writing anything and the failure looked like a worker bug. The test now writes a `.pth` file into the child venv that lists the outer venv's site-packages and `src/` (a no-op on a bare interpreter, so the test is unchanged for everyone running from a global Python), and probes `import pydantic` in the child first, so a missing link names the missing dependency instead of surfacing an empty exit. Two hermetic tests pin the helper on every platform, including the POSIX layout on CI: a package that exists only in a fake parent site-packages becomes importable in a child venv after the helper runs and not before, and nothing is written on a bare interpreter. Test-only; no user-visible change. Verified: the test file gives 19 passed from a bare interpreter and from a venv over a dependency-free CPython; clean `main` from that venv gives 1 failed, 16 passed (the reported error); four mutations of the helper (never write the `.pth`, `src/` only, wrong suffix, inverted early return) each fail the new tests, and removing the helper's call is caught from the dependency-free venv with a message naming pydantic. Closes #1408. commit 200fc21af339705f960bdd997db7cd7da2d48862 Author: dymux <[email protected]> Date: Thu Oct 1 18:23:48 2026 +0700 fix(cli): write machine-readable JSON straight to stdout (#1480) (#1485) `soup version --json --full`, `soup cost --json` and `soup eval against --json-only` printed their machine-readable JSON through Rich's `console.print`, which folds text at the console width (80 columns when piped) and reads `[...]` in a string as markup. A `version --json --full` record was split across lines, a `cost --json` value such as `[v2]` was dropped while the command exited 0, and `[/x]` (in a cost value or an `eval against` baseline id) aborted the command with `MarkupError`. All three now write with `typer.echo`, the convention the other machine-readable paths use; the human-readable branches still go through the Rich console. A repo-wide AST ratchet fails on any `console.print(<serialiser>.dumps(...))` under `src/soup_cli/`, with a self-test so a detector that stops matching cannot pass as a clean tree. `soup eval against --json-only` is a third site the issue did not name; it is reached through a `_json` alias that a grep misses. Users notice: one JSON document on one line, and bracketed text in a value survives. Verified: the new file is 6 passed (5 failed / 1 passed on `main`); 355 passed and 1 skipped across 12 related files with `main` merged (the one other local failure, `tests/test_cli.py::test_version_full_json`, fails the same way on `main`); each of the three sites reverted on its own fails its own behavioural test plus the ratchet (2, 3 and 2 failures), and both detector mutations fail the self-test. A real piped `soup version --json --full` and `soup cost --json` both parse with `json.load`. Closes #1480. commit 0f7ab958525d619a7aaa55d46743c2130a52dad8 Author: Younes Beriane <[email protected]> Date: Thu Oct 1 13:23:37 2026 +0200 fix(serve): honour --device in the transformers loader (#1479) `soup serve --device` was ignored by the transformers loader: `_load_model` (both the merged-model and the base + adapter branch) and the speculative draft loader hard-coded `device_map="auto"`, so `--device cpu` could still put the model on a visible GPU, and an unknown value such as `tpu` was accepted and silently dropped (`--device "[/bold]"` died with a Rich `MarkupError`). `soup serve` now places the model with the same rule as `soup infer` / `chat` / `diff` (`resolve_inference_device_map_and_dtype`): `cpu` and `mlx` load on the CPU in float32, `mps` and `cuda:N` are pinned to that device in float16, `cuda` keeps `device_map="auto"` in float16, and an explicit `--kv-cache-type bf16` / `f16` still sets the dtype on every device. `--device` is validated before anything loads, on every backend, and an unknown value exits 2 naming `cpu, cuda, cuda:<index> or mps`. The draft model is placed like the main model. Users notice: `--device cpu` really runs on the CPU; on a CPU-only host the default load is now float32 pinned to the CPU, which doubles its RAM (a 7B model needs about 28 GB instead of 14 GB); `cuda:N` is honoured (it was accepted and ignored before); the `--device` help describes all of it. Verified: the new file is 34 passed (19 failed / 15 passed on `main`), 627 passed across 17 related files, 34 passed again under forced colour at 60 columns; 23 of 24 mutations of the fix fail a test (the survivor is validation gated on the transformers backend, which the CLI does not do on vLLM or SGLang); the real CLI and a tiny real Llama loaded on CPU give the placements and dtypes above. Closes #1478. commit 2aeda018094c242dfbec11a059c922c08be005d0 Author: Momoyeyu <[email protected]> Date: Thu Oct 1 19:23:25 2026 +0800 fix(eval): score custom tasks by stated answer; token-bound contains (#1342) (#1490) Custom eval tasks scored a model's stated answer wrongly in both directions. `scoring: "contains"` was a raw substring test, so an expected `4` was paid by `14` and an expected `42` by `#### 420`; `scoring: "exact"` compared the whole output, so `The answer is 4.` scored 0 against `4`. `contains` now requires the expected string to appear as a whole alphanumeric-bounded token (case-insensitive), so `14` no longer contains `4` and `#### 420` no longer contains `42`. A new `scoring: "answer"` compares the final answer the model states, read with the same parser GRPO's `accuracy` reward uses (`#### 4`, `\boxed{4}`, `The answer is 4.`; numbers compare by value), and refuses at load time an `expected` that states no single answer, naming the line, instead of silently scoring 0. The eval gate's `scorer:` override accepts `answer` too. `exact` keeps its whole-output meaning and is now documented as such. This changes scores. A gate baseline (`soup eval gate --baseline` file or `registry://` row) or a `soup eval custom --run-id` result recorded with the old substring `contains` can be higher than the same outputs score now, and the baseline's scorer stamp covers only the bundled suites, so it does not warn: re-measure those before comparing. The changelog and `docs/evaluation.md` say so. Verified: 30 new tests (15 fail on `main`), 17 of 18 review mutations caught, 433 related tests passed, `ruff` clean. Closes #1342. commit a07d6ba2f9a12c028090db251ddbe4b5ed6bbb25 Author: Srinivasan R <[email protected]> Date: Thu Oct 1 16:53:13 2026 +0530 fix(eval): retry judge requests and report an unreachable judge instead of scoring it as a tie (#1447) (#1484) Every LLM-judge failure was either scored as a tie or aborted the whole batch, and nothing was ever retried. `compare_pair` turned any judge exception into a tie, so `soup ship --task-mode pairwise` with the judge down scored a measured 0.5 win-rate and printed DON'T SHIP; `soup eval judge` died on the first transport error and lost the scores already collected; a training eval gate logged an unreachable judge as a regression. Every judge request now goes through one helper. It retries 429, 5xx and transport errors twice (3 attempts, pause 1 s then 2 s, or `Retry-After` when it is a finite number, capped at 30 s), fails at once on a 4xx that will not heal or a reply it cannot decode, treats a reply with no usable content as a judge error, and raises `JudgeUnavailableError` naming the judge (scheme, host, port and path only, never userinfo or a query string) once the retries are spent. `soup ship --task-mode pairwise` exits 1 with that message instead of a verdict; `soup eval judge` skips and counts the failing row and still scores, shows and saves the rest (`1/3 items skipped`); the eval gate still fails closed, and its log line says which task could not be scored and why rather than calling it a regression. A judge that answers "tie" still scores 0.5, and the Online DPO adapter keeps leaving a failed pair unranked, as the ranking trainer expects. Review found three more ways to escape that design, all fixed before merge: `Retry-After: nan` reached `time.sleep` and raised a raw `ValueError`; the judge URL, userinfo and `?api_key=` included, went into the message, `GateTaskResult.error` and the training log; and an undecodable reply (`httpx.DecodingError`) aborted a run. Verified against a real local HTTP server as well as through the CLIs: 45 new tests, of which 40 fail on `main` with the new names shimmed in; 17 of 19 review mutations caught; related suites 470 passed. Closes #1447. commit e967397f585efb488f9464801eb2786efcbe38dc Author: Ashutosh Panda <[email protected]> Date: Thu Oct 1 16:53:01 2026 +0530 fix(trainer): read trl PPO per-step loss from loss/policy_avg (#1413) (#1416) A completed `task: ppo` run printed `Loss: unavailable` even though training worked: `summarize_training_loss` only read the `loss` key of `trainer.state.log_history`, while trl 0.29's `PPOTrainer` logs its per-step policy loss under `loss/policy_avg` and never writes `loss` or `train_loss`, so every entry was skipped and the run ended with "unavailable". `summarize_training_loss` now takes the per-step key from the caller (`loss_key`, default `"loss"`, so SFT/DPO summaries are byte-identical), and the trl path in `trainer/ppo.py` declares `loss_key="loss/policy_avg"` instead of the summary guessing it; the manual PPO loop keeps the default because it normalises to `loss`. A non-default key is recorded in the result as `loss_key` on the `delta` and `single` branches only (never on the `train_loss` mean fallback or an `unavailable` result), and the completion panel shows it, e.g. `Loss: 1.4200 -> 0.8700 (loss/policy_avg)`, so a PPO policy loss is not read as an SFT cross-entropy. SFT panel lines are unchanged. Verified: against the real trl 0.29.1 `PPOTrainer` through `PPOTrainerWrapper.train()` on a tiny CPU Llama the result is `delta` with `loss_key: loss/policy_avg` (main: `unavailable`); the 9 new tests fail 6 of 9 on main and kill 11 of 11 mutations of the fix and the panel (including moving the key to the manual-loop call and summarising an empty history, via a behavioural test that drives `_train_builtin` on a trl-0.29-shaped `log_history`); 24 related test files give 1268 passed, 0 failed on the merged tree. Three optional mutations (re-tagging the mean fallback or the unavailable result) are not pinned by a test; trl's PPOTrainer never logs `train_loss`, so nothing is wrong today. The live per-step display still does not see `loss/policy_avg` (`SoupTrainerCallback.on_log` reads only `loss`); that is a separate gap. Closes #1413. commit f562f79d4af4ec0a52c0a785c7522d3f80c38989 Author: Bakdaulet <[email protected]> Date: Thu Oct 1 16:22:51 2026 +0500 fix(diagnose): report unmeasured probes as NOT_RUN instead of OK (#1489) fix(diagnose): report unmeasured probes as NOT_RUN instead of OK 1.00 (part 1 of #1435) `soup diagnose` scored a live probe that never produced a measurement as a perfect pass: a `--dataset` with no usable rows, a probe that raised (for example a 9000-character prompt over the 8192 limit), a typo in `--tokenizer`, or memorization on `question`/`answer`, `query`/`response` or `input`/`output` rows, which `score_memorization` cannot read. The overall verdict, the exit code and the badge all treated those as OK 1.00 and the exception text was dropped. This adds a `NOT_RUN` verdict (score 0.0 only, ranked above MINOR and below MAJOR, grey in the badge and the table, shown as `not run` / `—` instead of a score). All six converted probe sites keep the exception text in the evidence (NUL-stripped, capped, Rich-escaped), and memorization reports `NOT_RUN` when none of the first rows has text to split. `soup diagnose` exits 3 when the overall verdict is `NOT_RUN`; `--allow-not-run` keeps exit 0 for runs that knowingly skip probes; MAJOR still exits 2 and is never hidden. `--tokenizer` is resolved before any model loads, so a bad or empty value is an input error (exit 3). `train --diagnose-gate`, the `--evidence` path and the MCP `diagnose_evidence` tool understand `NOT_RUN` through one shared helper. Probes that do not apply (no `--dataset`, non-JSON format targets, no benchmark corpus, modes omitted from an evidence file) stay a neutral OK. Users notice: live runs that used to exit 0 on an unmeasured probe now exit 3, and ShareGPT / LLaVA / plaintext datasets read as `NOT_RUN` until part 2 of #1435 teaches the row builders those shapes. Known and declared gap: memorization still reads OK when every text is too short to split; it waits on #1481. Verified on the PR head merged with main (CPU): 175 passed across the two touched test files (39 of them fail on clean main), 1212 passed across 17 diagnose-related test files, ruff clean, 19 of 20 mutations of the new logic killed (the survi…
What does this PR do?
Writes the
soup reward synth --forcecandidate verifier to a.soup.reward-candidate.*.pytemp sibling instead of overwriting-oin place, loads and calibrates that file, andos.replaces it onto-oonly once calibration accepts it.-ois never moved or overwritten before that final replace, so a refusal, a calibration error, or a crash at any point before it leaves the previous verifier exactly as it was. Also guardsmerge_adapter_to_dense(shared bysoup shrinkandsoup draft distill): it now renamesout_diraside as a backup before swapping in the merged model, and restores that backup if the swap fails, rather than deletingout_dirup front. Fixes #1450.Type of change
Checklist
ruff check src/soup_cli/ scripts/ tests/passespytest tests/ -vpassesREADME.mdand the matching page underdocs/) if neededNotes: ran the issue's test file plus related regression tests (472 passed) and ruff clean; full
pytest tests/ -vleft unchecked rather than claimed since the heavy GPU/torch stack wasn't run.