Repository navigation
fix(infer,chat,diff): honor --device instead of hard-coding device_map=auto - #1460
Conversation
…p=auto
soup infer, soup chat and soup diff each resolve --device (explicit or via
detect_device()) but every from_pretrained call in their _load_model
hard-coded device_map="auto" and torch_dtype=torch.float16, ignoring it.
device_map="auto" lets accelerate place the model on any visible
accelerator, which is exactly wrong when the user asked for the CPU on
purpose (to leave VRAM free for a training job, or because the model does
not fit on the GPU), and float16 is not a CPU-appropriate dtype.
Add soup_cli.utils.device_resolve.resolve_device_map_and_dtype(), a small
shared helper (alongside the existing trust_remote.py pattern) that maps a
resolved device string to (device_map, torch_dtype): device_map={"":
device} once the device is known, "auto" only when it is genuinely unset;
float32 on cpu, float16 elsewhere. All three loaders now pass its result to
both their full-model and adapter-base from_pretrained calls.
Also drop chat.py's _generate(..., device="cuda") parameter, unused since
input_ids/attention_mask are already moved with model.device.
Closes MakazhanAlpamys#1443.
…_map signature change TestDiffModelLoading.test_load_model_uses_torch_dtype (MakazhanAlpamys#478) asserted the literal "torch_dtype=torch.float16" in diff.py's _load_model source; MakazhanAlpamys#1443 resolves that value per --device instead of hard-coding it, so the check now asserts the invariant it actually guards (both from_pretrained calls still spell the kwarg "torch_dtype=", not "dtype="), by count instead of by the old literal value. test_issue781_inference_bos.py's _via_chat() called chat.py's _generate() with a now-removed device="cpu" kwarg; drop it to match the signature.
…y, add adapter tests Review on this branch flagged two gaps in the MakazhanAlpamys#1443 fix: - utils/device_resolve.py duplicated the "what device_map does this device string become" decision that utils/gpu.py's resolve_device_map() already makes for training, inviting drift between the two. Move the inference helper into gpu.py itself, next to resolve_device_map, as resolve_inference_device_map_and_dtype(); its docstring cross-references resolve_device_map() and explains why they stay separate functions rather than one (distributed rank pinning via LOCAL_RANK/WORLD_SIZE vs. a single named/auto-detected device), and folds in its established convention that a plain, unindexed accelerator name is left as "auto" rather than pinned. MLX (any case) is now explicitly treated as the CPU bucket, since it has no torch device and must never reach from_pretrained's device_map as a literal string. The dtype annotation is now "torch.dtype" instead of the untyped "object". - test_issue1443_device_map.py's three regression tests only ever exercised each loader's full-model branch: their model_path is not a real local directory, so is_adapter is always False. Both branches already read the same device_map/torch_dtype pair, but nothing pinned that down — reverting device_map back to "auto" in only the adapter branch of any one loader left every test green. Add one adapter-branch test per loader (tmp_path with adapter_config.json, patched peft.PeftModel.from_pretrained) that asserts on the base model's from_pretrained call, and confirmed each one fails if that loader's adapter branch is reverted while its full-model test keeps passing. Also add a CLI-level test that omits --device entirely, so the auto-detected device from detect_device() is what the panel shows and what actually reaches from_pretrained (acceptance criterion 3), reading the panel through the shared strip_ansi() helper (criterion 4).
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Hi @drakeo338, thank you for this. The core fix works and I confirmed it by execution. The remaining problems are that a good part of the new behaviour has no test that would notice if it broke, and that the default path changes in ways the changelog does not say.
What is good
- The CPU case is fixed at the source: with
--device cpu,chat,diffandinfernow handfrom_pretraineddevice_map={"": "cpu"}andtorch.float32, on both the full-model and the adapter branch (I ran the real CLI with the loaders mocked and recorded the kwargs).--device mpspins{"": "mps"}; with no flag and a CPU-only box the panel saysDevice: cpuand so does the placement. - One shared helper instead of three copies, put next to
resolve_device_mapwith a docstring that explains why the two must not be merged (rank-pinning vs a single named device). Removing the unuseddeviceparameter fromchat._generateand adjusting the two existing tests was also right; thetest_bugfixeschange keeps its purpose (thetorch_dtype=spelling guard) because the following lines still check for a baredtype=. - The adapter-branch tests are the detail I liked most: a test that only exercises the full-model branch would let a revert of the base-model load slip through, and yours does not. On clean
origin/mainyour 7 tests give 7 failed; on your head they pass.
What I ran
pytest tests/test_issue1443_device_map.py tests/test_bugfixes.py tests/test_issue781_inference_bos.py tests/test_infer.py tests/test_chat.py tests/test_bench.py tests/test_diff.py tests/test_v0404_part_a.py tests/test_v0717.py tests/test_issue531_stream_base_provenance.py tests/test_cuda_graph_cli.py on the merge with current main -> 368 passed, ruff clean. Then 19 mutations, one at a time, against your 7 tests. 11 are killed: restoring device_map="auto" in any one of the six loader branches (your acceptance criterion 2, all six caught), passing None to the helper from any of the three _load_models, chat's CLI no longer passing the device, and making the CPU branch return "auto". Eight survive, and 7 of those also survive 332 adjacent tests:
| mutation | result |
|---|---|
infer.py:248 CLI passes None to _load_model instead of the resolved device |
7 passed |
diff.py:117 same, model A |
7 passed |
gpu.py:145 None/empty device no longer returns the old ("auto", float16) |
7 passed |
gpu.py:151 CPU dtype float32 -> bfloat16 |
7 passed (the test only asserts != float16) |
gpu.py:156 mps / cuda:N pin disabled |
7 passed |
gpu.py:153 bf16 choice on CUDA disabled |
7 passed |
gpu.py:150 mlx no longer mapped to CPU |
7 passed |
The first two are the regression this issue is about: if soup infer or soup diff stops handing the resolved device to _load_model, the panel still says Device: cpu and the model goes back to device_map="auto", and nothing fails. Your "without --device" CLI test only covers chat. (The eighth, pinning plain --device cuda to {"": device} instead of "auto", is a design choice, see the last item.) Every mutation was undone; git diff is empty afterwards.
Please fix before merge
- Add tests for the unpinned behaviour (blocking). These 10 pass on your head and kill all seven survivors above (1-3 tests fail for each). Add to
tests/test_issue1443_device_map.py(needsimport pytest; the file already importsjson,patch,torch,CliRunner,strip_ansi):
@pytest.mark.parametrize(
("device", "expected"),
[
("cpu", ({"": "cpu"}, torch.float32)),
("CPU", ({"": "cpu"}, torch.float32)),
("mlx", ({"": "cpu"}, torch.float32)),
("mps", ({"": "mps"}, torch.float16)),
("cuda:1", ({"": "cuda:1"}, torch.float16)),
(None, ("auto", torch.float16)),
("", ("auto", torch.float16)),
],
)
def test_resolver_table(monkeypatch, device, expected):
from soup_cli.utils import gpu
monkeypatch.setattr(gpu, "cuda_supports_bf16", lambda: False)
assert gpu.resolve_inference_device_map_and_dtype(device) == expected
def test_resolver_uses_bf16_only_on_cuda_when_supported(monkeypatch):
from soup_cli.utils import gpu
monkeypatch.setattr(gpu, "cuda_supports_bf16", lambda: True)
assert gpu.resolve_inference_device_map_and_dtype("cuda") == ("auto", torch.bfloat16)
assert gpu.resolve_inference_device_map_and_dtype("cuda:0") == (
{"": "cuda:0"}, torch.bfloat16,
)
assert gpu.resolve_inference_device_map_and_dtype("mps") == ({"": "mps"}, torch.float16)
def _prep(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
model_dir = tmp_path / "merged"
model_dir.mkdir()
(model_dir / "config.json").write_text(json.dumps({}))
(tmp_path / "prompts.jsonl").write_text(json.dumps({"prompt": "hi"}) + "\n")
return str(model_dir)
def test_infer_cli_hands_the_resolved_device_to_the_loader(tmp_path, monkeypatch):
from soup_cli.cli import app
model_dir = _prep(tmp_path, monkeypatch)
with patch(
"soup_cli.commands.infer._load_model", side_effect=RuntimeError("stop after the load call"),
) as load:
result = runner.invoke(
app,
["infer", "-m", model_dir, "-i", "prompts.jsonl", "-o", "o.jsonl", "--device", "cpu"],
)
assert "Device: cpu" in " ".join(strip_ansi(result.output).split()), result.output
assert load.call_args.args[2] == "cpu", load.call_args
def test_diff_cli_hands_the_resolved_device_to_the_loader(tmp_path, monkeypatch):
from soup_cli.cli import app
model_dir = _prep(tmp_path, monkeypatch)
with patch(
"soup_cli.commands.diff._load_model", side_effect=RuntimeError("stop after the load call"),
) as load:
result = runner.invoke(
app, ["diff", "-a", model_dir, "-b", model_dir, "--prompt", "hi", "--device", "cpu"],
)
assert "Device: cpu" in " ".join(strip_ansi(result.output).split()), result.output
assert load.call_args_list[0].args[2] == "cpu", load.call_args_listAlso tighten _assert_cpu_placement: assert dtype == torch.float32 instead of != torch.float16.
-
State what changes on the default path (changelog, PR text, docs). The CLIs always resolve a device before
_load_model(chat.py:105-108,infer.py:228-231,diff.py:99-102), so the helper'sNonebranch never runs from the CLI and "the old default is unchanged without--device" holds only for placement. Without any flag: on an Ampere or newer GPU the dtype goes from float16 to bfloat16 (withcuda_supports_bf16patched to True:("auto", torch.bfloat16)), so the changelog sentence "float32on CPU,float16elsewhere" is not accurate; and a CPU-only host now loads float32 pinned to the CPU instead of"auto"/float16, which doubles RAM (a 7B model is about 28 GB instead of 14 GB).soup benchpicks both up as well, becausebench.py:228callsinfer._load_modelwith the auto-detected device. The issue allows bf16 on CUDA, so this can stay as long as it is said; otherwise keep float16 there. -
An unknown or mis-cased
--deviceis still echoed and then ignored. Executed:soup infer ... --device fooprintsDevice: fooand loads with("auto", float16); the same forgpu,tpu,xpuandauto. That is the bug this issue is about, for invalid values. Separately,MPSandCUDA:0are normalised for the branch choice but the raw string goes into the map ({"": "MPS"}), andtorch.device("MPS")raises, whileCPUandCUDAwork. Verified fix inresolve_inference_device_map_and_dtype, replacing thenormalized = ...line and thempsreturn:
normalized = str(device).strip().lower()
if not re.fullmatch(r"cpu|mlx|mps|cuda(:\d+)?", normalized):
raise ValueError(
f"Unsupported --device {device!r}: use cpu, cuda, cuda:<index> or mps."
)
...
if normalized == "mps" or ":" in normalized:
return {"": normalized}, dtype(re is already imported in gpu.py.) With this change plus the tests from item 1 and these two below, your 7 tests, the adjacent test_infer/test_chat/test_diff/test_bench files and the new ones give 88 passed. Tests for it:
@pytest.mark.parametrize("device", ["foo", "gpu", "tpu", "xpu", "auto", "cuda:abc", "cuda:0,1"])
def test_resolver_refuses_an_unknown_device(device):
from soup_cli.utils import gpu
with pytest.raises(ValueError, match="cpu, cuda"):
gpu.resolve_inference_device_map_and_dtype(device)
@pytest.mark.parametrize(("device", "expected"), [("MPS", {"": "mps"}), ("CUDA:0", {"": "cuda:0"})])
def test_pinned_device_map_uses_the_normalised_name(monkeypatch, device, expected):
from soup_cli.utils import gpu
monkeypatch.setattr(gpu, "cuda_supports_bf16", lambda: False)
device_map, dtype = gpu.resolve_inference_device_map_and_dtype(device)
assert device_map == expected and dtype == torch.float16
assert torch.device(device_map[""]) # what accelerate will hand to torchIf you prefer a friendlier message than a ValueError from the loader, validate in the three commands right after the if not device: block and exit with code 2; the helper check is then the safety net.
-
Docs.
--deviceonsoup infer,soup chatandsoup diffis not described anywhere indocs/. A short note indocs/commands.md(next to the three commands) would do:cpuis pinned to the CPU in float32;cudalets accelerate use every visible GPU (bf16 where supported);cuda:Nandmpsare pinned to that device;mlxruns on the CPU on this path. -
Design note, not blocking. Plain
--device cudakeepsdevice_map="auto"(all visible GPUs, and accelerate may spill to the CPU when it does not fit), whereas the issue suggested{"": device}whenever the user names a device. Both are defensible; please pick one, say so in the docs, and pin it with a test so it cannot drift.
Merge state: all 11 required checks are green on b61ad913, and the branch merges cleanly with current main. #1083 (batched infer) also edits infer.py; it merges cleanly with this head today, but whichever lands second may need a rebase.
Thank you again: with items 1-3 in, the gaps I found are closed.
… tests Addresses review on MakazhanAlpamys#1460: the resolver now rejects anything other than cpu, mlx, mps, cuda or cuda:N and lower-cases the name it pins, so MPS and CUDA:0 reach torch in a form it accepts. New tests cover the resolver table, bf16 on CUDA, the infer and diff CLIs handing the resolved device to the loader, and CPU loads in float32. The changelog states the default-path dtype/placement change and docs/commands.md documents --device.
|
Thanks, all eight surviving mutations are now killed, unknown or mis-cased --device values are handled, and the changelog and docs state the default-path dtype and placement change. |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Hi @drakeo338, thank you for the fast follow-up. I re-checked every item from my review by execution on the current head (34f1fa06, merged with current main), and all of them are closed. This PR now waits only on the required checks.
What is good
- You took the default-path change on the honest side: the changelog and the new table in
docs/commands.mdsay that the dtype becomesbfloat16on a GPU that supports it, that a CPU-only host now loadsfloat32pinned to the CPU (about 28 GB instead of 14 GB for a 7B model), and thatsoup benchinherits both. I checked every row of the table against what the helper returns. - Refusing unknown
--devicevalues is done at the right place and reads well for a user. Through the real CLI,soup infer|chat|diff --device foonow stops with one line,Error: ValueError: Unsupported --device 'foo': use cpu, cuda, cuda:<index> or mps., before any model is loaded.CPU,CUDA,CUDA:0andMPSare normalised and work ({"": "cuda:0"}and{"": "mps"}are valid torch devices). I also checked thatdetect_device()only returnscpu,cuda,mpsormlx, so the refusal can never fire on an auto-detected value. _assert_cpu_placementnow assertsfloat32exactly, and you added the("cuda", ("auto", float16))row, so the "plaincudakeepsauto" choice can no longer drift unnoticed.
What I ran
- On the tree merged with current
main(no conflicts):pytest tests/test_issue1443_device_map.py tests/test_bugfixes.py tests/test_issue781_inference_bos.py tests/test_infer.py tests/test_chat.py tests/test_bench.py tests/test_diff.py tests/test_inference_advanced.py-> 291 passed; the wider set withtest_v0404_part_a,test_v0717,test_issue531_stream_base_provenanceandtest_cuda_graph_cli(previousmain) -> 456 passed; the gpu / startup-light guards (test_cli_startup_is_light,test_gpu,test_device_map_distributed,test_issue1355_accelerator_device,test_issue423_mlx_device_detection,test_multi_gpu) -> 317 passed.ruffis clean and the changelog fragment validates. - Mutations: 27, one at a time, against your 27 tests alone (no other guard files). 26 fail a test. Last round 11 of 19 did. Every survivor from last round is now caught:
Nonehanded to_load_modelfrominfer(infer.py:248) and fromdiffmodel A (diff.py:117), the helper'sNonedefault, the CPU dtype, themps/cuda:Npin, the bf16 pick, themlxmapping, and plaincudapinned instead ofauto. The new refusal and normalisation are pinned too (removing the regex gate -> 7 failed, dropping.lower()-> 3 failed, acceptingcuda:abc-> 2 failed, using the raw string in the map -> 2 failed).git diffis empty afterwards. - On clean
mainyour new file gives 25 failed, 2 passed (the two are the CLI hand-off tests, which only guard against a future regression).
One optional tightening, not a blocker
One mutation survives, and it is a gap in the test I suggested, not in your work: diff.py:121 (model B) passing None instead of the device. test_diff_cli_hands_the_resolved_device_to_the_loader makes the first _load_model call raise, so model B's call never happens. If you want to close it, this replacement passes on your head and fails under that mutation (I ran both):
def test_diff_cli_hands_the_resolved_device_to_both_loaders(tmp_path, monkeypatch):
from soup_cli.cli import app
model_dir = _prep(tmp_path, monkeypatch)
calls: list = []
def fake_load(*args, **kwargs):
calls.append(args)
if len(calls) == 2:
raise RuntimeError("stop after the second load call")
return MagicMock(), MagicMock()
with patch("soup_cli.commands.diff._load_model", side_effect=fake_load):
runner.invoke(
app, ["diff", "-a", model_dir, "-b", model_dir, "--prompt", "hi", "--device", "cpu"],
)
assert len(calls) == 2, calls
assert [call[2] for call in calls] == ["cpu", "cpu"], calls(MagicMock is already imported in that file.) The call site is code this PR does not change and model A is pinned, so I would not hold the merge for it.
Merge state: the branch merges cleanly with current main. The required checks on 34f1fa06 are still running at the time of writing (lint, mlx-smoke, transformers-floor, pytorch-smoke and two of the test cells are green, nothing has failed), so the merge waits on them. #1083 (batched infer) also edits infer.py; it merges cleanly with this head today, but whichever lands second may need a rebase.
Thank you again for the thorough fix and for stating the default-path change plainly.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
@drakeo338, thank you for the fast rounds: every item from my last review is closed, and the device map, the validation and the tests are what I wanted. I am now asking for one change, and it is a project decision, not a defect in your work.
Decision on the default dtype. The change this PR makes without any flag (an Ampere or newer GPU going from float16 to bfloat16) is a change of default behaviour for everyone running soup infer, soup chat, soup diff and soup bench, and it should not ride along with a --device fix. It also has a cost we have measured elsewhere: batched generation diverges from one-by-one generation more in bfloat16 than in float16. So:
- CPU (
cpu,mlxon this path): keepfloat32, pinned to the CPU.float16on the CPU is the defect here, and the doubled RAM use (about 28 GB instead of 14 GB for a 7B model) stays declared in the fragment and the docs, as you already did. - GPU (
cuda,cuda:N,mps): keepfloat16, exactly as before. Only the device map and the validation change there.
If you think bfloat16 should be the GPU default, please open a separate issue with measurements (outputs against float16 on a few models); that is a decision I would want to make on its own.
What changes in the PR:
src/soup_cli/utils/gpu.py,resolve_inference_device_map_and_dtype: drop theuse_bf16pick, socuda,cuda:Nandmpsall returntorch.float16.cuda_supports_bf16()is no longer used there. Update the docstring's "bf16/fp16 pick" wording.- Tests: the exact-dtype assertions for
cuda,cuda:Nandmpsexpectfloat16, and add one case wherecuda_supports_bf16()is monkeypatched toTrueand the result is stillfloat16. Keep thefloat32pins forcpuandmlx. changelog.d/0.75.1/1460.fixed.md: remove the sentences saying the dtype isbfloat16where the GPU supports it and that the Ampere default moves fromfloat16tobfloat16; keep the CPUfloat32and RAM-doubling paragraph.docs/commands.md(the table around lines 388-389): thecudaandcuda:Nrows sayfloat16.
After that push, the PR is ready again: the required checks will run on the new head, and I will merge it as soon as they are green. #1479 (soup serve --device) will reuse this helper, so it gets the same rule.
|
Done in d99d94c: cuda, cuda:N and mps return float16 again (with a test that forces cuda_supports_bf16() to True), the changelog fragment drops the bfloat16 sentences, and the docs table says float16. |
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Hi @drakeo338, thank you for the quick turnaround on the dtype decision. I checked all four requested changes by execution on the current head (d99d94ca, merged with current main, no conflicts), and all of them are in. This PR now waits only on the required checks.
What is good
- The change is exactly the size of the decision.
git diff 34f1fa06 d99d94catouches four files (the resolver, its test, the changelog fragment and the docs table) and nothing ininfer.py,chat.py,diff.pyorbench.py, so the device map, the validation and the adapter-branch coverage I approved last round are untouched. - The new test is the right shape:
test_resolver_keeps_float16_even_when_cuda_supports_bf16forcescuda_supports_bf16()toTrueand asserts the exact dtype forcuda,cuda:0andmps. When I restored the old bf16 code in the resolver, that one test fails, so it carries the whole decision, and it is exact rather than!= bfloat16. - The fragment and the table in
docs/commands.mdnow say what the code does: CPUfloat32with the RAM cost (about 28 GB instead of 14 GB for a 7B model) andsoup benchinheriting it, andfloat16oncuda,cuda:Nandmps. I checked each of the five table rows against the real helper.
What I ran
- Real helper, 23 device strings, with
cuda_supports_bf16()forcedFalseand thenTrue: the dtype is the same in both cases for every valid device.cpu/CPU/mlxgive({"": "cpu"}, float32);cuda/CUDAgive("auto", float16);cuda:0/CUDA:0/cuda:12give({"": "cuda:N"}, float16);mps/MPSgive({"": "mps"}, float16); unset or empty gives("auto", float16);foo,gpu,auto,cuda:abc,cuda:0,1,xpu,tpustill raiseValueError. - Through the real CLI with the loaders mocked, on both the full-model and the adapter branch, with and without
--device, and withbf16forced on and off:soup infer,chat,diffandbenchhandfrom_pretrainedonly the pairs above, anddiffhands the same pair to both models. None of the 89 recordedfrom_pretrainedcalls carriesbfloat16.--device fooand--device bfloat16through the real CLI stop with one line and load nothing. - Tests on the tree merged with
main: the infer/chat/diff/bench/bugfix set -> 291 passed; the gpu and startup-light guards -> 455 passed; the docs guard tests -> 105 passed;ruffclean; the changelog fragment validates. - Mutations, one at a time, against your 27 tests alone: 40, of which 38 fail a test. That includes every way I could put bf16 back: plain
cudareturningbfloat16(2 failed),mpsandcuda:Nreturningbfloat16(5 failed), onlycuda:N(3 failed), the previous resolver code restored exactly (1 failed),cuda_supports_bf16()choosing on plaincudaor oncuda:N/mps(1 failed each), a hard-codedtorch.bfloat16in any of the six loader branches (1-2 failed each), andfloat16on the CPU, which is the original defect (10 failed).git diffis empty afterwards. - On clean
mainyour test file gives 25 failed, 2 passed.
Two optional tightenings, not needed for this PR
No need to push for these, and I am not holding the merge for them:
- The unset/empty branch of the resolver (
gpu.py:147-148) could returnbfloat16throughcuda_supports_bf16()and no test would notice: thebf16 = Truetest does not includeNoneor"". None of the four commands can reach that branch today (they all resolve a device first), so it only matters for a future caller. A parametrized case coveringNone,"",cuda,CUDA,cuda:0,cuda:1,mps,cpuandmlxundercuda_supports_bf16 -> Truepasses on your head and fails under that mutation (I ran both). - The gate
cuda(:\d+)?acceptscuda:007and non-ASCII digits, which torch then rejects at load (Invalid device string).cuda(:(0|[1-9][0-9]{0,2}))?would refuse them up front. It is a typo-grade edge that was already there when I approved.
Merge state: the branch merges cleanly with current main (still 89dc498a when I looked). The required checks on d99d94ca are still running at the time of writing: pytorch-smoke is green, three test cells are in progress, the rest are queued, and nothing has failed. #1083 (batched infer) also edits infer.py; it merges cleanly with this head today, but whichever lands second may need a rebase.
Thank you again for the fast fix and for taking the decision without a fuss.
|
Merged as |
commit 4e3ab348994dba22bff836fe255936b5212a14a2 Author: Alpamys Makazhan <[email protected]> Date: Fri Oct 2 22:40:37 2026 +0500 fix: soup init and the MCP plan snapshot refuse links at their write target (#1553) soup init wrote the starter config with Path.write_text, which follows a symbolic link or junction at the target, and a dangling link skipped the overwrite prompt. It now checks the target before prompting, creates a new file exclusively (O_CREAT | O_EXCL with O_NOFOLLOW on POSIX; a staging file renamed into place on Windows, where an exclusive create still follows a dangling link), opens an existing file through open_no_follow, and truncates it only after the open-time check has passed. A refusal names the path and exits 1; ordinary paths write the same bytes and mode as before. The MCP plan snapshot (.soup/mcp-runs/<run_id>/config.yaml) also refuses a link at the run directory itself, re-checks the parents after creating them, creates the run directory with exist_ok=False and config.yaml with O_EXCL through open_no_follow(check_parent=True), and requires the run id to be a single path component. Refusals stay path-free ExecutionErrors. Co-Authored-By: Claude Opus 5.5 <[email protected]> commit 12807838df1604f8873949ab2ca7340afaefd358 Author: Mohd Uwaish <[email protected]> Date: Fri Oct 2 18:24:07 2026 +0200 fix(config): refuse LoRA+ with model-only optimizers and GaLore at load (#1210) (#1255) `training.loraplus_lr_ratio` combined with `optimizer: apollo_adamw`, `lomo` or `adalomo`, or with `use_galore`, passed config load and failed only after the base model had been downloaded and loaded. LoRA+ builds its optimizer through `build_loraplus_optimizer`, which asks `Trainer.get_optimizer_cls_and_kwargs(args)` for it without a model, and transformers can only build those three optimizers (and GaLore) from one; the GaLore combination was refused only at runtime. A new `TrainingConfig._validate_loraplus_compat`, next to `_validate_lorafa_compat`, refuses both combinations at load with a message that names both fields, and suggests an optimizer transformers can build without the model. The runtime GaLore check in `peft_wiring.py` stays as a backstop. The refusal does not depend on the task, and LoRA+ with any other optimizer (`adamw_torch`, `adamw_8bit`, `adafactor`, ...) and each of the refused settings on its own still load. Users of LoRA+ with one of those optimizers now see the error from `soup train` in about a second instead of after the model download. `docs/peft-and-efficiency.md` gains an `Optimizers` bullet in the LoRA+ section, under the existing `Compatibility` bullet. Verified: `tests/test_issue1210_loraplus_model_only_optimizers.py` (36 tests: three refusals, the case-insensitive refusal, GaLore and seven controls, each under `sft`, `dpo` and `ppo`) passes, and 15 of them fail on `main` without the change; ten single-point mutations, including moving the refusal into an `sft`-only validator (10 failures), all fail the tests; 1333 related tests pass on the tree merged with `main`; `ruff` is clean. Closes #1210. commit ef44c0c0af75e9aefe3b109a17fe15ec843ec878 Author: Pernav Jain <[email protected]> Date: Fri Oct 2 21:38:54 2026 +0530 fix(notebook): make proof-4gb reproducible (#1545) `notebooks/proof-4gb.ipynb`, the "run it yourself" proof of 4 GB layer streaming, did not run on a fresh hosted multi-GPU runtime (Kaggle 2x T4) with a current `huggingface-hub`, for two reasons in its install cell. (1) `huggingface_hub` 1.32 keeps blobs in a cache-wide shared store that soup-cli 0.75.0's snapshot materialization treats as outside the cache and refuses (#1509, fix open in #1510); the install now caps `huggingface-hub<1.32.0`, which resolves to 1.31.0 and satisfies transformers' `>=1.5.0,<2.0`, with a comment saying why and when to drop it. (2) With two visible CUDA devices, layer streaming refuses to start because it cannot run under `nn.DataParallel` (`refuse_if_data_parallel` names `CUDA_VISIBLE_DEVICES=0`); the cell now sets `os.environ.setdefault("CUDA_VISIBLE_DEVICES", "0")` before torch is imported, so a caller who pre-selected a card keeps it. Only cell 3's source changes; the outputs, metadata and execution counts of all 19 cells are identical. `tests/test_issue844_notebook_pin_guard.py` gains two tests: the GPU is selected in a cell at or before the first cell that imports torch, and the install cell caps the hub below 1.32. Verified: the new tests are 2 failed / 7 passed on `main` and 9 passed with the PR; mutations deleting or relocating the device block, deleting the cap, or loosening it each fail a test; cell 3 was run in isolation with the `%pip` lines recorded and left the variable at `'0'` when unset and unchanged when preset. The two-GPU run itself was done by the author and not reproduced here. Closes #1543. commit c6271bffe35455c1e675800344dfd9f3c6215e82 Author: qwist1233-cpu <[email protected]> Date: Fri Oct 2 19:07:18 2026 +0300 fix(advise): print the ROI table when --probe-model implies --probe (#1448 slice) (#1521) `soup advise run <data> --probe-model X` (without `--probe`) paid for the live probe and then never printed it. `--probe-model` implies `--probe`, and the probe ran under the derived `run_probe = probe or probe_model is not None`, but the `ROI deltas (probe)` table was gated on the raw `--probe` option, so the measured baselines, LoRA estimate and base-model proximity were computed and thrown away. The table is now gated on `run_probe`, the same flag that runs the probe, at `src/soup_cli/commands/advise.py:333`. Users notice: `soup advise run data.jsonl --probe-model org/model` now prints the ROI table, as `docs/evaluation.md` already said it implies `--probe`; plain `--probe` still prints the table and a run with no probe flag still prints none. Verified: the new `tests/test_issue1448_advise_probe_model_roi_table.py` (4 tests, probe helpers monkeypatched) has 1 failed / 3 passed on `main`; with the PR, 228 passed / 1 skipped across it, `tests/test_v0717.py`, `tests/test_v0540.py` and the changelog-fragment tests. Four mutations of the gate each fail a test (`if probe:`, `if probe_model is not None:`, `if True:`, `if False:`). Part of #1448. commit 255a5ecce26204367d5ce72a96e19029c256c0fd Author: Jillur Rahman <[email protected]> Date: Fri Oct 2 22:05:04 2026 +0600 fix(distill): leave unaligned student positions out of aligned ULD (#1426) (#1541) Aligned ULD (`training.uld_strategy: wasserstein_aligned`, the strategy `soup draft distill` picks for every cross-tokenizer pair) mean-pools teacher logits onto student positions through a character alignment. A student position that aligns to no teacher token (a byte-fallback piece decoding to U+FFFD, an empty decoded piece, a student-only special token) got an all-zero teacher row, which softmaxes to a uniform distribution, and stayed in the loss mask. Against a confident student each such position scored about V/2 (7999.4 for the reproduction at a 32k vocabulary) while aligned positions scored near 0, so the loss was dominated by positions with nothing to distill and its gradient pushed the student toward maximum entropy there. `uld_aligned_loss` now counts a student position only when its alignment names at least one existing teacher position (`aligned_position_mask`, the same in-range rule `aggregate_aligned_logits` uses). That mask multiplies the label mask or the attention mask and, with neither, is the whole mask, so the position leaves both the sum and the denominator; a sequence with no aligned supervised position gives a finite 0.0. The distill trainer decodes the student's per-token strings with `skip_special_tokens=True`, as the teacher text already is, so a student-only special token decodes to "" and aligns to nothing. Users notice: distillation loss across different tokenizers no longer carries a large constant per byte-fallback piece, empty piece or special token, and the student is no longer pulled toward a uniform distribution at those positions. With no `labels` and no `attention_mask` the aligned loss is now the mean over aligned positions instead of over every position; with every position aligned the value is bit-identical (800 old-vs-new comparisons across four mask modes). Verified: the issue reproduction gives 7999.4 on main and 0.0 on the branch under labels, attention mask and no mask; a batch of rows with different alignments and an all-unaligned row stays finite with finite gradients; 15 new tests (11 fail on main, 4 are controls) including a trainer-level test through the real `_DistillTrainer.compute_loss`; 331 passed across the new file and six neighbouring ULD/distill files, 565 across nine more; ten mutations of the fix, each caught. Closes #1426. commit ed50391a1711f6b5fc9903c785d9b4b3ecb9bb81 Author: Urooj Fatima Mukhtar <[email protected]> Date: Fri Oct 2 14:40:32 2026 +0500 fix(eval): make contains scoring linear again (#1528) (#1540) The custom-eval `contains` scorer stopped being linear when #1490 made it match `expected` only as a whole token. Its pattern began with a lookbehind and ran under `re.IGNORECASE`, which keeps `re` off its literal-prefix search, so the search restarted at every position and compared up to `len(expected)` characters at each word start: time proportional to `len(output) * len(expected)`. On repetitive token-separated text a 200,000-character output against a 100,000-character `expected` took 27 s on `main` (68 s at 400,000 against 200,000, as measured in the #1490 review), where v0.75.1's substring test took 0 s. `score_contains` now lowercases both sides once and puts the escaped literal first, so `re` finds candidates with its literal-prefix search. The left boundary is a fixed-width lookbehind after the literal and the right boundary is the existing lookahead, so each candidate costs O(1). The token bound from #1490 is unchanged and nothing is truncated: a gold token at character 60,000 of a 60,016-character output still scores `True`. What users notice: scores are unchanged for ASCII text, and the same 200,000-character case takes about 0.13 s. Case folding is `str.lower()` on both sides, as in v0.75.1 (the regex had not shipped in a release), so `istanbul` no longer matches `İstanbul` and `s` no longer matches `ſ`; the `contains` row in docs/evaluation.md says so. Verified: the 8 new tests fail 4/8 on `main` and pass on the branch; test_issue1342 (30) unchanged; 620 tests across the 11 files that touch contains-scoring pass; 120,000 ASCII cases agree with #1490's regex and 320,000 cases (Unicode included) agree with an explicit overlapping scan; 16 mutations of the function all fail the new tests; growth is linear on 8 input shapes up to 400,000 characters. Closes #1528. commit 6879f323f7c8b893c3679abd53a34d61f3616fc1 Author: Srinivasan R <[email protected]> Date: Fri Oct 2 14:57:53 2026 +0530 fix(expect): refuse top-level suite keys other than expectations (#1483) (#1499) `soup expect` read only the `expectations` key of a suite and dropped every other top-level key, so a suite with `fail_fast: true` (or any label or typo) beside `expectations:` loaded, ran and exited 0 with no message: the gate that ran was not the gate the user wrote. #1461 had already made `parse_suite_spec` refuse unknown keys inside an entry and unknown argument names; this closes the same gap one level up. `parse_suite_spec` now refuses every top-level key other than `expectations` before it reads anything else, naming each key through #1461's `repr()`-based `_key_repr` so a hostile key cannot reach the terminal as a control sequence, and `soup expect` exits 3 before the data file is opened. Users with a suite that carried a label such as `name:` beside `expectations:` now get exit 3 with the key named instead of a silent pass (documented in `docs/compliance.md`); one existing fixture in `test_issue1231_expect_all_formats.py` was such a suite and loses that line. Verified on the PR merged with main: 270 passed across the expect, gate-exit-code and ANSI-safety files, the new tests give 8 failed / 2 passed on main, and mutations removing the check, the `repr()`, the all-keys naming, or widening the allowlist to `name`, `description`, `version` or an `expectation*` prefix are each caught. Real CLI: `fail_fast:` and `name:` beside `expectations:` go from exit 0 to exit 3. Closes #1483. commit 8fcd8b90e320cf99d2abd3816b334611b1460707 Author: Alpamys Makazhan <[email protected]> Date: Thu Oct 1 23:33:22 2026 +0500 ci: run the full matrix at approval (ci:full), a quick subset on other PR pushes (#1381) A push to a pull request now runs a quick subset of `ci.yml` (`lint` and `test (ubuntu-latest, 3.12)`); the full matrix runs on pushes to `main` and `release/**` and on pull requests that carry the `ci:full` label, which a maintainer adds at approval. Why: the Actions account runs at its free-plan cap (20 concurrent jobs, 5 on macOS). Measured over 1,906 runs since 2026-09-19, one full `ci.yml` run costs 358 job-minutes, saturated service is 61-71 runs a day while weekday demand is 87-99, and most of that demand was review iterations: every push to a pull request paid for the full matrix even while the head waited on its author. Running the full matrix once per pull request, at approval, is worth about x1.36 of capacity on the binding limit; dropping Windows and macOS on 3.11 from the full run (they stay in the release matrix) takes a full run from 4 macOS jobs to 3. How: `pull_request.types` adds `labeled` (still a plain `pull_request`, never `pull_request_target`); a new `plan` job declares the RELEASE, FULL and QUICK matrices once and `test` takes `fromJSON(needs.plan.outputs.matrix)`, so a quick run never creates the other required `test (...)` contexts and branch protection keeps them Expected: nothing can be merged on a quick run. The concurrency key gives a `labeled` event for any label other than `ci:full` its own group, so labelling cannot cancel a running full matrix. `CONTRIBUTING.md` tells contributors what they will see. Verification: `tests/test_ci_full_matrix_on_approval.py` pins the decision table, the matrices, the gated jobs, the concurrency keys and the decide step's shell (the matrix literals are parsed by `tests/test_requires_python_bound.py`); its own full matrix ran green on this head (11 of 11 required checks). commit 4142fdd89889989e08f38de81394392a242af74a Author: UmranPros <[email protected]> Date: Thu Oct 1 23:49:08 2026 +0530 fix(train): a blend that cannot be computed refuses instead of scaling one loss (#1425) (#1513) `training.preference_loss_weights` promised a runtime blend of 2-5 preference losses, but no trl 0.29 trainer puts per-sequence log-probs on the batch, so the term loop never ran and the combine wrapper silently fell back to `primary_loss * primary_weight`. A config such as `{dpo: 0.7, simpo: 0.3}` printed `Multi-objective preference loss: 0.70·dpo + 0.30·simpo` at setup and then trained DPO alone scaled by 0.7 (measured on a real trl 0.29.1 trainer: exit 0, `Loss: 0.4852` = 0.7 x 0.693147), with nothing in the output saying so. The fallback is deleted (with the now-dead `_pick_primary` and the `primary_loss` unpack): when a requested term cannot be computed the wrapper raises, naming the terms it could not compute, the terms it did compute, and what was requested. What users notice: every `preference_loss_weights` config now stops at the first step instead of training one loss under a blend's name; to train one loss, remove the weights and set `training.preference_loss`. The weighted-sum path itself is unchanged, so a trl that does carry log-probs on the batch would still blend. `docs/training.md` no longer claims a true weighted sum. Verified on the PR merged with main: 183 passed across the five preference/combine files, a real-trainer test (tiny random-init Llama, CPU, two blends) that passes on the PR and fails on main, and mutations restoring the fallback, naming every term as missing, emptying the computed list, accepting a partial blend and changing the exception type each caught. This is partial progress on #1425: the real blend (deriving log-probs from the trainer's own forward pass) and the "changing only the secondary loss changes the loss" check are not in this change. Refs #1425. commit 909d86c48a8de062ebc1e48704fc0cb174980700 Author: Alpamys <[email protected]> Date: Thu Oct 1 23:02:17 2026 +0500 docs(contributors): #1496, #1481, #1482, #1514 [skip ci] Display names from `gh api users/<login>`. commit 92e6a1fd00405c10f60c2080bf6e04b2b43526bd Author: UmranPros <[email protected]> Date: Thu Oct 1 23:30:50 2026 +0530 fix(train): refuse the RL-signal flags PPO cannot feed (#1441) (#1514) fix(train): `task: ppo` refuses the reward-hacking and echo-trap flags it cannot feed (#1441) The trl PPO trainer this build supports (`trl.experimental.ppo.PPOTrainer`) takes no `reward_funcs`: its reward comes from the reward model it calls internally. Soup still built the callable reward wrappers and a shared signal buffer at setup and then dropped them, so the buffer was never written, and because the buffer object existed the detector's `on_log` fallback returned early too. A run with `reward_hack_detector`, `reward_hack_mitigation` or `echo_trap_enabled` was announced, loaded, and then never halted, mitigated or trapped anything. `PPOTrainerWrapper.setup` now refuses these three flags (the exact set that makes `rl_callbacks_need_buffer` create the buffer) on the experimental trl API, before `datasets`, the model, the reward model or the tokenizer is loaded. The message names every flag that was set and points at `task: grpo`, where the buffer is fed. `reward_hack_mitigation: off`, the other `reward_hack_*` / `echo_trap_*` tunables (already refused at config load unless one of the three is set) and the transitional trl API are unaffected, and none of the 200 shipped configs, templates and recipes is refused. `docs/training.md` and `docs/commands.md` no longer say PPO is supported for these flags. Feeding the buffer from the PPO rollout and the meaning of `reward_fn` on PPO are separate follow-ups. Verified: 21 tests in `tests/test_issue1441_ppo_refuses_unfed_rl_flags.py`, including setup-level tests that stub the loaders and assert nothing loaded, and a drift test pinning `_unsupported_rl_flags` to `rl_callbacks_need_buffer`; 11 mutations of the refusal (call removed, condition inverted, each flag dropped, refusal moved after each load, never raising, always raising, no flag names, drift) all fail a test; the neighbouring PPO suites (222 tests with the changelog and backend-support guards) pass on the merge with current `main`; and a real `soup train --config soup.yaml --yes` with all three flags stops at "Setting up model + trainer..." with the new message, while the same config without them is not refused. Closes #1441. commit 5c335b93c3e4993ead7e3f6ef552d4c3ad4f870e Author: Alpamys Makazhan <[email protected]> Date: Thu Oct 1 23:00:43 2026 +0500 fix(net): harden outbound endpoint validation (#1527) Outbound endpoint checks for the vLLM provider, `soup data generate --api-base`, judge URLs (flags, eval-gate suites, `soup ship --judge-model`, `training.online_dpo_judge`) and the Web UI chat proxy share one helper, `net_guard.refuse_private_ip_literal`, which refuses a private, link-local or reserved IP literal on every scheme. Loopback stays allowed; hostnames are not resolved. `0.0.0.0` is no longer loopback for `soup data generate` and the chat proxy. The shared predicate also counts 100.64.0.0/10 and fec0::/10 as non-public, and the shared parser classifies a non-ASCII host in the ASCII forms an HTTP client connects to (stdlib IDNA 2003 and the idna package), so webhooks, the OTLP endpoint, telemetry, hub endpoints and `soup ingest --pull` inherit both. Also: `soup ship` checks `--judge-model` before it builds the base and tuned models; the chat proxy answers 400, not 500, for an endpoint that does not parse; `online_dpo_judge` names itself on an unparsable URL. Docs say which addresses each setting accepts and name the two ranges. Tests: `tests/test_outbound_endpoint_ip_literals.py` (every entry point through its real surface, a differential test against `httpx.URL(url).host`, no network I/O) and the extended #616 ratchet. No changelog fragment: the change is recorded in the 0.75.2 section of CHANGELOG.md, which main gains at its next release. Co-Authored-By: Claude Opus 5.5 <[email protected]> commit 1d0e7384dcb5bfa9e900d6632e6efb0685c5b447 Author: Areen Agrawal <[email protected]> Date: Thu Oct 1 23:30:16 2026 +0530 fix(lora): map mistral3 target_modules and add modality to mistral-medium-3-5-sft (#1395) (#1482) `mistral-medium-3-5-sft` could not attach LoRA: its base, `mistralai/Mistral-Medium-3.5-128B`, is `Mistral3ForConditionalGeneration` (a Pixtral vision tower plus a `ministral3` text tower), but the recipe had no `modality: vision`, so SFT loaded it with `AutoModelForCausalLM`, which does not know `Mistral3Config` (`soup recipes verify` failed at stage `build`), and `training.lora.target_modules: auto` had no row for `mistral3` (stage `resolve`: no mapping for model_type ['ministral3']), so the architecture was refused even with the right loader. The recipe now sets `modality: vision` (SFT loads the image-text class), and `MOE_TEXT_LORA_TARGETS` gains a `mistral3` row: `.*language_model\..*\.self_attn\.(q_proj|k_proj|v_proj|o_proj)`, scoped to the language tower so the Pixtral vision tower, the multimodal projector and the MLP projections stay unadapted (a plain `q_proj/k_proj/v_proj/o_proj` suffix list would have adapted 16 modules, 8 of them in the vision tower). Measured on a 2-layer meta-device skeleton of the real config: 8 modules adapted, 0 in the vision tower, 0 experts. The recipe's `EXCEPTIONS` pin in `scripts/check_recipe_attach.py` is removed (the ratchet reports "0 pinned that now attach" without it, and flags the pin if it is put back), the recipe snapshot gains the `modality` line, and docs/peft-and-efficiency.md lists `mistral3` among the architectures `target_modules: auto` maps. The recipe count does not change. Verified: `soup recipes verify --config` on the real recipe moves from "0 attach, 1 cannot" (exit 2) to "1 attach, 0 cannot" (exit 0); 30 new tests (20 fail on the previous tree; the other 10 are documentation tests that build their own regex), 731 passed across the related files on the tree merged with main, `scripts/check_recipe_attach.py` exits 0 (and exits 2 when the pin is put back), and seven mutations of the fix are all caught (delete the row, drop the leading `.*`, plain suffix tuple, unscoped regex, drop `o_proj`, drop `modality: vision`, drop the snapshot line). Closes #1395. commit be0049993e2fa93ab26205cf6721d983aeb08547 Author: fukalous <[email protected]> Date: Fri Oct 2 01:59:43 2026 +0800 fix(diagnose): detect memorization in long rows (#1481) `soup diagnose` could not see memorization on long training rows. The live probe asks the adapter to continue the first quarter of a row and caps the generation at 64 tokens, but the memorization score compared that capped completion with the whole held-out suffix by set-Jaccard, so the longer the row the more the untouched suffix diluted the overlap: a verbatim 48-word echo read MAJOR on a 60 or 100-word row and OK from 150 words up, and a model that had memorized its long rows passed with a clean 1.00. The overlap is now clipped completion-bigram precision against the suffix: the share of the completion's adjacent token pairs that occur, counted at most as often as they occur, in the suffix. The default path pairs adjacent normalized words (stopwords and short ASCII words are still ignored, as before); `--tokenizer` pairs adjacent sub-word tokens. Bigrams, not single words, are what keep the score honest: a single-word precision made unrelated text flag as MAJOR more often the longer the row (59 percent of unrelated completions at 4000 words of real prose), flagged one-word completions, and treated a repetition loop of a common word as a verbatim echo. What users notice: a verbatim echo now reads MAJOR at every row length (60 to 4000 words, default and `--tokenizer`, Latin, Cyrillic and CJK text), and an unrelated continuation, a short completion or a repetition loop stays OK. The 0.5 threshold now means bigram precision rather than set-Jaccard, so a partial echo can read MAJOR slightly earlier than before (on a 60-word row a completion copying 55 percent of its words in order flips, where set-Jaccard needed 67 percent), and memorization results saved by earlier releases are not directly comparable; the changelog and `docs/evaluation.md` say so. Verified on the PR head merged with main (CPU): 34 new tests (13 of them fail on clean main) and 406 passed with the neighbouring diagnose and changelog-fragment tests; 7 of 8 mutations of the scoring killed (the survivor is the pre-existing `>=` versus `>` boundary at exactly the threshold); real-prose runs (docs/*.md, SmolLM2 tokenizer, 120 rows per length) show verbatim 1.00 and unrelated 0.00 to 0.03 from 60 to 4000 words on both paths. Closes #1437. commit b9ad2fb4f7d42b9a76ff417d000f8336abe6bd54 Author: Alpamys Makazhan <[email protected]> Date: Thu Oct 1 20:33:45 2026 +0500 test(serve): pin that --device is refused on every backend, not only transformers (#1478) (#1523) The review of #1479 (merged as `0f7ab958`) ran 24 mutations of the `soup serve --device` fix and one survived: validating `--device` only when `backend == "transformers"` passed every test, because the refusal tests all ran on the default backend. That mutation would be a real regression (an unknown device reaching vLLM/SGLang unchecked). This adds one parametrised case to `tests/test_issue1478_serve_device_map.py`: `--device tpu` must exit 2 on `--backend vllm` and `--backend sglang`, before either server is built or `uvicorn.run` is called. Mutation check: guarding the validation block in `serve.py` with `and backend == "transformers"` makes the two new cases fail (2 failed, 34 passed); unmutated, 36 pass. Test-only, no source change. commit 4d14b3ddc246f5fd2f3f5255c872d5ecfaa8f3dc Author: Chaitanya Reddy <[email protected]> Date: Thu Oct 1 19:47:24 2026 +0530 fix(data): detect format from a bounded prefix, not only the first row (closes #1424) (#1496) `format: auto` chose a dataset's format from the first row alone, so a first row that happened to lack an optional key picked the poorer converter for the whole file and silently dropped that key from every row that had it: a LLaVA file whose first row was text-only loaded as `sharegpt` and lost every image, an OpenAI-shape tools file whose first row had no `tools` lost the tool-schema turn, and audio and ASR files lost their audio path. Detection now reads a bounded prefix (`MAX_DETECT_ROWS = 100` rows). Each row resolves to the richest format its keys fit; a uniform file detects exactly as before (a golden comparison over all 22 `.jsonl` files in the tree is identical to main apart from the wording "Cannot detect format at row 0"). A mixed file either upgrades to the richer format of the same family (today: `chatml` rows mixed with `tool-calling` rows load as `tool-calling`), but only when the richer converter accepts every poorer row and converts it to exactly the messages the poorer converter would (so a chatml row carrying message-level `name` or `weight` is refused rather than silently stripped), or it is refused with a message naming both shapes and the row where each was first seen and telling the user to set `data.format`. Rows that match no signature, and non-object rows, are skipped as the converters skip them instead of crashing on `.keys()`. What users notice: a mixed file that used to load with a key stripped, or with rows dropped under a warning, now stops at load; `soup data validate` exits 3 on it with the refusal. The Web UI data explorer detects from the same window and reports the refusal in a new `format_error` field instead of answering 500, and `docs/data.md` documents the 100-row window, the upgrade rule and the refusals. Verified on the PR head merged with main (CPU): 34 new tests (19 fail on main with only the constant added so the module imports), 815 passed across the 22 existing modules that call `detect_format` or the explorer, 15 of 18 mutations killed (the three survivors are ancillary lines; their tests are optional), ruff clean. Closes #1424. commit 8d8abc933a50056bb8ce850f1fcdf7620dfbf3ec Author: t <t@t> Date: Thu Oct 1 17:20:25 2026 +0500 docs(contributors): #1398, #1455, #1457, #1419, #1516, #1517, #1469, #1498, #1306, #1454 Display names from `gh api users/<login>`. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> commit 1a67520c07669acdcd3ed76afbacf75cef2e0dc8 Author: Younes Beriane <[email protected]> Date: Thu Oct 1 14:18:54 2026 +0200 fix: keep the previous artifact when reward synth or adapter fuse fails (#1454) Two commands replaced an existing artifact in an order that removed the old copy before the new one was known to be valid and in place, so a failed replace could leave no good copy at all. `merge_adapter_to_dense` (used by `soup shrink` to fuse the heal adapter into the pruned model, and by `soup draft distill` to write the dense draft) deleted `out_dir` and then moved the staged model in, and its error handler also deleted the staged model, so a failure between the two steps (an antivirus or indexer lock, disk full) left neither copy. It now renames the previous model aside, swaps the merged model in, restores the previous model if the swap fails, and removes the aside copy only after the swap succeeded. If the restore also fails, both copies are kept and both paths are named in the error; if the rename-aside itself fails, `out_dir` is left unchanged and the staged model is kept and named. Interrupts (Ctrl-C) are no longer turned into a "the old model is unchanged" error. The docstring says what the swap guarantees. `soup reward synth -o reward.py --force` overwrote the existing verifier with the candidate before calibrating it, then deleted the file when calibration refused or failed, which destroyed a previously working verifier. It now writes the candidate to a `.soup.reward-candidate.*.py` sibling, loads and calibrates that file, re-validates `-o`, and replaces `-o` only once calibration accepts. A refusal (exit 2), a calibration error (exit 1), a failed candidate write or a failed final replace leaves the previous verifier byte-identical and removes the candidate. Verified: 11 new tests (the Windows open-handle case runs on Windows); 367 tests across the related files pass on the PR merged with `main`; the 11 mutations of the fix all fail a test; the new file gives 9 failed / 2 passed on `main`. Closes #1450. commit 2bcdbc9a7c6f50d7fe40199d6e86753b4320484e Author: Nurkhan Esenbek <[email protected]> Date: Thu Oct 1 17:18:42 2026 +0500 fix(data): wire classifier, reranker, and cross_encoder data loading (#1219) (#1306) `soup train` could not train the sequence-classification family. For `task: classifier` and `task: reranker` the loader kept source columns only for `grpo`, so `label` was dropped during format normalisation and the run died after the model loaded with `TypeError: label must be int / str / list, got NoneType`. For `task: cross_encoder`, `{text_a, text_b}` and `{question, answer}` pair rows could not be loaded at all. `--dry-run` said `Ready to train!` for all of them. Source columns are now kept for the whole family (`task_preserves_source_columns`, shared by `soup train` and `soup sweep`). `data.format: cross_encoder` exists, and `format: auto` resolves to it only when `task: cross_encoder` (`data_config_for_task`); `detect_format` and `FORMAT_SIGNATURES` are untouched, so GSM8K-shaped `{question, answer}` rows are still refused for `grpo`, `sft` and `pretrain` exactly as before. `data.format: cross_encoder` on any other task is refused at config load. Every classifier-family row (train and val) is checked on `--dry-run` and before the model loads, and an error names the row (`train row 3: missing required 'label' field for task 'cross_encoder'`). When `training.label_names` is set, `id2label` / `label2id` are written to the saved `config.json`. `docs/data.md` and `docs/training.md` document the row shapes per task. Verified: 791 tests across 16 files pass on the PR merged with `main`; 13 of 13 mutations of the fix fail a test; the new pin file gives 6 failed / 4 passed on `main`. Closes #1219. commit 387ccb3c63d9d0998bec239890ffe0cfcc4f3ab9 Author: UmranPros <[email protected]> Date: Thu Oct 1 17:48:02 2026 +0530 fix(data): from-traces reads the signals Soup's own producers write (#1440) (#1498) fix(data): from-traces reads the signals Soup's own producers write (#1440) `soup ingest` writes a top-level `signal` (`thumbs_up` / `thumbs_down` / `none`) into the trace stream that `soup data from-traces` and the loop harvest stage are documented to consume, but `parse_soup_serve` read the signal only from a nested `feedback.rating` that no Soup producer writes, and no parser ever set `edited_output`. So every ingest row read as `none`, `--signal thumbs_up` and the loop harvest built no pair, `--signal user_edit` could never match any log, and the command ended with a green `Wrote 0 preference pair(s)` and exit 0. `parse_soup_serve` now reads the top-level `signal` when it names a value of the closed vocabulary (`thumbs_up`, `thumbs_down`, `regenerated`, `user_edit`; anything else, including non-string values, reads as `none`) and keeps `feedback.rating` as the fallback for older logs; a top-level `signal` wins over a conflicting `feedback.rating`. `edited_output` is read from `edited_output`, `edited_response` or a nested `feedback.edited_output` (non-string and whitespace-only values ignored). When `from-traces` reads traces but builds no pair it now prints how many it read, which `--signal` it wanted and which signals were present (or that no trace carried one); an empty directory stays silent, and the `soup ingest` record-shape hint is printed only for `--format soup-serve`, the one parser that reads a top-level `signal`. `docs/data.md` is corrected: `--format soup-serve` and `--signal regenerations` are the accepted spellings, `--logs` for the soup-serve parser is a directory, and the record shape is documented with who writes which signal (`soup ingest` writes `thumbs_up` / `thumbs_down` / `none` under `trace_id`; `user_edit` and `regenerated` rows come from your own pipeline; `soup serve --trace-log` records carry no signal yet). Users notice that the round trip `soup ingest --source langfuse` then `soup data from-traces --format soup-serve --signal thumbs_up` now writes the pair instead of an empty file, and that a log with no usable signal says so instead of reporting a normal write of 0. Not in this change: `soup serve --trace-log` records still carry no signal and the harvest does not read the `--record-thumbs` database, so the loop still cannot learn from Soup's own serving; that producer half stays open on the issue. A single-file `--logs traces.jsonl` with `--format soup-serve` still ends with a green `Wrote 0` (the parser reads a directory); documented, not changed. Verified: the new test file fails 11 of 16 on the parent commit and passes 16 of 16 on the change; 414 tests across 9 related files pass on the merged tree; ruff clean; the round trip, the three `user_edit` shapes, a serve-trace-log shaped directory and hostile `signal` / `edited_output` / `feedback` / `id` values were executed through the real CLI; 19 mutations of the parsers and the message were applied one at a time and 15 are killed by the tests (the survivors are the openai/langchain hint gating, `regenerated` clamp membership and the non-dict `feedback` guard, each checked by hand). Refs #1440. commit d73e281187fdc6cd59dc64c4fb6f6767f21a6120 Author: Kratos <[email protected]> Date: Thu Oct 1 20:17:48 2026 +0800 Fix #1445: validate compile / GGUF-export / unlearn destinations before the expensive step (#1469) Three commands validated their destination only after the expensive step and then threw the finished work away. `soup compile -o <path outside cwd, or a symlink>` ran the whole optimizer and its LLM calls, then refused the path with an unhandled `ValueError` (exit 1). A quantized GGUF export into a directory that did not exist yet crashed in `tempfile.mkdtemp` after the adapter had been merged, and the cleanup deleted the merged model. `task: unlearn` loaded the policy and a frozen reference copy and ran every NPO / SimNPO / RMU step before refusing an `output` outside the working directory or a symlinked one. `build_compile_plan` now runs the containment and symlink check at plan time, so the optimizer is never called and the command exits 2 (the write-time check stays as the TOCTOU guard, and its `ValueError` is caught and exits 2 as well); `soup compile --plan-only` now refuses such a destination too instead of printing the plan and exiting 0. The export creates the destination's parent directory before the merge. `UnlearnTrainerWrapper.setup()` calls `_validated_output_dir` first, before torch, peft or any model is loaded; the check after the run stays as the TOCTOU guard. Verified: the new test file (9 tests, each with an in-cwd control) fails 6 of 9 on `main`; 4 of 4 mutations of the fix are killed (export `mkdir`, plan-time check, write-time `except`, unlearn `setup()` call); 1362 related tests pass; the real CLI run before and after for all three commands, including a Windows junction pointing outside the working directory. Closes #1445. commit 1bc20670c3c0245fb4a09f34b6e57340061e6960 Author: Mohammad Hijjawi <[email protected]> Date: Thu Oct 1 13:02:14 2026 +0100 fix(local-rl): render --min-pairs and -o into the nightly-train job (#1448) (#1517) Without `--once`, `soup local-rl train` renders a systemd `.service` / `.timer` and a launchd `.plist` that run `soup local-rl train --once` every night, but the rendered command carried `--db`, `--model` and `--train-method` only. The nightly run therefore trained on as few as 10 pairs into `local_rl_adapter/` whatever `--min-pairs` and `-o` the user had passed, and nothing said so. `build_train_argv`, `render_systemd_service`, `render_launchd_plist` and `write_scheduler_files` now take `min_pairs` and `output_dir` and always render `--min-pairs N --output DIR`, and the CLI passes both through, so the job does what `--once` with the same flags does. Both values are checked with the runner's own validators (`validate_min_pairs` / `validate_output_dir`, moved out of `run_nightly_train`), so a value the nightly run would refuse, such as `--min-pairs 0`, exits 2 before any scheduler directory or unit file is written, and a newline in the output path is refused like `--model` and `--db`. `docs/commands.md` lists the flags in the scheduled-mode line. `--backend` (the other half of this bullet in #1448) is unchanged. Verified: 14 new tests (12 fail on `main`), 309 passed across the neighbouring local-rl and docs-ratchet files, the issue's repro through the real CLI renders `--min-pairs 200 --output adapters/nightly` into both the `.service` and the `.plist`, and 15 of 16 mutations fail a test. Part of #1448. commit 65265a8b86abae367a3e5a0ad430749fb9f463d5 Author: Mohammad Hijjawi <[email protected]> Date: Thu Oct 1 13:01:56 2026 +0100 test(peft): count the MLX LoRA builder in the LoRA+ wiring scan (#1324 follow-up) (#1516) The #724 wiring scan counted only `get_peft_model(` and a `peft_config=` handed to trl as "builds a LoRA adapter", so `mlx_sft.py`, which builds its adapter with mlx-lm's `linear_to_lora_layers`, was neither wired nor exempt. That is how `backend: mlx` ignored `training.loraplus_lr_ratio` without a test noticing (#1324). The scan now counts `linear_to_lora_layers`, `mlx_sft.py` is exempt with its reason (the config-load refusal from #1383), and a new test fails when an exemption reason names a test file that does not exist. `tests/test_issue1324_mlx_loraplus_refusal.py` gains the cases #1383's file lacked: every ratio value, direct `SoupConfig(...)` construction, an explicit `null` ratio, the `transformers` / `unsloth` controls on `pretrain` and `dpo`, plain `soup train` stopping at config load, and `soup doctor --config` exiting 2. `docs/peft-and-efficiency.md` states that `backend: unsloth` gets the same LoRA+ optimizer as `transformers`. No source change and no behaviour change. Verified: 135 passed on the tree merged with `main`; six mutations (scan regex, exemption removed, exemption pointing at a renamed file, docs sentence, #1383 refusal deleted, refusal widened to `unsloth`) each fail 1 to 9 tests. Follow-up to #1324 / #1383. commit f47d6668ba0e08f790a7594ebd2f0ba8ff58c14c Author: Abdulwaarith Zakariyya <[email protected]> Date: Thu Oct 1 14:59:03 2026 +0300 fix(ab): decide soup ab with a variance-robust NIG Bayes factor, no burn-in (#1265) (#1419) `soup ab` estimated the row-to-row variance from the rows it was testing, so the #1227 two-sided mSPRT needed a burn-in of 30 or 40 rows per arm, was calibrated only up to 1000 rows and down to `--alpha 0.01`, and still sat near 0.011 at 0.01. It now decides with the normal-inverse-gamma mixture Bayes factor (right-Haar limit), which depends on the rows only through the two-sample t statistic, so its false-positive rate stays at most `--alpha` however often `soup ab` is re-run: the worst simulated rate is 0.033 at `--alpha 0.05` and 0.0066 at 0.01, against 0.051 and 0.011 before (22 ratios, 100,000 runs each; record and harness in `benchmarks/gate-1265-ab-nig.md`). The first 5 rows per arm set the scale for `--effect-size` and show as `prior_scale`; the burn-in table, the 1000-row limit and both calibration warnings are gone. The cost is stated in the docs and the fragment: with no difference, `accept_h0` takes several times as many rows (281 pairs against 43 at 0.5 standard deviations), and the "under 0.4%" wrong-accept figure is for simulated Gaussian rows. When the held-out rows barely vary (a warm cache, a judge that gives 1.0 early), `accept_h0` is held at `continue` while the tested rows' standard deviation is more than 3 times the held-out one (`ACCEPT_HOLD_SPREAD_RATIO`); a `reject_h0` is never held back, so the false-positive bound is unchanged. In a 2000-run simulation that start had ended 99.0% of runs in a wrong `accept_h0` at about 7 pairs, and with the hold it ends none that way; on Gaussian rows 15 of 6000 first verdicts change and no rejection does. The `--alpha + --beta < 1` refusal from #1339 stays with its reason restated, and #1339's and #1384's refusals of values a float cannot carry cover the new statistic, which is never compared with the boundaries unless finite; an `--effect-size` past about 1.9e151 standard deviations of the held-out rows, or held-out rows whose variance underflows to 0, is refused at every row count. Verified on the head merged with `main`: 208 passed across the four `soup ab` files and three neighbours, 16 of 17 mutations of the hold and the prior bound killed (the survivor, `>=` for `>`, differs only at exact float equality), the harness self-check matches the shipped code, and the new tests fail on `main` (50 failed / 12 passed in `tests/test_issue1265_ab_nig.py`). A faster accept rule is #1418. Closes #1265. commit 0ae08c7fbbf40f4744d90bdb7129b08147b4c780 Author: Kratos <[email protected]> Date: Thu Oct 1 19:56:14 2026 +0800 Fix #1434: soup expect fails closed on BOM, malformed and non-object JSONL rows (#1457) `soup expect` could pass a dataset it never checked. It opened the file as plain `utf-8`, so a UTF-8 BOM broke the first line's JSON and that row was silently skipped (the issue's repro passed with `Rows 1` where the same file without the BOM failed on the PII in row 1). Malformed JSON lines were counted into a yellow "skipped" note, valid JSON that was not an object (`[1, 2]`, `"text"`, `42`) was dropped without a word, and a file with no checkable row at all passed with `Rows 0`. As a CI gate, that is a run that checked nothing and exited 0. The loader now reads `utf-8-sig`, like the training loader, so a BOM is stripped. Any malformed line or non-object line makes the run refuse with exit 3 (input error, distinct from exit 2, a real suite failure), naming the count and the first ten line numbers with `(+N more)`. A file that yields no checkable rows is refused the same way. `docs/compliance.md` and `docs/data.md` now state exit 3, the BOM strip and the zero-row refusal. Reusing the training loader so `.json`/`.csv`/`.parquet` are read too stays out of scope: a `.json` array file is refused (fail-closed) rather than read. Verified on the PR merged with current `main` (which includes #1461): the 9 new tests (all fail on `main`) plus the existing expect suites, 247 passed; `ruff` clean; twelve mutations of `commands/expect.py` (encoding, zero-row check, uncheckable check, both recording branches, count, line number, ten-line cap, `(+N more)`, exit code) all killed; by hand against `main`, a BOM file goes from exit 0 to exit 2, array-per-line, empty and malformed files from exit 0 to exit 3, and a clean file still exits 0. Merged cleanly together with #1499 and #1505, which touch the same module. Closes #1434. commit 8bfeaa2e82887d533a1400cdb289a9da2cd4eb88 Author: Kratos <[email protected]> Date: Thu Oct 1 19:56:02 2026 +0800 Fix #1429: `tokenizer train` saves a complete byte-level BPE (decoder, byte alphabet, no hard-coded unk) (#1455) `soup tokenizer train` saved a byte-level BPE that could not round-trip text. The trainer attached no `ByteLevel` decoder, so `decode(encode(x))` returned `Ġ` for every space and `h ll o Ġ Ġ` for `héllo 😀 世界`. It also did not seed the 256-byte initial alphabet, so characters unseen in the corpus (accents, emoji, CJK) encoded to `<unk>`. And it hard-coded `unk_token="<unk>"` into the model, so passing `--special-token` without `<unk>` saved a tokenizer that names an unk token missing from its own vocabulary, and encoding then raised `Unk token <unk> not found in the vocabulary`. The trainer now attaches `decoders.ByteLevel()`, seeds `ByteLevel.alphabet()` as the initial alphabet, and builds the model with no `unk_token`. `--vocab-size` is checked after the special tokens are known: it must be at least `256 + len(special tokens) + 1`, which is the true boundary (measured with `tokenizers` alone: any size up to 256 + specials yields a vocabulary with zero merges and no error, so `main` accepted `-v 256` and saved a merge-less tokenizer). Below that the command exits 2 and names the minimum. Users notice that ASCII, accented, emoji and CJK text round-trips through `Tokenizer`, `PreTrainedTokenizerFast` and `AutoTokenizer`, and that a custom `--special-token` run encodes and saves `model.unk_token = null`. Verified on the PR merged with current `main`: the 9 new tests (7 fail on `main`, the other 2 are the sizes `main` also accepts) plus `tests/test_v0539.py` and the changelog-fragment tests, 87 passed; `ruff` clean; eight mutations of `tokenizer.py` (decoder removed, initial alphabet removed, hard-coded unk restored, floor disabled, `+ 1` dropped, floor ignoring specials, wrong message number), each killed by the new tests; a manual `soup tokenizer train` at `-v 260` (exit 2) and `-v 261` (exit 0, exact round trip). `tokenizer_config.json` is deliberately left out, as the issue asks. Closes #1429. commit 4b5f4a3cda36d482278bd4400da51a204cca2b39 Author: Srinivasan R <[email protected]> Date: Thu Oct 1 17:05:42 2026 +0530 fix(recipes): drop the recipes that cannot load; MiniMax-M3 SFT loads through the vision path (#1145) (#1398) Four shipped recipes named a repo that exists but a model that cannot load as written (#1145). `mistral-large-3-sft` and `mistral-large-3-dpo` are removed: `mistralai/Mistral-Large-3-675B-Instruct-2512` publishes Mistral's native `params.json` + `consolidated-*.safetensors` layout with no `config.json`, which `AutoConfig` cannot read; `docs/models.md` now says so and names the Hugging Face-format conversion on the Hub as a fact, not a recommendation. `minimax-m3-dpo` is removed, because the DPO trainer builds `AutoModelForCausalLM` for every modality and cannot build the `minimax_m3_vl` wrapper (#1393). `minimax-m3-sft` now sets `modality: vision`, so SFT builds the image-text wrapper and `target_modules: auto` keeps LoRA on the language tower; it drops `moe_lora` (refused on SFT vision by #1179) and `moe_aux_loss_coeff` (read only on the text path), and because that path loads `AutoProcessor`, which is custom code for this base, its description says to run `soup train --trust-remote-code` (verified: the flag reaches the vision processor). The recipe YAML also notes that the vision collator supervises every token, so `train_on_responses_only` is not applied on this path (#1156). The catalog loses three recipes (173 -> 170, 112 bases); every count site, both recipe fixtures, the MoE tables in `docs/performance-and-quantization.md` (29 recipes set `moe_lora`: 14 SFT, 8 GRPO, 7 DPO; MiniMax 0, kimi 7) and the two attach-ratchet pins for MiniMax are updated. A new test pins the `docs/models.md` paragraph. Users notice: `soup recipes list` shows three fewer recipes, and `minimax-m3-sft` attaches LoRA (16 modules on the meta device) instead of failing at load. Verified on CPU: 2446 tests over 20 recipe, count, snapshot, ratchet and docs-guard files, 0 failed; `scripts/check_recipe_attach.py` exits 0 (118 verified, 18 pinned, 0 new, 0 now attach); 9 of 9 mutations fail a test; the PR's tests fail on unmodified `main` (14 failed, 642 passed). Closes #1145. commit aae7508e88f6f7261d2e001df86e987bc2d5e870 Author: Alpamys <[email protected]> Date: Thu Oct 1 16:26:16 2026 +0500 docs(contributors): fix Bakdaulet's entry, the first-contribution text for #1489 was missing The entry written with the batch credit repeated the name line instead of the description. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> commit d70398d5113b106eb2405b096dcae8a2b97cbfe6 Author: Alpamys <[email protected]> Date: Thu Oct 1 16:25:33 2026 +0500 docs(contributors): #1308, #1460, #1461, #1168, #1366, #1462, #1463, #1364, #1492, #1493, #1489, #1416, #1484, #1490, #1479, #1485, #1495 Display names from `gh api users/<login>`. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> commit 7e681f07a72e529ce26276af4d8350011a19c0b2 Author: Ashutosh Panda <[email protected]> Date: Thu Oct 1 16:54:00 2026 +0530 fix(tests): pin #1408 helper with regression tests (#1495) When the suite ran from inside a virtualenv whose base interpreter has none of the dependencies (the CONTRIBUTING setup on, for example, Homebrew Python), `test_worker_verification_survives_a_real_venv_launcher` failed with an empty `RuntimeError: MLX teacher worker exited with 1:`. The child venv is created with `system_site_packages=True`, which from inside a venv exposes only the base interpreter's site-packages, not the outer venv's, so the worker died importing pydantic before writing anything and the failure looked like a worker bug. The test now writes a `.pth` file into the child venv that lists the outer venv's site-packages and `src/` (a no-op on a bare interpreter, so the test is unchanged for everyone running from a global Python), and probes `import pydantic` in the child first, so a missing link names the missing dependency instead of surfacing an empty exit. Two hermetic tests pin the helper on every platform, including the POSIX layout on CI: a package that exists only in a fake parent site-packages becomes importable in a child venv after the helper runs and not before, and nothing is written on a bare interpreter. Test-only; no user-visible change. Verified: the test file gives 19 passed from a bare interpreter and from a venv over a dependency-free CPython; clean `main` from that venv gives 1 failed, 16 passed (the reported error); four mutations of the helper (never write the `.pth`, `src/` only, wrong suffix, inverted early return) each fail the new tests, and removing the helper's call is caught from the dependency-free venv with a message naming pydantic. Closes #1408. commit 200fc21af339705f960bdd997db7cd7da2d48862 Author: dymux <[email protected]> Date: Thu Oct 1 18:23:48 2026 +0700 fix(cli): write machine-readable JSON straight to stdout (#1480) (#1485) `soup version --json --full`, `soup cost --json` and `soup eval against --json-only` printed their machine-readable JSON through Rich's `console.print`, which folds text at the console width (80 columns when piped) and reads `[...]` in a string as markup. A `version --json --full` record was split across lines, a `cost --json` value such as `[v2]` was dropped while the command exited 0, and `[/x]` (in a cost value or an `eval against` baseline id) aborted the command with `MarkupError`. All three now write with `typer.echo`, the convention the other machine-readable paths use; the human-readable branches still go through the Rich console. A repo-wide AST ratchet fails on any `console.print(<serialiser>.dumps(...))` under `src/soup_cli/`, with a self-test so a detector that stops matching cannot pass as a clean tree. `soup eval against --json-only` is a third site the issue did not name; it is reached through a `_json` alias that a grep misses. Users notice: one JSON document on one line, and bracketed text in a value survives. Verified: the new file is 6 passed (5 failed / 1 passed on `main`); 355 passed and 1 skipped across 12 related files with `main` merged (the one other local failure, `tests/test_cli.py::test_version_full_json`, fails the same way on `main`); each of the three sites reverted on its own fails its own behavioural test plus the ratchet (2, 3 and 2 failures), and both detector mutations fail the self-test. A real piped `soup version --json --full` and `soup cost --json` both parse with `json.load`. Closes #1480. commit 0f7ab958525d619a7aaa55d46743c2130a52dad8 Author: Younes Beriane <[email protected]> Date: Thu Oct 1 13:23:37 2026 +0200 fix(serve): honour --device in the transformers loader (#1479) `soup serve --device` was ignored by the transformers loader: `_load_model` (both the merged-model and the base + adapter branch) and the speculative draft loader hard-coded `device_map="auto"`, so `--device cpu` could still put the model on a visible GPU, and an unknown value such as `tpu` was accepted and silently dropped (`--device "[/bold]"` died with a Rich `MarkupError`). `soup serve` now places the model with the same rule as `soup infer` / `chat` / `diff` (`resolve_inference_device_map_and_dtype`): `cpu` and `mlx` load on the CPU in float32, `mps` and `cuda:N` are pinned to that device in float16, `cuda` keeps `device_map="auto"` in float16, and an explicit `--kv-cache-type bf16` / `f16` still sets the dtype on every device. `--device` is validated before anything loads, on every backend, and an unknown value exits 2 naming `cpu, cuda, cuda:<index> or mps`. The draft model is placed like the main model. Users notice: `--device cpu` really runs on the CPU; on a CPU-only host the default load is now float32 pinned to the CPU, which doubles its RAM (a 7B model needs about 28 GB instead of 14 GB); `cuda:N` is honoured (it was accepted and ignored before); the `--device` help describes all of it. Verified: the new file is 34 passed (19 failed / 15 passed on `main`), 627 passed across 17 related files, 34 passed again under forced colour at 60 columns; 23 of 24 mutations of the fix fail a test (the survivor is validation gated on the transformers backend, which the CLI does not do on vLLM or SGLang); the real CLI and a tiny real Llama loaded on CPU give the placements and dtypes above. Closes #1478. commit 2aeda018094c242dfbec11a059c922c08be005d0 Author: Momoyeyu <[email protected]> Date: Thu Oct 1 19:23:25 2026 +0800 fix(eval): score custom tasks by stated answer; token-bound contains (#1342) (#1490) Custom eval tasks scored a model's stated answer wrongly in both directions. `scoring: "contains"` was a raw substring test, so an expected `4` was paid by `14` and an expected `42` by `#### 420`; `scoring: "exact"` compared the whole output, so `The answer is 4.` scored 0 against `4`. `contains` now requires the expected string to appear as a whole alphanumeric-bounded token (case-insensitive), so `14` no longer contains `4` and `#### 420` no longer contains `42`. A new `scoring: "answer"` compares the final answer the model states, read with the same parser GRPO's `accuracy` reward uses (`#### 4`, `\boxed{4}`, `The answer is 4.`; numbers compare by value), and refuses at load time an `expected` that states no single answer, naming the line, instead of silently scoring 0. The eval gate's `scorer:` override accepts `answer` too. `exact` keeps its whole-output meaning and is now documented as such. This changes scores. A gate baseline (`soup eval gate --baseline` file or `registry://` row) or a `soup eval custom --run-id` result recorded with the old substring `contains` can be higher than the same outputs score now, and the baseline's scorer stamp covers only the bundled suites, so it does not warn: re-measure those before comparing. The changelog and `docs/evaluation.md` say so. Verified: 30 new tests (15 fail on `main`), 17 of 18 review mutations caught, 433 related tests passed, `ruff` clean. Closes #1342. commit a07d6ba2f9a12c028090db251ddbe4b5ed6bbb25 Author: Srinivasan R <[email protected]> Date: Thu Oct 1 16:53:13 2026 +0530 fix(eval): retry judge requests and report an unreachable judge instead of scoring it as a tie (#1447) (#1484) Every LLM-judge failure was either scored as a tie or aborted the whole batch, and nothing was ever retried. `compare_pair` turned any judge exception into a tie, so `soup ship --task-mode pairwise` with the judge down scored a measured 0.5 win-rate and printed DON'T SHIP; `soup eval judge` died on the first transport error and lost the scores already collected; a training eval gate logged an unreachable judge as a regression. Every judge request now goes through one helper. It retries 429, 5xx and transport errors twice (3 attempts, pause 1 s then 2 s, or `Retry-After` when it is a finite number, capped at 30 s), fails at once on a 4xx that will not heal or a reply it cannot decode, treats a reply with no usable content as a judge error, and raises `JudgeUnavailableError` naming the judge (scheme, host, port and path only, never userinfo or a query string) once the retries are spent. `soup ship --task-mode pairwise` exits 1 with that message instead of a verdict; `soup eval judge` skips and counts the failing row and still scores, shows and saves the rest (`1/3 items skipped`); the eval gate still fails closed, and its log line says which task could not be scored and why rather than calling it a regression. A judge that answers "tie" still scores 0.5, and the Online DPO adapter keeps leaving a failed pair unranked, as the ranking trainer expects. Review found three more ways to escape that design, all fixed before merge: `Retry-After: nan` reached `time.sleep` and raised a raw `ValueError`; the judge URL, userinfo and `?api_key=` included, went into the message, `GateTaskResult.error` and the training log; and an undecodable reply (`httpx.DecodingError`) aborted a run. Verified against a real local HTTP server as well as through the CLIs: 45 new tests, of which 40 fail on `main` with the new names shimmed in; 17 of 19 review mutations caught; related suites 470 passed. Closes #1447. commit e967397f585efb488f9464801eb2786efcbe38dc Author: Ashutosh Panda <[email protected]> Date: Thu Oct 1 16:53:01 2026 +0530 fix(trainer): read trl PPO per-step loss from loss/policy_avg (#1413) (#1416) A completed `task: ppo` run printed `Loss: unavailable` even though training worked: `summarize_training_loss` only read the `loss` key of `trainer.state.log_history`, while trl 0.29's `PPOTrainer` logs its per-step policy loss under `loss/policy_avg` and never writes `loss` or `train_loss`, so every entry was skipped and the run ended with "unavailable". `summarize_training_loss` now takes the per-step key from the caller (`loss_key`, default `"loss"`, so SFT/DPO summaries are byte-identical), and the trl path in `trainer/ppo.py` declares `loss_key="loss/policy_avg"` instead of the summary guessing it; the manual PPO loop keeps the default because it normalises to `loss`. A non-default key is recorded in the result as `loss_key` on the `delta` and `single` branches only (never on the `train_loss` mean fallback or an `unavailable` result), and the completion panel shows it, e.g. `Loss: 1.4200 -> 0.8700 (loss/policy_avg)`, so a PPO policy loss is not read as an SFT cross-entropy. SFT panel lines are unchanged. Verified: against the real trl 0.29.1 `PPOTrainer` through `PPOTrainerWrapper.train()` on a tiny CPU Llama the result is `delta` with `loss_key: loss/policy_avg` (main: `unavailable`); the 9 new tests fail 6 of 9 on main and kill 11 of 11 mutations of the fix and the panel (including moving the key to the manual-loop call and summarising an empty history, via a behavioural test that drives `_train_builtin` on a trl-0.29-shaped `log_history`); 24 related test files give 1268 passed, 0 failed on the merged tree. Three optional mutations (re-tagging the mean fallback or the unavailable result) are not pinned by a test; trl's PPOTrainer never logs `train_loss`, so nothing is wrong today. The live per-step display still does not see `loss/policy_avg` (`SoupTrainerCallback.on_log` reads only `loss`); that is a separate gap. Closes #1413. commit f562f79d4af4ec0a52c0a785c7522d3f80c38989 Author: Bakdaulet <[email protected]> Date: Thu Oct 1 16:22:51 2026 +0500 fix(diagnose): report unmeasured probes as NOT_RUN instead of OK (#1489) fix(diagnose): report unmeasured probes as NOT_RUN instead of OK 1.00 (part 1 of #1435) `soup diagnose` scored a live probe that never produced a measurement as a perfect pass: a `--dataset` with no usable rows, a probe that raised (for example a 9000-character prompt over the 8192 limit), a typo in `--tokenizer`, or memorization on `question`/`answer`, `query`/`response` or `input`/`output` rows, which `score_memorization` cannot read. The overall verdict, the exit code and the badge all treated those as OK 1.00 and the exception text was dropped. This adds a `NOT_RUN` verdict (score 0.0 only, ranked above MINOR and below MAJOR, grey in the badge and the table, shown as `not run` / `—` instead of a score). All six converted probe sites keep the exception text in the evidence (NUL-stripped, capped, Rich-escaped), and memorization reports `NOT_RUN` when none of the first rows has text to split. `soup diagnose` exits 3 when the overall verdict is `NOT_RUN`; `--allow-not-run` keeps exit 0 for runs that knowingly skip probes; MAJOR still exits 2 and is never hidden. `--tokenizer` is resolved before any model loads, so a bad or empty value is an input error (exit 3). `train --diagnose-gate`, the `--evidence` path and the MCP `diagnose_evidence` tool understand `NOT_RUN` through one shared helper. Probes that do not apply (no `--dataset`, non-JSON format targets, no benchmark corpus, modes omitted from an evidence file) stay a neutral OK. Users notice: live runs that used to exit 0 on an unmeasured probe now exit 3, and ShareGPT / LLaVA / plaintext datasets read as `NOT_RUN` until part 2 of #1435 teaches the row builders those shapes. Known and declared gap: memorization still reads OK when every text is too short to split; it waits on #1481. Verified on the PR head merged with main (CPU): 175 passed across the two touched test files (39 of them fail on clean main), 1212 passed across 17 diagnose-related test files, ruff clean, 19 of 20 mutations of the new logic killed (the survi…
Post-release documentation corrections for v0.76.0; no code change. `CHANGELOG.md`: the `[0.75.2]` Security entry now names GHSA-9f49-hqpf-c849 with its link instead of saying the advisory "will be published", and the second bullets of the #1360 (@taljeon) and #1460 (@drakeo338) entries carry the same `by @user in #PR` credit as their first bullets. `docs/performance-and-quantization.md`: the empty-adapter note says v0.75.1 and every later release carry the #1010 fix (the backport commit is an ancestor of the `v0.75.1` and `v0.75.2` tags), and two passages about the GGUF ladder and the 4-bit save formats are in the present tense. `docs/serving-and-export.md`: the live `/v1/messages` route is described as present, not "lands in v0.45.1". `changelog.d/README.md`: the example path uses `0.76.0`. `CONTRIBUTING.md`: the key-directories tree lists `src/soup_cli/bench/`, and 28 test-table rows no longer use internal lettered-section wording (every test file name is unchanged). Docs only, no changelog fragment.
What does this PR do?
Fixes #1443.
soup infer,soup chatandsoup diffecho the resolved--devicein their plan panel but their_load_modelhelpers always passdevice_map="auto"andtorch_dtype=torch.float16tofrom_pretrained. With a visible GPU,--device cputherefore still loads the model onto the GPU, becauseacceleratefills available accelerators first. This change pins the model to the resolved device and picks a device-appropriate dtype (float32on CPU,float16elsewhere) instead of always requesting"auto".Type of change
Checklist
ruff check src/soup_cli/ scripts/ tests/passespytest tests/ -vpasses — not run (heavy torch-dependent full suite); ran the targetedtests/test_issue1443_device_map.py(7 passed) plus the two adjusted pre-existing tests (both green)README.mdand the matching page underdocs/) if needed — no doc mentionsdevice_map/--deviceloading behavior🤖 Generated with Claude Code