Skip to content

fix(serve): honour --device in the transformers loader - #1479

Merged
MakazhanAlpamys merged 3 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1478-fix
Oct 1, 2026
Merged

MakazhanAlpamys merged 3 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1478-fix

Conversation

@drakeo338

Copy link
Copy Markdown
Contributor

Fixes #1478.

soup serve --device was accepted but the transformers loader always passed device_map="auto", so --device cpu could still put the model on a visible GPU. The loader now pins device_map="cpu" for --device cpu, loads float32 there (an explicit --kv-cache-type f16 is still honoured), and --device rejects unknown values with exit code 2. The new tests in tests/test_issue1478_serve_device_map.py fail on the base commit and pass here.

What does this PR do?

See above.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Tests

Checklist

  • ruff check src/soup_cli/ scripts/ tests/ passes
  • pytest tests/ -v passes (only the serve test file was run, 61 passed; not the full suite)
  • Updated relevant docs (README.md and the matching page under docs/) if needed (only the --device help text changed)
  • Added a changelog fragment, or this change has no user-visible impact

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drakeo338, thank you for picking this up so quickly. The fix for the case you targeted is correct and I confirmed it by running the real CLI. What is left is that four behaviours are not pinned by any test (two of them are the ones that would hurt users most), and that this PR should use the device helper from #1460 instead of a second rule.

What is good

  • --device cpu now reaches from_pretrained as device_map="cpu" and float32 on both loader branches. I ran soup serve --device cpu through the CLI with the loaders mocked, for a plain model and for --base + adapter, and recorded the kwargs: main gives device_map='auto' / float16 for both, your head gives 'cpu' / float32. I also loaded a tiny random Llama through the real _load_model on CPU with transformers 5.17: parameters on cpu, float32; with --kv-cache-type f16 / bf16 they come out float16 / bfloat16.
  • Testing the plain and the base + adapter branch separately is what makes the "either call going back to auto" criterion real: reverting either from_pretrained call fails a test (I checked both).
  • The validation runs before the "Loading model" panel. A side effect worth knowing: on main, soup serve --device "[/bold]" dies with a Rich MarkupError (exit 1); on your head it is refused with exit 2.
  • The fragment credit line has the right shape (#1478 by @drakeo338 in #1479), the fragment validator passes (236 fragments), and it states the CPU-only float32 default change. I confirmed that change by execution: no --device on a CPU-only host now gives 'cpu' / float32; with a GPU detected it is unchanged ('auto' / float16).
  • --backend vllm|sglang are unaffected: with both backends stubbed, device is still not passed to _serve_vllm / _serve_sglang; it only feeds the panel.

What I ran

tests/test_issue1478_serve_device_map.py plus 14 adjacent serve files (serve, speculative decoding, kv-cache, vLLM / SGLang / MII, multi-adapter, request guard, auto-quant, MoLE) on the merge with current main (9c6376b64): 556 passed, ruff clean. On clean main your 11 tests give 7 failed, 4 passed (the 4 are characterisation tests: explicit dtype, cuda/mps unchanged). Then 20 mutations, one at a time, against your 11 tests: 16 are killed, 4 survive, and all 4 also survive the 556 adjacent tests. Every mutation was undone (git diff empty).

mutation (serve.py) result
if device not in _SERVE_DEVICES: -> if device not in ("mps", "cpu"): (--device cuda exits 2) 11 passed
same with ("cuda", "cpu") (--device mps exits 2) 11 passed
on_cpu = device == "cpu" -> device in (None, "", "cpu") (no --device treated as CPU) 11 passed
drop the _rich_escape(...) around the echoed value (--device "[/bold]" exits 1 with a MarkupError) 11 passed

Please fix

  1. Blocking: nothing pins that the valid devices are accepted. The four refusal cases and " CPU " are all you test through the CLI, so soup serve --device cuda exiting 2 (every GPU user's spelling) passes all 11 tests and all 556 adjacent ones. The same goes for mps, for device=None reaching the loader as "not CPU" (your second acceptance criterion, "no --device behaves as today"), and for the escape on the echoed value. The test file at the end of this comment kills all four: against your head, its CLI-acceptance, markup and device=None cases pass (8 passed) and fail under the four mutations above (2, 1, 2 and 1 failed, in the table's order).

  2. Blocking: use the device helper from #1460 instead of a second rule. #1460 is approved at 34f1fa06 and is waiting for its last required checks, so it lands first; this PR should land after it, and #1478 asked for exactly that ("reuse the device-to-device_map helper that #1460 introduces"). Today the two rules disagree:

    --device this PR (soup serve) resolve_inference_device_map_and_dtype (infer / chat / diff)
    cpu "cpu", float32 {"": "cpu"}, float32
    cuda "auto", float16 "auto", bfloat16 where the GPU supports it, else float16
    cuda:1 exit 2 (it was accepted and ignored on main) {"": "cuda:1"}
    mps "auto", float16 {"": "mps"}, float16
    mlx exit 2 {"": "cpu"}, float32

    The cuda:N row is the one that matters: a script that passes --device cuda:0 to soup serve works on main (the value is ignored) and would start failing with exit 2, while the same spelling is honoured by soup infer. With the helper it keeps working and is pinned to that GPU. _load_draft_model (serve.py:1507-1508) has its own copy of the old rule too, so with pinned cuda:N / mps the main model and its draft would land on different devices again; it should call the helper as well. Once #1460 is on main, please rebase and apply:

    # the --device block in serve() (was serve.py:464-472)
    if device:
        from rich.markup import escape as _rich_escape
    
        from soup_cli.utils.gpu import resolve_inference_device_map_and_dtype
    
        device = device.strip().lower()
        try:
            resolve_inference_device_map_and_dtype(device)
        except ValueError as exc:
            console.print(f"[red]--device:[/] {_rich_escape(str(exc))}")
            raise typer.Exit(code=2) from exc
    
    # _load_model (was serve.py:1372-1383): drop on_cpu / device_map, and
    from soup_cli.utils.gpu import resolve_inference_device_map_and_dtype
    
    device_map, default_dtype = resolve_inference_device_map_and_dtype(device)
    if kv_cache_dtype == "bfloat16":
        load_dtype = torch.bfloat16
    elif kv_cache_dtype == "float16":
        load_dtype = torch.float16
    else:
        load_dtype = default_dtype
    
    # _load_draft_model (was serve.py:1507-1508); `import torch` there becomes unused
    device_map, dtype = resolve_inference_device_map_and_dtype(device)
    draft = AutoModelForCausalLM.from_pretrained(
        speculative_model, device_map=device_map, torch_dtype=dtype
    )

    tests/test_speculative_decoding.py:78 asserts device_map == "cpu" for the draft and becomes {"": "cpu"}; that edit is expected, the placement is the same. test_load_draft_model_cuda_dtype (same file, lines 98-113) asserts float16 for cuda; the helper returns bfloat16 when cuda_supports_bf16() is true, so pin it with monkeypatch.setattr("soup_cli.utils.gpu.cuda_supports_bf16", lambda: False) (the new test file below does this in an autouse fixture) or that assertion will depend on the machine running it. Delete _SERVE_DEVICES. Using the helper's dtype means soup serve on an Ampere or newer GPU loads bfloat16 instead of float16 by default (as infer / chat / diff do after #1460), so the fragment has to say that; see item 4.

  3. The --device message assertion is a raw-output assertion. assert "cuda, mps, cpu" in result.output (test file, last refusal test) passes at 80 columns and fails when Rich wraps the line: with COLUMNS=60 and forced colour the output is ... Accepted values: cuda, \nmps, cpu. and the substring is gone. Go through strip_ansi and collapse whitespace (from tests.conftest import strip_ansi; " ".join(strip_ansi(text).split())), as the file at the end does.

  4. Fragment (after the rebase). Say what the helper changes on the default path: bfloat16 on an Ampere or newer GPU; cuda:N and mps pinned to that device; mlx runs on the CPU; and on a CPU-only host the default load is now float32 pinned to the CPU, which doubles its RAM (a 7B model needs about 28 GB instead of 14 GB). The current text omits the RAM cost.

  5. Docs (not blocking). --device for soup serve is only in the help text. #1460 adds a "Choosing the device" table to docs/commands.md; adding soup serve to it once both are merged would keep one description of the rule.

What I ran with the fix

I merged the two heads in a scratch tree (they merge cleanly; no shared files), applied the patch above and replaced your test file with the one below (32 tests: the loader for both branches x five devices, the dtype overrides, the CLI accepting and normalising cpu / " CPU " / cuda / CUDA / cuda:1 / mps, refusing six bad values with the message, the markup case, vLLM still reaching its server without a device, and the draft placed like the main model). 796 passed across the two device files, test_infer / test_chat / test_diff / test_bench and the serve files before I added the draft-loader change, and 73 passed in the three files that change touches (this test file, test_speculative_decoding.py, test_issue1443_device_map.py); ruff clean. 10 mutations of the new code (loader ignoring the helper, dtype ignoring it, validation always / never refusing, no escape, no normalisation, CLI passing None, each explicit dtype ignored, exit code 1) are all killed, and so is reverting the draft loader to the old rule (3 failed).

tests/test_issue1478_serve_device_map.py (replaces yours)
"""Regression tests for #1478: ``soup serve --device`` reaches the transformers loader.

``serve._load_model`` used to load the base model and the merged model with a hard-coded
``device_map="auto"``, so ``--device cpu`` could still place the model on a visible GPU.
It now places the model with the same rule as ``soup infer`` / ``chat`` / ``diff``
(``utils.gpu.resolve_inference_device_map_and_dtype``).
"""

from __future__ import annotations

from unittest.mock import MagicMock, patch

import pytest
import torch
from typer.testing import CliRunner

from tests.conftest import strip_ansi

runner = CliRunner()


def _collapse(text: str) -> str:
    return " ".join(strip_ansi(text).split())


@pytest.fixture(autouse=True)
def _no_bf16(monkeypatch):
    # Pin the machine: the dtype rule is "bf16 on a CUDA GPU that supports it", which
    # depends on the box running the tests.
    from soup_cli.utils import gpu

    monkeypatch.setattr(gpu, "cuda_supports_bf16", lambda: False)


def _load(device, *, is_adapter=False, kv_cache_dtype=None):
    """Run the real ``_load_model`` with transformers/peft mocked; return the loader mock."""
    from soup_cli.commands.serve import _load_model

    tokenizer = MagicMock()
    tokenizer.pad_token = "<pad>"
    with patch("transformers.AutoTokenizer.from_pretrained", return_value=tokenizer), patch(
        "transformers.AutoModelForCausalLM.from_pretrained", return_value=MagicMock()
    ) as load, patch("peft.PeftModel.from_pretrained", return_value=MagicMock()):
        _load_model(
            model_path="some-org/some-model",
            base_model="some-org/base-model" if is_adapter else None,
            is_adapter=is_adapter,
            device=device,
            kv_cache_dtype=kv_cache_dtype,
        )
    return load


@pytest.mark.parametrize("is_adapter", [False, True], ids=["plain", "base+adapter"])
@pytest.mark.parametrize(
    ("device", "device_map", "dtype"),
    [
        ("cpu", {"": "cpu"}, torch.float32),
        ("mps", {"": "mps"}, torch.float16),
        ("cuda:1", {"": "cuda:1"}, torch.float16),
        ("cuda", "auto", torch.float16),
        (None, "auto", torch.float16),
    ],
)
def test_loader_places_the_model_by_the_shared_rule(is_adapter, device, device_map, dtype):
    load = _load(device, is_adapter=is_adapter)
    assert load.call_count == 1
    kwargs = load.call_args.kwargs
    assert kwargs["device_map"] == device_map, kwargs
    assert kwargs["torch_dtype"] == dtype, kwargs


@pytest.mark.parametrize(
    ("device", "kv_cache_dtype", "dtype"),
    [
        ("cpu", "float16", torch.float16),
        ("cpu", "bfloat16", torch.bfloat16),
        ("cuda", "float16", torch.float16),
        ("cuda", "bfloat16", torch.bfloat16),
    ],
)
def test_explicit_kv_cache_type_still_sets_the_load_dtype(device, kv_cache_dtype, dtype):
    assert _load(device, kv_cache_dtype=kv_cache_dtype).call_args.kwargs["torch_dtype"] == dtype


def _serve_to_the_loader(tmp_path, *args):
    """Invoke the real CLI and stop at ``_load_model``; return (result, loader mock)."""
    pytest.importorskip("fastapi")
    pytest.importorskip("uvicorn")
    from soup_cli.cli import app

    with patch(
        "soup_cli.commands.serve._load_model", side_effect=RuntimeError("stop after the load call")
    ) as load:
        result = runner.invoke(app, ["serve", "--model", str(tmp_path), *args])
    return result, load


@pytest.mark.parametrize(
    ("given", "passed"),
    [
        ("cpu", "cpu"),
        (" CPU ", "cpu"),
        ("cuda", "cuda"),
        ("CUDA", "cuda"),
        ("cuda:1", "cuda:1"),
        ("mps", "mps"),
    ],
)
def test_cli_hands_the_normalised_device_to_the_loader(tmp_path, given, passed):
    _, load = _serve_to_the_loader(tmp_path, "--device", given)
    assert load.call_args.kwargs["device"] == passed, load.call_args


@pytest.mark.parametrize("bad", ["tpu", "gpu", "auto", "cuda:abc", "cuda:0,1", "cpu0"])
def test_cli_refuses_an_unknown_device_naming_the_accepted_values(tmp_path, bad):
    result, load = _serve_to_the_loader(tmp_path, "--device", bad)
    assert result.exit_code == 2, (result.output, repr(result.exception))
    assert not load.called
    text = _collapse(result.output)
    assert "cpu, cuda, cuda:<index> or mps" in text, text
    assert repr(bad) in text, text


def test_cli_refuses_a_markup_like_device_instead_of_crashing(tmp_path):
    result, _ = _serve_to_the_loader(tmp_path, "--device", "[/bold]")
    assert result.exit_code == 2, (result.output, repr(result.exception))
    assert "[/bold]" in _collapse(result.output)


def test_vllm_backend_still_reaches_its_server_without_a_device_argument(tmp_path):
    pytest.importorskip("fastapi")
    pytest.importorskip("uvicorn")
    from soup_cli.cli import app

    with patch("soup_cli.utils.vllm.is_vllm_available", return_value=True), patch(
        "soup_cli.commands.serve._serve_vllm", return_value=MagicMock()
    ) as serve_vllm, patch("uvicorn.run"):
        result = runner.invoke(
            app, ["serve", "--model", str(tmp_path), "--backend", "vllm", "--device", "cuda"]
        )
    assert result.exit_code == 0, (result.output, repr(result.exception))
    assert "device" not in serve_vllm.call_args.kwargs


@pytest.mark.parametrize("device", ["cpu", "cuda", "cuda:1", "mps"])
def test_the_draft_model_is_placed_like_the_main_model(device):
    """One flag, one placement: the speculative draft must land where the main model does."""
    from soup_cli.commands.serve import _load_draft_model

    main = _load(device).call_args.kwargs
    with patch(
        "transformers.AutoModelForCausalLM.from_pretrained", return_value=MagicMock()
    ) as load:
        _load_draft_model("some-org/draft", device)
    draft = load.call_args.kwargs
    assert (draft["device_map"], draft["torch_dtype"]) == (
        main["device_map"], main["torch_dtype"],
    )

The CLI cases need fastapi and uvicorn; the file skips them with pytest.importorskip when either is missing, as your case-insensitivity test does.

Thank you again: once the four gaps are pinned and the placement goes through the shared helper, I found nothing else to change.

One open point I have to settle before you rebase on #1460: its helper changes the default dtype (on an Ampere or newer GPU from float16 to bfloat16, and on a CPU-only host to float32), and I am checking that default with the project owner before #1460 merges. Please hold off on adopting the helper's dtype here until I say so on #1460. The device map, the validation and the four test gaps above do not depend on it.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

@drakeo338, the dtype question I asked you to wait on is settled: #1460's helper will keep float16 on every GPU (cuda, cuda:N, mps) and use float32 only on the CPU. So once #1460 has merged, adopt the helper as it then stands (device map, validation and dtype), as the review says; for the serve loader that means float32 on the CPU and no change to the GPU dtype. Until #1460 merges there is nothing to rebase on.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

@drakeo338, #1460 has merged as 4eca01e2, so resolve_inference_device_map_and_dtype is on main and you can rebase now, as the review describes: call the helper in the serve loader (then apply the --kv-cache-type bf16/f16 override after it), validate --device early with exit 2, declare the Apple case (detect_device() can return mlx), reuse it in _load_draft_model, and pin the {"": "cpu"} shape in the tests. The dtype rule is the one settled on #1460: float32 on the CPU, float16 on every GPU.

One more thing you should know: #1491 by @wanjinhao1 was opened for the same issue (#1478) at 14:28 UTC; yours is the earlier PR and the one I am reviewing, so nothing changes for you unless I write otherwise here.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Hi @drakeo338, a short refreshed reminder for #1479 (head a05a355e, unchanged; all required checks green on it). #1460 merged as 4eca01e2, so resolve_inference_device_map_and_dtype is on main now. A second PR (#1491) has been opened for the same issue and is being closed, so yours is the one that lands once these are done. Two things from my earlier notes have changed, so please read these instead of the older list:

  • The merged helper keeps float16 on every GPU (cuda, cuda:N, mps); there is no bfloat16 pick. Ignore every bfloat16 remark in my earlier review: the fragment does not need to mention it, and the _no_bf16 autouse fixture in my test file is now unnecessary (harmless; I ran the file with it in place).
  • I re-ran the ready patch from my review (validation via the helper, _load_model, _load_draft_model, the one-line tests/test_speculative_decoding.py edit) plus the 32-test replacement file on #1479 merged with current main: it applies cleanly and 220 tests pass (new file, test_speculative_decoding, test_issue1443_device_map, test_serve, test_multi_adapter, test_v07114, test_serve_local_request_guard), ruff clean.

Remaining items, in order:

  1. Rebase onto main (git rebase origin/main; the two heads do not conflict).
  2. src/soup_cli/commands/serve.py:464-472: replace _SERVE_DEVICES (line 28) and the membership check with the helper, keeping exit code 2 and _rich_escape:
    if device:
        from rich.markup import escape as _rich_escape
    
        from soup_cli.utils.gpu import resolve_inference_device_map_and_dtype
    
        device = device.strip().lower()
        try:
            resolve_inference_device_map_and_dtype(device)
        except ValueError as exc:
            console.print(f"[red]--device:[/] {_rich_escape(str(exc))}")
            raise typer.Exit(code=2) from exc
  3. serve.py:1372-1383 (_load_model): drop on_cpu / device_map = ...; device_map, default_dtype = resolve_inference_device_map_and_dtype(device); keep the explicit bfloat16 / float16 overrides with load_dtype = default_dtype as the fallback.
  4. serve.py:1507-1508 (_load_draft_model): the same helper for device_map and torch_dtype (import torch there becomes unused). In tests/test_speculative_decoding.py:78 == "cpu" becomes == {"": "cpu"}.
  5. --device help text (serve.py:104-109), new: it currently says "Device: cuda, mps or cpu ... cuda and mps load with device_map=auto in float16". After the helper that is wrong for mps (pinned) and omits cuda:<index>. Suggested wording: "Device: cuda, cuda:<index>, mps or cpu. Auto-detected if not set. With the transformers backend, cpu loads on the CPU in float32 (unless --kv-cache-type bf16/f16 picks the dtype); cuda:<index> and mps are pinned to that device; cuda uses device_map=auto, all in float16."
  6. tests/test_issue1478_serve_device_map.py: replace with the file in my review comment (it pins the four surviving mutations and the draft placement, and uses strip_ansi + whitespace collapse instead of the raw "cuda, mps, cpu" in result.output). One optional addition that I checked passes on the adopted code, because the replacement has no equivalent: a loader-level refusal test,
    def test_load_model_refuses_an_unknown_device_even_when_called_directly():
        with pytest.raises(ValueError, match="cpu, cuda, cuda:<index> or mps"):
            _load("tpu")
  7. changelog.d/0.75.1/1479.fixed.md: say what the helper changes by default: cuda:N accepted and pinned to that GPU (it was accepted and ignored on main), mps pinned, mlx runs on the CPU, unknown values refused with exit 2 naming cpu, cuda, cuda:<index> or mps, and on a CPU-only host the default load is float32 pinned to the CPU, which doubles its RAM (a 7B model needs about 28 GB instead of 14 GB). The current text says cuda and mps keep device_map="auto", which stops being true for mps.
  8. Docs (optional): add soup serve to "Choosing the device" in docs/commands.md (added by fix(infer,chat,diff): honor --device instead of hard-coding device_map=auto #1460, docs/commands.md:381).

drakeo338 and others added 3 commits September 30, 2026 16:05
…lpamys#1478)

`soup serve --device cpu` was accepted but `_load_model` ignored its `device` argument and always passed `device_map="auto"`, so accelerate put the model on any visible GPU. The plain model and the base model under a LoRA adapter now load with `device_map="cpu"` and float32 when the device is `cpu`, matching `_load_draft_model`. `cuda` and `mps` keep `device_map="auto"` and float16, and an explicit bfloat16 KV-cache dtype is still honoured on CPU.

Verified: the new tests fail on the previous tree (the two CPU load tests) and pass with the change; the new file, `tests/test_vllm_serve.py`, `tests/test_serve.py` and the foreign-license-header test give 64 passed.

Closes MakazhanAlpamys#1478.
…nAlpamys#1478)

Review follow-up. `--device` is now stripped and lower-cased, and any value other than cuda, mps or cpu is refused with exit code 2 and a message naming the accepted values. The `--device` help text says what each device does. An explicit `--kv-cache-type f16` now loads float16 on CPU too, so the printed KV-cache note matches the loaded dtype. The changelog fragment names the float16 to float32 default change on CPU-only hosts. The CPU tests now require `device_map == "cpu"` exactly, and new tests cover the unknown-device refusal, case-insensitivity and f16 on CPU.
@drakeo338

Copy link
Copy Markdown
Contributor Author

Rebased on main; soup serve now uses the shared device helper for the main and draft models, with early exit-2 validation, updated help, fragment and tests (e618400).

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drakeo338, thank you for the quick rebase. I went through e618400d item by item against the list I sent, by running the real CLI and the tests rather than reading the diff, and everything on it is done. This is ready to merge and waits only on the required checks, which are all still queued on this head.

What is good

  • The head is based exactly on current main (8c10e2c6), so the rebase is complete and there is nothing left to resolve.
  • The helper is the single rule now. I drove soup serve with the loaders mocked and recorded the kwargs for a plain model and for --base + adapter: cpu gives {"": "cpu"} / float32, mlx the same, mps {"": "mps"} / float16, cuda:1 {"": "cuda:1"} / float16, cuda "auto" / float16. With no --device on a GPU host nothing changes ("auto" / float16); on a CPU-only host you get float32 pinned to the CPU, which the fragment says plainly, including the RAM doubling.
  • --kv-cache-type still wins: cpu + f16 is float16, cpu + bf16 is bfloat16, cuda + bf16 is bfloat16. I also loaded a tiny random Llama through the real _load_model on CPU with transformers 5.17: parameters on cpu, float32, and float16 / bfloat16 when the flag asks for them, so device_map={"": "cpu"} is accepted by transformers.
  • The draft model lands where the main model does for cpu, cuda, cuda:1 and mps, and the one touched assertion in test_speculative_decoding.py ("cpu" to {"": "cpu"}) is the expected placement change.
  • Validation is early and on every backend: --device tpu, gpu, auto, cuda:abc, cuda: and [/bold] exit 2 with Unsupported --device 'x': use cpu, cuda, cuda:<index> or mps., on transformers, vLLM and SGLang, with nothing loaded. On main all of them exit 0 and are ignored, and [/bold] dies with a Rich MarkupError.
  • The --device help renders correctly and matches what I measured, and the fragment text matches the behaviour clause by clause (cuda:N pinned, mps pinned, mlx on the CPU, exit 2 message, case-insensitive, RAM cost). The credit line is right and the fragment validator passes.

What I ran

  • tests/test_issue1478_serve_device_map.py: 34 passed, and 34 passed again under FORCE_COLOR=1 COLUMNS=60 (the assertion goes through strip_ansi + whitespace collapse now). On main the same file gives 19 failed, 15 passed.
  • That file plus 16 adjacent files (serve, speculative decoding, kv-cache, vLLM / SGLang / MII, multi-adapter, request guard, auto-quant, #1443's device tests, the changelog fragment tests): 627 passed. ruff check is clean.
  • 24 mutations of serve.py, one at a time, each undone afterwards: 23 fail a test. That includes the four that survived last round (--device cuda refused, --device mps refused, the escape dropped, no-device treated as CPU), the loader and the draft loader each ignoring the helper, the kv override applied before the helper, the exit code, and each of strip() / lower(). The one survivor is moving the validation inside the transformers branch only; the behaviour is right (I ran vLLM and SGLang, exit 2), it is just not pinned by a test. That gap comes from the test file I handed you, so please do not push anything for it.

Nothing further is needed from you on this PR. Thank you for taking the helper all the way through both loaders and the draft model instead of patching the one call that was reported.

@MakazhanAlpamys MakazhanAlpamys added the ci:full Run the full CI matrix on this PR (a maintainer adds it at approval) label Sep 30, 2026
@MakazhanAlpamys
MakazhanAlpamys merged commit 0f7ab95 into MakazhanAlpamys:main Oct 1, 2026
19 checks passed
@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Merged as 0f7ab958. Thank you, @drakeo338: you took the shared device helper all the way through soup serve, the merged-model and base + adapter loaders and the speculative draft model, with early exit-2 validation on every backend, instead of fixing only the call that was reported. The new file is 34 passed (19 failed / 15 passed on main); 627 passed across 17 related files; 23 of 24 mutations fail a test (including the four that survived the first round).

MakazhanAlpamys added a commit that referenced this pull request Oct 1, 2026
…1364, #1492, #1493, #1489, #1416, #1484, #1490, #1479, #1485, #1495

Display names from `gh api users/<login>`.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
MakazhanAlpamys added a commit that referenced this pull request Oct 1, 2026
…transformers (#1478) (#1523)

The review of #1479 (merged as `0f7ab958`) ran 24 mutations of the `soup serve --device` fix and one survived: validating `--device` only when `backend == "transformers"` passed every test, because the refusal tests all ran on the default backend. That mutation would be a real regression (an unknown device reaching vLLM/SGLang unchecked).

This adds one parametrised case to `tests/test_issue1478_serve_device_map.py`: `--device tpu` must exit 2 on `--backend vllm` and `--backend sglang`, before either server is built or `uvicorn.run` is called. Mutation check: guarding the validation block in `serve.py` with `and backend == "transformers"` makes the two new cases fail (2 failed, 34 passed); unmutated, 36 pass. Test-only, no source change.
Ercaner1988 added a commit to Ercaner1988/Soup that referenced this pull request Oct 3, 2026
commit 4e3ab348994dba22bff836fe255936b5212a14a2
Author: Alpamys Makazhan <[email protected]>
Date:   Fri Oct 2 22:40:37 2026 +0500

    fix: soup init and the MCP plan snapshot refuse links at their write target (#1553)

    soup init wrote the starter config with Path.write_text, which follows a
    symbolic link or junction at the target, and a dangling link skipped the
    overwrite prompt. It now checks the target before prompting, creates a
    new file exclusively (O_CREAT | O_EXCL with O_NOFOLLOW on POSIX; a
    staging file renamed into place on Windows, where an exclusive create
    still follows a dangling link), opens an existing file through
    open_no_follow, and truncates it only after the open-time check has
    passed. A refusal names the path and exits 1; ordinary paths write the
    same bytes and mode as before.

    The MCP plan snapshot (.soup/mcp-runs/<run_id>/config.yaml) also refuses
    a link at the run directory itself, re-checks the parents after creating
    them, creates the run directory with exist_ok=False and config.yaml with
    O_EXCL through open_no_follow(check_parent=True), and requires the run
    id to be a single path component. Refusals stay path-free
    ExecutionErrors.

    Co-Authored-By: Claude Opus 5.5 <[email protected]>

commit 12807838df1604f8873949ab2ca7340afaefd358
Author: Mohd Uwaish <[email protected]>
Date:   Fri Oct 2 18:24:07 2026 +0200

    fix(config): refuse LoRA+ with model-only optimizers and GaLore at load (#1210) (#1255)

    `training.loraplus_lr_ratio` combined with `optimizer: apollo_adamw`, `lomo` or `adalomo`, or with `use_galore`, passed config load and failed only after the base model had been downloaded and loaded. LoRA+ builds its optimizer through `build_loraplus_optimizer`, which asks `Trainer.get_optimizer_cls_and_kwargs(args)` for it without a model, and transformers can only build those three optimizers (and GaLore) from one; the GaLore combination was refused only at runtime.

    A new `TrainingConfig._validate_loraplus_compat`, next to `_validate_lorafa_compat`, refuses both combinations at load with a message that names both fields, and suggests an optimizer transformers can build without the model. The runtime GaLore check in `peft_wiring.py` stays as a backstop. The refusal does not depend on the task, and LoRA+ with any other optimizer (`adamw_torch`, `adamw_8bit`, `adafactor`, ...) and each of the refused settings on its own still load.

    Users of LoRA+ with one of those optimizers now see the error from `soup train` in about a second instead of after the model download. `docs/peft-and-efficiency.md` gains an `Optimizers` bullet in the LoRA+ section, under the existing `Compatibility` bullet.

    Verified: `tests/test_issue1210_loraplus_model_only_optimizers.py` (36 tests: three refusals, the case-insensitive refusal, GaLore and seven controls, each under `sft`, `dpo` and `ppo`) passes, and 15 of them fail on `main` without the change; ten single-point mutations, including moving the refusal into an `sft`-only validator (10 failures), all fail the tests; 1333 related tests pass on the tree merged with `main`; `ruff` is clean.

    Closes #1210.

commit ef44c0c0af75e9aefe3b109a17fe15ec843ec878
Author: Pernav Jain <[email protected]>
Date:   Fri Oct 2 21:38:54 2026 +0530

    fix(notebook): make proof-4gb reproducible (#1545)

    `notebooks/proof-4gb.ipynb`, the "run it yourself" proof of 4 GB layer streaming, did not run on a fresh hosted multi-GPU runtime (Kaggle 2x T4) with a current `huggingface-hub`, for two reasons in its install cell. (1) `huggingface_hub` 1.32 keeps blobs in a cache-wide shared store that soup-cli 0.75.0's snapshot materialization treats as outside the cache and refuses (#1509, fix open in #1510); the install now caps `huggingface-hub<1.32.0`, which resolves to 1.31.0 and satisfies transformers' `>=1.5.0,<2.0`, with a comment saying why and when to drop it. (2) With two visible CUDA devices, layer streaming refuses to start because it cannot run under `nn.DataParallel` (`refuse_if_data_parallel` names `CUDA_VISIBLE_DEVICES=0`); the cell now sets `os.environ.setdefault("CUDA_VISIBLE_DEVICES", "0")` before torch is imported, so a caller who pre-selected a card keeps it.

    Only cell 3's source changes; the outputs, metadata and execution counts of all 19 cells are identical. `tests/test_issue844_notebook_pin_guard.py` gains two tests: the GPU is selected in a cell at or before the first cell that imports torch, and the install cell caps the hub below 1.32.

    Verified: the new tests are 2 failed / 7 passed on `main` and 9 passed with the PR; mutations deleting or relocating the device block, deleting the cap, or loosening it each fail a test; cell 3 was run in isolation with the `%pip` lines recorded and left the variable at `'0'` when unset and unchanged when preset. The two-GPU run itself was done by the author and not reproduced here.

    Closes #1543.

commit c6271bffe35455c1e675800344dfd9f3c6215e82
Author: qwist1233-cpu <[email protected]>
Date:   Fri Oct 2 19:07:18 2026 +0300

    fix(advise): print the ROI table when --probe-model implies --probe (#1448 slice) (#1521)

    `soup advise run <data> --probe-model X` (without `--probe`) paid for the live probe and then never printed it. `--probe-model` implies `--probe`, and the probe ran under the derived `run_probe = probe or probe_model is not None`, but the `ROI deltas (probe)` table was gated on the raw `--probe` option, so the measured baselines, LoRA estimate and base-model proximity were computed and thrown away. The table is now gated on `run_probe`, the same flag that runs the probe, at `src/soup_cli/commands/advise.py:333`.

    Users notice: `soup advise run data.jsonl --probe-model org/model` now prints the ROI table, as `docs/evaluation.md` already said it implies `--probe`; plain `--probe` still prints the table and a run with no probe flag still prints none.

    Verified: the new `tests/test_issue1448_advise_probe_model_roi_table.py` (4 tests, probe helpers monkeypatched) has 1 failed / 3 passed on `main`; with the PR, 228 passed / 1 skipped across it, `tests/test_v0717.py`, `tests/test_v0540.py` and the changelog-fragment tests. Four mutations of the gate each fail a test (`if probe:`, `if probe_model is not None:`, `if True:`, `if False:`).

    Part of #1448.

commit 255a5ecce26204367d5ce72a96e19029c256c0fd
Author: Jillur Rahman <[email protected]>
Date:   Fri Oct 2 22:05:04 2026 +0600

    fix(distill): leave unaligned student positions out of aligned ULD (#1426) (#1541)

    Aligned ULD (`training.uld_strategy: wasserstein_aligned`, the strategy `soup draft distill` picks for every cross-tokenizer pair) mean-pools teacher logits onto student positions through a character alignment. A student position that aligns to no teacher token (a byte-fallback piece decoding to U+FFFD, an empty decoded piece, a student-only special token) got an all-zero teacher row, which softmaxes to a uniform distribution, and stayed in the loss mask. Against a confident student each such position scored about V/2 (7999.4 for the reproduction at a 32k vocabulary) while aligned positions scored near 0, so the loss was dominated by positions with nothing to distill and its gradient pushed the student toward maximum entropy there.

    `uld_aligned_loss` now counts a student position only when its alignment names at least one existing teacher position (`aligned_position_mask`, the same in-range rule `aggregate_aligned_logits` uses). That mask multiplies the label mask or the attention mask and, with neither, is the whole mask, so the position leaves both the sum and the denominator; a sequence with no aligned supervised position gives a finite 0.0. The distill trainer decodes the student's per-token strings with `skip_special_tokens=True`, as the teacher text already is, so a student-only special token decodes to "" and aligns to nothing.

    Users notice: distillation loss across different tokenizers no longer carries a large constant per byte-fallback piece, empty piece or special token, and the student is no longer pulled toward a uniform distribution at those positions. With no `labels` and no `attention_mask` the aligned loss is now the mean over aligned positions instead of over every position; with every position aligned the value is bit-identical (800 old-vs-new comparisons across four mask modes).

    Verified: the issue reproduction gives 7999.4 on main and 0.0 on the branch under labels, attention mask and no mask; a batch of rows with different alignments and an all-unaligned row stays finite with finite gradients; 15 new tests (11 fail on main, 4 are controls) including a trainer-level test through the real `_DistillTrainer.compute_loss`; 331 passed across the new file and six neighbouring ULD/distill files, 565 across nine more; ten mutations of the fix, each caught.

    Closes #1426.

commit ed50391a1711f6b5fc9903c785d9b4b3ecb9bb81
Author: Urooj Fatima Mukhtar <[email protected]>
Date:   Fri Oct 2 14:40:32 2026 +0500

    fix(eval): make contains scoring linear again (#1528) (#1540)

    The custom-eval `contains` scorer stopped being linear when #1490 made it match `expected` only as a whole token. Its pattern began with a lookbehind and ran under `re.IGNORECASE`, which keeps `re` off its literal-prefix search, so the search restarted at every position and compared up to `len(expected)` characters at each word start: time proportional to `len(output) * len(expected)`. On repetitive token-separated text a 200,000-character output against a 100,000-character `expected` took 27 s on `main` (68 s at 400,000 against 200,000, as measured in the #1490 review), where v0.75.1's substring test took 0 s.

    `score_contains` now lowercases both sides once and puts the escaped literal first, so `re` finds candidates with its literal-prefix search. The left boundary is a fixed-width lookbehind after the literal and the right boundary is the existing lookahead, so each candidate costs O(1). The token bound from #1490 is unchanged and nothing is truncated: a gold token at character 60,000 of a 60,016-character output still scores `True`.

    What users notice: scores are unchanged for ASCII text, and the same 200,000-character case takes about 0.13 s. Case folding is `str.lower()` on both sides, as in v0.75.1 (the regex had not shipped in a release), so `istanbul` no longer matches `İstanbul` and `s` no longer matches `ſ`; the `contains` row in docs/evaluation.md says so.

    Verified: the 8 new tests fail 4/8 on `main` and pass on the branch; test_issue1342 (30) unchanged; 620 tests across the 11 files that touch contains-scoring pass; 120,000 ASCII cases agree with #1490's regex and 320,000 cases (Unicode included) agree with an explicit overlapping scan; 16 mutations of the function all fail the new tests; growth is linear on 8 input shapes up to 400,000 characters.

    Closes #1528.

commit 6879f323f7c8b893c3679abd53a34d61f3616fc1
Author: Srinivasan R <[email protected]>
Date:   Fri Oct 2 14:57:53 2026 +0530

    fix(expect): refuse top-level suite keys other than expectations (#1483) (#1499)

    `soup expect` read only the `expectations` key of a suite and dropped every other top-level key, so a suite with `fail_fast: true` (or any label or typo) beside `expectations:` loaded, ran and exited 0 with no message: the gate that ran was not the gate the user wrote. #1461 had already made `parse_suite_spec` refuse unknown keys inside an entry and unknown argument names; this closes the same gap one level up. `parse_suite_spec` now refuses every top-level key other than `expectations` before it reads anything else, naming each key through #1461's `repr()`-based `_key_repr` so a hostile key cannot reach the terminal as a control sequence, and `soup expect` exits 3 before the data file is opened. Users with a suite that carried a label such as `name:` beside `expectations:` now get exit 3 with the key named instead of a silent pass (documented in `docs/compliance.md`); one existing fixture in `test_issue1231_expect_all_formats.py` was such a suite and loses that line. Verified on the PR merged with main: 270 passed across the expect, gate-exit-code and ANSI-safety files, the new tests give 8 failed / 2 passed on main, and mutations removing the check, the `repr()`, the all-keys naming, or widening the allowlist to `name`, `description`, `version` or an `expectation*` prefix are each caught. Real CLI: `fail_fast:` and `name:` beside `expectations:` go from exit 0 to exit 3. Closes #1483.

commit 8fcd8b90e320cf99d2abd3816b334611b1460707
Author: Alpamys Makazhan <[email protected]>
Date:   Thu Oct 1 23:33:22 2026 +0500

    ci: run the full matrix at approval (ci:full), a quick subset on other PR pushes (#1381)

    A push to a pull request now runs a quick subset of `ci.yml` (`lint` and `test (ubuntu-latest, 3.12)`); the full matrix runs on pushes to `main` and `release/**` and on pull requests that carry the `ci:full` label, which a maintainer adds at approval.

    Why: the Actions account runs at its free-plan cap (20 concurrent jobs, 5 on macOS). Measured over 1,906 runs since 2026-09-19, one full `ci.yml` run costs 358 job-minutes, saturated service is 61-71 runs a day while weekday demand is 87-99, and most of that demand was review iterations: every push to a pull request paid for the full matrix even while the head waited on its author. Running the full matrix once per pull request, at approval, is worth about x1.36 of capacity on the binding limit; dropping Windows and macOS on 3.11 from the full run (they stay in the release matrix) takes a full run from 4 macOS jobs to 3.

    How: `pull_request.types` adds `labeled` (still a plain `pull_request`, never `pull_request_target`); a new `plan` job declares the RELEASE, FULL and QUICK matrices once and `test` takes `fromJSON(needs.plan.outputs.matrix)`, so a quick run never creates the other required `test (...)` contexts and branch protection keeps them Expected: nothing can be merged on a quick run. The concurrency key gives a `labeled` event for any label other than `ci:full` its own group, so labelling cannot cancel a running full matrix. `CONTRIBUTING.md` tells contributors what they will see.

    Verification: `tests/test_ci_full_matrix_on_approval.py` pins the decision table, the matrices, the gated jobs, the concurrency keys and the decide step's shell (the matrix literals are parsed by `tests/test_requires_python_bound.py`); its own full matrix ran green on this head (11 of 11 required checks).

commit 4142fdd89889989e08f38de81394392a242af74a
Author: UmranPros <[email protected]>
Date:   Thu Oct 1 23:49:08 2026 +0530

    fix(train): a blend that cannot be computed refuses instead of scaling one loss (#1425) (#1513)

    `training.preference_loss_weights` promised a runtime blend of 2-5 preference losses, but no trl 0.29 trainer puts per-sequence log-probs on the batch, so the term loop never ran and the combine wrapper silently fell back to `primary_loss * primary_weight`. A config such as `{dpo: 0.7, simpo: 0.3}` printed `Multi-objective preference loss: 0.70·dpo + 0.30·simpo` at setup and then trained DPO alone scaled by 0.7 (measured on a real trl 0.29.1 trainer: exit 0, `Loss: 0.4852` = 0.7 x 0.693147), with nothing in the output saying so. The fallback is deleted (with the now-dead `_pick_primary` and the `primary_loss` unpack): when a requested term cannot be computed the wrapper raises, naming the terms it could not compute, the terms it did compute, and what was requested. What users notice: every `preference_loss_weights` config now stops at the first step instead of training one loss under a blend's name; to train one loss, remove the weights and set `training.preference_loss`. The weighted-sum path itself is unchanged, so a trl that does carry log-probs on the batch would still blend. `docs/training.md` no longer claims a true weighted sum. Verified on the PR merged with main: 183 passed across the five preference/combine files, a real-trainer test (tiny random-init Llama, CPU, two blends) that passes on the PR and fails on main, and mutations restoring the fallback, naming every term as missing, emptying the computed list, accepting a partial blend and changing the exception type each caught. This is partial progress on #1425: the real blend (deriving log-probs from the trainer's own forward pass) and the "changing only the secondary loss changes the loss" check are not in this change. Refs #1425.

commit 909d86c48a8de062ebc1e48704fc0cb174980700
Author: Alpamys <[email protected]>
Date:   Thu Oct 1 23:02:17 2026 +0500

    docs(contributors): #1496, #1481, #1482, #1514 [skip ci]

    Display names from `gh api users/<login>`.

commit 92e6a1fd00405c10f60c2080bf6e04b2b43526bd
Author: UmranPros <[email protected]>
Date:   Thu Oct 1 23:30:50 2026 +0530

    fix(train): refuse the RL-signal flags PPO cannot feed (#1441) (#1514)

    fix(train): `task: ppo` refuses the reward-hacking and echo-trap flags it cannot feed (#1441)

    The trl PPO trainer this build supports (`trl.experimental.ppo.PPOTrainer`) takes no `reward_funcs`: its reward comes from the reward model it calls internally. Soup still built the callable reward wrappers and a shared signal buffer at setup and then dropped them, so the buffer was never written, and because the buffer object existed the detector's `on_log` fallback returned early too. A run with `reward_hack_detector`, `reward_hack_mitigation` or `echo_trap_enabled` was announced, loaded, and then never halted, mitigated or trapped anything.

    `PPOTrainerWrapper.setup` now refuses these three flags (the exact set that makes `rl_callbacks_need_buffer` create the buffer) on the experimental trl API, before `datasets`, the model, the reward model or the tokenizer is loaded. The message names every flag that was set and points at `task: grpo`, where the buffer is fed. `reward_hack_mitigation: off`, the other `reward_hack_*` / `echo_trap_*` tunables (already refused at config load unless one of the three is set) and the transitional trl API are unaffected, and none of the 200 shipped configs, templates and recipes is refused. `docs/training.md` and `docs/commands.md` no longer say PPO is supported for these flags. Feeding the buffer from the PPO rollout and the meaning of `reward_fn` on PPO are separate follow-ups.

    Verified: 21 tests in `tests/test_issue1441_ppo_refuses_unfed_rl_flags.py`, including setup-level tests that stub the loaders and assert nothing loaded, and a drift test pinning `_unsupported_rl_flags` to `rl_callbacks_need_buffer`; 11 mutations of the refusal (call removed, condition inverted, each flag dropped, refusal moved after each load, never raising, always raising, no flag names, drift) all fail a test; the neighbouring PPO suites (222 tests with the changelog and backend-support guards) pass on the merge with current `main`; and a real `soup train --config soup.yaml --yes` with all three flags stops at "Setting up model + trainer..." with the new message, while the same config without them is not refused.

    Closes #1441.

commit 5c335b93c3e4993ead7e3f6ef552d4c3ad4f870e
Author: Alpamys Makazhan <[email protected]>
Date:   Thu Oct 1 23:00:43 2026 +0500

    fix(net): harden outbound endpoint validation (#1527)

    Outbound endpoint checks for the vLLM provider, `soup data generate --api-base`,
    judge URLs (flags, eval-gate suites, `soup ship --judge-model`,
    `training.online_dpo_judge`) and the Web UI chat proxy share one helper,
    `net_guard.refuse_private_ip_literal`, which refuses a private, link-local or
    reserved IP literal on every scheme. Loopback stays allowed; hostnames are not
    resolved. `0.0.0.0` is no longer loopback for `soup data generate` and the
    chat proxy.

    The shared predicate also counts 100.64.0.0/10 and fec0::/10 as non-public,
    and the shared parser classifies a non-ASCII host in the ASCII forms an HTTP
    client connects to (stdlib IDNA 2003 and the idna package), so webhooks, the
    OTLP endpoint, telemetry, hub endpoints and `soup ingest --pull` inherit both.

    Also: `soup ship` checks `--judge-model` before it builds the base and tuned
    models; the chat proxy answers 400, not 500, for an endpoint that does not
    parse; `online_dpo_judge` names itself on an unparsable URL. Docs say which
    addresses each setting accepts and name the two ranges.

    Tests: `tests/test_outbound_endpoint_ip_literals.py` (every entry point through
    its real surface, a differential test against `httpx.URL(url).host`, no network
    I/O) and the extended #616 ratchet. No changelog fragment: the change is
    recorded in the 0.75.2 section of CHANGELOG.md, which main gains at its next
    release.

    Co-Authored-By: Claude Opus 5.5 <[email protected]>

commit 1d0e7384dcb5bfa9e900d6632e6efb0685c5b447
Author: Areen Agrawal <[email protected]>
Date:   Thu Oct 1 23:30:16 2026 +0530

    fix(lora): map mistral3 target_modules and add modality to mistral-medium-3-5-sft (#1395) (#1482)

    `mistral-medium-3-5-sft` could not attach LoRA: its base, `mistralai/Mistral-Medium-3.5-128B`, is `Mistral3ForConditionalGeneration` (a Pixtral vision tower plus a `ministral3` text tower), but the recipe had no `modality: vision`, so SFT loaded it with `AutoModelForCausalLM`, which does not know `Mistral3Config` (`soup recipes verify` failed at stage `build`), and `training.lora.target_modules: auto` had no row for `mistral3` (stage `resolve`: no mapping for model_type ['ministral3']), so the architecture was refused even with the right loader. The recipe now sets `modality: vision` (SFT loads the image-text class), and `MOE_TEXT_LORA_TARGETS` gains a `mistral3` row: `.*language_model\..*\.self_attn\.(q_proj|k_proj|v_proj|o_proj)`, scoped to the language tower so the Pixtral vision tower, the multimodal projector and the MLP projections stay unadapted (a plain `q_proj/k_proj/v_proj/o_proj` suffix list would have adapted 16 modules, 8 of them in the vision tower). Measured on a 2-layer meta-device skeleton of the real config: 8 modules adapted, 0 in the vision tower, 0 experts. The recipe's `EXCEPTIONS` pin in `scripts/check_recipe_attach.py` is removed (the ratchet reports "0 pinned that now attach" without it, and flags the pin if it is put back), the recipe snapshot gains the `modality` line, and docs/peft-and-efficiency.md lists `mistral3` among the architectures `target_modules: auto` maps. The recipe count does not change. Verified: `soup recipes verify --config` on the real recipe moves from "0 attach, 1 cannot" (exit 2) to "1 attach, 0 cannot" (exit 0); 30 new tests (20 fail on the previous tree; the other 10 are documentation tests that build their own regex), 731 passed across the related files on the tree merged with main, `scripts/check_recipe_attach.py` exits 0 (and exits 2 when the pin is put back), and seven mutations of the fix are all caught (delete the row, drop the leading `.*`, plain suffix tuple, unscoped regex, drop `o_proj`, drop `modality: vision`, drop the snapshot line). Closes #1395.

commit be0049993e2fa93ab26205cf6721d983aeb08547
Author: fukalous <[email protected]>
Date:   Fri Oct 2 01:59:43 2026 +0800

    fix(diagnose): detect memorization in long rows (#1481)

    `soup diagnose` could not see memorization on long training rows. The live probe asks the adapter to continue the first quarter of a row and caps the generation at 64 tokens, but the memorization score compared that capped completion with the whole held-out suffix by set-Jaccard, so the longer the row the more the untouched suffix diluted the overlap: a verbatim 48-word echo read MAJOR on a 60 or 100-word row and OK from 150 words up, and a model that had memorized its long rows passed with a clean 1.00.

    The overlap is now clipped completion-bigram precision against the suffix: the share of the completion's adjacent token pairs that occur, counted at most as often as they occur, in the suffix. The default path pairs adjacent normalized words (stopwords and short ASCII words are still ignored, as before); `--tokenizer` pairs adjacent sub-word tokens. Bigrams, not single words, are what keep the score honest: a single-word precision made unrelated text flag as MAJOR more often the longer the row (59 percent of unrelated completions at 4000 words of real prose), flagged one-word completions, and treated a repetition loop of a common word as a verbatim echo.

    What users notice: a verbatim echo now reads MAJOR at every row length (60 to 4000 words, default and `--tokenizer`, Latin, Cyrillic and CJK text), and an unrelated continuation, a short completion or a repetition loop stays OK. The 0.5 threshold now means bigram precision rather than set-Jaccard, so a partial echo can read MAJOR slightly earlier than before (on a 60-word row a completion copying 55 percent of its words in order flips, where set-Jaccard needed 67 percent), and memorization results saved by earlier releases are not directly comparable; the changelog and `docs/evaluation.md` say so.

    Verified on the PR head merged with main (CPU): 34 new tests (13 of them fail on clean main) and 406 passed with the neighbouring diagnose and changelog-fragment tests; 7 of 8 mutations of the scoring killed (the survivor is the pre-existing `>=` versus `>` boundary at exactly the threshold); real-prose runs (docs/*.md, SmolLM2 tokenizer, 120 rows per length) show verbatim 1.00 and unrelated 0.00 to 0.03 from 60 to 4000 words on both paths.

    Closes #1437.

commit b9ad2fb4f7d42b9a76ff417d000f8336abe6bd54
Author: Alpamys Makazhan <[email protected]>
Date:   Thu Oct 1 20:33:45 2026 +0500

    test(serve): pin that --device is refused on every backend, not only transformers (#1478) (#1523)

    The review of #1479 (merged as `0f7ab958`) ran 24 mutations of the `soup serve --device` fix and one survived: validating `--device` only when `backend == "transformers"` passed every test, because the refusal tests all ran on the default backend. That mutation would be a real regression (an unknown device reaching vLLM/SGLang unchecked).

    This adds one parametrised case to `tests/test_issue1478_serve_device_map.py`: `--device tpu` must exit 2 on `--backend vllm` and `--backend sglang`, before either server is built or `uvicorn.run` is called. Mutation check: guarding the validation block in `serve.py` with `and backend == "transformers"` makes the two new cases fail (2 failed, 34 passed); unmutated, 36 pass. Test-only, no source change.

commit 4d14b3ddc246f5fd2f3f5255c872d5ecfaa8f3dc
Author: Chaitanya Reddy <[email protected]>
Date:   Thu Oct 1 19:47:24 2026 +0530

    fix(data): detect format from a bounded prefix, not only the first row (closes #1424) (#1496)

    `format: auto` chose a dataset's format from the first row alone, so a first row that happened to lack an optional key picked the poorer converter for the whole file and silently dropped that key from every row that had it: a LLaVA file whose first row was text-only loaded as `sharegpt` and lost every image, an OpenAI-shape tools file whose first row had no `tools` lost the tool-schema turn, and audio and ASR files lost their audio path.

    Detection now reads a bounded prefix (`MAX_DETECT_ROWS = 100` rows). Each row resolves to the richest format its keys fit; a uniform file detects exactly as before (a golden comparison over all 22 `.jsonl` files in the tree is identical to main apart from the wording "Cannot detect format at row 0"). A mixed file either upgrades to the richer format of the same family (today: `chatml` rows mixed with `tool-calling` rows load as `tool-calling`), but only when the richer converter accepts every poorer row and converts it to exactly the messages the poorer converter would (so a chatml row carrying message-level `name` or `weight` is refused rather than silently stripped), or it is refused with a message naming both shapes and the row where each was first seen and telling the user to set `data.format`. Rows that match no signature, and non-object rows, are skipped as the converters skip them instead of crashing on `.keys()`.

    What users notice: a mixed file that used to load with a key stripped, or with rows dropped under a warning, now stops at load; `soup data validate` exits 3 on it with the refusal. The Web UI data explorer detects from the same window and reports the refusal in a new `format_error` field instead of answering 500, and `docs/data.md` documents the 100-row window, the upgrade rule and the refusals.

    Verified on the PR head merged with main (CPU): 34 new tests (19 fail on main with only the constant added so the module imports), 815 passed across the 22 existing modules that call `detect_format` or the explorer, 15 of 18 mutations killed (the three survivors are ancillary lines; their tests are optional), ruff clean.

    Closes #1424.

commit 8d8abc933a50056bb8ce850f1fcdf7620dfbf3ec
Author: t <t@t>
Date:   Thu Oct 1 17:20:25 2026 +0500

    docs(contributors): #1398, #1455, #1457, #1419, #1516, #1517, #1469, #1498, #1306, #1454

    Display names from `gh api users/<login>`.

    Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

commit 1a67520c07669acdcd3ed76afbacf75cef2e0dc8
Author: Younes Beriane <[email protected]>
Date:   Thu Oct 1 14:18:54 2026 +0200

    fix: keep the previous artifact when reward synth or adapter fuse fails (#1454)

    Two commands replaced an existing artifact in an order that removed the old copy before the new one was known to be valid and in place, so a failed replace could leave no good copy at all.

    `merge_adapter_to_dense` (used by `soup shrink` to fuse the heal adapter into the pruned model, and by `soup draft distill` to write the dense draft) deleted `out_dir` and then moved the staged model in, and its error handler also deleted the staged model, so a failure between the two steps (an antivirus or indexer lock, disk full) left neither copy. It now renames the previous model aside, swaps the merged model in, restores the previous model if the swap fails, and removes the aside copy only after the swap succeeded. If the restore also fails, both copies are kept and both paths are named in the error; if the rename-aside itself fails, `out_dir` is left unchanged and the staged model is kept and named. Interrupts (Ctrl-C) are no longer turned into a "the old model is unchanged" error. The docstring says what the swap guarantees.

    `soup reward synth -o reward.py --force` overwrote the existing verifier with the candidate before calibrating it, then deleted the file when calibration refused or failed, which destroyed a previously working verifier. It now writes the candidate to a `.soup.reward-candidate.*.py` sibling, loads and calibrates that file, re-validates `-o`, and replaces `-o` only once calibration accepts. A refusal (exit 2), a calibration error (exit 1), a failed candidate write or a failed final replace leaves the previous verifier byte-identical and removes the candidate.

    Verified: 11 new tests (the Windows open-handle case runs on Windows); 367 tests across the related files pass on the PR merged with `main`; the 11 mutations of the fix all fail a test; the new file gives 9 failed / 2 passed on `main`.

    Closes #1450.

commit 2bcdbc9a7c6f50d7fe40199d6e86753b4320484e
Author: Nurkhan Esenbek <[email protected]>
Date:   Thu Oct 1 17:18:42 2026 +0500

    fix(data): wire classifier, reranker, and cross_encoder data loading (#1219) (#1306)

    `soup train` could not train the sequence-classification family. For `task: classifier` and `task: reranker` the loader kept source columns only for `grpo`, so `label` was dropped during format normalisation and the run died after the model loaded with `TypeError: label must be int / str / list, got NoneType`. For `task: cross_encoder`, `{text_a, text_b}` and `{question, answer}` pair rows could not be loaded at all. `--dry-run` said `Ready to train!` for all of them.

    Source columns are now kept for the whole family (`task_preserves_source_columns`, shared by `soup train` and `soup sweep`). `data.format: cross_encoder` exists, and `format: auto` resolves to it only when `task: cross_encoder` (`data_config_for_task`); `detect_format` and `FORMAT_SIGNATURES` are untouched, so GSM8K-shaped `{question, answer}` rows are still refused for `grpo`, `sft` and `pretrain` exactly as before. `data.format: cross_encoder` on any other task is refused at config load. Every classifier-family row (train and val) is checked on `--dry-run` and before the model loads, and an error names the row (`train row 3: missing required 'label' field for task 'cross_encoder'`). When `training.label_names` is set, `id2label` / `label2id` are written to the saved `config.json`. `docs/data.md` and `docs/training.md` document the row shapes per task.

    Verified: 791 tests across 16 files pass on the PR merged with `main`; 13 of 13 mutations of the fix fail a test; the new pin file gives 6 failed / 4 passed on `main`.

    Closes #1219.

commit 387ccb3c63d9d0998bec239890ffe0cfcc4f3ab9
Author: UmranPros <[email protected]>
Date:   Thu Oct 1 17:48:02 2026 +0530

    fix(data): from-traces reads the signals Soup's own producers write (#1440) (#1498)

    fix(data): from-traces reads the signals Soup's own producers write (#1440)

    `soup ingest` writes a top-level `signal` (`thumbs_up` / `thumbs_down` / `none`) into the trace stream that `soup data from-traces` and the loop harvest stage are documented to consume, but `parse_soup_serve` read the signal only from a nested `feedback.rating` that no Soup producer writes, and no parser ever set `edited_output`. So every ingest row read as `none`, `--signal thumbs_up` and the loop harvest built no pair, `--signal user_edit` could never match any log, and the command ended with a green `Wrote 0 preference pair(s)` and exit 0.

    `parse_soup_serve` now reads the top-level `signal` when it names a value of the closed vocabulary (`thumbs_up`, `thumbs_down`, `regenerated`, `user_edit`; anything else, including non-string values, reads as `none`) and keeps `feedback.rating` as the fallback for older logs; a top-level `signal` wins over a conflicting `feedback.rating`. `edited_output` is read from `edited_output`, `edited_response` or a nested `feedback.edited_output` (non-string and whitespace-only values ignored). When `from-traces` reads traces but builds no pair it now prints how many it read, which `--signal` it wanted and which signals were present (or that no trace carried one); an empty directory stays silent, and the `soup ingest` record-shape hint is printed only for `--format soup-serve`, the one parser that reads a top-level `signal`. `docs/data.md` is corrected: `--format soup-serve` and `--signal regenerations` are the accepted spellings, `--logs` for the soup-serve parser is a directory, and the record shape is documented with who writes which signal (`soup ingest` writes `thumbs_up` / `thumbs_down` / `none` under `trace_id`; `user_edit` and `regenerated` rows come from your own pipeline; `soup serve --trace-log` records carry no signal yet).

    Users notice that the round trip `soup ingest --source langfuse` then `soup data from-traces --format soup-serve --signal thumbs_up` now writes the pair instead of an empty file, and that a log with no usable signal says so instead of reporting a normal write of 0.

    Not in this change: `soup serve --trace-log` records still carry no signal and the harvest does not read the `--record-thumbs` database, so the loop still cannot learn from Soup's own serving; that producer half stays open on the issue. A single-file `--logs traces.jsonl` with `--format soup-serve` still ends with a green `Wrote 0` (the parser reads a directory); documented, not changed.

    Verified: the new test file fails 11 of 16 on the parent commit and passes 16 of 16 on the change; 414 tests across 9 related files pass on the merged tree; ruff clean; the round trip, the three `user_edit` shapes, a serve-trace-log shaped directory and hostile `signal` / `edited_output` / `feedback` / `id` values were executed through the real CLI; 19 mutations of the parsers and the message were applied one at a time and 15 are killed by the tests (the survivors are the openai/langchain hint gating, `regenerated` clamp membership and the non-dict `feedback` guard, each checked by hand).

    Refs #1440.

commit d73e281187fdc6cd59dc64c4fb6f6767f21a6120
Author: Kratos <[email protected]>
Date:   Thu Oct 1 20:17:48 2026 +0800

    Fix #1445: validate compile / GGUF-export / unlearn destinations before the expensive step (#1469)

    Three commands validated their destination only after the expensive step and then threw the finished work away. `soup compile -o <path outside cwd, or a symlink>` ran the whole optimizer and its LLM calls, then refused the path with an unhandled `ValueError` (exit 1). A quantized GGUF export into a directory that did not exist yet crashed in `tempfile.mkdtemp` after the adapter had been merged, and the cleanup deleted the merged model. `task: unlearn` loaded the policy and a frozen reference copy and ran every NPO / SimNPO / RMU step before refusing an `output` outside the working directory or a symlinked one.

    `build_compile_plan` now runs the containment and symlink check at plan time, so the optimizer is never called and the command exits 2 (the write-time check stays as the TOCTOU guard, and its `ValueError` is caught and exits 2 as well); `soup compile --plan-only` now refuses such a destination too instead of printing the plan and exiting 0. The export creates the destination's parent directory before the merge. `UnlearnTrainerWrapper.setup()` calls `_validated_output_dir` first, before torch, peft or any model is loaded; the check after the run stays as the TOCTOU guard.

    Verified: the new test file (9 tests, each with an in-cwd control) fails 6 of 9 on `main`; 4 of 4 mutations of the fix are killed (export `mkdir`, plan-time check, write-time `except`, unlearn `setup()` call); 1362 related tests pass; the real CLI run before and after for all three commands, including a Windows junction pointing outside the working directory.

    Closes #1445.

commit 1bc20670c3c0245fb4a09f34b6e57340061e6960
Author: Mohammad Hijjawi <[email protected]>
Date:   Thu Oct 1 13:02:14 2026 +0100

    fix(local-rl): render --min-pairs and -o into the nightly-train job (#1448) (#1517)

    Without `--once`, `soup local-rl train` renders a systemd `.service` / `.timer` and a launchd `.plist` that run `soup local-rl train --once` every night, but the rendered command carried `--db`, `--model` and `--train-method` only. The nightly run therefore trained on as few as 10 pairs into `local_rl_adapter/` whatever `--min-pairs` and `-o` the user had passed, and nothing said so. `build_train_argv`, `render_systemd_service`, `render_launchd_plist` and `write_scheduler_files` now take `min_pairs` and `output_dir` and always render `--min-pairs N --output DIR`, and the CLI passes both through, so the job does what `--once` with the same flags does. Both values are checked with the runner's own validators (`validate_min_pairs` / `validate_output_dir`, moved out of `run_nightly_train`), so a value the nightly run would refuse, such as `--min-pairs 0`, exits 2 before any scheduler directory or unit file is written, and a newline in the output path is refused like `--model` and `--db`. `docs/commands.md` lists the flags in the scheduled-mode line. `--backend` (the other half of this bullet in #1448) is unchanged. Verified: 14 new tests (12 fail on `main`), 309 passed across the neighbouring local-rl and docs-ratchet files, the issue's repro through the real CLI renders `--min-pairs 200 --output adapters/nightly` into both the `.service` and the `.plist`, and 15 of 16 mutations fail a test. Part of #1448.

commit 65265a8b86abae367a3e5a0ad430749fb9f463d5
Author: Mohammad Hijjawi <[email protected]>
Date:   Thu Oct 1 13:01:56 2026 +0100

    test(peft): count the MLX LoRA builder in the LoRA+ wiring scan (#1324 follow-up) (#1516)

    The #724 wiring scan counted only `get_peft_model(` and a `peft_config=` handed to trl as "builds a LoRA adapter", so `mlx_sft.py`, which builds its adapter with mlx-lm's `linear_to_lora_layers`, was neither wired nor exempt. That is how `backend: mlx` ignored `training.loraplus_lr_ratio` without a test noticing (#1324). The scan now counts `linear_to_lora_layers`, `mlx_sft.py` is exempt with its reason (the config-load refusal from #1383), and a new test fails when an exemption reason names a test file that does not exist. `tests/test_issue1324_mlx_loraplus_refusal.py` gains the cases #1383's file lacked: every ratio value, direct `SoupConfig(...)` construction, an explicit `null` ratio, the `transformers` / `unsloth` controls on `pretrain` and `dpo`, plain `soup train` stopping at config load, and `soup doctor --config` exiting 2. `docs/peft-and-efficiency.md` states that `backend: unsloth` gets the same LoRA+ optimizer as `transformers`. No source change and no behaviour change. Verified: 135 passed on the tree merged with `main`; six mutations (scan regex, exemption removed, exemption pointing at a renamed file, docs sentence, #1383 refusal deleted, refusal widened to `unsloth`) each fail 1 to 9 tests. Follow-up to #1324 / #1383.

commit f47d6668ba0e08f790a7594ebd2f0ba8ff58c14c
Author: Abdulwaarith Zakariyya <[email protected]>
Date:   Thu Oct 1 14:59:03 2026 +0300

    fix(ab): decide soup ab with a variance-robust NIG Bayes factor, no burn-in (#1265) (#1419)

    `soup ab` estimated the row-to-row variance from the rows it was testing, so the #1227 two-sided mSPRT needed a burn-in of 30 or 40 rows per arm, was calibrated only up to 1000 rows and down to `--alpha 0.01`, and still sat near 0.011 at 0.01. It now decides with the normal-inverse-gamma mixture Bayes factor (right-Haar limit), which depends on the rows only through the two-sample t statistic, so its false-positive rate stays at most `--alpha` however often `soup ab` is re-run: the worst simulated rate is 0.033 at `--alpha 0.05` and 0.0066 at 0.01, against 0.051 and 0.011 before (22 ratios, 100,000 runs each; record and harness in `benchmarks/gate-1265-ab-nig.md`). The first 5 rows per arm set the scale for `--effect-size` and show as `prior_scale`; the burn-in table, the 1000-row limit and both calibration warnings are gone. The cost is stated in the docs and the fragment: with no difference, `accept_h0` takes several times as many rows (281 pairs against 43 at 0.5 standard deviations), and the "under 0.4%" wrong-accept figure is for simulated Gaussian rows. When the held-out rows barely vary (a warm cache, a judge that gives 1.0 early), `accept_h0` is held at `continue` while the tested rows' standard deviation is more than 3 times the held-out one (`ACCEPT_HOLD_SPREAD_RATIO`); a `reject_h0` is never held back, so the false-positive bound is unchanged. In a 2000-run simulation that start had ended 99.0% of runs in a wrong `accept_h0` at about 7 pairs, and with the hold it ends none that way; on Gaussian rows 15 of 6000 first verdicts change and no rejection does. The `--alpha + --beta < 1` refusal from #1339 stays with its reason restated, and #1339's and #1384's refusals of values a float cannot carry cover the new statistic, which is never compared with the boundaries unless finite; an `--effect-size` past about 1.9e151 standard deviations of the held-out rows, or held-out rows whose variance underflows to 0, is refused at every row count. Verified on the head merged with `main`: 208 passed across the four `soup ab` files and three neighbours, 16 of 17 mutations of the hold and the prior bound killed (the survivor, `>=` for `>`, differs only at exact float equality), the harness self-check matches the shipped code, and the new tests fail on `main` (50 failed / 12 passed in `tests/test_issue1265_ab_nig.py`). A faster accept rule is #1418.

    Closes #1265.

commit 0ae08c7fbbf40f4744d90bdb7129b08147b4c780
Author: Kratos <[email protected]>
Date:   Thu Oct 1 19:56:14 2026 +0800

    Fix #1434: soup expect fails closed on BOM, malformed and non-object JSONL rows (#1457)

    `soup expect` could pass a dataset it never checked. It opened the file as plain `utf-8`, so a UTF-8 BOM broke the first line's JSON and that row was silently skipped (the issue's repro passed with `Rows 1` where the same file without the BOM failed on the PII in row 1). Malformed JSON lines were counted into a yellow "skipped" note, valid JSON that was not an object (`[1, 2]`, `"text"`, `42`) was dropped without a word, and a file with no checkable row at all passed with `Rows 0`. As a CI gate, that is a run that checked nothing and exited 0.

    The loader now reads `utf-8-sig`, like the training loader, so a BOM is stripped. Any malformed line or non-object line makes the run refuse with exit 3 (input error, distinct from exit 2, a real suite failure), naming the count and the first ten line numbers with `(+N more)`. A file that yields no checkable rows is refused the same way. `docs/compliance.md` and `docs/data.md` now state exit 3, the BOM strip and the zero-row refusal. Reusing the training loader so `.json`/`.csv`/`.parquet` are read too stays out of scope: a `.json` array file is refused (fail-closed) rather than read.

    Verified on the PR merged with current `main` (which includes #1461): the 9 new tests (all fail on `main`) plus the existing expect suites, 247 passed; `ruff` clean; twelve mutations of `commands/expect.py` (encoding, zero-row check, uncheckable check, both recording branches, count, line number, ten-line cap, `(+N more)`, exit code) all killed; by hand against `main`, a BOM file goes from exit 0 to exit 2, array-per-line, empty and malformed files from exit 0 to exit 3, and a clean file still exits 0. Merged cleanly together with #1499 and #1505, which touch the same module.

    Closes #1434.

commit 8bfeaa2e82887d533a1400cdb289a9da2cd4eb88
Author: Kratos <[email protected]>
Date:   Thu Oct 1 19:56:02 2026 +0800

    Fix #1429: `tokenizer train` saves a complete byte-level BPE (decoder, byte alphabet, no hard-coded unk) (#1455)

    `soup tokenizer train` saved a byte-level BPE that could not round-trip text. The trainer attached no `ByteLevel` decoder, so `decode(encode(x))` returned `Ġ` for every space and `h ll o Ġ Ġ` for `héllo 😀 世界`. It also did not seed the 256-byte initial alphabet, so characters unseen in the corpus (accents, emoji, CJK) encoded to `<unk>`. And it hard-coded `unk_token="<unk>"` into the model, so passing `--special-token` without `<unk>` saved a tokenizer that names an unk token missing from its own vocabulary, and encoding then raised `Unk token <unk> not found in the vocabulary`.

    The trainer now attaches `decoders.ByteLevel()`, seeds `ByteLevel.alphabet()` as the initial alphabet, and builds the model with no `unk_token`. `--vocab-size` is checked after the special tokens are known: it must be at least `256 + len(special tokens) + 1`, which is the true boundary (measured with `tokenizers` alone: any size up to 256 + specials yields a vocabulary with zero merges and no error, so `main` accepted `-v 256` and saved a merge-less tokenizer). Below that the command exits 2 and names the minimum. Users notice that ASCII, accented, emoji and CJK text round-trips through `Tokenizer`, `PreTrainedTokenizerFast` and `AutoTokenizer`, and that a custom `--special-token` run encodes and saves `model.unk_token = null`.

    Verified on the PR merged with current `main`: the 9 new tests (7 fail on `main`, the other 2 are the sizes `main` also accepts) plus `tests/test_v0539.py` and the changelog-fragment tests, 87 passed; `ruff` clean; eight mutations of `tokenizer.py` (decoder removed, initial alphabet removed, hard-coded unk restored, floor disabled, `+ 1` dropped, floor ignoring specials, wrong message number), each killed by the new tests; a manual `soup tokenizer train` at `-v 260` (exit 2) and `-v 261` (exit 0, exact round trip). `tokenizer_config.json` is deliberately left out, as the issue asks.

    Closes #1429.

commit 4b5f4a3cda36d482278bd4400da51a204cca2b39
Author: Srinivasan R <[email protected]>
Date:   Thu Oct 1 17:05:42 2026 +0530

    fix(recipes): drop the recipes that cannot load; MiniMax-M3 SFT loads through the vision path (#1145) (#1398)

    Four shipped recipes named a repo that exists but a model that cannot load as written (#1145). `mistral-large-3-sft` and `mistral-large-3-dpo` are removed: `mistralai/Mistral-Large-3-675B-Instruct-2512` publishes Mistral's native `params.json` + `consolidated-*.safetensors` layout with no `config.json`, which `AutoConfig` cannot read; `docs/models.md` now says so and names the Hugging Face-format conversion on the Hub as a fact, not a recommendation. `minimax-m3-dpo` is removed, because the DPO trainer builds `AutoModelForCausalLM` for every modality and cannot build the `minimax_m3_vl` wrapper (#1393). `minimax-m3-sft` now sets `modality: vision`, so SFT builds the image-text wrapper and `target_modules: auto` keeps LoRA on the language tower; it drops `moe_lora` (refused on SFT vision by #1179) and `moe_aux_loss_coeff` (read only on the text path), and because that path loads `AutoProcessor`, which is custom code for this base, its description says to run `soup train --trust-remote-code` (verified: the flag reaches the vision processor). The recipe YAML also notes that the vision collator supervises every token, so `train_on_responses_only` is not applied on this path (#1156). The catalog loses three recipes (173 -> 170, 112 bases); every count site, both recipe fixtures, the MoE tables in `docs/performance-and-quantization.md` (29 recipes set `moe_lora`: 14 SFT, 8 GRPO, 7 DPO; MiniMax 0, kimi 7) and the two attach-ratchet pins for MiniMax are updated. A new test pins the `docs/models.md` paragraph. Users notice: `soup recipes list` shows three fewer recipes, and `minimax-m3-sft` attaches LoRA (16 modules on the meta device) instead of failing at load. Verified on CPU: 2446 tests over 20 recipe, count, snapshot, ratchet and docs-guard files, 0 failed; `scripts/check_recipe_attach.py` exits 0 (118 verified, 18 pinned, 0 new, 0 now attach); 9 of 9 mutations fail a test; the PR's tests fail on unmodified `main` (14 failed, 642 passed). Closes #1145.

commit aae7508e88f6f7261d2e001df86e987bc2d5e870
Author: Alpamys <[email protected]>
Date:   Thu Oct 1 16:26:16 2026 +0500

    docs(contributors): fix Bakdaulet's entry, the first-contribution text for #1489 was missing

    The entry written with the batch credit repeated the name line instead of the description.

    Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

commit d70398d5113b106eb2405b096dcae8a2b97cbfe6
Author: Alpamys <[email protected]>
Date:   Thu Oct 1 16:25:33 2026 +0500

    docs(contributors): #1308, #1460, #1461, #1168, #1366, #1462, #1463, #1364, #1492, #1493, #1489, #1416, #1484, #1490, #1479, #1485, #1495

    Display names from `gh api users/<login>`.

    Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

commit 7e681f07a72e529ce26276af4d8350011a19c0b2
Author: Ashutosh Panda <[email protected]>
Date:   Thu Oct 1 16:54:00 2026 +0530

    fix(tests): pin #1408 helper with regression tests (#1495)

    When the suite ran from inside a virtualenv whose base interpreter has none of the dependencies (the CONTRIBUTING setup on, for example, Homebrew Python), `test_worker_verification_survives_a_real_venv_launcher` failed with an empty `RuntimeError: MLX teacher worker exited with 1:`. The child venv is created with `system_site_packages=True`, which from inside a venv exposes only the base interpreter's site-packages, not the outer venv's, so the worker died importing pydantic before writing anything and the failure looked like a worker bug.

    The test now writes a `.pth` file into the child venv that lists the outer venv's site-packages and `src/` (a no-op on a bare interpreter, so the test is unchanged for everyone running from a global Python), and probes `import pydantic` in the child first, so a missing link names the missing dependency instead of surfacing an empty exit. Two hermetic tests pin the helper on every platform, including the POSIX layout on CI: a package that exists only in a fake parent site-packages becomes importable in a child venv after the helper runs and not before, and nothing is written on a bare interpreter. Test-only; no user-visible change.

    Verified: the test file gives 19 passed from a bare interpreter and from a venv over a dependency-free CPython; clean `main` from that venv gives 1 failed, 16 passed (the reported error); four mutations of the helper (never write the `.pth`, `src/` only, wrong suffix, inverted early return) each fail the new tests, and removing the helper's call is caught from the dependency-free venv with a message naming pydantic.

    Closes #1408.

commit 200fc21af339705f960bdd997db7cd7da2d48862
Author: dymux <[email protected]>
Date:   Thu Oct 1 18:23:48 2026 +0700

    fix(cli): write machine-readable JSON straight to stdout (#1480) (#1485)

    `soup version --json --full`, `soup cost --json` and `soup eval against --json-only` printed their machine-readable JSON through Rich's `console.print`, which folds text at the console width (80 columns when piped) and reads `[...]` in a string as markup. A `version --json --full` record was split across lines, a `cost --json` value such as `[v2]` was dropped while the command exited 0, and `[/x]` (in a cost value or an `eval against` baseline id) aborted the command with `MarkupError`. All three now write with `typer.echo`, the convention the other machine-readable paths use; the human-readable branches still go through the Rich console. A repo-wide AST ratchet fails on any `console.print(<serialiser>.dumps(...))` under `src/soup_cli/`, with a self-test so a detector that stops matching cannot pass as a clean tree. `soup eval against --json-only` is a third site the issue did not name; it is reached through a `_json` alias that a grep misses. Users notice: one JSON document on one line, and bracketed text in a value survives. Verified: the new file is 6 passed (5 failed / 1 passed on `main`); 355 passed and 1 skipped across 12 related files with `main` merged (the one other local failure, `tests/test_cli.py::test_version_full_json`, fails the same way on `main`); each of the three sites reverted on its own fails its own behavioural test plus the ratchet (2, 3 and 2 failures), and both detector mutations fail the self-test. A real piped `soup version --json --full` and `soup cost --json` both parse with `json.load`. Closes #1480.

commit 0f7ab958525d619a7aaa55d46743c2130a52dad8
Author: Younes Beriane <[email protected]>
Date:   Thu Oct 1 13:23:37 2026 +0200

    fix(serve): honour --device in the transformers loader (#1479)

    `soup serve --device` was ignored by the transformers loader: `_load_model` (both the merged-model and the base + adapter branch) and the speculative draft loader hard-coded `device_map="auto"`, so `--device cpu` could still put the model on a visible GPU, and an unknown value such as `tpu` was accepted and silently dropped (`--device "[/bold]"` died with a Rich `MarkupError`). `soup serve` now places the model with the same rule as `soup infer` / `chat` / `diff` (`resolve_inference_device_map_and_dtype`): `cpu` and `mlx` load on the CPU in float32, `mps` and `cuda:N` are pinned to that device in float16, `cuda` keeps `device_map="auto"` in float16, and an explicit `--kv-cache-type bf16` / `f16` still sets the dtype on every device. `--device` is validated before anything loads, on every backend, and an unknown value exits 2 naming `cpu, cuda, cuda:<index> or mps`. The draft model is placed like the main model. Users notice: `--device cpu` really runs on the CPU; on a CPU-only host the default load is now float32 pinned to the CPU, which doubles its RAM (a 7B model needs about 28 GB instead of 14 GB); `cuda:N` is honoured (it was accepted and ignored before); the `--device` help describes all of it. Verified: the new file is 34 passed (19 failed / 15 passed on `main`), 627 passed across 17 related files, 34 passed again under forced colour at 60 columns; 23 of 24 mutations of the fix fail a test (the survivor is validation gated on the transformers backend, which the CLI does not do on vLLM or SGLang); the real CLI and a tiny real Llama loaded on CPU give the placements and dtypes above. Closes #1478.

commit 2aeda018094c242dfbec11a059c922c08be005d0
Author: Momoyeyu <[email protected]>
Date:   Thu Oct 1 19:23:25 2026 +0800

    fix(eval): score custom tasks by stated answer; token-bound contains (#1342) (#1490)

    Custom eval tasks scored a model's stated answer wrongly in both directions. `scoring: "contains"` was a raw substring test, so an expected `4` was paid by `14` and an expected `42` by `#### 420`; `scoring: "exact"` compared the whole output, so `The answer is 4.` scored 0 against `4`.

    `contains` now requires the expected string to appear as a whole alphanumeric-bounded token (case-insensitive), so `14` no longer contains `4` and `#### 420` no longer contains `42`. A new `scoring: "answer"` compares the final answer the model states, read with the same parser GRPO's `accuracy` reward uses (`#### 4`, `\boxed{4}`, `The answer is 4.`; numbers compare by value), and refuses at load time an `expected` that states no single answer, naming the line, instead of silently scoring 0. The eval gate's `scorer:` override accepts `answer` too. `exact` keeps its whole-output meaning and is now documented as such.

    This changes scores. A gate baseline (`soup eval gate --baseline` file or `registry://` row) or a `soup eval custom --run-id` result recorded with the old substring `contains` can be higher than the same outputs score now, and the baseline's scorer stamp covers only the bundled suites, so it does not warn: re-measure those before comparing. The changelog and `docs/evaluation.md` say so.

    Verified: 30 new tests (15 fail on `main`), 17 of 18 review mutations caught, 433 related tests passed, `ruff` clean.

    Closes #1342.

commit a07d6ba2f9a12c028090db251ddbe4b5ed6bbb25
Author: Srinivasan R <[email protected]>
Date:   Thu Oct 1 16:53:13 2026 +0530

    fix(eval): retry judge requests and report an unreachable judge instead of scoring it as a tie (#1447) (#1484)

    Every LLM-judge failure was either scored as a tie or aborted the whole batch, and nothing was ever retried. `compare_pair` turned any judge exception into a tie, so `soup ship --task-mode pairwise` with the judge down scored a measured 0.5 win-rate and printed DON'T SHIP; `soup eval judge` died on the first transport error and lost the scores already collected; a training eval gate logged an unreachable judge as a regression.

    Every judge request now goes through one helper. It retries 429, 5xx and transport errors twice (3 attempts, pause 1 s then 2 s, or `Retry-After` when it is a finite number, capped at 30 s), fails at once on a 4xx that will not heal or a reply it cannot decode, treats a reply with no usable content as a judge error, and raises `JudgeUnavailableError` naming the judge (scheme, host, port and path only, never userinfo or a query string) once the retries are spent. `soup ship --task-mode pairwise` exits 1 with that message instead of a verdict; `soup eval judge` skips and counts the failing row and still scores, shows and saves the rest (`1/3 items skipped`); the eval gate still fails closed, and its log line says which task could not be scored and why rather than calling it a regression. A judge that answers "tie" still scores 0.5, and the Online DPO adapter keeps leaving a failed pair unranked, as the ranking trainer expects.

    Review found three more ways to escape that design, all fixed before merge: `Retry-After: nan` reached `time.sleep` and raised a raw `ValueError`; the judge URL, userinfo and `?api_key=` included, went into the message, `GateTaskResult.error` and the training log; and an undecodable reply (`httpx.DecodingError`) aborted a run. Verified against a real local HTTP server as well as through the CLIs: 45 new tests, of which 40 fail on `main` with the new names shimmed in; 17 of 19 review mutations caught; related suites 470 passed.

    Closes #1447.

commit e967397f585efb488f9464801eb2786efcbe38dc
Author: Ashutosh Panda <[email protected]>
Date:   Thu Oct 1 16:53:01 2026 +0530

    fix(trainer): read trl PPO per-step loss from loss/policy_avg (#1413) (#1416)

    A completed `task: ppo` run printed `Loss: unavailable` even though training worked: `summarize_training_loss` only read the `loss` key of `trainer.state.log_history`, while trl 0.29's `PPOTrainer` logs its per-step policy loss under `loss/policy_avg` and never writes `loss` or `train_loss`, so every entry was skipped and the run ended with "unavailable".

    `summarize_training_loss` now takes the per-step key from the caller (`loss_key`, default `"loss"`, so SFT/DPO summaries are byte-identical), and the trl path in `trainer/ppo.py` declares `loss_key="loss/policy_avg"` instead of the summary guessing it; the manual PPO loop keeps the default because it normalises to `loss`. A non-default key is recorded in the result as `loss_key` on the `delta` and `single` branches only (never on the `train_loss` mean fallback or an `unavailable` result), and the completion panel shows it, e.g. `Loss: 1.4200 -> 0.8700 (loss/policy_avg)`, so a PPO policy loss is not read as an SFT cross-entropy. SFT panel lines are unchanged.

    Verified: against the real trl 0.29.1 `PPOTrainer` through `PPOTrainerWrapper.train()` on a tiny CPU Llama the result is `delta` with `loss_key: loss/policy_avg` (main: `unavailable`); the 9 new tests fail 6 of 9 on main and kill 11 of 11 mutations of the fix and the panel (including moving the key to the manual-loop call and summarising an empty history, via a behavioural test that drives `_train_builtin` on a trl-0.29-shaped `log_history`); 24 related test files give 1268 passed, 0 failed on the merged tree. Three optional mutations (re-tagging the mean fallback or the unavailable result) are not pinned by a test; trl's PPOTrainer never logs `train_loss`, so nothing is wrong today. The live per-step display still does not see `loss/policy_avg` (`SoupTrainerCallback.on_log` reads only `loss`); that is a separate gap.

    Closes #1413.

commit f562f79d4af4ec0a52c0a785c7522d3f80c38989
Author: Bakdaulet <[email protected]>
Date:   Thu Oct 1 16:22:51 2026 +0500

    fix(diagnose): report unmeasured probes as NOT_RUN instead of OK (#1489)

    fix(diagnose): report unmeasured probes as NOT_RUN instead of OK 1.00 (part 1 of #1435)

    `soup diagnose` scored a live probe that never produced a measurement as a perfect pass: a `--dataset` with no usable rows, a probe that raised (for example a 9000-character prompt over the 8192 limit), a typo in `--tokenizer`, or memorization on `question`/`answer`, `query`/`response` or `input`/`output` rows, which `score_memorization` cannot read. The overall verdict, the exit code and the badge all treated those as OK 1.00 and the exception text was dropped.

    This adds a `NOT_RUN` verdict (score 0.0 only, ranked above MINOR and below MAJOR, grey in the badge and the table, shown as `not run` / `—` instead of a score). All six converted probe sites keep the exception text in the evidence (NUL-stripped, capped, Rich-escaped), and memorization reports `NOT_RUN` when none of the first rows has text to split. `soup diagnose` exits 3 when the overall verdict is `NOT_RUN`; `--allow-not-run` keeps exit 0 for runs that knowingly skip probes; MAJOR still exits 2 and is never hidden. `--tokenizer` is resolved before any model loads, so a bad or empty value is an input error (exit 3). `train --diagnose-gate`, the `--evidence` path and the MCP `diagnose_evidence` tool understand `NOT_RUN` through one shared helper. Probes that do not apply (no `--dataset`, non-JSON format targets, no benchmark corpus, modes omitted from an evidence file) stay a neutral OK.

    Users notice: live runs that used to exit 0 on an unmeasured probe now exit 3, and ShareGPT / LLaVA / plaintext datasets read as `NOT_RUN` until part 2 of #1435 teaches the row builders those shapes. Known and declared gap: memorization still reads OK when every text is too short to split; it waits on #1481.

    Verified on the PR head merged with main (CPU): 175 passed across the two touched test files (39 of them fail on clean main), 1212 passed across 17 diagnose-related test files, ruff clean, 19 of 20 mutations of the new logic killed (the survi…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full Run the full CI matrix on this PR (a maintainer adds it at approval)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

soup serve --device is accepted, but the transformers loader always places the model with device_map="auto"

2 participants