Skip to content

fix: thread --trust-remote-code to every load it was silently dropped from - #1456

Merged
MakazhanAlpamys merged 4 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1449-fix
Sep 30, 2026
Merged

MakazhanAlpamys merged 4 commits into
MakazhanAlpamys:mainfrom
drakeo338:claude/1449-fix

Conversation

@drakeo338

Copy link
Copy Markdown
Contributor

Fixes #1449.

What does this PR do?

--trust-remote-code was accepted by data download, eval auto, train --find-lr, draft distill and shrink but never actually reached the load it was meant to unblock: two commands ignored the flag entirely, train --find-lr hardcoded trust_remote_code=False and silently wrote a synthetic LR curve on refusal, and draft distill/shrink resolved the flag but dropped it when spawning the child soup train process. This change threads the flag through all five paths to the actual from_pretrained/load_dataset/pipeline calls and child argv, and also refuses trust_remote_code outright on datasets>=4 (which dropped script execution) instead of pretending it still works.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Tests

Checklist

  • ruff check src/soup_cli/ scripts/ tests/ passes
  • pytest tests/ -v passes — not run in full; ran the discriminating subset (tests/test_trust_remote_code.py tests/test_issue1449_trust_remote_code_dropped.py, 37 passed) that covers every path this change touches
  • Updated relevant docs (README.md and the matching page under docs/) if needed — docs/performance-and-quantization.md already documents the intended behavior; no doc content changed
  • Added a changelog fragment, or this change has no user-visible impact

🤖 Generated with Claude Code

batch_size=8,
run_id=self.run_id,
device=None,
trust_remote_code=False,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff plus config/schema.py and the trainer classes. The 5 sites from #1449 all look correctly threaded and the new test file exercises each one, so I don't have a defect on those.

One spot this PR also had to touch: monitoring/callback.py:581 and :598 now pass trust_remote_code=False explicitly into the auto-eval benchmark()/custom() calls, because those functions gained the parameter. SoupConfig still has no trust_remote_code field (confirmed in config/schema.py), so there's no value to read here yet.

That means a run started with soup train --trust-remote-code for a custom-code base model can still load fine for training, but its periodic auto-eval will fail to load the same model, caught by the surrounding except Exception: logger.exception(...) and silently skipped rather than surfaced. Not a regression from this PR, just the same #1449 gap in a spot the issue didn't name.

Is that a deliberate scope cut, or worth its own follow-up issue?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deliberate: SoupConfig has no field for it yet, so auto-eval has nothing to read. Happy to thread the --trust-remote-code flag into the callback here or in a follow-up, whichever you prefer.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks for confirming. Agreed it belongs in its own follow-up rather than this PR.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drakeo338, thank you for this. The fix itself is right: I checked all five sites by execution and every one now does what #1449 asks. One important thing is still missing before it can merge: the tests do not pin the safe default, and that matters more than usual here because the flag runs code from a model repo.

What is good

  • All five sites are threaded correctly and each has its own test that records what actually reaches the consumer (load_dataset kwargs, lm-eval model_args, the pipeline kwargs, both find-lr from_pretrained calls, the child argv). On clean origin/main your new file gives 7 failed / 6 passed; on your head 13 passed.
  • Acceptance criterion 4 holds for real, not only in argv: I ran _run_distill against a local auto_map draft base and then ran the parent's argv through the child soup train in-process. With trc=True the child gets past the trust check (it stops later only because my stub directory has no tokenizer). With trc=False it raises the "requires trust_remote_code=True" error, as it should.
  • The injection guard survives: with --trust-remote-code set, an adapter whose base_model_name_or_path is x/y,trust_remote_code=True is still refused and lm-eval is never called. Through the real CLI, eval benchmark, eval custom and eval auto without the flag produce pretrained=...,peft=... with no trust_remote_code and trust_remote_code=False for the generator; with the flag they get True. Nothing enables remote code implicitly today.
  • Refusing the flag on datasets>=4 instead of forwarding something that would be ignored is exactly the "refused or explained, not ignored" the issue asked for, and the explicit trust_remote_code=trust_remote_code (instead of leaving pipeline at None) is a small tightening too.

What I ran

pytest tests/test_issue1449_trust_remote_code_dropped.py tests/test_issue752_cli_commands_called_as_functions.py tests/test_typer_defaults_do_not_leak.py tests/test_trust_remote_code.py -> 59 passed. Then 22 mutations, one at a time, against your 13 new tests: 15 are killed (every "drop the argument at site X" mutation, for all five sites, including hard-coding False in either find-lr load, either child argv, the pipeline call and the >= 4 gate). Seven survive. Two of those (removing the kwarg from auto() -> benchmark(), and from the training callback) are caught by the repo's typer-default guards. Five survive everything, including those guards (105 tests):

mutation result
eval.py:119 if trust_remote_code: -> if True: (always append trust_remote_code=True) 13 passed, 105 passed
eval.py:80 benchmark() option default False -> True 13 passed, 105 passed
callback.py:581 training auto-eval calls benchmark(..., trust_remote_code=True) 13 passed, 105 passed
callback.py:598 same for custom(...) 13 passed, 105 passed
data.py:1536 print the "Remote Code Warning" panel even without the flag 13 passed, 105 passed

Each mutation was undone and git diff is empty afterwards.

Please fix before merge

  1. Pin the default-deny direction (blocking). The issue says "Without the flag, the same tests see False or no flag", and the eval tests only have the with-flag case. The callback rows above are the serious ones: if someone flips either to True, every soup train with eval.auto_eval: true would start running remote code without anyone opting in, and no test would notice. The following 8 tests are verified: they pass on your head, and between them they kill all five rows above plus the dropped-kwarg mutation on auto(). Add to tests/test_issue1449_trust_remote_code_dropped.py (needs import pytest and from tests.conftest import strip_ansi):
def _flat(text: str) -> str:
    return " ".join(strip_ansi(text).split())


class TestDefaultsStayDeny:
    def test_eval_auto_without_flag_trusts_nothing(self, tmp_path, monkeypatch):
        from soup_cli.cli import app

        seen_model_args, seen_generator_calls = TestEvalAutoTrust()._setup(tmp_path, monkeypatch)
        result = runner.invoke(app, ["eval", "auto", "-c", "soup.yaml"])
        assert result.exit_code == 0, result.output
        assert seen_model_args and "trust_remote_code" not in seen_model_args[0], seen_model_args
        assert seen_generator_calls[0][1].get("trust_remote_code") is False, seen_generator_calls

    @pytest.mark.parametrize(("flag", "expected"), [([], False), (["--trust-remote-code"], True)])
    def test_eval_benchmark_cli_model_args(self, tmp_path, monkeypatch, flag, expected):
        from soup_cli.cli import app

        seen_model_args, _ = TestEvalAutoTrust()._setup(tmp_path, monkeypatch)
        result = runner.invoke(app, ["eval", "benchmark", "-m", "out", "-b", "mmlu", *flag])
        assert result.exit_code == 0, result.output
        assert ("trust_remote_code=True" in seen_model_args[0]) is expected, seen_model_args

    @pytest.mark.parametrize(("flag", "expected"), [([], False), (["--trust-remote-code"], True)])
    def test_eval_custom_cli_generator_kwargs(self, tmp_path, monkeypatch, flag, expected):
        from soup_cli.cli import app

        _, seen_generator_calls = TestEvalAutoTrust()._setup(tmp_path, monkeypatch)
        result = runner.invoke(app, ["eval", "custom", "-t", "tasks.jsonl", "-m", "out", *flag])
        assert result.exit_code == 0, result.output
        assert seen_generator_calls[0][1].get("trust_remote_code") is expected, seen_generator_calls

    def test_training_auto_eval_callback_never_opts_in(self, monkeypatch):
        import soup_cli.commands.eval as ce
        from soup_cli.monitoring.callback import SoupTrainerCallback

        calls: dict = {}
        monkeypatch.setattr(ce, "benchmark", lambda **kw: calls.__setitem__("benchmark", kw))
        monkeypatch.setattr(ce, "custom", lambda **kw: calls.__setitem__("custom", kw))
        callback = SoupTrainerCallback.__new__(SoupTrainerCallback)
        callback.eval_config = type(
            "EvalCfg", (),
            {"auto_eval": True, "benchmarks": ["mmlu"], "custom_tasks": "tasks.jsonl"},
        )()
        callback.output_dir = "out"
        callback.run_id = "run-1"
        callback._run_auto_eval()
        assert calls["benchmark"]["trust_remote_code"] is False, calls
        assert calls["custom"]["trust_remote_code"] is False, calls

    @pytest.mark.parametrize(("flag", "panel"), [([], False), (["--trust-remote-code"], True)])
    def test_data_download_warning_panel_only_with_flag(self, tmp_path, monkeypatch, flag, panel):
        import datasets

        from soup_cli.cli import app

        monkeypatch.chdir(tmp_path)
        monkeypatch.setattr(datasets, "__version__", "3.6.0")
        monkeypatch.setattr(datasets, "load_dataset", lambda *a, **k: iter([{"text": "x"}]))
        result = runner.invoke(
            app, ["data", "download", "org/ds", "-n", "1", "-o", "o.jsonl", *flag],
        )
        assert result.exit_code == 0, result.output
        assert ("Remote Code Warning" in _flat(result.output)) is panel, _flat(result.output)

(The last test covers the "panel only when it is true" acceptance criterion, which no test checks today.)

  1. --trust-remote-code with datasets missing now raises a traceback (data.py:1228-1235). _datasets_major_version does an unguarded import datasets, before the existing except ImportError at data.py:1568. Executed: flag plus missing datasets -> exit 1 with a raw ModuleNotFoundError; without the flag you still get "datasets library not available. Install with: pip install datasets". datasets is part of the [train] extra, so a base install can hit it. Fix:
    try:
        import datasets
    except ImportError:
        return None

Test (fails on your head, passes with the fix; 14 passed together with your file):

    def test_flag_without_datasets_installed_gets_the_install_hint(self, tmp_path, monkeypatch):
        from soup_cli.cli import app

        monkeypatch.chdir(tmp_path)
        monkeypatch.setitem(sys.modules, "datasets", None)  # `import datasets` raises ImportError
        result = runner.invoke(
            app, ["data", "download", "org/ds", "-n", "1", "-o", "o.jsonl", "--trust-remote-code"],
        )
        assert result.exit_code == 1, (result.output, repr(result.exception))
        assert "datasets library not available" in _flat(result.output)
  1. Docs and fragment. This PR adds a public --trust-remote-code option to soup eval benchmark and soup eval custom and turns soup data download --trust-remote-code into an error on datasets>=4. Please add to docs/performance-and-quantization.md (the "Coverage" list under "--trust-remote-code opt-in"): soup eval benchmark / eval custom, soup train --find-lr, soup draft distill, soup shrink, and one sentence saying the flag is refused for data download when the installed datasets is 4 or newer. Add a clause about the two new options to changelog.d/0.75.1/1456.fixed.md.

Optional, not blocking: the eval and find-lr paths forward the raw flag and print no REMOTE CODE WARNING panel, while the docs say other commands print one for non-first-party models. A follow-up could route them through resolve_trust_remote_code.

@AmirF194's question on callback.py:581: good catch, thank you. Reading the code confirms it: SoupConfig has no field for this, so the training auto-eval passes False and, for a custom-code base trained with --trust-remote-code, its load fails and is skipped by the surrounding except. I would keep that out of this PR: it needs a decision about how the flag travels from train() into the callback, which is a different change from the five sites in #1449. In this PR, keep the explicit False and pin it with test_training_auto_eval_callback_never_opts_in above. The follow-up can then flip that one test deliberately.

Merge state: all 11 required checks are green on 12c3e86b, and the branch merges cleanly with current main. One heads-up: #1334 (the --find-lr refusal work) edits the same region of train.py and conflicts with this PR, so whichever lands second needs a rebase.

With item 1 and the ImportError guard in, the gaps I found are closed. Thank you for the careful per-site tests.

@drakeo338

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. Pushed e8ce9cf: the safe default is now pinned at every site, all five surviving mutations (plus the two dropped-kwarg ones) are killed, the missing-datasets ImportError is guarded, and docs and the fragment are updated.

@MakazhanAlpamys

Copy link
Copy Markdown
Owner

@drakeo338, a heads-up for your rebase: #1334 merged as 636d3efb. It renames the function your --find-lr change edits in train.py (_run_live_lr_sweep_or_synth) and deletes the synthetic curve, so your branch will conflict there. Take main's version of that function and re-apply the trust_remote_code forwarding at its current name. The "writes a synthetic curve" sentence in #1449 is obsolete after #1334. The items in my review are unchanged.

… from (MakazhanAlpamys#1449)

`soup data download`, `soup eval auto` and `soup train --find-lr` accepted
`--trust-remote-code` but never passed it to the load that needed it, and
`soup draft distill` / `soup shrink` (heal step) resolved the flag but never
forwarded it to the child `soup train` subprocess they spawn. Opting in
changed nothing on any of the five paths.

- `data.py`: `_hf_download_dataset` takes and forwards `trust_remote_code` to
  `load_dataset`; `download_dataset` passes the resolved flag through and
  only prints the "Remote Code Warning" panel when it is set.
- `eval.py`: `benchmark()` and `custom()` each gain the flag. `benchmark()`
  appends `,trust_remote_code=True` to the lm-eval `model_args` from the
  resolved flag only (never from `adapter_config.json`'s
  `base_model_name_or_path`, which `_reject_lm_eval_injection` still
  refuses if it carries `,`/`='`). `auto()` forwards its own flag to both.
- `eval/custom.py`: `_create_default_generator` forwards the flag to the
  `pipeline(...)` call.
- `train.py`: `_live_lr_sweep_from_config` receives the flag from `train()`
  via `_run_live_lr_sweep_or_synth` instead of hardcoding `False` on both
  `from_pretrained` calls.
- `draft.py` / `shrink.py`: `_run_distill` / `_run_heal` append
  `--trust-remote-code` to the child `soup train` argv when the already-
  resolved `trc` is true, instead of the child refusing and telling the
  user to re-run with a flag they already passed.

`monitoring/callback.py`'s post-training auto-eval calls `benchmark()` and
`custom()` as plain functions and has no trust context of its own, so it now
passes `trust_remote_code=False` explicitly to keep every typer parameter
filled (MakazhanAlpamys#752).

Adds `tests/test_issue1449_trust_remote_code_dropped.py`, one test per site
plus the injection-guard control, each asserting what the consumer actually
receives with and without the flag.

Closes MakazhanAlpamys#1449.
…s#1449's tests

Review round on MakazhanAlpamys#1449 found one real gap and several weak tests:

- `soup data download --trust-remote-code` forwarded the flag to
  `load_dataset` unconditionally. On datasets>=4, `load_dataset` pops
  `trust_remote_code` from its kwargs and only logs an error — it no
  longer supports it at all — so the flag silently did nothing, the same
  bug this issue was filed for, one layer down. `download_dataset` now
  checks the installed `datasets` major version and refuses the flag
  outright when it is >= 4, instead of forwarding it into the void.

- The find-lr test only ever exercised `AutoTokenizer.from_pretrained`:
  both Auto* classes shared one stub that raised immediately, so the
  tokenizer call raised before `AutoModelForCausalLM.from_pretrained`
  was ever reached, and the assertions read `seen` as if the model call
  had been checked too. Split them into two stubs (tokenizer succeeds,
  model raises) and assert `len(seen) == 2`. Also stopped the test's
  fake `torch` module from being injected into `sys.modules` before the
  Auto* attributes were first read: this transformers version's lazy
  module replaces `sys.modules["transformers"]` the first time certain
  attributes are resolved, and doing that while a fake `torch` is
  already registered risks caching a broken class under those names for
  the rest of the test session. Reading the attributes first (and only
  injecting the fake when real torch isn't installed) avoids it. The
  same module-identity quirk turned up while adding the new
  `_create_default_generator` test below, so it gets the same fix.

- Added a unit test for `_create_default_generator` itself, stubbing
  `transformers.pipeline` directly and asserting the kwarg it receives
  with and without the flag.

- Tightened the lm-eval injection-guard test to assert on "Refusing to
  evaluate" rather than accepting any non-zero exit code, and added the
  missing `shrink --heal` negative test (`trc=False` omits the flag from
  the child `soup train` argv), mirroring the one `draft distill` already
  had.

The changelog fragment's filename should be this fix's own PR number per
changelog.d/README.md, not the issue number — it can't be renamed until
that PR exists, so it stays as `1449.fixed.md` for now and needs renaming
once the PR is opened.
… document coverage

Without --trust-remote-code every site must pass trust_remote_code=False:
eval benchmark/custom/auto, the training auto-eval callback, the data
download warning panel and _create_default_generator's default are now
pinned by tests. _datasets_major_version no longer raises when datasets
is not installed, so the flag gets the usual install hint instead of a
traceback. Docs list the new coverage and the datasets>=4 refusal.
@drakeo338

Copy link
Copy Markdown
Contributor Author

Rebased on 636d3ef; the find-lr trust_remote_code forwarding now sits on _run_live_lr_sweep / _live_lr_sweep_from_config and main's version of the function is otherwise unchanged.

@MakazhanAlpamys MakazhanAlpamys left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drakeo338, thank you for the quick and complete round. I re-checked every item from my review by execution on the current head (15e886bc, merged with current main), and all of them are closed. This PR now waits only on the required checks.

What is good

  • The default-deny tests are exactly what the issue's acceptance criterion asked for, and you went one better with test_default_generator_defaults_to_false (a caller that omits the kwarg gets an explicit False, not None). That is the test that would catch a silent change of the pipeline(...) default later.
  • The ImportError guard is in _datasets_major_version and the test drives the real CLI with datasets made unimportable, so it checks the user-visible message ("datasets library not available") and not just the helper.
  • The rebase over #1334 was done the careful way: the forwarding sits on _run_live_lr_sweep / _live_lr_sweep_from_config and no synthetic-curve code came back.
  • Docs and fragment now say what the PR does, including the two new --trust-remote-code options on eval benchmark / eval custom and the datasets>=4 refusal.

What I ran

  • On your head: pytest tests/test_issue1449_trust_remote_code_dropped.py tests/test_issue752_cli_commands_called_as_functions.py tests/test_typer_defaults_do_not_leak.py tests/test_trust_remote_code.py -> 69 passed. On the tree merged with current main (4f51716f, no conflicts) the same four files plus test_eval, test_code_review_critical, test_code_review_high, test_v0401_part_c, test_dataset_hub, test_part_a_wave1, test_v07129, test_v07133 -> 423 passed. ruff is clean and the changelog fragment validates.
  • Mutations: 24, one at a time, against your 23 tests alone (no other guard files). All 24 fail a test. Last round 15 of 22 did. The five that survived everything are now caught (if True: at eval.py:119 -> 2 failed; benchmark() option default True -> 1 failed; callback passing True at callback.py:581 and :598 -> 1 failed each; the warning panel without the flag at data.py:1539 -> 1 failed), and so are the new ones I added this time (custom() option default True, the unguarded import datasets at data.py:1233). git diff is empty afterwards.
  • On clean main your new file gives 14 failed, 9 passed (the 9 are the without-flag and no-op cases).
  • Through the real CLI with the installed datasets 5.0.1: soup data download org/ds -n 1 -o o.jsonl --trust-remote-code prints the refusal and stops before any network call, and soup eval benchmark --help / soup eval custom --help both list --trust-remote-code with the default-deny text.

Two small notes, neither blocks the merge

  1. soup eval human (eval.py:938-939) and run_eval (eval/custom.py:338) still build their generator with the default False, so a custom-code model cannot be evaluated through them even with the flag. They were not among the five sites in #1449, so I left them out of this PR.
  2. As agreed with @AmirF194 on callback.py:581, threading the flag from soup train into the training auto-eval is a separate change. When it lands it will flip test_training_auto_eval_callback_never_opts_in on purpose, which is what that test is for.

Merge state: the branch is rebased on 636d3efb and merges cleanly with current main. The required checks on 15e886bc are still queued or running at the time of writing (mlx-smoke and pytorch-smoke are green, nothing has failed), so the merge waits on them.

Thank you for the careful per-site tests and for taking the review points one by one.

@MakazhanAlpamys MakazhanAlpamys added the ci:full Run the full CI matrix on this PR (a maintainer adds it at approval) label Sep 30, 2026
@MakazhanAlpamys
MakazhanAlpamys merged commit 5068bb6 into MakazhanAlpamys:main Sep 30, 2026
19 checks passed
@MakazhanAlpamys

Copy link
Copy Markdown
Owner

Merged as 5068bb6c. Thank you, @drakeo338: the per-site tests record what actually reaches each consumer (the load_dataset kwargs, the lm-eval model_args, the pipeline kwargs, both find-lr from_pretrained calls, the child argv), and in the review round you pinned the default-deny direction at every site, including the training auto-eval callback, which is the one place a wrong default would have run remote code unprompted on every soup train with eval.auto_eval. 23 new tests, all 24 mutations I ran fail a test (the five that survived the first round are all caught), 14 of the new file's tests fail on clean main, and the real CLI refuses --trust-remote-code on the installed datasets 5.0.1 before any network call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full Run the full CI matrix on this PR (a maintainer adds it at approval)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--trust-remote-code is dropped by data download, eval auto, train --find-lr, draft distill and shrink

3 participants