Repository navigation
fix: thread --trust-remote-code to every load it was silently dropped from - #1456
Conversation
| batch_size=8, | ||
| run_id=self.run_id, | ||
| device=None, | ||
| trust_remote_code=False, |
There was a problem hiding this comment.
Read the full diff plus config/schema.py and the trainer classes. The 5 sites from #1449 all look correctly threaded and the new test file exercises each one, so I don't have a defect on those.
One spot this PR also had to touch: monitoring/callback.py:581 and :598 now pass trust_remote_code=False explicitly into the auto-eval benchmark()/custom() calls, because those functions gained the parameter. SoupConfig still has no trust_remote_code field (confirmed in config/schema.py), so there's no value to read here yet.
That means a run started with soup train --trust-remote-code for a custom-code base model can still load fine for training, but its periodic auto-eval will fail to load the same model, caught by the surrounding except Exception: logger.exception(...) and silently skipped rather than surfaced. Not a regression from this PR, just the same #1449 gap in a spot the issue didn't name.
Is that a deliberate scope cut, or worth its own follow-up issue?
There was a problem hiding this comment.
Deliberate: SoupConfig has no field for it yet, so auto-eval has nothing to read. Happy to thread the --trust-remote-code flag into the callback here or in a follow-up, whichever you prefer.
There was a problem hiding this comment.
Makes sense, thanks for confirming. Agreed it belongs in its own follow-up rather than this PR.
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Hi @drakeo338, thank you for this. The fix itself is right: I checked all five sites by execution and every one now does what #1449 asks. One important thing is still missing before it can merge: the tests do not pin the safe default, and that matters more than usual here because the flag runs code from a model repo.
What is good
- All five sites are threaded correctly and each has its own test that records what actually reaches the consumer (
load_datasetkwargs, lm-evalmodel_args, thepipelinekwargs, both find-lrfrom_pretrainedcalls, the child argv). On cleanorigin/mainyour new file gives 7 failed / 6 passed; on your head 13 passed. - Acceptance criterion 4 holds for real, not only in argv: I ran
_run_distillagainst a localauto_mapdraft base and then ran the parent's argv through the childsoup trainin-process. Withtrc=Truethe child gets past the trust check (it stops later only because my stub directory has no tokenizer). Withtrc=Falseit raises the "requires trust_remote_code=True" error, as it should. - The injection guard survives: with
--trust-remote-codeset, an adapter whosebase_model_name_or_pathisx/y,trust_remote_code=Trueis still refused and lm-eval is never called. Through the real CLI,eval benchmark,eval customandeval autowithout the flag producepretrained=...,peft=...with notrust_remote_codeandtrust_remote_code=Falsefor the generator; with the flag they getTrue. Nothing enables remote code implicitly today. - Refusing the flag on
datasets>=4instead of forwarding something that would be ignored is exactly the "refused or explained, not ignored" the issue asked for, and the explicittrust_remote_code=trust_remote_code(instead of leavingpipelineatNone) is a small tightening too.
What I ran
pytest tests/test_issue1449_trust_remote_code_dropped.py tests/test_issue752_cli_commands_called_as_functions.py tests/test_typer_defaults_do_not_leak.py tests/test_trust_remote_code.py -> 59 passed. Then 22 mutations, one at a time, against your 13 new tests: 15 are killed (every "drop the argument at site X" mutation, for all five sites, including hard-coding False in either find-lr load, either child argv, the pipeline call and the >= 4 gate). Seven survive. Two of those (removing the kwarg from auto() -> benchmark(), and from the training callback) are caught by the repo's typer-default guards. Five survive everything, including those guards (105 tests):
| mutation | result |
|---|---|
eval.py:119 if trust_remote_code: -> if True: (always append trust_remote_code=True) |
13 passed, 105 passed |
eval.py:80 benchmark() option default False -> True |
13 passed, 105 passed |
callback.py:581 training auto-eval calls benchmark(..., trust_remote_code=True) |
13 passed, 105 passed |
callback.py:598 same for custom(...) |
13 passed, 105 passed |
data.py:1536 print the "Remote Code Warning" panel even without the flag |
13 passed, 105 passed |
Each mutation was undone and git diff is empty afterwards.
Please fix before merge
- Pin the default-deny direction (blocking). The issue says "Without the flag, the same tests see
Falseor no flag", and the eval tests only have the with-flag case. The callback rows above are the serious ones: if someone flips either toTrue, everysoup trainwitheval.auto_eval: truewould start running remote code without anyone opting in, and no test would notice. The following 8 tests are verified: they pass on your head, and between them they kill all five rows above plus the dropped-kwarg mutation onauto(). Add totests/test_issue1449_trust_remote_code_dropped.py(needsimport pytestandfrom tests.conftest import strip_ansi):
def _flat(text: str) -> str:
return " ".join(strip_ansi(text).split())
class TestDefaultsStayDeny:
def test_eval_auto_without_flag_trusts_nothing(self, tmp_path, monkeypatch):
from soup_cli.cli import app
seen_model_args, seen_generator_calls = TestEvalAutoTrust()._setup(tmp_path, monkeypatch)
result = runner.invoke(app, ["eval", "auto", "-c", "soup.yaml"])
assert result.exit_code == 0, result.output
assert seen_model_args and "trust_remote_code" not in seen_model_args[0], seen_model_args
assert seen_generator_calls[0][1].get("trust_remote_code") is False, seen_generator_calls
@pytest.mark.parametrize(("flag", "expected"), [([], False), (["--trust-remote-code"], True)])
def test_eval_benchmark_cli_model_args(self, tmp_path, monkeypatch, flag, expected):
from soup_cli.cli import app
seen_model_args, _ = TestEvalAutoTrust()._setup(tmp_path, monkeypatch)
result = runner.invoke(app, ["eval", "benchmark", "-m", "out", "-b", "mmlu", *flag])
assert result.exit_code == 0, result.output
assert ("trust_remote_code=True" in seen_model_args[0]) is expected, seen_model_args
@pytest.mark.parametrize(("flag", "expected"), [([], False), (["--trust-remote-code"], True)])
def test_eval_custom_cli_generator_kwargs(self, tmp_path, monkeypatch, flag, expected):
from soup_cli.cli import app
_, seen_generator_calls = TestEvalAutoTrust()._setup(tmp_path, monkeypatch)
result = runner.invoke(app, ["eval", "custom", "-t", "tasks.jsonl", "-m", "out", *flag])
assert result.exit_code == 0, result.output
assert seen_generator_calls[0][1].get("trust_remote_code") is expected, seen_generator_calls
def test_training_auto_eval_callback_never_opts_in(self, monkeypatch):
import soup_cli.commands.eval as ce
from soup_cli.monitoring.callback import SoupTrainerCallback
calls: dict = {}
monkeypatch.setattr(ce, "benchmark", lambda **kw: calls.__setitem__("benchmark", kw))
monkeypatch.setattr(ce, "custom", lambda **kw: calls.__setitem__("custom", kw))
callback = SoupTrainerCallback.__new__(SoupTrainerCallback)
callback.eval_config = type(
"EvalCfg", (),
{"auto_eval": True, "benchmarks": ["mmlu"], "custom_tasks": "tasks.jsonl"},
)()
callback.output_dir = "out"
callback.run_id = "run-1"
callback._run_auto_eval()
assert calls["benchmark"]["trust_remote_code"] is False, calls
assert calls["custom"]["trust_remote_code"] is False, calls
@pytest.mark.parametrize(("flag", "panel"), [([], False), (["--trust-remote-code"], True)])
def test_data_download_warning_panel_only_with_flag(self, tmp_path, monkeypatch, flag, panel):
import datasets
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
monkeypatch.setattr(datasets, "__version__", "3.6.0")
monkeypatch.setattr(datasets, "load_dataset", lambda *a, **k: iter([{"text": "x"}]))
result = runner.invoke(
app, ["data", "download", "org/ds", "-n", "1", "-o", "o.jsonl", *flag],
)
assert result.exit_code == 0, result.output
assert ("Remote Code Warning" in _flat(result.output)) is panel, _flat(result.output)(The last test covers the "panel only when it is true" acceptance criterion, which no test checks today.)
--trust-remote-codewithdatasetsmissing now raises a traceback (data.py:1228-1235)._datasets_major_versiondoes an unguardedimport datasets, before the existingexcept ImportErroratdata.py:1568. Executed: flag plus missingdatasets-> exit 1 with a rawModuleNotFoundError; without the flag you still get "datasets library not available. Install with: pip install datasets".datasetsis part of the[train]extra, so a base install can hit it. Fix:
try:
import datasets
except ImportError:
return NoneTest (fails on your head, passes with the fix; 14 passed together with your file):
def test_flag_without_datasets_installed_gets_the_install_hint(self, tmp_path, monkeypatch):
from soup_cli.cli import app
monkeypatch.chdir(tmp_path)
monkeypatch.setitem(sys.modules, "datasets", None) # `import datasets` raises ImportError
result = runner.invoke(
app, ["data", "download", "org/ds", "-n", "1", "-o", "o.jsonl", "--trust-remote-code"],
)
assert result.exit_code == 1, (result.output, repr(result.exception))
assert "datasets library not available" in _flat(result.output)- Docs and fragment. This PR adds a public
--trust-remote-codeoption tosoup eval benchmarkandsoup eval customand turnssoup data download --trust-remote-codeinto an error ondatasets>=4. Please add todocs/performance-and-quantization.md(the "Coverage" list under "--trust-remote-codeopt-in"):soup eval benchmark/eval custom,soup train --find-lr,soup draft distill,soup shrink, and one sentence saying the flag is refused fordata downloadwhen the installeddatasetsis 4 or newer. Add a clause about the two new options tochangelog.d/0.75.1/1456.fixed.md.
Optional, not blocking: the eval and find-lr paths forward the raw flag and print no REMOTE CODE WARNING panel, while the docs say other commands print one for non-first-party models. A follow-up could route them through resolve_trust_remote_code.
@AmirF194's question on callback.py:581: good catch, thank you. Reading the code confirms it: SoupConfig has no field for this, so the training auto-eval passes False and, for a custom-code base trained with --trust-remote-code, its load fails and is skipped by the surrounding except. I would keep that out of this PR: it needs a decision about how the flag travels from train() into the callback, which is a different change from the five sites in #1449. In this PR, keep the explicit False and pin it with test_training_auto_eval_callback_never_opts_in above. The follow-up can then flip that one test deliberately.
Merge state: all 11 required checks are green on 12c3e86b, and the branch merges cleanly with current main. One heads-up: #1334 (the --find-lr refusal work) edits the same region of train.py and conflicts with this PR, so whichever lands second needs a rebase.
With item 1 and the ImportError guard in, the gaps I found are closed. Thank you for the careful per-site tests.
|
Thanks for the thorough review. Pushed e8ce9cf: the safe default is now pinned at every site, all five surviving mutations (plus the two dropped-kwarg ones) are killed, the missing-datasets ImportError is guarded, and docs and the fragment are updated. |
|
@drakeo338, a heads-up for your rebase: #1334 merged as |
… from (MakazhanAlpamys#1449) `soup data download`, `soup eval auto` and `soup train --find-lr` accepted `--trust-remote-code` but never passed it to the load that needed it, and `soup draft distill` / `soup shrink` (heal step) resolved the flag but never forwarded it to the child `soup train` subprocess they spawn. Opting in changed nothing on any of the five paths. - `data.py`: `_hf_download_dataset` takes and forwards `trust_remote_code` to `load_dataset`; `download_dataset` passes the resolved flag through and only prints the "Remote Code Warning" panel when it is set. - `eval.py`: `benchmark()` and `custom()` each gain the flag. `benchmark()` appends `,trust_remote_code=True` to the lm-eval `model_args` from the resolved flag only (never from `adapter_config.json`'s `base_model_name_or_path`, which `_reject_lm_eval_injection` still refuses if it carries `,`/`='`). `auto()` forwards its own flag to both. - `eval/custom.py`: `_create_default_generator` forwards the flag to the `pipeline(...)` call. - `train.py`: `_live_lr_sweep_from_config` receives the flag from `train()` via `_run_live_lr_sweep_or_synth` instead of hardcoding `False` on both `from_pretrained` calls. - `draft.py` / `shrink.py`: `_run_distill` / `_run_heal` append `--trust-remote-code` to the child `soup train` argv when the already- resolved `trc` is true, instead of the child refusing and telling the user to re-run with a flag they already passed. `monitoring/callback.py`'s post-training auto-eval calls `benchmark()` and `custom()` as plain functions and has no trust context of its own, so it now passes `trust_remote_code=False` explicitly to keep every typer parameter filled (MakazhanAlpamys#752). Adds `tests/test_issue1449_trust_remote_code_dropped.py`, one test per site plus the injection-guard control, each asserting what the consumer actually receives with and without the flag. Closes MakazhanAlpamys#1449.
…s#1449's tests Review round on MakazhanAlpamys#1449 found one real gap and several weak tests: - `soup data download --trust-remote-code` forwarded the flag to `load_dataset` unconditionally. On datasets>=4, `load_dataset` pops `trust_remote_code` from its kwargs and only logs an error — it no longer supports it at all — so the flag silently did nothing, the same bug this issue was filed for, one layer down. `download_dataset` now checks the installed `datasets` major version and refuses the flag outright when it is >= 4, instead of forwarding it into the void. - The find-lr test only ever exercised `AutoTokenizer.from_pretrained`: both Auto* classes shared one stub that raised immediately, so the tokenizer call raised before `AutoModelForCausalLM.from_pretrained` was ever reached, and the assertions read `seen` as if the model call had been checked too. Split them into two stubs (tokenizer succeeds, model raises) and assert `len(seen) == 2`. Also stopped the test's fake `torch` module from being injected into `sys.modules` before the Auto* attributes were first read: this transformers version's lazy module replaces `sys.modules["transformers"]` the first time certain attributes are resolved, and doing that while a fake `torch` is already registered risks caching a broken class under those names for the rest of the test session. Reading the attributes first (and only injecting the fake when real torch isn't installed) avoids it. The same module-identity quirk turned up while adding the new `_create_default_generator` test below, so it gets the same fix. - Added a unit test for `_create_default_generator` itself, stubbing `transformers.pipeline` directly and asserting the kwarg it receives with and without the flag. - Tightened the lm-eval injection-guard test to assert on "Refusing to evaluate" rather than accepting any non-zero exit code, and added the missing `shrink --heal` negative test (`trc=False` omits the flag from the child `soup train` argv), mirroring the one `draft distill` already had. The changelog fragment's filename should be this fix's own PR number per changelog.d/README.md, not the issue number — it can't be renamed until that PR exists, so it stays as `1449.fixed.md` for now and needs renaming once the PR is opened.
… document coverage Without --trust-remote-code every site must pass trust_remote_code=False: eval benchmark/custom/auto, the training auto-eval callback, the data download warning panel and _create_default_generator's default are now pinned by tests. _datasets_major_version no longer raises when datasets is not installed, so the flag gets the usual install hint instead of a traceback. Docs list the new coverage and the datasets>=4 refusal.
|
Rebased on 636d3ef; the find-lr trust_remote_code forwarding now sits on |
e8ce9cf to
15e886b
Compare
MakazhanAlpamys
left a comment
There was a problem hiding this comment.
Hi @drakeo338, thank you for the quick and complete round. I re-checked every item from my review by execution on the current head (15e886bc, merged with current main), and all of them are closed. This PR now waits only on the required checks.
What is good
- The default-deny tests are exactly what the issue's acceptance criterion asked for, and you went one better with
test_default_generator_defaults_to_false(a caller that omits the kwarg gets an explicitFalse, notNone). That is the test that would catch a silent change of thepipeline(...)default later. - The
ImportErrorguard is in_datasets_major_versionand the test drives the real CLI withdatasetsmade unimportable, so it checks the user-visible message ("datasets library not available") and not just the helper. - The rebase over #1334 was done the careful way: the forwarding sits on
_run_live_lr_sweep/_live_lr_sweep_from_configand no synthetic-curve code came back. - Docs and fragment now say what the PR does, including the two new
--trust-remote-codeoptions oneval benchmark/eval customand thedatasets>=4refusal.
What I ran
- On your head:
pytest tests/test_issue1449_trust_remote_code_dropped.py tests/test_issue752_cli_commands_called_as_functions.py tests/test_typer_defaults_do_not_leak.py tests/test_trust_remote_code.py-> 69 passed. On the tree merged with currentmain(4f51716f, no conflicts) the same four files plustest_eval,test_code_review_critical,test_code_review_high,test_v0401_part_c,test_dataset_hub,test_part_a_wave1,test_v07129,test_v07133-> 423 passed.ruffis clean and the changelog fragment validates. - Mutations: 24, one at a time, against your 23 tests alone (no other guard files). All 24 fail a test. Last round 15 of 22 did. The five that survived everything are now caught (
if True:ateval.py:119-> 2 failed;benchmark()option defaultTrue-> 1 failed; callback passingTrueatcallback.py:581and:598-> 1 failed each; the warning panel without the flag atdata.py:1539-> 1 failed), and so are the new ones I added this time (custom()option defaultTrue, the unguardedimport datasetsatdata.py:1233).git diffis empty afterwards. - On clean
mainyour new file gives 14 failed, 9 passed (the 9 are the without-flag and no-op cases). - Through the real CLI with the installed
datasets5.0.1:soup data download org/ds -n 1 -o o.jsonl --trust-remote-codeprints the refusal and stops before any network call, andsoup eval benchmark --help/soup eval custom --helpboth list--trust-remote-codewith the default-deny text.
Two small notes, neither blocks the merge
soup eval human(eval.py:938-939) andrun_eval(eval/custom.py:338) still build their generator with the defaultFalse, so a custom-code model cannot be evaluated through them even with the flag. They were not among the five sites in #1449, so I left them out of this PR.- As agreed with @AmirF194 on
callback.py:581, threading the flag fromsoup traininto the training auto-eval is a separate change. When it lands it will fliptest_training_auto_eval_callback_never_opts_inon purpose, which is what that test is for.
Merge state: the branch is rebased on 636d3efb and merges cleanly with current main. The required checks on 15e886bc are still queued or running at the time of writing (mlx-smoke and pytorch-smoke are green, nothing has failed), so the merge waits on them.
Thank you for the careful per-site tests and for taking the review points one by one.
|
Merged as |
Display names from `gh api users/<login>`. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Fixes #1449.
What does this PR do?
--trust-remote-codewas accepted bydata download,eval auto,train --find-lr,draft distillandshrinkbut never actually reached the load it was meant to unblock: two commands ignored the flag entirely,train --find-lrhardcodedtrust_remote_code=Falseand silently wrote a synthetic LR curve on refusal, anddraft distill/shrinkresolved the flag but dropped it when spawning the childsoup trainprocess. This change threads the flag through all five paths to the actualfrom_pretrained/load_dataset/pipeline calls and child argv, and also refusestrust_remote_codeoutright ondatasets>=4(which dropped script execution) instead of pretending it still works.Type of change
Checklist
ruff check src/soup_cli/ scripts/ tests/passespytest tests/ -vpasses — not run in full; ran the discriminating subset (tests/test_trust_remote_code.py tests/test_issue1449_trust_remote_code_dropped.py, 37 passed) that covers every path this change touchesREADME.mdand the matching page underdocs/) if needed —docs/performance-and-quantization.mdalready documents the intended behavior; no doc content changed🤖 Generated with Claude Code