Repository navigation
ml_model_loaded is true for a corrupt checkpoint #191
Description
Guard.ml_model_loaded returns True when active_model points at a checkpoint whose config.json does not parse.
Two tests in tests/unit/ml/test_model_store.py assert False and fail on dev:
test_guard_survives_corrupt_checkpoint and test_guard_require_ml_corrupt_checkpoint_raises_model_error.
Repro: point a model entry at a directory containing a config.json holding the text not-json, build a Guard with require_ml=false, read ml_model_loaded.
Not a cache artefact. It reproduces with HF_HOME and XDG_CACHE_HOME redirected to an empty directory and UNPLUG_MODEL_PATH unset.
ml_model_loaded reads self._ml_provider.loaded (guard.py:902), so the provider appears to set loaded before the checkpoint is validated. Scanning then runs with a model that is not there, and the failure is silent rather than closed.
Activity
AaliyaKhan25 commentedon Sep 27, 2026
I'd like to work on this. Traced the full path through guard.py, ml/models.py, ml/span_model.py, ml/providers.py, core/runtime/model_runtime.py, and ml/registry.py, then read the actual failing test.
Based on the code as written, test_guard_survives_corrupt_checkpoint's fixture (a directory with only a config.json containing "not-json", no weight/tokenizer files) should already fail loudly enough to make the test pass: is_valid_checkpoint() correctly flags it as invalid and prepare_active_model_spec leaves the bad path in place rather than substituting the cache; load_active_model_provider only checks Path(path).is_dir() (true) and proceeds to construct TransformersSpanProvider; SpanInferenceModel._load_unlocked() should then hit AutoTokenizer.from_pretrained(...) against a directory with no tokenizer files and local_files_only=True, which I'd expect to raise — and the fallback retry (use_fast=False) isn't wrapped in another except, so it should propagate up through provider.load() (which only sets _loaded = True after success) to Guard.init's exception handler, leaving ml_model_loaded correctly False.
Since the test fails in CI, my read is that AutoTokenizer.from_pretrained (or the use_fast=False fallback) isn't actually raising the way the code assumes — likely falling back to some default/generic tokenizer instead of erroring on a checkpoint with no tokenizer files at all, which would let execution reach AutoConfig.from_pretrained next, or skip the intended failure entirely. I want to confirm this against the real transformers library behavior before proposing a fix . Working on getting a local ML environment set up to run the test directly and confirm.
chiruu12 commentedon Oct 6, 2026
Yours. Confirming the tokenizer behaviour against real transformers before proposing a fix is the right call, that is the part the issue does not settle.
Two things I want in the PR.
ml_model_loadedgoes false on any load failure, and the underlying checkpoint error reaches the caller instead of only a log line. Plus a test that feeds a truncated or corrupt checkpoint and asserts both.Holding #193 until this one has a PR up.
AaliyaKhan25 commentedon Oct 11, 2026
#207 solves this