Defer some imports by jayzuccarelli · Pull Request #49663 · huggingface/transformers

jayzuccarelli · 2026-10-16T10:56:38Z

What

Loading a config or tokenizer pulled in the generation stack and scikit-learn through module-level imports that only matter once a model generates, uses a watermark, or loads a GGUF file. This moves them to first use:

generation/configuration_utils.py: WatermarkLogitsProcessor / SynthIDTextWatermarkLogitsProcessor now import inside construct_processor (type hint kept under TYPE_CHECKING).
models/auto/auto_factory.py: GenerationMixin imports inside add_generation_mixin_to_remote_model, the only place it is used.
generation/candidate_generator.py: sklearn.metrics.roc_curve imports where the ROC threshold is computed. The is_sklearn_available() check is unchanged.
configuration_utils.py, models/auto/tokenization_auto.py: load_gguf_checkpoint imports in the GGUF branch only.

No behavior change for code that uses these features.

Numbers

AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B") and AutoConfig.for_model("llama"), fresh process each, warm cache, offline, 6 interleaved runs, median wall time:

	main (`683b721`)	this PR
tokenizer	7.6 s, sklearn imported	4.7 s, sklearn not imported
config (llama)	6.8 s, sklearn imported	5.4 s, sklearn not imported

-X importtime on main: auto_factory 2.9 s cumulative, of which candidate_generator -> sklearn.metrics 1.5 s. With this PR auto_factory is 1 ms.

Old, slow box: Intel i5-2430M (2011, 4 threads), Python 3.12.13, torch 2.14.1+cpu, scikit-learn 1.9.1, tokenizers 0.23.2. Absolute times will be much lower on modern hardware; the ratio is what matters.

Not in this PR: torch itself

torch is still imported when available. On current main it comes in through other module-level imports on the config path, e.g. modeling_rope_utils.py:28 and utils/type_validators.py:17-19 (which imports ACT2FN from activations). The type_validators -> activations -> integrations/hub_kernels -> integrations/flash_attention -> generation.continuous_batching chain also still loads generation.logits_process for model configs like llama. Removing those touches a lot more modules, so I'd keep it for a follow-up if you want it. (My earlier comment on the issue said torch stays out; that was inaccurate.)

Tests

tests/utils/test_import_utils.py: new subprocess test that importing AutoConfig/AutoTokenizer and building a PreTrainedConfig leaves sklearn, generation.logits_process, generation.candidate_generator and modeling_gguf_pytorch_utils out of sys.modules, and that AutoConfig.for_model("llama") leaves sklearn, candidate_generator and the gguf loader out. Fails on main, passes here.
First-use tests: both watermark configs still construct their processors, and AssistedCandidateGenerator.update_candidate_strategy still reaches roc_curve and updates the threshold (skipped without sklearn).
Ran: tests/utils/test_import_utils.py 168 passed; tests/generation/test_watermarking.py + test_candidate_generator.py 37 passed; test_logits_process.py -k "atermark or SynthID" 12 passed; test_configuration_utils.py -k "atermark or SynthID" 1 passed; ruff check / ruff format --check (0.14.10) clean; utils/check_dummies.py, check_inits.py, check_import_complexity.py clean.
Not run locally: GGUF loading tests (tests/quantization/ggml, need gguf and large downloads). The GGUF change is only moving the import into the branch that calls it.

cc @ArthurZucker @itazap

AI disclosure: I used Claude Code to write the change and tests; I reviewed the diff and the measurements.

…ading Importing AutoConfig/AutoTokenizer pulled in the generation stack (watermark processors, GenerationMixin) and the GGUF loader, which import torch, plus sklearn.metrics via candidate_generator. Each is only needed on one code path, so import it there. Fixes huggingface#49378

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Also check that AutoConfig.for_model('llama') keeps sklearn, the candidate generator and the gguf loader out of sys.modules, and that the assisted candidate generator still reaches sklearn.metrics.roc_curve on first use. Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-actions · 2026-10-16T10:57:23Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/ — this covers files the PR deletes or renames, not only the ones it edits
A new security issue was detected in the changed Python files (Bandit check) — a high-severity one, shell execution, exec/eval, a URL fetch, or disabled TLS verification
The PR touches a path this repository protects from untrusted PRs, such as the file that decides who reviews it — a maintainer must make that change in a separate PR

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

…orts Co-Authored-By: Claude Opus 5.5 <[email protected]> # Conflicts: # tests/utils/test_import_utils.py

ArthurZucker · 2026-10-22T00:49:14Z

    run(torch.zeros(3))  # a graph break inside the helper would raise here


+def test_auto_classes_do_not_import_generation_or_sklearn():


nope none of the tests are actually usefull they only cover some part.
What we could do is a hook and do a simple code checking there are no imports in globals. We can have this contract on stuff that should not require torch to AutoTokenizer and potentially AutoConfig.

Now if the import can just "release the gil" I think a lot of this would not be a pb.
Anyways The changes are good because minimal effort!

Makes sense, I dropped the tests. Happy to do the import-contract check as a follow-up PR, maybe extending utils/check_import_complexity.py?

HuggingFaceDocBuilderDev · 2026-10-22T00:58:19Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-actions · 2026-10-22T01:09:19Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

Cyrilvallez

Yep, fine with deferring those!

jayzuccarelli and others added 4 commits October 6, 2026 20:32

Test that deferred watermark imports still load on first use

d199661

Merge branch 'main' into lazy-generation-imports

6caad15

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Use the run_command test helper instead of calling subprocess directly

757c4c2

Co-Authored-By: Claude Opus 5.5 <[email protected]>

AshishKumar4 mentioned this pull request Oct 19, 2026

Loading a tokenizer imports torch and scikit-learn when they are installed (about 2 s of a 3 s import) #49378

Closed

ArthurZucker requested a review from Cyrilvallez October 21, 2026 01:40

Merge remote-tracking branch 'upstream/main' into lazy-generation-imp…

6667ca9

…orts Co-Authored-By: Claude Opus 5.5 <[email protected]> # Conflicts: # tests/utils/test_import_utils.py

ArthurZucker reviewed Oct 22, 2026

View reviewed changes

Drop the import tests per review

4a46a42

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Cyrilvallez approved these changes Oct 22, 2026

View reviewed changes

Cyrilvallez changed the title ~~Defer scikit-learn, GenerationMixin, watermark processors and the gguf loader out of config/tokenizer loading~~ Defer some imports Oct 22, 2026

Cyrilvallez merged commit 30a8894 into huggingface:main Oct 22, 2026
111 checks passed

jayzuccarelli deleted the lazy-generation-imports branch October 22, 2026 11:02

stevhliu mentioned this pull request Oct 23, 2026

[docs] Translation workflow #49889

Closed

Navigation Menu

Repository navigation

Defer some imports - #49663

Defer some imports#49663
Cyrilvallez merged 7 commits into
huggingface:mainfrom
jayzuccarelli:lazy-generation-imports

jayzuccarelli commented Oct 16, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

ArthurZucker Oct 22, 2026

Uh oh!

jayzuccarelli Oct 22, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Cyrilvallez left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		run(torch.zeros(3)) # a graph break inside the helper would raise here


		def test_auto_classes_do_not_import_generation_or_sklearn():

Conversation

jayzuccarelli commented Oct 16, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Numbers

Not in this PR: torch itself

Tests

Uh oh!

github-actions Bot commented Oct 16, 2026

CI Security Gate — automatic approval blocked

Uh oh!

ArthurZucker Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

jayzuccarelli Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026

Uh oh!

Cyrilvallez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jayzuccarelli commented Oct 16, 2026 •

edited by github-actions Bot

Loading