Skip to content

Fix OptimizerRegistry.create_optimizer crash on layer_decay without an explicit min scale - #2754

Merged
rwightman merged 1 commit into
huggingface:mainfrom
winklemad:fix-registry-layer-decay-min-scale
Aug 26, 2026
Merged

rwightman merged 1 commit into
huggingface:mainfrom
winklemad:fix-registry-layer-decay-min-scale

Conversation

@winklemad

Copy link
Copy Markdown
Contributor

What

Minor, but a hard crash: calling the registry's create_optimizer with layer_decay set and the min scale left at its default raises a TypeError.

from timm.optim._optim_factory import default_registry
model = torch.nn.Sequential(torch.nn.Linear(10, 5), torch.nn.ReLU(), torch.nn.Linear(5, 2))

default_registry.create_optimizer(model, 'adamw', lr=1e-3, weight_decay=0.05, layer_decay=0.75)
# TypeError: '>' not supported between instances of 'float' and 'NoneType'

The common entry point create_optimizer_v2 is not affected — it passes 0.0.

Why

OptimizerRegistry.create_optimizer defaults layer_decay_min_scale to None and forwards it unguarded as min_scale into param_groups_layer_decay, whose scale line is:

layer_scales = list(max(min_scale, layer_decay ** (layer_max - i)) for i in range(num_layers))

max(None, <float>) raises in Python 3. Both create_optimizer_v2 (layer_decay_min_scale: float = 0.0) and param_groups_layer_decay itself (min_scale: float = 0.) already default this to 0.0; only the registry method defaulted it to None.

Fix

Default layer_decay_min_scale to 0.0 in the registry method, matching its siblings. 0.0 is a no-op clamp (the scales are already >= 0), so this only removes the crash — it doesn't change behavior for callers that passed a value.

layer_decay_no_opt_scale keeps its Optional[float] = None default on purpose: it's consumed behind a truthiness guard (if no_opt_scale and ...), so None is safe there. Verified that the same call with only no_opt_scale still crashes on min_scale, confirming min_scale is the sole culprit.

Tests

Added test_registry_create_optimizer_layer_decay_default_min_scale, which builds layer-decay param groups through the registry path with the min scale omitted. It fails on main (the TypeError) and passes with the fix; the related layer-decay/param-group tests stay green.

…t min_scale

The registry create_optimizer defaulted layer_decay_min_scale to None and forwarded
it unguarded as min_scale into param_groups_layer_decay, whose layer_scales line does
max(min_scale, layer_decay ** ...). With layer_decay set and min_scale left default,
that evaluates max(None, float) and raises TypeError. The sibling create_optimizer_v2
and param_groups_layer_decay itself both default this to 0.0; match that.

layer_decay_no_opt_scale keeps its Optional[None] default because it is used behind a
truthiness guard (if no_opt_scale and ...), so None is safe there.
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@rwightman
rwightman merged commit a4075cd into huggingface:main Aug 26, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants