Skip to content

fix: normalize scalar seed from a config file the way the CLI does - #4003

Merged
baberabb merged 4 commits into
EleutherAI:mainfrom
winklemad:fix-config-seed-normalization
Aug 21, 2026
Merged

baberabb merged 4 commits into
EleutherAI:mainfrom
winklemad:fix-config-seed-normalization

Conversation

@winklemad

Copy link
Copy Markdown
Contributor

Fixes #4002.

A scalar seed from a YAML --config file was passed through to run.py:420-423 unexpanded, so cfg.seed[0] raised TypeError: 'int' object is not subscriptable. The CLI never hit this because argparse converts --seed through _int_or_none_list_arg_type (run.py:313) before the value is stored; the config path had no equivalent step.

EvaluatorConfig now normalizes seed in _configure(), mirroring what that converter already accepts:

input result
42 [42, 42, 42, 42]
[1, 2, 3] [1, 2, 3, 1234], with the same "filled with defaults" warning the CLI emits
[1, 2, 3, 4] unchanged
null unchanged, still means "do not seed"
[1, 2] ValueError naming the field, instead of an IndexError four lines later

That covers three failures that all came from the missing expansion:

  • seed: 42 crashed, which is the documented list/int type in docs/config_files.md
  • seed: 0 silently disabled seeding entirely, because 0 is falsy and every if cfg.seed guard fell to None. That one is the reason I'd treat this as more than a crash fix: a run pinned to seed 0 for reproducibility was not seeded at all, and said nothing.
  • seed: [42] raised IndexError

I also handled a quoted scalar (seed: "42"), which was quietly the worst of the set: cfg.seed[0] indexed the string and seeded with the character "4".

run.py is untouched. Once the value is normalized, [0, 0, 0, 0] is a truthy list, so the existing if cfg.seed guards behave correctly and there was no reason to change them.

Tests

Added to tests/test_cli_subcommands.py, alongside the existing seed cases:

  • scalar 42 from a config, asserting all four accessors in run.py see 42
  • seed: 0 actually seeds with 0 rather than disabling seeding
  • [42] expands, [1,2,3] fills the last from defaults like the CLI does
  • null still disables seeding
  • [1,2] and a non-integer entry raise a ValueError mentioning the field
  • quoted "42" parses to integers

The first four fail on main. Suite is 77 passed (68 before this change); ruff check and ruff format --check are clean on both files.

`--seed` is run through argparse's converter, which turns `42` into
`[42, 42, 42, 42]`, but a seed read from a YAML config is handed to
EvaluatorConfig untouched. So the documented `seed: 42` dies with
"'int' object is not subscriptable" when run.py unpacks `cfg.seed`,
`seed: [42]` raises IndexError, and `seed: 0` is silently read as
"don't seed" by the `if cfg.seed` guard, which costs reproducibility.

Normalize the value in EvaluatorConfig so both entry points end up with
the same shape, and reject a wrong-length list up front rather than
letting it fail later with an IndexError.
@CLAassistant

CLAassistant commented Aug 15, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@baberabb baberabb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, and for the tests. Merged!

@baberabb
baberabb merged commit 9067fdb into EleutherAI:main Aug 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scalar seed in a YAML config crashes with TypeError; seed: 0 silently disables seeding

3 participants