Repository navigation
fix: normalize scalar seed from a config file the way the CLI does - #4003
Merged
baberabb merged 4 commits intoAug 21, 2026
Merged
Conversation
`--seed` is run through argparse's converter, which turns `42` into `[42, 42, 42, 42]`, but a seed read from a YAML config is handed to EvaluatorConfig untouched. So the documented `seed: 42` dies with "'int' object is not subscriptable" when run.py unpacks `cfg.seed`, `seed: [42]` raises IndexError, and `seed: 0` is silently read as "don't seed" by the `if cfg.seed` guard, which costs reproducibility. Normalize the value in EvaluatorConfig so both entry points end up with the same shape, and reject a wrong-length list up front rather than letting it fail later with an IndexError.
# Conflicts: # lm_eval/config/evaluate_config.py
baberabb
approved these changes
Aug 21, 2026
baberabb
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the PR, and for the tests. Merged!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4002.
A scalar
seedfrom a YAML--configfile was passed through torun.py:420-423unexpanded, socfg.seed[0]raisedTypeError: 'int' object is not subscriptable. The CLI never hit this because argparse converts--seedthrough_int_or_none_list_arg_type(run.py:313) before the value is stored; the config path had no equivalent step.EvaluatorConfignow normalizesseedin_configure(), mirroring what that converter already accepts:42[42, 42, 42, 42][1, 2, 3][1, 2, 3, 1234], with the same "filled with defaults" warning the CLI emits[1, 2, 3, 4]null[1, 2]ValueErrornaming the field, instead of anIndexErrorfour lines laterThat covers three failures that all came from the missing expansion:
seed: 42crashed, which is the documentedlist/inttype indocs/config_files.mdseed: 0silently disabled seeding entirely, because0is falsy and everyif cfg.seedguard fell toNone. That one is the reason I'd treat this as more than a crash fix: a run pinned to seed 0 for reproducibility was not seeded at all, and said nothing.seed: [42]raisedIndexErrorI also handled a quoted scalar (
seed: "42"), which was quietly the worst of the set:cfg.seed[0]indexed the string and seeded with the character"4".run.pyis untouched. Once the value is normalized,[0, 0, 0, 0]is a truthy list, so the existingif cfg.seedguards behave correctly and there was no reason to change them.Tests
Added to
tests/test_cli_subcommands.py, alongside the existing seed cases:42from a config, asserting all four accessors inrun.pysee42seed: 0actually seeds with0rather than disabling seeding[42]expands,[1,2,3]fills the last from defaults like the CLI doesnullstill disables seeding[1,2]and a non-integer entry raise aValueErrormentioning the field"42"parses to integersThe first four fail on
main. Suite is 77 passed (68 before this change);ruff checkandruff format --checkare clean on both files.