Skip to content

fix: bind model_index by value in MegatronEngine's param_sync_func callbacks - #4273

Closed
kratos0718 wants to merge 1 commit into
huggingface:mainfrom
kratos0718:fix/closure-capture-param-sync-func-megatron
Closed

kratos0718 wants to merge 1 commit into
huggingface:mainfrom
kratos0718:fix/closure-capture-param-sync-func-megatron

Conversation

@kratos0718

Copy link
Copy Markdown

What does this PR do?

MegatronEngine.get_module_config builds one callback per model chunk:

config.param_sync_func = [
    lambda x: self.optimizer.finish_param_sync(model_index, x)
    for model_index in range(len(self.module))
]

The lambda captures model_index by reference, not by value - a closure binds the variable, not whatever it happened to hold when the lambda was created. By the time any of these callbacks is actually invoked (later, once overlap_param_gather and delay_param_gather training kicks in), the comprehension has already finished and model_index holds its final value - len(self.module) - 1 - for every callback in the list. Whichever model chunk's callback fires, it reports the last chunk's index to finish_param_sync, not its own.

Fixed with the standard default-argument capture (model_index=model_index), which evaluates the current value at lambda-creation time instead of call time, so each callback gets its own bound copy - this is the same list used one line above for no_sync_func/grad_sync_func, but those bind plain method references (model_chunk.no_sync) rather than a lambda closing over the loop variable, so they aren't affected by this.

Found via codehound, a static analyzer I built that flags exactly this pattern (a lambda inside a loop that gets stored rather than called immediately).

Fixes # (no issue filed - a self-contained, narrowly-scoped correctness fix)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? (not applicable - internal training-loop wiring, no public API change)
  • Did you write any new necessary tests?

get_module_config() itself needs the real megatron-core package to exercise directly (get_args/get_model_config import straight from megatron.core, which isn't available in this dev/test environment), so the new test mirrors the exact list-comprehension shape in isolation with a lightweight optimizer stand-in, and asserts each callback reports its own index. Verified it fails against the pre-fix code (all three callbacks report index 2) and passes against the fix.

Who can review?

Tagging @SunMarc per the "Core parts of the library" note above.

config.param_sync_func = [lambda x: self.optimizer.finish_param_sync(model_index, x)
                           for model_index in range(len(self.module))]

builds one callback per model chunk, but the lambda captures model_index
by reference, not value - Python closures bind the variable, not its
current value at creation time. By the time any of these callbacks
actually runs, the comprehension has finished and model_index holds its
final value (len(self.module) - 1) for every single one of them. Whichever
chunk's callback fires, it reports the last chunk's index to
finish_param_sync, not its own.

Fixed with the standard default-argument capture: model_index=model_index
evaluates the current value at lambda-creation time, giving each callback
its own bound copy.

get_module_config() itself needs the real megatron-core package to
exercise directly (get_args/get_model_config are imported straight from
megatron.core), so the added test mirrors the exact list-comprehension
shape in isolation with a lightweight optimizer stand-in, rather than
constructing a full MegatronEngine.
kratos0718 added a commit to kratos0718/codehound that referenced this pull request Sep 17, 2026
CH008 asyncio-run-in-running-loop: asyncio.run() called from inside an
async function always raises RuntimeError immediately - scoped to the
immediate enclosing function only, since whether a nested sync function
is safe to call depends on what thread it runs on.

CH009 floating-thread: the thread analog of CH006 - a non-daemon
threading.Thread that's started but never joined. Building it surfaced
a real false positive in llama_index: six chat-engine stream_chat
methods hand the thread off through a *different* object's attribute
(chat_response.write_response_to_history_thread = thread), not self,
with that object itself returned and the thread joined later once the
caller finishes consuming the stream (chat_engine/types.py). Broadened
the escape check from self.<attr> to any object's attribute.

CH010 loop-closure-capture: a lambda inside a for loop or comprehension
that captures the loop variable by reference, so every *stored* instance
ends up sharing the loop's final value. Found and fixed a genuine bug
this way in HuggingFace's accelerate (huggingface/accelerate#4273) -
MegatronEngine.get_module_config's param_sync_func list, where every
distributed-training sync callback reported the same (last) model chunk
index. Also caught its own false positive first, in marimo:
sorted(key=lambda row: row[sort_arg.by]) inside a for loop looks
identical at the AST level, but sorted() consumes the lambda immediately
within the same iteration. Rewrote the check to require the lambda be
directly stored (appended/assigned/returned), not merely passed as a
callback argument that gets consumed on the spot.

CH011 exception-chaining (except X as e: raise Y(...) with no `from e`)
was built, tested, and then not shipped: it fired 1,911 times across the
same ~20-framework corpus used to validate the other checks. That volume
says the pattern is too common to be a defensible per-finding signal for
this tool, not that the check is broken. Documented in FINDINGS.md as a
deliberate decision rather than silently dropped.

Version bump to 1.2.0.

@SunMarc SunMarc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't accept fixes for megatron anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants