Repository navigation
fix: bind model_index by value in MegatronEngine's param_sync_func callbacks - #4273
Closed
kratos0718 wants to merge 1 commit into
Closed
kratos0718 wants to merge 1 commit into
kratos0718 wants to merge 1 commit into
Conversation
config.param_sync_func = [lambda x: self.optimizer.finish_param_sync(model_index, x)
for model_index in range(len(self.module))]
builds one callback per model chunk, but the lambda captures model_index
by reference, not value - Python closures bind the variable, not its
current value at creation time. By the time any of these callbacks
actually runs, the comprehension has finished and model_index holds its
final value (len(self.module) - 1) for every single one of them. Whichever
chunk's callback fires, it reports the last chunk's index to
finish_param_sync, not its own.
Fixed with the standard default-argument capture: model_index=model_index
evaluates the current value at lambda-creation time, giving each callback
its own bound copy.
get_module_config() itself needs the real megatron-core package to
exercise directly (get_args/get_model_config are imported straight from
megatron.core), so the added test mirrors the exact list-comprehension
shape in isolation with a lightweight optimizer stand-in, rather than
constructing a full MegatronEngine.
kratos0718
added a commit
to kratos0718/codehound
that referenced
this pull request
Sep 17, 2026
CH008 asyncio-run-in-running-loop: asyncio.run() called from inside an async function always raises RuntimeError immediately - scoped to the immediate enclosing function only, since whether a nested sync function is safe to call depends on what thread it runs on. CH009 floating-thread: the thread analog of CH006 - a non-daemon threading.Thread that's started but never joined. Building it surfaced a real false positive in llama_index: six chat-engine stream_chat methods hand the thread off through a *different* object's attribute (chat_response.write_response_to_history_thread = thread), not self, with that object itself returned and the thread joined later once the caller finishes consuming the stream (chat_engine/types.py). Broadened the escape check from self.<attr> to any object's attribute. CH010 loop-closure-capture: a lambda inside a for loop or comprehension that captures the loop variable by reference, so every *stored* instance ends up sharing the loop's final value. Found and fixed a genuine bug this way in HuggingFace's accelerate (huggingface/accelerate#4273) - MegatronEngine.get_module_config's param_sync_func list, where every distributed-training sync callback reported the same (last) model chunk index. Also caught its own false positive first, in marimo: sorted(key=lambda row: row[sort_arg.by]) inside a for loop looks identical at the AST level, but sorted() consumes the lambda immediately within the same iteration. Rewrote the check to require the lambda be directly stored (appended/assigned/returned), not merely passed as a callback argument that gets consumed on the spot. CH011 exception-chaining (except X as e: raise Y(...) with no `from e`) was built, tested, and then not shipped: it fired 1,911 times across the same ~20-framework corpus used to validate the other checks. That volume says the pattern is too common to be a defensible per-finding signal for this tool, not that the check is broken. Documented in FINDINGS.md as a deliberate decision rather than silently dropped. Version bump to 1.2.0.
SunMarc
reviewed
Sep 23, 2026
SunMarc
left a comment
Member
There was a problem hiding this comment.
We don't accept fixes for megatron anymore
1 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
MegatronEngine.get_module_configbuilds one callback per model chunk:The lambda captures
model_indexby reference, not by value - a closure binds the variable, not whatever it happened to hold when the lambda was created. By the time any of these callbacks is actually invoked (later, onceoverlap_param_gatheranddelay_param_gathertraining kicks in), the comprehension has already finished andmodel_indexholds its final value -len(self.module) - 1- for every callback in the list. Whichever model chunk's callback fires, it reports the last chunk's index tofinish_param_sync, not its own.Fixed with the standard default-argument capture (
model_index=model_index), which evaluates the current value at lambda-creation time instead of call time, so each callback gets its own bound copy - this is the same list used one line above forno_sync_func/grad_sync_func, but those bind plain method references (model_chunk.no_sync) rather than a lambda closing over the loop variable, so they aren't affected by this.Found via codehound, a static analyzer I built that flags exactly this pattern (a lambda inside a loop that gets stored rather than called immediately).
Fixes # (no issue filed - a self-contained, narrowly-scoped correctness fix)
Before submitting
get_module_config()itself needs the realmegatron-corepackage to exercise directly (get_args/get_model_configimport straight frommegatron.core, which isn't available in this dev/test environment), so the new test mirrors the exact list-comprehension shape in isolation with a lightweight optimizer stand-in, and asserts each callback reports its own index. Verified it fails against the pre-fix code (all three callbacks report index2) and passes against the fix.Who can review?
Tagging @SunMarc per the "Core parts of the library" note above.