chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) by yaoyu-33 · Pull Request #1568 · NVIDIA-NeMo/RL

yaoyu-33 · 2025-11-25T18:35:43Z

What does this PR do ?

update 3rdparty versions

Issues

List issues that this PR closes (syntax):

Closes #1575

Closes #1577

closes #1343 (no longer ooms, but metrics do fail though, but this is an improvement from before)

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

...

Summary by CodeRabbit

Chores
- Updated project submodule dependencies to latest compatible versions.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

github-actions · 2025-11-25T18:36:39Z

✅ Submodule Fast-Forward Check Results

Check based on commit: d98aae4 (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

coderabbitai · 2025-11-25T18:38:49Z

📝 Walkthrough

Walkthrough

Updated two third-party submodule pointers: Megatron-Bridge to commit 1e9a459b43aa1f62ca1356e554d2b0196ebdd546 and Megatron-LM to commit 0d401602bf48046683adfc2542a70613f6e772e6. No functional or behavioral changes.

Changes

Cohort / File(s)	Summary
Megatron-Bridge submodule `3rdparty/Megatron-Bridge-workspace/Megatron-Bridge`	Updated submodule pointer from 8aa287df3ca6833c78733460f0c0f0bcfb79f5de to 1e9a459b43aa1f62ca1356e554d2b0196ebdd546
Megatron-LM submodule `3rdparty/Megatron-LM-workspace/Megatron-LM`	Updated submodule pointer from 76065f17e1e1e2850d1e9009ae5f601007aeeeb3 to 0d401602bf48046683adfc2542a70613f6e772e6

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Suggested labels

mcore

Suggested reviewers

yfw

Pre-merge checks and finishing touches

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Test Results For Major Changes	✅ Passed	PR contains only minor submodule pointer updates with no functional changes; test documentation not required for maintenance changes.
Title check	✅ Passed	The title accurately describes the main change: updating two third-party submodule pointers (Megatron and Megatron-Bridge) with specific dates, directly matching the pull request's core objective.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch update-3rdparty-mcore

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6639a40 and d98aae4.

📒 Files selected for processing (2)

3rdparty/Megatron-Bridge-workspace/Megatron-Bridge (1 hunks)
3rdparty/Megatron-LM-workspace/Megatron-LM (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

!(**/tests/**|**/test_*.py|**/test_*.sh)

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Add the NVIDIA copyright header to all Python files and shell scripts (excluding tests). The header should include the current year

Files:

3rdparty/Megatron-Bridge-workspace/Megatron-Bridge
3rdparty/Megatron-LM-workspace/Megatron-LM

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

GitHub Check: Post automodel integration comment / Comment on PR

🔇 Additional comments (2)

3rdparty/Megatron-Bridge-workspace/Megatron-Bridge (1)

1-1: Verify pre-check items before merging.

The PR template shows unchecked boxes for contributor guidelines, tests, and documentation. Before approving, ensure:

All contributor guidelines have been followed

Tests pass with the updated dependencies

Any necessary documentation has been updated to reflect version changes

3rdparty/Megatron-LM-workspace/Megatron-LM (1)

1-1: Verify test compatibility with updated submodule.

This is a submodule pointer update to Megatron-LM. Ensure that the full test suite passes with the new commit hash.

Please confirm:

Full test suite has been run against the new submodule version.

The updated commit hash 0d401602bf48046683adfc2542a70613f6e772e6 is accessible and stable.

Any breaking changes in the submodule have been addressed (if applicable).

github-actions · 2025-11-25T18:42:28Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 8807048 (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

yfw · 2025-11-26T22:36:41Z

Closes #1575

github-actions · 2025-12-02T21:24:30Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 0dfa133 (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2025-12-03T06:47:34Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 29f8685 (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2025-12-03T07:15:54Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 1176efc (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

github-actions · 2025-12-04T09:12:23Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 95150ad (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

terrykong · 2025-12-10T08:12:04Z

closes #1577 in 81530f3, which is a simpler solution since there was only one test that used 2 gpu

terrykong · 2025-12-10T08:22:10Z

the last remaining mcore test passed, this pr is good to go now

github-actions · 2025-12-10T08:22:39Z

✅ Submodule Fast-Forward Check Results

Check based on commit: f6da561 (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

terrykong · 2025-12-14T08:36:51Z

disabling mcore inference test and summarized current failure here #1633 (cc @shanmugamr1992 )

github-actions · 2025-12-14T08:37:04Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 009f1c0 (PR #1568 from update-3rdparty-mcore)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-LM: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]> Signed-off-by: Brian Yu <[email protected]>

…A-NeMo#1568) Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]>

…A-NeMo#1568) Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]> Signed-off-by: yuanhangs <[email protected]>

Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]>

yaoyu-33 requested a review from a team as a code owner November 25, 2025 18:35

coderabbitai Bot reviewed Nov 25, 2025

View reviewed changes

Comment thread 3rdparty/Megatron-Bridge-workspace/Megatron-Bridge

ahmadki mentioned this pull request Dec 1, 2025

feat: Support qwen3-next, mcore path #1530

Open

1 task

terrykong force-pushed the update-3rdparty-mcore branch from 8807048 to 0dfa133 Compare December 2, 2025 21:23

terrykong requested a review from a team as a code owner December 3, 2025 06:46

shanmugamr1992 requested a review from a team as a code owner December 4, 2025 09:11

terrykong changed the title ~~Update megatron bridge and megatron-core~~ chore: update megatron (10/21/2025) / mbridge (10/28/2025) Dec 5, 2025

yaoyu-33 changed the title ~~chore: update megatron (10/21/2025) / mbridge (10/28/2025)~~ chore: update megatron (11/21/2025) / mbridge (11/28/2025) Dec 5, 2025

yaoyu-33 changed the title ~~chore: update megatron (11/21/2025) / mbridge (11/28/2025)~~ chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) Dec 5, 2025

terrykong requested a review from a team as a code owner December 6, 2025 06:52

terrykong mentioned this pull request Dec 9, 2025

Add mixed precision validation between the model config and optimizer config NVIDIA-NeMo/Megatron-Bridge#1665

Closed

terrykong requested review from a team as code owners December 10, 2025 06:35

terrykong force-pushed the update-3rdparty-mcore branch 2 times, most recently from 8ab85e4 to f6da561 Compare December 10, 2025 08:21

terrykong requested a review from a team as a code owner December 10, 2025 08:21

terrykong added the CI:L1 Run doctests, unit tests, and functional tests label Dec 10, 2025

terrykong temporarily deployed to nemo-ci December 10, 2025 08:22 — with GitHub Actions Inactive

terrykong previously approved these changes Dec 10, 2025

View reviewed changes

terrykong temporarily deployed to nemo-ci December 14, 2025 08:36 — with GitHub Actions Inactive

terrykong approved these changes Dec 14, 2025

View reviewed changes

terrykong temporarily deployed to nemo-ci December 14, 2025 08:58 — with GitHub Actions Inactive

terrykong temporarily deployed to nemo-ci December 14, 2025 10:45 — with GitHub Actions Inactive

terrykong merged commit 5d04b36 into main Dec 14, 2025
40 of 41 checks passed

terrykong deleted the update-3rdparty-mcore branch December 14, 2025 12:02

bxyu-nvidia pushed a commit that referenced this pull request Dec 16, 2025

chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) (#1568)

13cf814

Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]> Signed-off-by: Brian Yu <[email protected]>

This was referenced Dec 28, 2025

fix: on GB200 use single-thread checkpoint save to avoid Cpu OOM #1703

Merged

Add FP8 support to nano-v3 branch #1704

Open

coderabbitai Bot mentioned this pull request Jan 5, 2026

cp: fix: on GB200 use single-thread checkpoint save to avoid Cpu OOM (1703) into r0.5.0 #1714

Merged

DeL-TaiseiOzaki pushed a commit to DeL-TaiseiOzaki/RL that referenced this pull request Jan 8, 2026

chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) (NVIDI…

e11a493

…A-NeMo#1568) Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]>

This was referenced Feb 4, 2026

chore: Update Megatron submodule pins #1787

Closed

chore: bump mcore and mbridge #1902

Merged

feat: Megatron LoRA GRPO w/ Weight Merging #1889

Merged

This was referenced Feb 24, 2026

chore: Switch to mcore upstream main #1990

Merged

chore: bumpup Megatron-Bridge submodule to main #2039

Merged

This was referenced Mar 5, 2026

fix: update Megatron-LM submodule to bfd160ba to fix checkpoint SIGSEGV #2070

Closed

fix: bump Megatron-LM to fix checkpoint save segfault (fork → spawn) #2084

Closed

fix: bump Megatron-LM submodule to include upstream checkpoint segfault fix #2085

Closed

seonjinn pushed a commit that referenced this pull request Mar 8, 2026

chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) (#1568)

6b0e91a

Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]>

seonjinn pushed a commit that referenced this pull request Mar 8, 2026

chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) (#1568)

a786c46

Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]>

seonjinn pushed a commit that referenced this pull request Mar 9, 2026

chore: update megatron dev (11/21/2025) / mbridge (11/28/2025) (#1568)

716569b

Signed-off-by: Terry Kong <[email protected]> Co-authored-by: Terry Kong <[email protected]>

This was referenced May 5, 2026

Update tools/launch to take into account number of gpus to avoid reaper #1577

Closed

Qwen2.5-vl crash when using mcore backend #1575

Closed

Dtensor memory regression with TP and long sequence length present in transformers>=4.54,<4.56 #1343

Closed

Conversation

yaoyu-33 commented Nov 25, 2025 • edited by terrykong Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Nov 25, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

coderabbitai Bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Pre-merge checks and finishing touches

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Nov 25, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

yfw commented Nov 26, 2025

Uh oh!

github-actions Bot commented Dec 2, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Dec 3, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Dec 3, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Dec 4, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

terrykong commented Dec 10, 2025

Uh oh!

terrykong commented Dec 10, 2025

Uh oh!

github-actions Bot commented Dec 10, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

terrykong commented Dec 14, 2025

Uh oh!

github-actions Bot commented Dec 14, 2025

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yaoyu-33 commented Nov 25, 2025 •

edited by terrykong

Loading

coderabbitai Bot commented Nov 25, 2025 •

edited

Loading