Skip to content
NVIDIA-NeMoPublic

About

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

Topics

Resources

Contributing

Security policy

Stars

1.4k stars

Watchers

6 watching

Forks

Latest commit

ย 

History

367 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

๐Ÿฆ‹ Molt

A Hugging Face-native, minimal, and scalable framework for agentic RL research.

Ray ยท vLLM ยท NeMo AutoModel: the smallest Hugging Face-native stack for fully asynchronous, multimodal, multi-turn agentic RL at 1T-class scale.


Project Page Docs

License PyTorch Hugging Face NeMo AutoModel vLLM Ray RL code Tech Report Ask DeepWiki.com


Project Page ยท Docs ยท Technical Report ยท Video ยท News ยท Architecture ยท Why Molt ยท Installation ยท Quick Start ยท Comparison ยท Documentation ยท Contributing

Package SFT RL Runtime
molt molt.cli.train_sft molt.cli.train_rl_ray vLLM

Molt is agentic-first, Hugging Face-native, minimal, and scalable. The agent is the program: the reward is whatever Python you write in an Env or a ChatAgent, whether a grader, a multi-turn tool loop, a VLM environment, or an LLM judge. Three components do the rest. Ray places the processes and runs the async queues, vLLM generates the rollouts, and NeMo AutoModel + FSDP2 trains a single actor. The model stays an ordinary PyTorch module in the Hugging Face layout, so one checkpoint directory serves both vLLM and the trainer, with no bridge to write and no weights to convert. The whole RL path is ~9.2K lines, and it scales to 1T-class MoE with TP / EP / CP: the entry point that trains a 4B model on one node also trains GLM-5.2 (753B) at --fsdp.ep_size 256, with Adam CPU offload for the largest actors.

๐Ÿ“ฐ News

๐Ÿงฉ Architecture

One loop. Six stages.

Molt architecture. Top: one loop in six stages, where agent workers request tokens from the vLLM engines, return a reward, and send trajectories to a Ray queue, and the trainer takes batches and refits the engines with the new weights. Middle: what each of the four parts provides. Bottom: tensor, expert, and context parallelism, one flag each, and model sizes from 4B to 2.8T parameters on the same trainer.

Read the figure top to bottom: the loop, what each part provides, and how the trainer scales. Ray places three kinds of processes (agent workers, vLLM engines, and one trainer) and carries the queue between them. That is the entire runtime. In asynchronous mode the six stages overlap, so a slow episode delays only itself. The contract is token-first: token ids, log-probs, action ranges, rewards, and multimodal tensors stay aligned from rollout to training. Anything you can compute in Python is a valid reward, including LLM-as-judge calls back through the same vLLM engines that drive rollout.

โœจ Why Molt

What you get Why it matters for research
๐Ÿค– Agentic-first One Gymnasium-aligned API, Env.step() or ChatAgent.run(), covers graders, multi-turn tools, VLM environments, and existing agents that speak the OpenAI or Anthropic API The agent is the program. Iterate on environments in plain Python while the trainer stays untouched
โš™๏ธ Fully asynchronous runtime Ray placement, async rollout queues, vLLM engines, partial rollout, weight sync Rollout, training, and weight sync overlap, so a DeepSeek-V3-class actor stays fed without bespoke infrastructure
๐Ÿ”ฅ Hugging Face-native, PyTorch end to end The model is an ordinary PyTorch module in the Hugging Face layout (NeMo AutoModel + FSDP2), and one checkpoint directory feeds both vLLM and the trainer Models load as released, with no bridge to write and no checkpoint to convert. Changing one means editing plain PyTorch
๐ŸŽฏ Single-actor simplicity One actor and an optional KL reference. The whole RL graph fits on a page Every gradient is explicit, and every loss term is one file away
๐Ÿš€ Frontier-scale MoE AutoModel + FSDP2 with TP / EP / CP and Adam CPU offload, MoE-native. GLM-5.2 (753B) runs at --fsdp.ep_size 256 The 4B and 753B recipes share the entry point, the agent, and the algorithm. Nothing is rewritten between scales
๐Ÿ”— Token-first contract Token ids, log-probs, action ranges, rewards, and multimodal tensors stay aligned Multi-turn, VLM, and tool-call traces share one format end to end
๐Ÿชถ Small, hackable surface ~9.2K lines of RL code in three thin layers. Each mechanism (advantage, loss, token capture, training loop) is one file The function you edit is the function that runs. Read it in an afternoon, or hand it whole to a coding agent

๐Ÿ“ฆ Installation

Clone the repository first. The launch scripts, agents, and recipes live in it, and the container mounts this checkout.

git clone https://github.com/NVIDIA-NeMo/labs-molt.git
cd labs-molt

๐Ÿณ Container (recommended)

dockerfile/Dockerfile bakes the whole CUDA 13 stack (torch 2.14, vLLM, TransformerEngine, flash-attn, mamba, DeepEP, NeMo AutoModel) for A100, H100, H200, B200, and GB200. SFT and RL run in it as-is.

docker pull hijkzzz/molt:latest                                 # or a pinned release: hijkzzz/molt:0.1.10
docker build -f dockerfile/Dockerfile -t hijkzzz/molt:latest .  # to change the CUDA / vLLM / AutoModel pins

Tags from 0.1.9 are multi-arch (amd64 and arm64), so the same pull works on x86 and on Grace-Blackwell hosts. One Dockerfile builds both: publish a tag with docker buildx build --platform linux/amd64,linux/arm64 --push.

๐Ÿ’ป Local install

For development outside the container. It installs the exact git-pinned AutoModel this repository is validated against (setup.py, AUTOMODEL), so R3 routing replay and Muon work out of the box.

pip install -e ".[vllm]"

CUDA 13 is required (torch==2.14.0+cu130). A CUDA 12 environment will not work. With a host driver older than 580, use the container: it ships the CUDA forward-compatibility layer and enables it automatically.

The molt-rl package on PyPI cannot carry the git-pinned AutoModel, so it lags the pin and is not a supported install path. Use the container or the local install.

๐Ÿชถ Optional backend: AutoModel-Slim

The default is upstream NeMo AutoModel, pinned in setup.py. automodel-slim is Molt's own trimmed copy of that pinned commit, with the same package name, import paths, and API. To opt in, see Installation in the guide.

๐Ÿš€ Quick Start

๐Ÿ“˜ SFT

torchrun --standalone --nproc_per_node=8 -m molt.cli.train_sft \
  --model.model_name_or_path /path/to/automodel \
  --data.dataset /path/to/sft.jsonl \
  --data.input_key input \
  --data.output_key output \
  --ckpt.output_dir ./ckpt/sft \
  --fsdp.attn_implementation te

SFT uses the same AutoModel and FSDP2 model-loading path as RL. Example script: examples/scripts/quick_start/sft_qwen3_4b.sh.

๐ŸŽฎ RL

python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/automodel \
  --data.prompt_dataset /path/to/prompts.jsonl \
  --data.input_key input \
  --data.label_key answer \
  --train.agent_path examples/python/agents/math.py \
  --vllm.num_engines 2 \
  --vllm.tensor_parallel_size 2 \
  --rollout.batch_size 128 \
  --train.batch_size 128 \
  --train.micro_batch_size 1 \
  --algo.advantage.estimator reinforce \
  --fsdp.attn_implementation te \
  --ckpt.output_dir ./ckpt/rl

Example script: examples/scripts/quick_start/rl_qwen3_4b.sh. Every launch script, single-node and Slurm, is in examples/scripts/.

Sample groups, KL, asynchronous training, IS correction, and evaluation are each a flag or two away. The Quick start guide lists them, and the RL flags reference has every flag.

๐Ÿ“Š How It Compares

The RL ecosystem optimizes for breadth. Molt optimizes for agentic research velocity at scale: the smallest Hugging Face-native stack that still drives fully asynchronous agentic RL at frontier MoE scale.

๐Ÿฆ‹ Molt OpenRLHF verl slime
Training backend PyTorch / FSDP2 + NeMo AutoModel DeepSpeed ZeRO-3 FSDP / FSDP2 / Megatron Megatron (FSDP exp.)
Hugging Face models native native native on FSDP; bridged on Megatron bridged (rebuilt in Megatron, weights mapped both ways)
Rollout engine vLLM (Ray) vLLM (Ray) vLLM / SGLang / TRT-LLM SGLang only
RL topology actor (+ optional PPO critic) actor + critic + RM actor + critic + RM actor + critic + RM
Reward source agent Python agent / endpoint / RM agent / RM / endpoint rollout fn / RM
Parallelism TP / EP / CP, MoE-native ZeRO-3 / FSDP TP / PP / EP / SP TP / PP / DP / CP / EP
Multimodal VLM RL, multi-turn tool calls VLM RL (v0.10+) Qwen2.5-VL, Kimi-VL geo3k VLM
Config surface CLI flags only CLI + scripts Hydra + YAML CLI + YAML
RL code sizeยน ~9.2K LOC ~7.2K ~62K ~25K
Design center agentic-first research RLHF coverage production breadth Megatron throughput

One framework, one job. Molt takes a checkpoint from SFT to frontier-scale agentic RL on vLLM, with no bridge in between. Read every line that touches your gradients, in plain PyTorch.

ยน How the RL-code line counts were measured

ยน RL code = every Python file the framework's RL path uses โ€” online trainer, rollout, Ray orchestration, experience/advantage/reward/KL/loss, actor/critic/RM inference, plus shared models, utils, parallelism, and kernels the RL training command depends on. Excludes pure SFT, DPO/KTO/IPO trainers, reward-model training, distillation, vendored third-party code, tests, examples, scripts, and docs. Counts code lines only (blank and comment-only lines excluded). Measured by tracing the import graph from each RL entry point (molt.cli.train_rl_ray, openrlhf.cli.train_ppo_ray, verl.trainer.main_ppo); slime loads its Megatron/SGLang backends lazily, so its core slime/ package plus its slime_plugins/ model-zoo (+~4.7K โ€” the in-repo model code its RL path uses, counted on the same basis as molt's models/) are counted, minus SFT/distillation. Molt measured 2026-07-20 on this repo; the others measured 2026-06-16 at each repo's then-latest main HEAD (verl 86e8123, slime 243773c, OpenRLHF b3d2927).

๐Ÿ“– Documentation

This README is the overview. The guide covers each area in depth.

Area Pages
Agents and rewards Agent contract ยท Recipes
Training Algorithms ยท Asynchronous training ยท IS correction ยท Evaluate and debug
Scaling Scaling ยท MoE routing ยท MTP rollout ยท LoRA
Reference RL flags ยท SFT flags ยท molt.agents API

The pages are Markdown files in docs/guide/pages/.

โœ… Validation

Fast local checks:

python -m compileall -q molt examples/python tests
pytest -q

Container checks:

SKIP_BUILD=1 DOCKER_GPUS=all DOCKER_SHM_SIZE=32g \
  bash examples/scripts/docker_run.sh "pytest -q"

๐Ÿ™ Acknowledgement

Molt builds on OpenRLHF and keeps its Python package layout where practical. The architecture is deliberately minimal: a Gymnasium-aligned agent (Env or ChatAgent), a single trainable actor, optional KL reference workers, vLLM generation, and online policy optimization, all on PyTorch and NeMo AutoModel.

๐Ÿ“š Citation

If you use Molt in your research, please cite:

@techreport{hu2026molt,
  title       = {Molt: A Hugging Face-Native, Minimal, and Scalable Framework for Agentic Reinforcement Learning},
  author      = {Jian Hu and Molt Contributors},
  institution = {NVIDIA},
  year        = {2026},
  url         = {https://www.researchgate.net/publication/409325071_Molt_A_Scalable_PyTorch-Native_Training_Framework_for_Agentic_Reinforcement_Learning}
}

๐Ÿค Contributing

External contributions are welcome. See CONTRIBUTING.md. Every commit must be signed off with git commit -s, per the Developer Certificate of Origin (DCO).

๐Ÿ“„ License

Apache License 2.0. Copyright and third-party attributions are in NOTICE and THIRD_PARTY_NOTICES.md.

About

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

Topics

Resources

Contributing

Security policy

Stars

1.4k stars

Watchers

6 watching

Forks

Releases

Packages

Contributors

Languages