GitHub - ROCm/mori: Modular RDMA Interface

Documentation | Installation | Quickstart | API reference

News

[2026/10] 🔥 MoRI UMBP Empowers AMD Instinct™ MI355X GPU to Demonstrate Token-per-Dollar TCO Leadership on the Public AgentX Leaderboard (blog).
[2026/07] 🔥 MORI powers Kimi's agentic AI serving solution with UMBP, co-designed by AMD and Moonshot AI for AMD Instinct™ GPUs (blog).
[2026/05] 🔥 MORI powers SGLang on AMD Instinct™ MI355X achieves competitive TCO for large-scale DeepSeek disaggregated inference (blog).
[2026/05] 🔥 MORI becomes the primary EP communication library for AMD platforms in Alibaba RTP-LLM (MORI-EP PR).
[2026/05] MORI's SDMA-based AllGather collective is integrated into DeepSpeed for ZeRO-3 optimization on AMD GPUs, delivering up to 10% end-to-end training speedup by offloading AllGather traffic to dedicated SDMA copy engines (example, post).
[2026/04] 🔥 Tencent OpenUCL adopts the Mori ecosystem, using Mori's EP-style dispatch/combine pattern in AMD GPU deployments and leveraging MORI-SHMEM for GPU-initiated communication.
[2026/03] 🔥 MORI-SHMEM powers ByteDance Triton-distributed EP dispatch/combine kernels as the backend, delivering seamless integration and high performance on AMD GPUs (EP Kernels, MORI-SHMEM Integration).
[2026/02] 🔥 MORI powers AMD's WideEP and PD disaggregation in SemiAnalysis InferenceX v2 benchmark (PR, InferenceX, blog).
[2026/01] 🔥 MORI-EP and MORI-IO integrated into SGLang and vLLM for MoE Expert Parallelism and PD Disaggregation on AMD GPUs (sglang & MORI-EP, sglang & MORI-IO, vllm & MORI-EP, vllm & MORI-IO).
[2025/12] MORI adds support for AMD's AINIC (Pollara) with SOTA performance (AINIC & MORI-EP, AINIC & MORI-IO).
[2025/09] MORI-EP now seamlessly scales to 64 GPUs with SOTA performance (multiple optimizations, multi-QP support, low-latency kernel).
[2025/09] MORI adds Broadcom BNXT (Thor2) IBGDA support (PR).

Introduction

MORI (Modular RDMA Interface) is a bottom-up, modular, and composable framework for building high-performance communication applications with a strong focus on RDMA + GPU integration. Inspired by the role of MLIR in compiler infrastructure, MORI provides reusable and extensible building blocks that make it easier for developers to adopt advanced techniques such as IBGDA (Infiniband GPUDirect Async) and GDS (GPUDirect Storage).

To help developers get started quickly, MORI also includes a suite of optimized libraries—MORI-EP (MoE dispatch & combine kernels, in two generations: EPv1 on MORI-SHMEM and EPv2 on MORI-CCO), MORI-IO (p2p communication for KVCache transfer), MORI-CCL (collective communication), MORI-CCO (Next-Generation Collective Communication Object for GPUs) and MORI-FUSED (GEMMs fused with collectives, built on MORI-CCO)—that deliver out-of-the-box performance, with support for AMD Pensando DSC, Broadcom Thor2, and NVIDIA Mellanox ConnectX-7 NICs.

Features summary

Applications
- MORI-EP: intra and inter-node dispatch/combine kernels with SOTA performance. Two generations:
  - EPv1: built on MORI-SHMEM.
  - EPv2: built on MORI-CCO. Designed natively for gfx1250 and rack-scale EP.
- MORI-FUSED: GEMMs fused with the all-reduce, reduce-scatter, all-to-all or all-gather next to them, overlapping the transfer with the GEMM over the SDMA copy engines
- MORI-IO: point-to-point communication library with ultra-low overhead
- MORI-CCL: lightweight and flexible collective communication library designed for highly customized use cases such as latency-sensitive or resource-constrained environment
- MORI-UMBP: unified memory & bandwidth pool with tiered storage and distributed key-value access for scalable memory management
- MORI-CCO: Next-Generation Collective Communication Object for GPUs. Natively designed for compute-communication fusion, with symmetric memory support.
Framework
- High-performance building blocks for IBGDA / P2P and more
- Modular & composable components for developing communication applications, such as transport management, topology detection and etc.
- Open-Shmem-style APIs
- C++ and Python level APIs

Documentation

📖 https://rocm.github.io/mori/ — the full documentation, with navigation and search. It is built from docs/ on every push to main, so it describes the development tree; each page's title names the commit.

Topic	Description	Docs
Getting started	Installation, fabric checks and first steps	Installation · Quickstart
MORI-EP (EPv1, EPv2)	EPv1 (built on SHMEM) and EPv2 (built on CCO) dispatch/combine: API, kernel types and backends, configuration, usage examples, how to choose	EP Guide · Benchmark · RDMA bandwidth
MORI-FUSED	GEMMs fused with the all-reduce, reduce-scatter, all-to-all or all-gather next to them, built on MORI-CCO	Guide
MORI-CCO	Next-generation collective communication object: communicator, symmetric-memory windows, LSA / GDA / SDMA device APIs	CCO Guide
MORI-IO	P2P communication concepts, engine/backend/session design	IO Guide · Benchmark
MORI-UMBP	Unified memory & bandwidth pool: tiered storage, distributed KV access, transfer simulation	Single-Node Smoke Test · Simulation · Calibration · PD Benchmark · Tier-Management Benchmark
MORI-VIZ	Warp-level kernel profiler with Perfetto integration	Profiler

Benchmarks

MORI-EP

Benchmark on DeepSeek V3 model configurations:

Bandwidth (4096 tokens, 7168 hidden, top-8 experts, FP8 dispatch + BF16 combine)

Hardware	Kernels	Dispatch XGMI	Dispatch RDMA	Combine XGMI	Combine RDMA
MI300X + CX7	EP8	307 GB/s	x	330 GB/s	x
	EP16-V1	171 GB/s	52 GB/s	219 GB/s	67 GB/s
	EP32-V1	103 GB/s*	57 GB/s*	91 GB/s*	50 GB/s*
MI355X + AINIC	EP8	345 GB/s	x	420 GB/s	x
	EP16-V1	179 GB/s	54 GB/s	234 GB/s	71 GB/s
	EP32-V1	85 GB/s	46 GB/s	110 GB/s	61 GB/s

Latency (128 tokens, 7168 hidden, top-8 experts, FP8 dispatch + BF16 combine)

Hardware	Kernels	Dispatch Latency	Dispatch BW	Combine Latency	Combine BW
MI300X + CX7	EP8	35 us	134 GB/s	47 us	204 GB/s
	EP16-V1-LL	76 us	96 GB/s	122 us	121 GB/s
	EP32-V1-LL	157 us*	48 GB/s*	280 us*	55 GB/s*
MI355X + AINIC	EP8	31 us	142 GB/s	36 us	276 GB/s
	EP16-V1-LL	84 us	87 GB/s	108 us	139 GB/s
	EP32-V1-LL	152 us	45 GB/s	187 us	76 GB/s

* Stale data from previous kernel version; updated numbers pending re-benchmarking.

MORI-IO

NOTE: This is the preview version of MORI-IO benchmark performance.

GPU Direct RDMA READ, pairwise, 128 consecutive transfers, 1 GPU, MI300X + Thor2:

+--------------------------------------------------------------------------------------------------------+
|                                            Initiator Rank 0                                            |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
| MsgSize (B) | BatchSize | TotalSize (MB) | Max BW (GB/s) | Avg Bw (GB/s) | Min Lat (us) | Avg Lat (us) |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+
|      8      |    128    |      0.00      |      0.03     |      0.03     |    33.38     |    36.33     |
|      16     |    128    |      0.00      |      0.06     |      0.06     |    34.09     |    36.35     |
|      32     |    128    |      0.00      |      0.12     |      0.11     |    34.57     |    36.33     |
|      64     |    128    |      0.01      |      0.24     |      0.23     |    33.62     |    36.33     |
|     128     |    128    |      0.02      |      0.49     |      0.45     |    33.62     |    36.49     |
|     256     |    128    |      0.03      |      0.94     |      0.89     |    34.81     |    36.99     |
|     512     |    128    |      0.07      |      1.86     |      1.77     |    35.29     |    37.01     |
|     1024    |    128    |      0.13      |      3.84     |      3.53     |    34.09     |    37.09     |
|     2048    |    128    |      0.26      |      7.33     |      6.96     |    35.76     |    37.65     |
|     4096    |    128    |      0.52      |     12.94     |     12.46     |    40.53     |    42.09     |
|     8192    |    128    |      1.05      |     20.75     |     20.12     |    50.54     |    52.11     |
|    16384    |    128    |      2.10      |     29.03     |     28.33     |    72.24     |    74.02     |
|    32768    |    128    |      4.19      |     36.50     |     35.91     |    114.92    |    116.81    |
|    65536    |    128    |      8.39      |     41.74     |     41.39     |    200.99    |    202.70    |
|    131072   |    128    |     16.78      |     45.14     |     44.85     |    371.69    |    374.10    |
|    262144   |    128    |     33.55      |     46.93     |     46.76     |    715.02    |    717.56    |
|    524288   |    128    |     67.11      |     47.94     |     47.81     |   1399.99    |   1403.64    |
|   1048576   |    128    |     134.22     |     48.44     |     48.32     |   2770.90    |   2777.76    |
+-------------+-----------+----------------+---------------+---------------+--------------+--------------+

Hardware Support Matrix

GPU

	MORI-EP	MORI-IO	MORI-SHMEM	MORI-CCO
MI308X	✅	✅	✅	✅
MI300X	✅	✅	✅	✅
MI325X	✅	✅	✅	✅
MI355X	✅	✅	✅	✅
MI450X	🚧	🚧	🚧	🚧

NIC

	MORI-EP	MORI-IO	MORI-SHMEM	MORI-CCO
Pollara	✅	✅	✅	✅
CX7	✅	✅	✅	✅
Thor2	✅	✅	✅	✅
Volcano	🚧	🚧	🚧	🚧

✅ Supported 🚧 Under Development

Installation

Prerequisites

ROCm >= 6.4 (hipcc needed at runtime for JIT kernel compilation, not at install time)
- On RHEL/Rocky/AlmaLinux also install gcc-c++: that JIT needs the C++ standard headers, which ROCm's RPMs do not pull in. Without them every kernel compile fails with Could not find standard C++ header 'cmath'.
System packages (required for pip install; not bundled in wheels). On Debian/Ubuntu install at least:
- libibverbs-dev, ibverbs-utils See docker/Dockerfile.dev for the full apt list used in CI/dev images.
Optional: libopenmpi-dev, openmpi-bin — only needed when building C++ examples (BUILD_EXAMPLES=ON) or enabling MPI bootstrap (MORI_WITH_MPI=ON)

Or build docker image with:

cd mori && docker build -t rocm/mori:dev -f docker/Dockerfile.dev .

IBGDA NIC support (optional, for GPU-direct RDMA — auto-detected, no manual configuration needed):

NIC	User library
AMD Pollara (AINIC)	`libionic.so`
Mellanox ConnectX	`libmlx5.so` (typically pre-installed)
Broadcom Thor2	`libbnxt_re.so`

Note: IBGDA requires vendor-specific DV (Direct Verbs) libraries. Mellanox libmlx5 is typically pre-installed with the kernel OFED stack. For Thor2 and Pollara, install the corresponding userspace library from your NIC vendor.

Recommended NIC firmware/driver versions (check with mori check or tools/env_check.sh):

Vendor	Recommendation
AMD Pollara (AINIC)	`>= 1.117.5-a-45` (`1.117.1` major lacks IBGDA support)
Broadcom (Thor2)	`237.1.137.x` / `235.2.86.x`; known bad: `231.x`, `232.x`
Mellanox (ConnectX, mlx5)	No known minimum; tested on `ConnectX-7`

Validate the environment

Before a cross-node run, check the fabric:

mori check <peer_ip>          # or: tools/env_check.sh <peer_ip>

Seven steps — kernel PCI P2PDMA support (CONFIG_PCI_P2PDMA, read from /proc/kallsyms; warns only), NIC firmware and driver, QoS/PFC, DCQCN, then intra-node, inter-node bandwidth and latency meshes over every NIC pair, on host memory and on GPU memory. Exits non-zero if any check failed, and reports the fabric topology, which decides what can run on it.

Install

MoRI can be installed in three ways: from PyPI (stable), nightly pre-built wheels (latest dev), or from source.

From PyPI (stable release)

pip install amd_mori

Nightly (pre-built, tested daily)

pip install --pre amd-mori-nightly

Deprecated: the GitHub Pages nightly index (https://rocm.github.io/mori/nightly/) is being retired. Please install nightly builds from PyPI (amd-mori-nightly) as shown above.

Note: amd-mori and amd-mori-nightly both provide the mori Python module. Do not install both at the same time — uninstall one before installing the other.

From source

git clone --recursive https://github.com/ROCm/mori.git
cd mori
pip install .

For a venv build, add --no-build-isolation — and install the build dependencies first, because that flag is exactly what stops pip from reading them out of pyproject.toml:

pip install -r requirements-build.txt
pip install . --no-build-isolation

Skipping the first line leaves the build without Cython, and it only warns before dropping the mori.cco.cco extension, taking the flydsl EP backend with it — on an install that otherwise reports success. python tools/verify_install.py checks for exactly that.

No hipcc needed at install time — host code compiles with a standard C++ compiler. GPU kernels are JIT-compiled on first use and cached to ~/.mori/jit/. If a GPU is detected during install, kernel precompilation starts automatically in the background.

To manually precompile all kernels (e.g. in a Docker image build):

MORI_PRECOMPILE=1 python -c "import mori"

Verify installation

python -c "import mori; print(mori.__version__)"

Testing

Test MORI-EP (dispatch / combine)

EPv2 (MORI-CCO)

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH
python -c "import mori; print(mori.__file__)"

# Test correctness (8 GPUs, gfx942 or gfx950: on gfx125x the default HIP backend
# rejects some of the cases these run, such as scatter combine and StdMoE)
pytest tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_intranode.py -q
torchrun --standalone --nproc_per_node=8 tests/python/ops/dispatch_combine_v2/test_ep_backend_parity.py

# Benchmark performance
torchrun --standalone --nproc_per_node=8 tests/python/ops/dispatch_combine_v2/bench_ep.py

Two-node runs and their restrictions: see the EP Guide.

EPv1 (MORI-SHMEM)

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH
python -c "import mori; print(mori.__file__)"

# Test correctness (8 GPUs)
pytest tests/python/ops/test_dispatch_combine_intranode.py -q
pytest tests/python/ops/test_dispatch_combine_async_ll.py -q
pytest tests/python/ops/test_dispatch_combine_internode_v1.py -q

# Benchmark performance
python tests/python/ops/bench_dispatch_combine.py

Test MORI-IO

cd /path/to/mori
export PYTHONPATH=/path/to/mori:$PYTHONPATH

# Correctness tests
pytest tests/python/io/

# Benchmark performance (two nodes). The runner defaults to the native C++
# benchmark (tests/cpp/io/bench_engine); pass --engine python for the torchrun path.
tools/run_internode_io_benchmark.sh --rank 0 --master-addr "<master_ip>" --ifname ens14np0 \
  -- --op write --enable-batch-transfer --enable-sess --buffer-size 32768 --transfer-batch-size 128
# (run the same command on the second node with --rank 1)

See the MORI-IO benchmark guide for both engines and tuning.

Contribution Guide

Welcome to MORI! We appreciate your interest in contributing. Whether you're fixing bugs, adding features, improving documentation, or sharing feedback, your contributions help make MORI better for everyone.

Code Quality

MORI uses pre-commit hooks to maintain code quality. After cloning the repository:

pip install pre-commit
cd /path/to/mori
pre-commit install

# Run on all files (first time)
pre-commit run --all-files

Pre-commit automatically checks code formatting, linting, license headers, and other quality checks on commit. To skip checks when necessary: git commit --no-verify

Name		Name	Last commit message	Last commit date
.claude/skills		.claude/skills
.github		.github
3rdparty		3rdparty
benchmark		benchmark
cmake		cmake
docker		docker
docs		docs
examples		examples
include/mori		include/mori
python/mori		python/mori
src		src
tests		tests
tools		tools
.clang-format		.clang-format
.dockerignore		.dockerignore
.gitignore		.gitignore
.gitmodules		.gitmodules
.pre-commit-config.yaml		.pre-commit-config.yaml
.typos.toml		.typos.toml
CMakeLists.txt		CMakeLists.txt
CPPLINT.cfg		CPPLINT.cfg
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
pyproject.toml		pyproject.toml
requirements-build.txt		requirements-build.txt
setup.py		setup.py

Navigation Menu

Repository navigation

About

Repository files navigation

News

Introduction

Features summary

Documentation

Benchmarks

MORI-EP

MORI-IO

Hardware Support Matrix

Installation

Prerequisites

Validate the environment

Install

From PyPI (stable release)

Nightly (pre-built, tested daily)

From source

Verify installation

Testing

Test MORI-EP (dispatch / combine)

EPv2 (MORI-CCO)

EPv1 (MORI-SHMEM)

Test MORI-IO

Contribution Guide

Code Quality

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

News

Introduction

Features summary

Documentation

Benchmarks

MORI-EP

MORI-IO

Hardware Support Matrix

Installation

Prerequisites

Validate the environment

Install

From PyPI (stable release)

Nightly (pre-built, tested daily)

From source

Verify installation

Testing

Test MORI-EP (dispatch / combine)

EPv2 (MORI-CCO)

EPv1 (MORI-SHMEM)

Test MORI-IO

Contribution Guide

Code Quality

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages