MORI UMBP Single-Node Smoke Test
This guide documents how to run the single-node UMBP + SGLang correctness smoke test that ships with Mori. It launches the run_umbp_single_node_hicache.sh helper script inside a ROCm container.
Use the placeholders in this document to map the workflow to any target node. Every environment-dependent value is surfaced as an argument or environment variable; update those inputs and the script can run unchanged.
For the underlying architecture see
src/umbp/doc/design-master-control-plane.md;
for the full env-var inventory see
src/umbp/doc/runtime-env-vars.md.
Overview
The script spins up the Docker image, rebuilds Mori with UMBP enabled, launches a single-node SGLang server backed by UMBP hierarchical cache, issues a probe completion, and collects logs under ~/umbp_single_node_results. It is intended as a fast health check for DP+EP configurations.
Prerequisites
Access to the target node (for example
<node-name>), including an active reservation if required by the cluster scheduler.Mori and SGLang checkouts on the node.
Access to a checkpoint directory when running with real weights.
Docker privileges on the node.
Configuration Inputs
Script constants
The script declares a few host-specific defaults near the top. Update them before running if your environment differs.
Variable |
Purpose |
Typical override |
|---|---|---|
|
Home directory mounted into the container. |
Set to your |
|
Base path containing the repositories. |
Point to the shared filesystem path that holds Mori and SGLang. |
|
Additional |
Remove entries that do not exist or add the host paths you require. |
|
ROCm container tag. |
Change if you need a different toolchain. |
Runtime environment variables
Export the variables below to adapt the run to your environment. The script reads these values at launch time.
Variable |
Default in script |
When to change |
Example |
|---|---|---|---|
|
a site-specific path |
Real checkpoint location. |
|
|
|
Where pip builds Mori. Leave it: the server runs Mori from |
|
|
|
Where to store logs. |
|
|
|
Enable DP+EP mode. |
|
|
|
Data parallel group size. |
|
|
|
Expert parallel group size. |
|
|
|
Tensor parallel group size. |
|
|
|
Disable if you already manage the master. |
|
|
|
Master listener address. |
|
|
|
IP advertised to clients. |
|
|
|
Hicache IO engine port. |
|
|
|
Peer service port. |
|
|
|
Skip checkpoint validation. |
|
|
|
Fraction reserved for static allocations. |
|
|
|
Timeout for probe request in seconds. |
|
Keep both checkouts under NFS_BASE, as sglang/ and mori/: the server’s
PYTHONPATH and the umbp_master binary are taken from ${NFS_BASE}/sglang and
${NFS_BASE}/mori whatever SGLANG_REPO and MORI_REPO say, so pointing those
elsewhere builds one checkout and runs another.
The script builds Mori with pip install . --no-build-isolation -v inside the
container; MORI_PIP_FLAGS set on the host does not reach that command. By default
it runs two GSM8K benchmark rounds after the probe; set RUN_GSM8K=false to skip
them.
Launch Steps
SSH to the target node:
ssh -o StrictHostKeyChecking=no <node-name>
Review script constants:
sed -n '1,120p' run_umbp_single_node_hicache.sh
Update
USER_HOME,NFS_BASE,EXTRA_MOUNTS, andDOCKER_IMAGEif the defaults do not match your environment.Export runtime variables (fill placeholders with your values):
export MODEL_PATH=<path-to-model-checkpoints> export SGLANG_REPO=<path-to-sglang> export MORI_REPO=<path-to-mori> export RESULTS_DIR=<path-for-results> export ENABLE_DP=<true|false> export DP_SIZE=<dp-size> export EP_SIZE=<ep-size> export TP_SIZE=<tp-size> export START_UMBP_MASTER=true export UMBP_NODE_ADDRESS=$(hostname -I | awk '{print $1}') export USE_DUMMY_WEIGHTS=false
Add any other overrides from the table above as needed.
Run the script with the desired weight mode:
bash run_umbp_single_node_hicache.sh --real-weights # or bash run_umbp_single_node_hicache.sh --use-dummy-weights
The script will:
Start or recycle the
umbp-single-nodecontainer.Rebuild Mori with
BUILD_UMBP=ONviapip install.Compute network interface settings for NCCL/Gloo.
Launch the SGLang server with the configured parallelism values and hicache settings.
Monitor the probe and cleanup:
The script waits up to one hour for
http://127.0.0.1:30000/healthto report ready.It issues a probe request and stores the output in
${RESULTS_DIR}/probe_response.json.On success, the script terminates the SGLang server and UMBP master, leaving artifacts in
${RESULTS_DIR}.
Core Dumps
By default the container routes crashes through Apport (/usr/share/apport/apport). If raw core files are required, disable Apport inside the container and set:
sysctl -w kernel.core_pattern=/nfs/users/<user>/cores/core.%e.%p
ulimit -c unlimited
Make sure the directory exists and has enough space.
Troubleshooting
Container launch:
docker psshould showumbp-single-node. If it exits immediately, checkdocker logs.Server logs: Available under
${RESULTS_DIR}/server_*.log.UMBP master logs: Saved alongside server logs when
START_UMBP_MASTER=true.Model validation: The script validates that
MODEL_PATHcontainsmodel-*.safetensorswhen running with real weights.Probe failures: Inspect the tail of the server log; the script prints it automatically on errors.
Refer back to this document whenever the single-node smoke test needs to be run or automated.