MORI UMBP Single-Node Smoke Test

This guide documents how to run the single-node UMBP + SGLang correctness smoke test that ships with Mori. It launches the run_umbp_single_node_hicache.sh helper script inside a ROCm container.

Use the placeholders in this document to map the workflow to any target node. Every environment-dependent value is surfaced as an argument or environment variable; update those inputs and the script can run unchanged.

For the underlying architecture see src/umbp/doc/design-master-control-plane.md; for the full env-var inventory see src/umbp/doc/runtime-env-vars.md.

Overview

The script spins up the Docker image, rebuilds Mori with UMBP enabled, launches a single-node SGLang server backed by UMBP hierarchical cache, issues a probe completion, and collects logs under ~/umbp_single_node_results. It is intended as a fast health check for DP+EP configurations.

Prerequisites

  • Access to the target node (for example <node-name>), including an active reservation if required by the cluster scheduler.

  • Mori and SGLang checkouts on the node.

  • Access to a checkpoint directory when running with real weights.

  • Docker privileges on the node.

Configuration Inputs

Script constants

The script declares a few host-specific defaults near the top. Update them before running if your environment differs.

Variable

Purpose

Typical override

USER_HOME

Home directory mounted into the container.

Set to your ${HOME} path on the node.

NFS_BASE

Base path containing the repositories.

Point to the shared filesystem path that holds Mori and SGLang.

EXTRA_MOUNTS

Additional docker run bind mounts.

Remove entries that do not exist or add the host paths you require.

DOCKER_IMAGE

ROCm container tag.

Change if you need a different toolchain.

Runtime environment variables

Export the variables below to adapt the run to your environment. The script reads these values at launch time.

Variable

Default in script

When to change

Example

MODEL_PATH

a site-specific path

Real checkpoint location.

/path/to/DeepSeek-V3-0324

MORI_REPO

${NFS_BASE}/mori

Where pip builds Mori. Leave it: the server runs Mori from ${NFS_BASE}/mori (see below).

${NFS_BASE}/mori

RESULTS_DIR

${USER_HOME}/umbp_single_node_results

Where to store logs.

/tmp/umbp_results

ENABLE_DP

false

Enable DP+EP mode.

true

DP_SIZE

8

Data parallel group size.

4

EP_SIZE

8

Expert parallel group size.

2

TP_SIZE

8

Tensor parallel group size.

8

START_UMBP_MASTER

true

Disable if you already manage the master.

false

UMBP_MASTER_ADDRESS

127.0.0.1:15558

Master listener address.

<host-ip>:15558

UMBP_NODE_ADDRESS

127.0.0.1

IP advertised to clients.

<node-ip>

UMBP_IO_ENGINE_PORT

16000

Hicache IO engine port.

16010

UMBP_PEER_SERVICE_PORT

17000

Peer service port.

17010

USE_DUMMY_WEIGHTS

false

Skip checkpoint validation.

true

MEM_FRACTION_STATIC

0.7

Fraction reserved for static allocations.

0.6

PROBE_MAX_TIME

300

Timeout for probe request in seconds.

600

Keep both checkouts under NFS_BASE, as sglang/ and mori/: the server’s PYTHONPATH and the umbp_master binary are taken from ${NFS_BASE}/sglang and ${NFS_BASE}/mori whatever SGLANG_REPO and MORI_REPO say, so pointing those elsewhere builds one checkout and runs another.

The script builds Mori with pip install . --no-build-isolation -v inside the container; MORI_PIP_FLAGS set on the host does not reach that command. By default it runs two GSM8K benchmark rounds after the probe; set RUN_GSM8K=false to skip them.

Launch Steps

  1. SSH to the target node:

    ssh -o StrictHostKeyChecking=no <node-name>
    
  2. Review script constants:

    sed -n '1,120p' run_umbp_single_node_hicache.sh
    

    Update USER_HOME, NFS_BASE, EXTRA_MOUNTS, and DOCKER_IMAGE if the defaults do not match your environment.

  3. Export runtime variables (fill placeholders with your values):

    export MODEL_PATH=<path-to-model-checkpoints>
    export SGLANG_REPO=<path-to-sglang>
    export MORI_REPO=<path-to-mori>
    export RESULTS_DIR=<path-for-results>
    export ENABLE_DP=<true|false>
    export DP_SIZE=<dp-size>
    export EP_SIZE=<ep-size>
    export TP_SIZE=<tp-size>
    export START_UMBP_MASTER=true
    export UMBP_NODE_ADDRESS=$(hostname -I | awk '{print $1}')
    export USE_DUMMY_WEIGHTS=false
    

    Add any other overrides from the table above as needed.

  4. Run the script with the desired weight mode:

    bash run_umbp_single_node_hicache.sh --real-weights
    # or
    bash run_umbp_single_node_hicache.sh --use-dummy-weights
    

    The script will:

    • Start or recycle the umbp-single-node container.

    • Rebuild Mori with BUILD_UMBP=ON via pip install.

    • Compute network interface settings for NCCL/Gloo.

    • Launch the SGLang server with the configured parallelism values and hicache settings.

  5. Monitor the probe and cleanup:

    • The script waits up to one hour for http://127.0.0.1:30000/health to report ready.

    • It issues a probe request and stores the output in ${RESULTS_DIR}/probe_response.json.

    • On success, the script terminates the SGLang server and UMBP master, leaving artifacts in ${RESULTS_DIR}.

Core Dumps

By default the container routes crashes through Apport (/usr/share/apport/apport). If raw core files are required, disable Apport inside the container and set:

sysctl -w kernel.core_pattern=/nfs/users/<user>/cores/core.%e.%p
ulimit -c unlimited

Make sure the directory exists and has enough space.

Troubleshooting

  • Container launch: docker ps should show umbp-single-node. If it exits immediately, check docker logs.

  • Server logs: Available under ${RESULTS_DIR}/server_*.log.

  • UMBP master logs: Saved alongside server logs when START_UMBP_MASTER=true.

  • Model validation: The script validates that MODEL_PATH contains model-*.safetensors when running with real weights.

  • Probe failures: Inspect the tail of the server log; the script prints it automatically on errors.

Refer back to this document whenever the single-node smoke test needs to be run or automated.