GPT-OSS 20B (Mixture of Experts)
This benchmark trains a 20B parameter GPT model with Mixture of Experts (MoE) architecture using the Primus framework on AMD and NVIDIA GPUs.
Run the following build command from this directory. The build process will take a while to complete.
# From gpt-oss-20b/primus directory
docker build -t rocm/amd-mlperf:gpt_oss_20b_training_5.1 .The current codebase uses the c4/en/3.0.1 dataset from HuggingFace/AllenAI for training and evaluation.
The pre-tokenized dataset is available for download. Navigate to your desired download directory and run the following commands:
# Create desired download directory with the right permission
cd /data/gpt_oss_20b
# Download training and validation data
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d data https://training.mlcommons-storage.org/metadata/llama-3-1-8b-preprocessed-c4-dataset.uriAfter download, you should see files with the following naming conventions:
- Training:
c4-train.en_6_text_document.binand.idx - Validation:
c4-validation-91205-samples.en_text_document.binand.idx
The data directory is approximately 80 GB.
Set the directory for data and results. Ensure $LOGDIR has write access.
export DATADIR=/data/gpt_oss_20b/data
export MODELDIR=/data/gpt_oss_20b/model
export LOGDIR=/data/gpt_oss_20b/results
export CONT=rocm/amd-mlperf:gpt_oss_20b_training_5.1
# Create results directory
mkdir -p $LOGDIR
sudo chmod -R 777 $LOGDIRSet appropriate configuration and system-specific hyperparameters based on hardware type:
| Config File | System | GPUs |
|---|---|---|
config_MI355X_1x8x1.sh |
MI355X | 1 node × 8 GPUs |
config_B200_1x8x1.sh |
B200 | 1 node × 8 GPUs |
source config_MI355X_1x8x1.shexport NEXP=1
bash run_with_docker.shexport NEXP=10
bash run_with_docker.shsbatch -A <account> -p <partition> -t <time_limit> run.subAfter completion, logs will be available under $LOGDIR.
3.34
Validation loss (log perplexity)
Evaluation every 12,288 samples (768 iterations with GBS=16)
We evaluate using the first 1,024 samples from the validation dataset.
| Parameter | Value |
|---|---|
| Model Size | 20B parameters |
| Architecture | GPT with Mixture of Experts |
| Sequence Length | 8192 |
| Expert Parallelism | 8 |
| Hyperparameter | Value |
|---|---|
| Micro Batch Size | 2 |
| Global Batch Size | 16 |
| Learning Rate | 8e-4 |
| LR Schedule | Cosine decay with warmup |
| Weight Decay | 0.1 |
| Adam β1, β2, eps | 0.9, 0.95, 1e-5 |
| Max Training Iterations | 1,200,000 |
gpt-oss-20b/primus/
├── conf/ # Configuration files
│ └── gpt_oss_20B-pretrain.yaml
├── src/ # Training source code
│ └── train.py
├── config_MI355X_1x8x1.sh # System configuration (MI355 - AMD)
├── config_B200_1x8x1.sh # System configuration (B200 - NVIDIA)
├── Dockerfile # Dockerfile (MI355 - AMD)
├── Dockerfile.nvidia # Dockerfile (B200 - NVIDIA)
└── requirements.txt # Python dependencies (includes primus-mllog)
Approximate train time to convergence is ~6.5 hours.