The resolution of global weather and climate models is typically too coarse to resolve localized weather phenomena, such as severe thunderstorms and mesoscale convective systems, at sufficient detail. Therefore, high-resolution regional models are commonly used to model the area of interest in more detail. Modeling the atmosphere at this scale is both challenging and expensive with traditional numerical weather prediction. Simple loss functions also often limit the ability of AI-based atmospheric models to reproduce extreme events and characterize uncertainty. High-resolution diffusion models offer a solid foundation for realistic and probabilistic AI-based modeling of local weather.
This recipe provides a toolkit for training and simple inference for regional AI high-resolution models using diffusion. Examples of the types of models that can be trained with it include:
- CorrDiff,
a downscaling model that generates local high-resolution weather samples probabilistically from a lower-resolution input at the same time. The regression+diffusion setup used in the original CorrDiff is supported, while purely diffusion-based downscaling can also be trained. This recipe is now better supported and recommended over the original CorrDiff recipe. CorrDiff-ERA5-HRRR, which downscales ERA5 to the 3 km HRRR grid over the contiguous US, is a diffusion-only DiT trained with flow matching and can be replicated with the
replication_corrdiff_conusconfig.
- StormCast, a generative diffusion model designed to emulate NOAA’s High-Resolution Rapid Refresh (HRRR) model, a 3 km operational convection-allowing model. StormCast autoregressively predicts multiple atmospheric state variables. The original StormCast uses a regression+diffusion U-Net configuration similar to CorrDiff, while the full-CONUS version uses a diffusion transformer without a regression stage.
- StormScope, a nowcasting/nearcasting model for clouds and precipitation, using satellite radiances and radar measurements as training data. StormScope's generative model architecture allows the uncertainty of the predicted variables to be accurately quantified.
All of the above models can be formulated in terms of a high-resolution state update guided by a conditioning. In the code, we refer to the high-resolution training target in generic terms as state and to everything the model conditions on as background, including any past state history a forecasting model needs as input. See Model Inputs and Outputs and Adding Custom Datasets for details.
- Install PhysicsNeMo (if not already installed) with the
datapipes-extrasandnn-extrasoptional dependency groups, along with the packages inrequirements.txt. - Copy this folder (
examples/weather/regional_weather_diffusion) to a system with a GPU available.
To install the requirements not covered by PhysicsNeMo with the above-mentioned optional dependency groups, you can run:
pip install -r requirements.txtnatten (used by the DiT architecture) is built from source unless a pre-built wheel matching your PyTorch and CUDA versions is available. To install a pre-built wheel, use the natten-cu12 or natten-cu13 PhysicsNeMo extra, or see the NATTEN installation instructions.
To check that everything works before preparing your own data, generate a small example dataset filled with random data and run a short training on it:
python -c "from datasets.xarray import write_example_files; write_example_files('data', image_shape=(128, 128))"
python train.py --config-name liteThe first command writes data files in the format read by the default XarrayDataset to data/. The second trains the default reference_downscaling model (a diffusion transformer) for 20 steps with a batch size of 2, using the lite config. The outputs, including checkpoints and validation plots, are saved in rundir/stormcast-training/0. Rerunning the same command continues from the latest checkpoint in that directory; delete it to start from scratch (see Checkpoints and Resuming).
To train a model on your own data:
- Convert your data to the format described in docs/XARRAY_DATASET.md, or implement a custom dataset.
- Copy the
reference_*config closest to your problem (see Configuration Basics), e.g.cp config/reference_downscaling.yaml config/my_model.yaml, and edit itsdatasetsection to point to your data and select your variables. - Test the config by setting it in
config/lite.yamlin place ofreference_downscalingand runningpython train.py --config-name lite. - Launch the training with
python train.py --config-name my_model, or withtorchrunto use multiple GPUs (see Distributed Training).
To test the functionality of the training code, you can run the pytest test suite in test_training.py in a single-GPU or multi-GPU environment.
To run the single-GPU tests, run:
pytest test_training.pyIf no tests fail, that indicates that the training environment is functional. It is normal for many tests to be skipped because not all parameter combinations are supported and some tests are meant to be run with multiple GPUs. Multi-GPU tests need to be run using torchrun, with tests of 1, 2, and 4 GPUs included. For instance, to run all 2-GPU tests:
torchrun --standalone --nproc_per_node=2 --no-python pytest test_training.pyThe multi-GPU tests might produce some clutter because all processes write to the terminal, but the output should be readable.
The tests take a while to run (around 15 min on an H100 for the single-GPU tests). Potentially useful pytest options include -k "test_training" (run only the tests from the test_training function) and -x (stop at first failure).
Training is handled by train.py, configured using Hydra based on the contents of the config directory and validated using pydantic. Hydra allows for YAML-based modular and hierarchical configuration management and supports command-line overrides for quick testing and experimentation. The config directory includes the following subdirectories:
dataset: specifies the dataset used for training and the parameters of the dataset, such as the location of the data filesmodel: specifies the model type and model-specific hyperparameterssampler: specifies hyperparameters used in the sampling process for diffusion modelstraining: specifies training-specific hyperparameters and settings like checkpoint/log frequency and where to save training outputsinference: specifies settings forinference.py, such as which model checkpoints and dataset samples to usehydra: specifies basic Hydra settings, like where to store outputs (based on the training outputs directory)
Also in the config directory are several top-level configs:
reference_*: recommended starting points for new projects:reference_downscaling: diffusion-only downscaling with a DiT (the default used bytrain.py)reference_regression_diffusion_downscaling: CorrDiff-style regression+diffusion downscaling with a U-Netreference_stormcast: StormCast-style forecasting guided by low-resolution conditioning, with a DiTreference_stormscope: StormScope-style nowcasting with a DiT
replication_*: configs for reproducing the training of published modelslite: a minimal config with a small batch size and frequent printouts and checkpoints, for quickly testing that training runs without errorstest_*: configs for testing training with synthetic data, without needing any data files
You can select any of these by specifying it as a config name at the command line using --config-name. Optionally, you can also override any specific items of interest using the command line args. For example:
python train.py --config-name reference_stormcast training.batch_size=4More extensive configuration modifications can be made by creating a new top-level configuration file. The reference_* configs show how to specify a top-level config that uses default configuration settings with additional custom modifications added on top.
At runtime, Hydra parses the config subdirectory and command-line overrides into a runtime configuration object cfg, which has all settings accessible through both attribute and dictionary-like interfaces. For example, the total training batch size can be accessed either as cfg.training.batch_size or cfg['training']['batch_size']. The actual content of the config is validated using pydantic based on the guidelines specified in utils/config.py.
The training script train.py initializes the training experiment and launches the main training loop, which is defined in utils/trainer.py. Outputs (training logs, checkpoints) are saved to a directory specified by the following training config items:
training.outdir: 'rundir' # Root path under which to save training outputs
training.experiment_name: 'stormcast-training' # Name for the training experiment
training.run_id: '0' # Unique ID to use for this training run
training.rundir: ${training.outdir}/${training.experiment_name}/${training.run_id} # Path where experiment outputs will be savedThe training.run_id setting can be used for distinguishing between different runs of the same configuration. The final training output directory is constructed by composing together the training.outdir root path (defaults to rundir), the training.experiment_name, and the training.run_id. By default, the outputs of inference.py are saved in the inference subdirectory of this directory.
This recipe can be used to train different configurations of diffusion models: nowcasting and downscaling models, hybrid nowcasting/downscaling models that use both a past state and a forcing condition, as well as unconditional diffusion models. The type of model is not set explicitly; instead, it depends on how the dataset provides input (conditioning) and target data.
The input and output data are contained in the dict returned by the __getitem__ function of the dataset. The expected model output (i.e. the training target) is always contained in the "state" key of the dict.
The types of input the model uses are configured in the model.diffusion_conditions (if training a diffusion model) or model.regression_conditions (if training a regression model) configuration settings. These are lists where the valid items are "background", "invariant" and (for model.diffusion_conditions only) "regression".
"background" is the condition tensor returned by your dataset and contains all variables that the model is conditioned on, typically past state and low-resolution conditioning, depending on your model type. It must be returned in the "background" key of the dict returned by the __getitem__ function of your dataset. For unconditional diffusion models, "background" may be omitted from the model.*_conditions list and from the __getitem__ dict. The following table shows typical settings for each type of model:
| Model type | Example model | "background" contains |
"state" contains |
|---|---|---|---|
| Hybrid | StormCast | cat(past_state, low_res_conditioning) |
future_state |
| Nowcasting | StormScope | past_state |
future_state |
| Downscaling | CorrDiff | low_res_conditioning |
high_res_state |
| Unconditional | Nothing (omit "background" key) |
state_tensor |
If you want to use invariants (that is, conditions that are the same for each sample; a surface elevation map is a typical example), return the invariants as a NumPy array from the get_invariants function of your dataset and add "invariant" to your condition list.
If training a regression-diffusion model like CorrDiff, add "regression" to model.diffusion_conditions. The training loop will then automatically run the regression model supplied in model.regression_weights. Conversely, model.regression_conditions is not used and setting it is not necessary if training a pure diffusion model.
DiT-type models additionally support scalar conditions that are single numbers that apply to the entire grid. To add these, return the conditions as a 1D NumPy array in the "scalar_conditions" key of the dictionary returned by __getitem__. Additionally, implement the scalar_condition_channels method for the dataset.
- Diffusion-only (
reference_downscaling,reference_stormcast,reference_stormscope): a single diffusion model generates"state"directly. This is simpler to train and run and has worked well in StormCast-CONUS and StormScope, but may require tuning the noise distribution to your dataset. - Regression+diffusion (
reference_regression_diffusion_downscaling, as in CorrDiff and the original StormCast): a regression model trained with a mean squared error loss predicts the expected"state", and a diffusion model generates the residual on top of it. This can make the diffusion task easier, at the cost of training two models and running both at inference. Train the regression model first, then the diffusion model.
We recommend starting with diffusion-only training for most new projects.
The architecture is selected with model.architecture:
dit(recommended for new projects): a diffusion transformer with NATTEN neighborhood attention. Supports scalar conditions and mask tokens, but currently only diffusion model training.unet: a SongUNet-family U-Net, as used in CorrDiff and the original StormCast. Required for regression models and lead time labels.
model.hyperparameters is passed to the DiT or SongUNet class via utils/nn.py; see their docstrings for all options. The main ones are:
- DiT:
hidden_size,depth,num_heads,patch_sizeandattn_kernel_size(neighborhood size in tokens). The grid dimensions must be divisible bypatch_size. - U-Net:
model_channels,channel_mult(per-level channel multipliers; its length sets the depth),num_blocksandattn_resolutions. The grid dimensions must be divisible by2**(len(channel_mult) - 1).
See config/model/reference_*.yaml for starting points and config/model/replication_*.yaml for the settings of published models.
You can ignore this section if you plan to train a diffusion-only model such as StormScope.
Regression models are trained by setting training.loss.type to regression. For example, to train the regression model of the original StormCast, you can use the replication_stormcast_v1 config (uncomment the regression-specific settings in that config first) and an optional name for the training experiment:
python train.py --config-name replication_stormcast_v1 training.loss.type=regression training.experiment_name=regressionDiffusion model training is launched by setting training.loss.type to edm (the default), or to flow_matching to train with flow matching.
To train a diffusion model that uses the regression model output (unnecessary for diffusion-only models), there are two config items that must be defined:
'regression'included inmodel.diffusion_conditions.model.regression_weightsset to the path of a PhysicsNeMo (.mdlus) checkpoint with model weights for the pre-trained regression model. These are saved in the checkpoint directory during training (see Checkpoints and Resuming).
The reference_regression_diffusion_downscaling and replication_stormcast_v1 top-level configs show examples of how to specify these settings.
With that, launching diffusion training for the original StormCast configuration looks something like:
python train.py --config-name replication_stormcast_v1 training.experiment_name=diffusionBy default, diffusion models are trained with the EDM denoising score matching loss. They can instead be trained with rectified flow matching, as used by CorrDiff-ERA5-HRRR. Three settings define how a diffusion model is trained:
-
training.loss.type: the noise schedule and the loss.edmtrains with denoising score matching on the EDM schedule (EDMNoiseScheduler,WeightedMSEDSMLoss);flow_matchingtrains with rectified flow matching (RectifiedFlowNoiseScheduler,WeightedFlowMatchingLoss), with times drawn uniformly from[training.loss.t_min, training.loss.t_max], where$t = 0$ is clean data and$t = 1$ is pure noise. -
training.loss.prediction_type: what the network outputs:x0(the clean state, the default),epsilon(the noise),score, orflow(the velocity). The loss and the sampler convert between these with the noise scheduler. Withflow_matchingandx0, the network predicts the clean state (x-prediction) and the loss is the plain mean squared error on it. -
model.preconditioner: how the network is wrapped.edm(the default) applies the EDM input, output and noise-level scalings withEDMPreconditioner, so the wrapped network predictsx0; it requiresprediction_type: x0.time_scaledonly multiplies the times bymodel.time_scale(default 999) before the network's timestep embedding (TimeScaledModelinutils/nn.py), so the network output is the prediction itself. Usetime_scaledwithflow_matching.
To train one of the reference_* diffusion models with flow matching, make these changes in your copy of the config:
defaults:
...
- sampler/rectified_flow # in place of sampler/edm_deterministic
...
model:
preconditioner: time_scaled
training:
loss:
type: 'flow_matching' # remove the EDM sigma_* settings here and in sampler.argsThe rectified_flow sampler (config/sampler/rectified_flow.yaml) integrates the rectified-flow ODE from pure noise with num_steps steps of the heun or euler solver. Its time_shift time_shift: 1 when sampling at the training resolution.
The replication_corrdiff_conus config trains CorrDiff-ERA5-HRRR with x-prediction flow matching:
-
Data: same-time pairs of ERA5 and HRRR over a 1024 × 1792 pixel CONUS domain, read by the custom dataset
corrdiff_conus.CorrDiffConusDataset(see docs/CORRDIFF_CONUS_DATASET.md). The"state"holds 99 HRRR variables; the"background"holds 26 ERA5 variables interpolated to the HRRR grid, the cosine of the solar zenith angle, and three static fields. The scalar conditions are the sine and cosine of the time of year and presence flags fortcwvandsp, which the dataset drops with probability 0.5 during training so that the model also runs without them. The model trains on random 256 × 256 patches (dataset.patch_shape). -
Model (
config/model/replication_corrdiff_conus.yaml): a 678M-parameter DiT (28 blocks, hidden size 1152, 18 heads, 2 × 2 pixel patches) with 31 × 31 token neighborhood attention, 2D rotary position embeddings and normalized queries and keys, wrapped withpreconditioner: time_scaled. The query and key normalization uses NVIDIA Apex, which the PhysicsNeMo and NGC PyTorch containers include. -
Training: 781,250 steps at batch size 64 (50M samples) with AdamW, learning rate
$10^{-4}$ , a 1000-step warmup followed by cosine decay, BF16 mixed precision, and gradient clipping at 1. With 2 samples per GPU, training uses about 68 GB of memory per GPU. Each checkpoint takes 8 GB, sotraining.checkpoint_freqis set to 7812 steps (0.5M samples).
To train it with 8 GPUs on one node:
torchrun --standalone --nproc_per_node=8 train.py --config-name replication_corrdiff_conus dataset.data_path=/path/to/dataA smaller model, such as model.hyperparameters.hidden_size=768 model.hyperparameters.depth=16 model.hyperparameters.num_heads=12, trains about four times faster.
The trained model samples the full 1024 × 1792 domain, which has 28 times more pixels than the training patches. The released model uses time_shift: 32 with 50 Heun steps on the full domain (see Inference), while the validation during training samples the training patches with time_shift: 1. The released checkpoint has the following ensemble-mean RMSE and spread against HRRR for 8 members on 2025-10-24 00 UTC:
| Variable | Ensemble-mean RMSE | Ensemble spread |
|---|---|---|
u10m |
1.05 m/s | 0.95 m/s |
v10m |
1.15 m/s | 1.01 m/s |
t2m |
0.70 K | 0.58 K |
mslp |
49.2 Pa | 38.9 Pa |
refc |
4.70 dBZ | 4.30 dBZ |
To test flow matching training without data, use --config-name test_flow_matching (a small DiT on mock data), or run the test_flow_matching* and test_rectified_flow* tests in test_training.py.
To run a quick end-to-end test of training, copy the config/lite.yaml config file and swap reference_downscaling for your training config file. Then launch training using your modified lite.yaml config file. It applies overrides that make the training run for a few steps and go through all essential steps (initialization, training, validation and checkpoint saving). This verifies that training runs without obvious problems, though it is not expected to provide a useful model and will likely not reveal problems that only appear after longer training, such as training instabilities.
Checkpoints are saved every training.checkpoint_freq steps and at the end of training, in {training.rundir}/checkpoints_regression or {training.rundir}/checkpoints_diffusion. Each consists of two files:
<ModelClass>.0.<step>.mdlus(e.g.StormCastUNet.0.16000.mdlus,EDMPreconditioner.0.16000.mdlus, orTimeScaledModel.0.16000.mdluswithmodel.preconditioner: time_scaled): the model weights and architecture, loadable withphysicsnemo.Module.from_checkpoint(path). Use the best one (typically the one with the lowest validation loss) for inference,model.regression_weightsortraining.initial_weights.checkpoint.0.<step>.pt: the optimizer and scheduler states, needed to resume training but not for inference.
Training automatically resumes from the latest checkpoint in training.rundir, which makes it easy to continue interrupted runs (training.max_run_steps can be used to stop a job after a given number of steps). To start a new run instead, change training.experiment_name or training.run_id, or delete the old run directory. Set training.resume_checkpoint to a step number to resume from a specific checkpoint.
To initialize a new run from an earlier model, e.g. for fine-tuning, set training.initial_weights to its .mdlus file. This loads only the weights and is ignored when resuming.
Both regression and diffusion training can be distributed easily with data parallelism through torchrun or other launchers (for example, SLURM srun). As long as GPU memory is sufficient, the same configuration file can be used regardless of the number of GPUs. Each training batch is distributed evenly between the GPUs; you just need to ensure the configuration being run has a batch size that is divisible by the number of available GPUs/processes. For example, distributed training over 8 GPUs on one node would look something like:
torchrun --standalone --nnodes=1 --nproc_per_node=8 train.py --config-name <your_training_config>As an advanced form of distributed training, the training code also supports domain parallelism through PhysicsNeMo ShardTensor. It distributes individual training samples across multiple GPUs, and therefore can be used to train large models and/or large domains when even a single sample will not fit in the memory of one GPU. Domain parallelism is controlled by the training.domain_parallel_size config setting that specifies how many GPUs each sample should be distributed to. For example, if you have at least two GPUs:
torchrun --standalone --nnodes=1 --nproc_per_node=2 train.py --config-name <your_training_config> training.domain_parallel_size=2will train using a domain parallel size of 2 (use test_diffusion as your_training_config to run a quick test of domain parallel training). When domain parallelism is used, the local batch size should be 1, so the global batch size will be smaller than the number of GPUs: batch_size * domain_parallel_size == number_of_gpus.
Domain parallelism introduces some additional communication overhead and therefore should only be used if even a local batch size of 1 cannot fit on a single GPU.
The default configuration uses a batch size of 64 (controlled by training.batch_size), as used in the StormCast paper. Many datasets can be trained with a smaller batch size; for instance, StormCast-CONUS and StormScope use a batch size of 16, as set in the reference_stormcast, reference_stormscope and corresponding replication_* config files. If your data grid is large, you may be able to train with a smaller batch size since the larger grid already provides more training data per batch.
If you have few GPUs and/or GPUs with limited memory, the default setting of training.batch_size_per_gpu: 'auto' may cause you to run out of memory. In that case, you can reduce the per-GPU memory utilization by manually setting training.batch_size_per_gpu to training.batch_size // (number_of_gpus * n) where n is an integer. The training code will automatically employ gradient accumulation to maintain the desired effective batch size specified by training.batch_size while using less memory, at the cost of longer training time.
Another way to reduce memory usage is to enable 16-bit training by setting training.perf.fp_optimizations to amp-bf16. While the default is fp32, our experience indicates that BF16 training works as well as 32-bit training and therefore it is set by the reference configuration files. To train DiT models in BF16, it is necessary to set model.hyperparameters.attn_kwargs.qk_norm=true, likewise already enabled in the reference configs.
If even a single sample is too big to fit on one GPU, consider using domain parallelism.
For diffusion models, you can enable per-sigma-bin loss logging to diagnose which noise levels the model struggles with. This logs the mean training loss for equal-probability bins of the sampled sigma values to TensorBoard and/or WandB (under loss/train_sigma_bin/).
To enable it, add the following to your diffusion training config:
training:
loss:
track_sigma_bin_loss: true
sigma_bin_count: 8 # number of equal-probability bins (default: 8)
# sigma_bin_edges: [] # optional: explicit strictly-increasing bin edgesWhen sigma_bin_edges is not provided, bins are computed automatically so that each bin has equal probability mass under the configured sigma distribution (lognormal or loguniform). With flow_matching, the bins instead divide the time range [t_min, t_max] evenly. This feature is ignored for regression models.
The dataset used by the training process is set by the dataset.name configuration option, in the form module.DatasetClass where the training will use the class DatasetClass found in module.py within the datasets directory. For example, dataset.name=xarray.XarrayDataset will use XarrayDataset from datasets/xarray.py. You can either use one of the provided datasets or implement your own.
xarray.XarrayDataset: a flexible Xarray-based dataset that can be configured to train many types of models. For more complex use cases or for loading data from custom data formats, it can be used as a starting point and modified or subclassed. See docs/XARRAY_DATASET.md.mock.MockDataset: supplies random data in different configurations; useful for testing training loops.data_loader_hrrr_era5.HrrrEra5Dataset: legacy dataloader for StormCast; reads HRRR and ERA5 data from zarr-formatted files. See docs/HRRR_ERA5_DATASET.md.corrdiff_conus.CorrDiffConusDataset: dataloader for CorrDiff-ERA5-HRRR; serves same-time ERA5 and HRRR pairs over the contiguous US, with random training patches and randomly dropped inputs. See docs/CORRDIFF_CONUS_DATASET.md.
When xarray.XarrayDataset is not sufficient, you can define a custom dataset class. See docs/CUSTOM_DATASET.md for the interface it must implement, how to configure it, and the optional spatial mask and loading strategies.
Every training.print_progress_freq steps, the trainer prints a few training statistics such as the training loss.
This recipe also supports both TensorBoard and Weights & Biases (WandB), which can be enabled by setting training.log_to_tensorboard=True and training.log_to_wandb=True, respectively. TensorBoard is free to use; WandB requires a license, but academic accounts are free to create at wandb.ai. After you set up an account, you can adjust entity and project in utils/logging.py to the appropriate names for your WandB workspace. TensorBoard logs are written to {training.rundir}/tensorboard while WandB logs are in {training.rundir}/wandb.
Validation runs every training.validation_freq steps and evaluates training.validation_steps batches (16 by default) of the validation set, using the same samples and noise every time so that the validation loss is comparable between steps. Increase training.validation_steps if these samples don't represent the validation set well. Images of generated samples and power spectra are also saved in the logs and in {training.rundir}/images for variables listed in training.validation_plot_variables.
Diffusion losses typically drop quickly at first and then slowly. Their late improvements are small even while sample quality still improves. The validation images and spectra are therefore usually more informative than the loss alone: a ratio of generated to target spectral power close to 1 means that the generated fields have realistic small-scale detail. If the loss increases or spikes, see Training Is Unstable.
inference.py generates samples from a trained diffusion model with any dataset, using the training config to build the validation dataset and the model inputs. The inference options are added to the training config with +inference=default (see config/inference/default.yaml). For example:
python inference.py --config-name reference_downscaling +inference=default \
inference.diffusion_checkpoint=<checkpoint_dir>/EDMPreconditioner.0.<step>.mdlus \
inference.indices=[0,10,20] inference.ensemble_size=4 inference.plot_variables=[t2m_t+0]This generates an ensemble of 4 samples for each of the validation dataset samples 0, 10 and 20, and saves them along with the corresponding targets, in physical units, to samples.zarr in inference.output_dir (by default {training.rundir}/inference). It also saves a plot of t2m for each sample, comparing the first ensemble member to the target. For regression+diffusion models, the regression model is loaded from model.regression_weights unless inference.regression_checkpoint is given, and its prediction is also saved if inference.save_regression=true. Other parts of the config can be overridden as in training, e.g. sampler.args.num_steps=32.
Models trained with flow matching are sampled with the rectified-flow sampler of their config. For example, to sample CorrDiff-ERA5-HRRR on the full domain as the released model does:
python inference.py --config-name replication_corrdiff_conus +inference=default \
inference.diffusion_checkpoint=<checkpoint_dir>/TimeScaledModel.0.<step>.mdlus \
dataset.patch_shape=null sampler.args.time_shift=32 sampler.args.num_steps=50 \
inference.indices=[0] inference.ensemble_size=4 inference.plot_variables=[t2m]When the inference samples are larger than the training samples, as here, inference.py rebuilds the DiT for the larger size with the same weights. This requires rotary position embeddings (attention_backend: "natten2d_rope"); a DiT with a learned absolute positional embedding can only run at its training size.
This is not autoregressive rollout. inference.py does not feed its predictions back into the model as input for a following step; each sample uses its own input from the dataset. (Prior versions of this recipe carried predictions forward through the two-element state convention; that convention has been removed -- see Adding Custom Datasets.) For autoregressive rollout or more complex inference workflows, inference.py can be used as a starting point, or you can bring your checkpoints to Earth2Studio for further analysis and visualizations. The Earth2Studio wrapper for StormCast can be used as a starting point for custom implementations. Earth2Studio also provides model creation skills (for coding agents) that help with writing wrappers for new models.
Possible solutions include decreasing the batch size, training with more GPUs and using domain parallelism. See the section Memory Management.
Supply the past state from the dataset in "background" and the future state in "state". Refer to Model Inputs and Outputs.
Supply the low-resolution conditioning from the dataset in "background" and the high-resolution target in "state". Refer to Model Inputs and Outputs.
Ensure that the model.diffusion_conditions and model.regression_conditions settings for your model do not include "background". Your dataset may omit the "background" key entirely from the dict returned by __getitem__; if it provides one anyway, it will be ignored. See Model Inputs and Outputs.
If training seems slow, check your GPU utilization. If it's low, the problem is likely a bottleneck with the data loading. Another way to identify a data bottleneck is to test your model using MockDataset (set dataset.name=mock.MockDataset); if training with MockDataset is much faster than with real data, it likely indicates a data loading bottleneck. Many datasets used in regional high-resolution models contain lots of data. Depending on the dataset size and format, consider the following optimizations:
- Increasing
training.num_data_workers(ordataset.loader.num_workers, which overrides it) - If your dataset implements one, switching to the PhysicsNeMo
datapipesloading strategy (dataset.loader.backend: "datapipes") -- see Loading Strategy:torchvsdatapipes - Parallel loading of data in multiple threads within the dataset; controlled by
dataset.num_threadsif using the referenceXarrayDataset - Optimized data preprocessing using e.g. Numba
- Data compression, if bandwidth from storage is the limiting factor
- Caching data, particularly if your dataset is small enough to be preloaded to memory (see the
cache_locationoption ofXarrayDataset)
Enabling BF16 training (training.perf.fp_optimizations=amp-bf16) speeds up training considerably; if using a DiT model, model.hyperparameters.attn_kwargs.qk_norm=true should be set to ensure stability.
Compiling the training loss with torch.compile (training.perf.torch_compile=True), which will also compile the model forward pass, also typically improves performance. This also works under domain parallelism (ShardTensor).
Consider the following solutions:
- Ensure that your data is normalized to near zero mean and unit variance.
- Skewed and heavy-tailed data distributions may introduce instability. Consider applying a nonlinear transformation in preprocessing, or in the dataset (see Normalization Statistics for
XarrayDataset), to bring the variable distribution closer to a normal distribution. For instance, some variables become better behaved when log-transformed. Ensure that the distribution of the transformed variable is near zero mean and unit variance. - The default lognormal sampling of noise levels may introduce instability in some cases. Consider setting
training.loss.sigma_distributiontologuniform, andtraining.loss.sigma_minandtraining.loss.sigma_maxto the ends of the noise level range (for example, 0.01 and 200, respectively). - If using DiT, we recommend setting
model.hyperparameters.attn_kwargs.qk_norm=true; without this option, DiT has proven to be unstable in the current implementation. It is already set in the reference configurations. - If you have many state variables, the diffusion model may not have the capacity to denoise them all. Increasing the model width (
model.hyperparameters.hidden_sizefor DiT,model.hyperparameters.model_channelsfor U-Net) may help, at the cost of increased compute and memory requirements.
- Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling
- Elucidating the design space of diffusion-based generative models
- Score-Based Generative Modeling through Stochastic Differential Equations
- Learning Accurate Storm-Scale Evolution from Observations
- Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling (CorrDiff)
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Back to Basics: Let Denoising Generative Models Denoise (x-prediction)
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Stable Diffusion 3)

