Skip to content

Fix UnboundLocalError in the CRPS losses for single member forecasts with spatial weights - #204

Merged
azrael417 merged 4 commits into
NVIDIA:mainfrom
MaxFreedomPollard:fix/crps-single-member
Nov 2, 2026
Merged

azrael417 merged 4 commits into
NVIDIA:mainfrom
MaxFreedomPollard:fix/crps-single-member

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

CRPSLoss, GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss raise UnboundLocalError for a single member forecast with spatial weights. The E == 1 path skips the ensemble branch, and that branch is the only place that sets spatial_weights_split. MetricsHandler passes weights as soon as a target contains NaN, so the CRPS metric fails this way for deterministic models. I noticed this in #202 and split it out from there as requested.

import torch
from makani.utils.losses import CRPSLoss

crps = CRPSLoss(img_shape=(32, 64), crop_shape=(32, 64), crop_offset=(0, 0), channel_names=["t2m"], grid_type="equiangular")
fc = torch.randn(2, 1, 1, 32, 64)  # a single member
obs = torch.randn(2, 1, 32, 64)
crps(fc, obs, torch.ones_like(obs))
# main: UnboundLocalError: cannot access local variable 'spatial_weights_split' where it is not associated with a value

The fix flattens the weights before the branch, so both paths have them, and splits them in ensemble direction only when the loss is ensemble distributed. That is what CRPSLoss already did, and it is how the observations are handled. GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss used to split the weights whenever the ensemble group had more than one rank, even when the observations stayed whole. With ensemble_distributed=True nothing changes.

test_e1_spatial_weights in tests/test_losses.py checks two things for all four losses. The weighted score of a single member must equal that of two identical members, whose spread is zero. Weights of one must leave the score unchanged. On main it fails with the UnboundLocalError for all four.

Testing, on CPU only: tests/test_losses.py passes (533 passed). I ran it with TORCHDYNAMO_DISABLE=1 because torch.compile cannot invoke the compiler in my environment. tests/distributed/test_distributed_losses.py passes on two gloo ranks, with the ensemble split and with the latitude split. black, ruff and the license header check are clean on the changed files.

Not changed here: with weights per channel, KernelScoreLoss and GradientCRPSLoss(absolute=False) fail with a shape mismatch for any ensemble size. They expand the channels, into the kernel basis and into the two gradient components, but the weights keep the original channel count. Weights that broadcast over the channels work. I can look at that separately if it is useful.

🤖 Generated with Claude Code

CRPSLoss, GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss raised
UnboundLocalError for a single member forecast with spatial weights:
the E == 1 path skips the ensemble branch, which is the only place that
flattened the weights into spatial_weights_split. MetricsHandler passes
weights as soon as a target contains NaN, so the CRPS metric failed this
way for deterministic models.

Flatten the weights before the branch, so that both paths have them,
and split them in ensemble direction only when the loss is ensemble
distributed, as CRPSLoss already did and as the observations are.
GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss split the weights
whenever the ensemble group had more than one rank, also when the
observations stayed whole.

The new test checks for all four losses that the weighted score of a
single member equals that of two identical members, whose spread is
zero, and that weights of one leave it unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@greptile-apps

greptile-apps Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was found.

Summary

Fixes weighted single-member forecasts in CRPSLoss, GradientCRPSLoss, VortDivCRPSLoss, and KernelScoreLoss.

  • Single-member CRPS-style losses now use spatial weights.

Reviews (3) · Last reviewed commit: "Use a batch of 4 in the distributed weig..." · Reviewed by Greptile

@azrael417

Copy link
Copy Markdown
Collaborator

Thanks, the fix looks good, also the change to split the weights in GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss only when the loss is ensemble distributed. The test fits nicely into TestEnsembleLossE1FastPath. Two points on the tests:

1. The test does not check that the weights are applied

Both checks would also pass if the E=1 path ignored the weights: E=1 and two identical members agree when neither uses them, and unit weights equal no weights either way. Could you add a check that the weights actually act, for example by changing the observations in the zero weight region and asserting the score stays the same:

obs_changed = obs.clone()
obs_changed[..., : _IMG_W // 3] += 10.0
self.assertTrue(
    compare_tensors(f"{name} E=1 masked region", fn(fc1, obs_changed, wgt), out_e1, atol=1e-5, verbose=verbose)
)

A comparison against a manual weighted average of |obs - fc| for CRPSLoss would work as well.

2. The ensemble split change is not covered by the distributed tests

tests/distributed/test_distributed_losses.py never passes spatial weights and has no tests for GradientCRPSLoss, VortDivCRPSLoss or KernelScoreLoss, so the two rank runs do not exercise the second part of the fix. The existing setup would guard it naturally: the reference instance there is built with ensemble_distributed=False and runs on every rank, so with an ensemble group of 2 it hits exactly the old shape mismatch. Could you pass spatial weights in test_distributed_crps, split along the spatial dims like the observations, and add a weighted GradientCRPSLoss case, along the lines of the "local weights are not split" subtest in test_distributed_spectral_coherence_loss?

azrael417 and others added 2 commits November 2, 2026 03:48
…ghted CRPS losses distributed

The serial test now checks that the scores for two weights adding up to one sum to the unweighted score, which
fails if the weights are ignored, and that CRPSLoss does not see observations in the zero weight region.
test_distributed_weighted_crps runs CRPSLoss and GradientCRPSLoss with spatial weights against a reference that
is not ensemble distributed, which guards splitting the weights only for an ensemble distributed loss.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@azrael417 azrael417 self-assigned this Nov 2, 2026
With a batch of 32, the full size GradientCRPSLoss reference with gradients on every rank pushed the 16 rank CI
grids past the runner memory. The test checks how the weights are split, which does not depend on the batch size.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@azrael417
azrael417 merged commit bd50d3a into NVIDIA:main Nov 2, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants