Repository navigation
Fix UnboundLocalError in the CRPS losses for single member forecasts with spatial weights - #204
Conversation
CRPSLoss, GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss raised UnboundLocalError for a single member forecast with spatial weights: the E == 1 path skips the ensemble branch, which is the only place that flattened the weights into spatial_weights_split. MetricsHandler passes weights as soon as a target contains NaN, so the CRPS metric failed this way for deterministic models. Flatten the weights before the branch, so that both paths have them, and split them in ensemble direction only when the loss is ensemble distributed, as CRPSLoss already did and as the observations are. GradientCRPSLoss, VortDivCRPSLoss and KernelScoreLoss split the weights whenever the ensemble group had more than one rank, also when the observations stayed whole. The new test checks for all four losses that the weighted score of a single member equals that of two identical members, whose spread is zero, and that weights of one leave it unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
[Medium impact] The PR appears safe to merge; no new actionable issue was found. SummaryFixes weighted single-member forecasts in
Reviews (3) · Last reviewed commit: "Use a batch of 4 in the distributed weig..." · Reviewed by Greptile |
|
Thanks, the fix looks good, also the change to split the weights in 1. The test does not check that the weights are applied Both checks would also pass if the E=1 path ignored the weights: E=1 and two identical members agree when neither uses them, and unit weights equal no weights either way. Could you add a check that the weights actually act, for example by changing the observations in the zero weight region and asserting the score stays the same: obs_changed = obs.clone()
obs_changed[..., : _IMG_W // 3] += 10.0
self.assertTrue(
compare_tensors(f"{name} E=1 masked region", fn(fc1, obs_changed, wgt), out_e1, atol=1e-5, verbose=verbose)
)A comparison against a manual weighted average of 2. The ensemble split change is not covered by the distributed tests
|
…ghted CRPS losses distributed The serial test now checks that the scores for two weights adding up to one sum to the unweighted score, which fails if the weights are ignored, and that CRPSLoss does not see observations in the zero weight region. test_distributed_weighted_crps runs CRPSLoss and GradientCRPSLoss with spatial weights against a reference that is not ensemble distributed, which guards splitting the weights only for an ensemble distributed loss. Co-Authored-By: Claude Opus 5.5 <[email protected]>
With a batch of 32, the full size GradientCRPSLoss reference with gradients on every rank pushed the 16 rank CI grids past the runner memory. The test checks how the weights are split, which does not depend on the batch size. Co-Authored-By: Claude Opus 5.5 <[email protected]>
CRPSLoss,GradientCRPSLoss,VortDivCRPSLossandKernelScoreLossraiseUnboundLocalErrorfor a single member forecast with spatial weights. TheE == 1path skips the ensemble branch, and that branch is the only place that setsspatial_weights_split.MetricsHandlerpasses weights as soon as a target contains NaN, so the CRPS metric fails this way for deterministic models. I noticed this in #202 and split it out from there as requested.The fix flattens the weights before the branch, so both paths have them, and splits them in ensemble direction only when the loss is ensemble distributed. That is what
CRPSLossalready did, and it is how the observations are handled.GradientCRPSLoss,VortDivCRPSLossandKernelScoreLossused to split the weights whenever the ensemble group had more than one rank, even when the observations stayed whole. Withensemble_distributed=Truenothing changes.test_e1_spatial_weightsintests/test_losses.pychecks two things for all four losses. The weighted score of a single member must equal that of two identical members, whose spread is zero. Weights of one must leave the score unchanged. On main it fails with theUnboundLocalErrorfor all four.Testing, on CPU only:
tests/test_losses.pypasses (533 passed). I ran it withTORCHDYNAMO_DISABLE=1because torch.compile cannot invoke the compiler in my environment.tests/distributed/test_distributed_losses.pypasses on two gloo ranks, with the ensemble split and with the latitude split. black, ruff and the license header check are clean on the changed files.Not changed here: with weights per channel,
KernelScoreLossandGradientCRPSLoss(absolute=False)fail with a shape mismatch for any ensemble size. They expand the channels, into the kernel basis and into the two gradient components, but the weights keep the original channel count. Weights that broadcast over the channels work. I can look at that separately if it is useful.🤖 Generated with Claude Code