System Info
transformers version: 5.13.0.dev0 (main, commit c96378c413)- Platform: macOS, Python 3.11 (code-level bug, no specific hardware required)
Who can help?
@yonigozlan @zucchini-nlp (PPFormulaNet / Florence2 area)
Description
PPFormulaNetForConditionalGeneration.forward builds decoder_input_ids by right-shifting the target with shift_tokens_right (modular_pp_formulanet.py:386), then computes the training loss with self.loss_function (which maps to ForCausalLMLoss). ForCausalLMLoss shifts the labels a second time (labels[..., 1:], loss/loss_utils.py:24-25), so the model trains against labels[..., 1:] instead of labels — an off-by-one in the training loss.
This is the same class of bug that was fixed for Moonshine in #46784 (and is being fixed for CohereASR / Florence2). PPFormulaNet right-shifts input_ids rather than labels, but the effect is identical when input_ids == labels — the standard encoder-decoder training setup, and exactly what the model's own test inputs use (input_ids is set equal to the decoder target).
Reproduction
import torch
from transformers import PPFormulaNetConfig, PPFormulaNetForConditionalGeneration
vision_config = {
"image_size": 768, "patch_size": 16, "hidden_size": 48, "windows_size": 14,
"num_hidden_layers": 1, "output_channels": 16, "num_attention_heads": 2,
"global_attn_indexes": [1, 1, 1, 1], "mlp_dim": 1,
"post_conv_in_channels": 16, "post_conv_mid_channels": 16,
"post_conv_out_channels": 16, "decoder_hidden_size": 48,
}
text_config = {"decoder_ffn_dim": 16, "decoder_layers": 1, "d_model": 48, "vocab_size": 99}
config = PPFormulaNetConfig(text_config=text_config, vision_config=vision_config, num_hidden_layers=1)
model = PPFormulaNetForConditionalGeneration(config).eval()
V = config.text_config.vocab_size
torch.manual_seed(0)
target = torch.tensor([[5, 11, 23, 7, 42, 2]]) # input_ids == labels == decoder target
pixel_values = torch.rand(1, 3, 768, 768)
with torch.no_grad():
out = model(pixel_values=pixel_values, input_ids=target, labels=target)
aligned = torch.nn.functional.cross_entropy(out.logits.reshape(-1, V), target.reshape(-1))
shifted = torch.nn.functional.cross_entropy(out.logits[..., :-1, :].reshape(-1, V), target[..., 1:].reshape(-1))
print("model loss :", out.loss.item()) # 4.612360
print("CE vs labels :", aligned.item()) # 4.542673 (correct)
print("CE vs labels[1:] :", shifted.item()) # 4.612360 (what the model actually computes)The model loss matches the double-shifted CE, not the aligned CE.
Expected behavior
Loss should be computed directly against labels with a plain CrossEntropyLoss (matching WhisperForConditionalGeneration, BartForConditionalGeneration, and the fixed Moonshine), with no second shift. Logits and generate() are unchanged, and -100 stays ignored.
System Info
transformersversion: 5.13.0.dev0 (main, commitc96378c413)Who can help?
@yonigozlan @zucchini-nlp (PPFormulaNet / Florence2 area)
Description
PPFormulaNetForConditionalGeneration.forwardbuildsdecoder_input_idsby right-shifting the target withshift_tokens_right(modular_pp_formulanet.py:386), then computes the training loss withself.loss_function(which maps toForCausalLMLoss).ForCausalLMLossshifts the labels a second time (labels[..., 1:],loss/loss_utils.py:24-25), so the model trains againstlabels[..., 1:]instead oflabels— an off-by-one in the training loss.This is the same class of bug that was fixed for Moonshine in #46784 (and is being fixed for CohereASR / Florence2). PPFormulaNet right-shifts
input_idsrather thanlabels, but the effect is identical wheninput_ids == labels— the standard encoder-decoder training setup, and exactly what the model's own test inputs use (input_idsis set equal to the decoder target).Reproduction
The model loss matches the double-shifted CE, not the aligned CE.
Expected behavior
Loss should be computed directly against
labelswith a plainCrossEntropyLoss(matchingWhisperForConditionalGeneration,BartForConditionalGeneration, and the fixed Moonshine), with no second shift. Logits andgenerate()are unchanged, and-100stays ignored.