We have ForSequenceClassification variants for several of our CLMs, and this line is commonly copied between them:
sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1
However, this line assumes right-padding, because it treats the length of the sequence as the number of tokens before the first pad token (when multiple values have the max value, then argmax returns the first index with that value).
If we have left-padding instead, then this will break because it will compute a sequence length of -1. This was reported as a bug in Gemma here, but it likely affects other models as well.
tokenizer.padding_side is probably not accessible to the model, because it is stored in tokenizer_config and not config. As a result, I think the solution here will have to be rewriting this code so that it can handle either left- or right-padding.
We have
ForSequenceClassificationvariants for several of our CLMs, and this line is commonly copied between them:sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1However, this line assumes right-padding, because it treats the length of the sequence as the number of tokens before the first pad token (when multiple values have the max value, then
argmaxreturns the first index with that value).If we have left-padding instead, then this will break because it will compute a sequence length of -1. This was reported as a bug in Gemma here, but it likely affects other models as well.
tokenizer.padding_sideis probably not accessible to the model, because it is stored intokenizer_configand notconfig. As a result, I think the solution here will have to be rewriting this code so that it can handle either left- or right-padding.