System Info
main branch
Who can help?
@patrickvonplaten, @Narsil, @gante
Information
Tasks
Reproduction
In generate when output_scores=True, the behavior is inconsistent. In greedy_search mode, the scores are raw logits
|
next_token_logits = outputs.logits[:, -1, :] |
|
|
|
# Store scores, attentions and hidden_states when required |
|
if return_dict_in_generate: |
|
if output_scores: |
|
scores += (next_token_logits,) |
but in sample mode (and various beam search modes), the scores are processed logits
|
next_token_logits = outputs.logits[:, -1, :] |
|
|
|
# pre-process distribution |
|
next_token_scores = logits_processor(input_ids, next_token_logits) |
|
next_token_scores = logits_warper(input_ids, next_token_scores) |
|
|
|
# Store scores, attentions and hidden_states when required |
|
if return_dict_in_generate: |
|
if output_scores: |
|
scores += (next_token_scores,) |
Expected behavior
In generate when output_scores=True, the returned scores should be consistent. It could either be raw logits or the processed logits. While for my usecase, I only need raw logits. There might be some usecases which require the processed logits. So there're multiple options:
- Return raw logits when
output_scores=True
- Return processed logits when
output_scores=True
- Return processed logits when
output_scores=True, and raw logits when output_raw_scores=True
System Info
main branch
Who can help?
@patrickvonplaten, @Narsil, @gante
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
In
generatewhenoutput_scores=True, the behavior is inconsistent. Ingreedy_searchmode, the scores are raw logitstransformers/src/transformers/generation_utils.py
Lines 1690 to 1695 in 740a157
but in
samplemode (and various beam search modes), the scores are processed logitstransformers/src/transformers/generation_utils.py
Lines 1945 to 1954 in 740a157
Expected behavior
In
generatewhenoutput_scores=True, the returned scores should be consistent. It could either be raw logits or the processed logits. While for my usecase, I only need raw logits. There might be some usecases which require the processed logits. So there're multiple options:output_scores=Trueoutput_scores=Trueoutput_scores=True, and raw logits whenoutput_raw_scores=True