ENH: added new output_logits option to generate function#28667
Conversation
|
Hi @mbaak 👋 Thank you for opening the PR 🤗 I haven't seen a request for this feature (or I have no recollection of it). Personally, I am also not very convinced that it is very useful -- at the end of the day, the model will select the next token after applying the logits processors, if there are any. Since merging this PR would increase the complexity of our codebase AND I'm not convinced, I won't be merging it for now. However, I may have a poor perception of this feature's usefulness. I'm going to do my standard bargain: if this comment gets 20 reactions, it means there are users looking for this feature, in which case I will add the feature 🤗 (Whoever does the 20th reaction, please tag me) |
|
Hi @gante, Let me try to explain the use-case better, my description was a bit short and perhaps unclear. I'm using a RAG setup for question-answering on large documents. (For example, a document describes a project, and a question can be: what's the location of the project?) As causallm model I'm using llama2 (13B) with default settings, meaning with logit processing and warping turned on. The answers are generated as usual, so based on the processed logits to generate the best possible next tokens. However, these answers need to be reviewed for mistakes/hallucinations. So we want a confidence score for each answer - one that helps in the review. A causallm model does not provide that of course, just generated tokens, so we got to be creative and came up with a possible solution. (We find this setup with an LLM is significantly more accurate than a RAG using dedicated bert-based QA models, which do provide scores.) We do the following: But for this to work we need the unprocessed logits. B/c after warping (and using this query) normally only one token remains, yes or no, which has the (renormalized) probability 1, so the confidence score is always 0 or 1. That does not help us. So I'm using a causallm model in a somewhat unconventional way. But one that I believe is very useful, in the sense that it provides a (much needed!) functionality that is otherwise missing for RAGs (as far as I have seen). Hope this makes it more clear! 🙂 There may be a different way to do this, but I'm afraid I am not aware of it. Hence the PR for unprocessed logits, which for the reason above I think this would a useful addition to have. |
|
This would be very beneficial for my current project, I would love this PR to be merged. |
|
upvote |
|
It seems I was wrong, and several people do want it :) Reverting my decision, I'll review the PR so as to include the feature. @mbaak indeed, with sampling the @vwxyzjn given your comment in the other issue, this might be useful to you :) |
gante
left a comment
gante
left a comment
There was a problem hiding this comment.
Thank you for the contribution 💛
I've commented a few suggestions, they should be simple to implement!
|
Thanks for the comments! I'll pick it up. |
|
@gante I've implemented your feedback, the code should be good to go I think! |
|
@mbaak To make our CI go green, you will need to:
|
|
@gante Done! |
|
@mbaak there seems a missing |
amyeroberts
left a comment
amyeroberts
left a comment
There was a problem hiding this comment.
Thanks for adding this feature - LGTM!
Just some small nit comments to address, but otherwise looks ready for merge
|
@gante Fyi I'm working on fixing the tests. (By default they're skipped locally, that's why I missed them earlier.) |
output_logits option behaves like output_scores, but returns the raw, unprocessed prediction logit scores, ie. the values before they undergo logit processing and/or warping. The latter happens by default for the regular output scores. It's useful to have the unprocessed logit scores in certain circumstances. For example, unprocessed logit scores are very useful with causallm models when one wants to determine the probability of a certain answer, e.g. when asking a question with a yes/no answer. In that case getting the next-token probabilities of both "yes" and "no" (and/or their relative ratio) is of interest for classification. The reason for getting these _before_ logit processing and/or warping is b/c a) that can change the probabilities or b) reject the tokens of interest / reduce the number of tokens to just 1. For an example use-case see paper TabLLM: Few-shot Classification of Tabular Data with Large Language Models by Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. https://arxiv.org/abs/2210.10723 In addition: - added dedicated unit test: tests/generation/test_utils/test_return_unprocessed_logit_scores which tests return of logics with output_logits=True in generation. - set output_logits=True in all other generation unit tests, that also have output_scores=True. Implemented @gante's and @amyeroberts review feedback
|
@gante I fixed the CI tests (I had only run the generation/ tests locally, not the model ones), and have implemented @amyeroberts' feedback. I think it's good to go now! |
|
@mbaak thank you for iterating and making the library richer 💛 |
6 participants
What does this PR do?
output_logits option behaves like output_scores, but returns the raw, unprocessed prediction logit scores, ie. the values before they undergo logit processing and/or warping. The latter happens by default for the regular output scores.
It's useful to have the unprocessed logit scores in certain circumstances. For example, unprocessed logit scores are very useful with causallm models when one wants to determine the probability of a certain answer, e.g. when asking a question with a yes/no answer. In that case getting the next-token probabilities of both "yes" and "no" (and/or their relative ratio) is of interest for classification. The reason for getting these before logit processing and/or warping is b/c a) that can change the probabilities or b) reject the tokens of interest / reduce the number of tokens to just 1.
In practice this can be used to generate confidence / classification scores when eg. using causallm models for question-answering tasks. Query your language model with: "Is the {statement} correct? Answer yes or no:", take the raw logit scores and softmax them, and calculate the score: prob(yes) / (prob(yes) + prob(no)) to get a useful classification score.
For an example use-case see paper TabLLM: Few-shot Classification of Tabular Data with Large Language Models by Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. https://arxiv.org/abs/2210.10723
In addition:
Fixes # (issue)
NA
Before submitting
Pull Request section?
to it if that's the case.
(Yes, I've seen it discussed but now cannot refind the link.)
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.