Skip to content

fix return_token_log_probs on vLLM > 0.3.3 endpoints#498

Merged
saiatmakuri merged 3 commits intomainfrom
saiatmakuri/fix-return_token_log_probs
Apr 23, 2024
Merged

fix return_token_log_probs on vLLM > 0.3.3 endpoints#498
saiatmakuri merged 3 commits intomainfrom
saiatmakuri/fix-return_token_log_probs

Conversation

@saiatmakuri
Copy link
Copy Markdown
Contributor

@saiatmakuri saiatmakuri commented Apr 23, 2024

Pull Request Summary

since vLLM > 0.3.3, logprobs are returned as as type Logprob instead as direct float's. it's an API-breaking change (see vllm-project/vllm#3065 (comment))

change vllm_server to return output in the same previous format

Test Plan and Usage Guide

create new llama-2-7b with vLLM==0.4.0.post1

Completion.create(
    model="llama-2-7b",
    prompt="Hello ",
    max_new_tokens=10,
    temperature=0.0,
    return_token_log_probs=True,
)

@saiatmakuri saiatmakuri self-assigned this Apr 23, 2024
@saiatmakuri saiatmakuri requested a review from dmchoiboi April 23, 2024 04:30
yunfeng-scale
yunfeng-scale approved these changes Apr 23, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@yunfeng-scale yunfeng-scale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the fix! i think we need to do some backfills also?

@saiatmakuri
Copy link
Copy Markdown
Contributor Author

saiatmakuri commented Apr 23, 2024

thanks for the fix! i think we need to do some backfills also?

will update endpoints to latest image post merge

@saiatmakuri saiatmakuri merged commit 10d84ca into main Apr 23, 2024
@saiatmakuri saiatmakuri deleted the saiatmakuri/fix-return_token_log_probs branch April 23, 2024 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@seanshi-scale seanshi-scale Awaiting requested review from seanshi-scale

@dmchoiboi dmchoiboi Awaiting requested review from dmchoiboi

1 more reviewer

@yunfeng-scale yunfeng-scale yunfeng-scale approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@saiatmakuri saiatmakuri

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@saiatmakuri @yunfeng-scale