Skip to content

Define non-positive top_k; top_k range check #779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ggerganov merged 2 commits into from
Apr 5, 2023
Merged

Define non-positive top_k; top_k range check #779

ggerganov merged 2 commits into from
Apr 5, 2023

Conversation

@ivanstepanovftw
Copy link
Contributor

@ivanstepanovftw ivanstepanovftw commented Apr 5, 2023

  1. With --top_k 0 expected that all logits will be sampled (by top_p)
  2. Range check added to prevent from resizing logits more than vocab have. Resizing to more than vocab have causes token 0 to be appended to logits to be sampled

ggerganov
ggerganov approved these changes Apr 5, 2023
View reviewed changes
@ggerganov ggerganov merged commit 5a8c4f6 into ggml-org:master Apr 5, 2023
@ivanstepanovftw ivanstepanovftw deleted the non-positive-topk branch April 5, 2023 16:20
@Piezoid
Copy link
Contributor

Piezoid commented Apr 6, 2023

@ivanstepanovftw @ggerganov
Correct me if I'm mistaken, but I think the top_p filter needs the logits to be sorted. With this PR, disabling the top_k filter bypasses the sorting.

Instead, the whole array should be sorted when there is no top_k filtering:

sample_top_k(logits_id, top_k > 0 ? std::min(top_k, n_logits) : n_logits);

@ivanstepanovftw
Copy link
Contributor Author

ivanstepanovftw commented Apr 6, 2023

You are right!

@ivanstepanovftw ivanstepanovftw mentioned this pull request Apr 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ivanstepanovftw @Piezoid @ggerganov