Skip to content

guided decoding with grammar#488

Merged
saiatmakuri merged 4 commits intomainfrom
saiatmakuri/vllm-guided-decoding-grammar
Apr 29, 2024
Merged

guided decoding with grammar#488
saiatmakuri merged 4 commits intomainfrom
saiatmakuri/vllm-guided-decoding-grammar

Conversation

@saiatmakuri
Copy link
Copy Markdown
Contributor

Pull Request Summary

Support guided_grammar from vLLM

Test Plan and Usage Guide

tested within vllm entrypoint and from model engine

simple_sql_grammar = """
start: select_statement

select_statement: "SELECT" column "from" table "where" condition

column: "col_1" | "col_2"
table: "table_1" | "table_2"
condition: column "=" number

number: "1" | "2"
"""

requests.post(
  "http://localhost:5000/v1/llm/completions-sync?model_endpoint_name=mixtral-8x7b-instruct-test",
  json={
    "prompt": "Generate a sql state that select col_1 from "",
    "max_new_tokens": 10,
    "temperature": 0.2,
    "guided_grammar": simple_sql_grammar
  },
  auth=("<USER_ID>", '')
).json()

output:

{
	"request_id": "343b5e82-524c-4100-ba0d-57af51d33e48",
	"output": {
		"text": "SELECTcol_1fromtable_1wherecol_2=1",
		"num_prompt_tokens": 12,
		"num_completion_tokens": 17,
		"tokens": null
	}
}

@saiatmakuri saiatmakuri marked this pull request as ready for review April 23, 2024 05:36
@@ -1,2 +1,2 @@
vllm==0.4.0.post1
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happened to post1?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

saw this bug in 0.4.0.post1 - vllm-project/vllm#3953

@saiatmakuri saiatmakuri merged commit 866bcd1 into main Apr 29, 2024
@saiatmakuri saiatmakuri deleted the saiatmakuri/vllm-guided-decoding-grammar branch April 29, 2024 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants