Repository navigation
Conversation
`_ensure_json_quoted` wrapped `String` values in quote characters by hand, so values containing `"`, `\` or control characters were quoted but not escaped. The generated regex then only matched the unescaped text: `List[Literal['say "hi"']]` accepted `["say "hi""]`, which no JSON parser can read, and rejected `["say \"hi\""]`, the encoding `json.dumps` produces. Dict keys/values and Tuple elements use the same helper. Encode the value with `json.dumps(..., ensure_ascii=False)` so escapes match JSON (`ensure_ascii=False` keeps non-ASCII characters literal, as the `outlines_core` JSON-Schema path and the previous behaviour both do).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 Thank you for opening a PR!
maincommit;pre-commitis installed and configured on your machine, and you ran it before opening the PR;Summary
_ensure_json_quotednow encodesStringvalues withjson.dumpsinstead ofwrapping them in quote characters by hand, so a literal containing
",\ora control character produces the escape sequence JSON requires.
Why
_ensure_json_quoted(src/outlines/types/dsl.py:880) built the quoted form asString(f'"{term.value}"'). That adds the delimiters but never escapes thevalue, so the generated regex only matched the unescaped text. For
list[Literal['say "hi"']]the pattern came out as:re.fullmatchon["say \"hi\""]— the encodingjson.dumpsproduces, and theonly form a JSON parser accepts — returns
None. The pattern instead accepts["say "hi""], which no parser can read.dict[Literal['say "hi"'], str]iswrong the same way for its keys, and
Tuple[Literal[...]]for its elements,since all three go through the same helper.
A literal containing a backslash, a tab or a newline is in the same position:
_json_string_builtin_patternsin #2015 and the corresponding section of #1962already note that these values need escaping on top of quoting; that PR scopes
itself to built-in
Regexterms and leaves theStringbranch untouched, sothis is the other half.
Changes
src/outlines/types/dsl.py:_ensure_json_quotedreturnsString(json.dumps(term.value, ensure_ascii=False)).tests/types/test_dsl.py: two tests coveringList,DictandTuple.ensure_ascii=Falsekeeps non-ASCII characters literal, matching the previousoutput and the
outlines_coreJSON-Schema path;json.dumpswith ASCII valuesreproduces the old string exactly, so nothing outside the
"/\/control-character cases changes.
Testing
No issue is filed for this; it is the
Stringhalf of the escaping gapdescribed in #1962, tracked there and explicitly left open by #2015.