Add schemas support to OpenAI models#16
Conversation
MattKotsenas
force-pushed
the
feature/schemas
branch
from
6aba92a to
05e80d2
Compare
|
@tonybaloney, would you mind taking a look at this? If you have any questions, please let me know. Thanks! |
|
Yes, sorry I've been really busy the last week or so |
|
No worries! |
|
I see now that this should all be driven off the models.json file. I have a script incoming to automate downloading a new version, and then I'll update this to match. |
|
Can you update the script to populate this field, also I added an extra formatting step ( |
|
Updating the script and re-running it will make rebasing a lot easier |
MattKotsenas
force-pushed
the
feature/schemas
branch
from
05e80d2 to
4764577
Compare
|
Updated! |
tonybaloney
left a comment
tonybaloney
left a comment
There was a problem hiding this comment.
Thanks for this. Works really well on 4.1 series models
There was a problem hiding this comment.
Pull Request Overview
Adds support for JSON schema-based responses in GitHub-hosted OpenAI models, enabling schema validation for newer models.
- Introduces
supports_schemasin the model‐parsing tool - Extends
CHAT_MODELStuples andGitHubModelsto include asupports_schemaflag - Updates
executeto generateJsonSchemaFormatwhenprompt.schemais provided - Adds tests for supported and unsupported schema usage
- Bumps
llmdependency to>=0.23
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tools/parse_models_json.py | Added supports_schemas helper |
| tests/test_llm_github_models.py | Imported BaseModel, added schema tests |
| pyproject.toml | Bumped llm dependency version |
| llm_github_models.py | Expanded model tuple entries, added supports_schema handling in execute |
Comments suppressed due to low confidence (5)
tools/parse_models_json.py:17
- The function name
supports_schemasdoesn't match the flagsupports_schemaused elsewhere. Consider renaming it tosupports_schemafor consistency.
def supports_schemas(name):
llm_github_models.py:268
- There's no test covering the branch where a raw dict schema is passed. Add a test to verify
prompt.schemaas a dict is correctly forwarded toJsonSchemaFormat.
schema=prompt.schema # type: ignore[variable]
llm_github_models.py:281
- The
chunkslist is created but never used. Consider removing it or utilizing it to populateresponse.response_jsoninstead of theTODOplaceholder.
chunks = []
pyproject.toml:12
- The code now imports and uses
pydantic.BaseModelbutpydanticisn't declared as a dependency. Please addpydantic>=1.xtodependencies.
"llm>=0.23",
llm_github_models.py:250
- Changing the
api_versionglobally may break compatibility for models that required a special preview version previously (e.g.,o3-mini). Confirm that the new version works for all models or reintroduce model-specific overrides.
extra["api_version"] = "2025-03-01-preview" # Use latest version
Co-authored-by: Copilot <[email protected]>
2 participants
Fixes #15
Add schemas support. Via a quick test, only the newer OpenAI models support JsonSchemaFormat currently.