Make it so callable JSON schema extra works by dmontagu · Pull Request #6798 · pydantic/pydantic

dmontagu · 2023-07-21T20:28:55Z

Note that as mentioned here, the better solution for FastAPI is to use a custom GenerateJsonSchema that generates the JSON schema differently for Optional query params, etc.. But this at least makes it possible to express what you want on the pydantic side via:

from typing import Annotated

from pydantic import BaseModel, Field
from pydantic.json_schema import SkipJsonSchema


class Model(BaseModel):
    x: int | SkipJsonSchema[None] = Field(default=None, json_schema_extra=lambda x: x.pop('default'))

print(Model.model_json_schema())
#> {'properties': {'x': {'title': 'X', 'type': 'integer'}}, 'title': 'Model', 'type': 'object'}

Selected Reviewer: @samuelcolvin

cloudflare-workers-and-pages · 2023-07-21T20:30:46Z

Deploying with Cloudflare Pages

Latest commit:	`6745019`
Status:	✅ Deploy successful!
Preview URL:	https://19f679d4.pydantic-docs2.pages.dev
Branch Preview URL:	https://field-json-schema-extra-call.pydantic-docs2.pages.dev

View logs

dmontagu · 2023-07-21T20:58:01Z

please review

dmontagu · 2023-07-21T20:59:20Z

            new_kwargs.update(field_info._attributes_set)
+            for x in field_info.metadata:
+                if not isinstance(x, FieldInfo):
+                    metadata[type(x)] = x


I ran into a case where this was necessary to get proper merging of FieldInfos that use gt, lt, etc.

This stuff is getting really annoying. @samuelcolvin not having to do this stuff is one of the big advantages of the Output enum thing we were discussing the other day.

dmontagu · 2023-07-21T21:00:32Z

            return maybe_updated_schema
        return original_schema

+    def apply_single_annotation_json_schema(


I had to break this out as a separate function in order to ensure the callable JSON schema ran after all modifications to the core schema that influence the json schema are performed. Otherwise, for example, you can't remove the default in the callable.

dmontagu · 2023-07-21T21:01:26Z

        def new_handler(source: Any) -> core_schema.CoreSchema:
            schema = metadata_get_schema(source, get_inner_schema)
            schema = self.apply_single_annotation(schema, annotation)
+            schema = self.apply_single_annotation_json_schema(schema, annotation)


As you can see here, everything related to JSON schema now runs after everything not related to JSON schema.

No typo as far as I can tell — the self.apply_single_annotation does everything not related to JSON schema, then after that, self.apply_single_annotation_json_schema does everything related to json schema. I think that's what I said in the comment above?

dmontagu · 2023-07-21T21:04:14Z

+        d: Annotated[int, Field(default=4), Field(json_schema_extra=pop_default)]
+        # TODO: it doesn't work properly to have both annotation and assigned value of FieldInfo:
+        # e: Annotated[int, Field(json_schema_extra=pop_default)] = Field(default=5)
+        # f: Annotated[int, Field(default=6)] = Field(json_schema_extra=pop_default)


I think it would be possible to make this and the previous case work (i.e., e and f), but I tried and it ended up being a significantly bigger change and I wasn't able to see it all the way through and get all tests passing. So rather than wasting time on it for now, I just added a note here. Happy to create an issue for this if there isn't one already. (CC @adriangb)

While it might be tempting to try to make this an error, for various reasons, I think it would be preferable if assigning a FieldInfo was equivalent to adding it as the last annotation on the field (and that does work, as you can see in the handling of attributes c and d).

I do think we should make it work at some point. Is it not working now? I thought it was.

It works for most things, it just doesn't work for json_schema_extra and default specifically, annoyingly.

Also, even on main, this doesn't work:

from pydantic import BaseModel, Field from typing_extensions import Annotated class Model(BaseModel): a: Annotated[int, Field(default=1)] = Field() Model()

pydantic_core._pydantic_core.ValidationError: 1 validation error for Model a Field required [type=missing, input_value={}, input_type=dict]

I was able to fix this through an appropriate change in FieldInfo.from_annotated_attribute 💪

adriangb · 2023-07-22T10:29:04Z

Can’t this go in

pydantic/pydantic/json_schema.py

Line 2174 in 6ef959f

json_schema: JsonSchemaValue | None

? To avoid making Field even more complex.

samuelcolvin

Looks fine, I'm a bit lost on a few things but if you're confident, let's merge.

samuelcolvin · 2023-07-24T19:36:51Z

+        if isinstance(field_info.json_schema_extra, dict):
+            json_schema_updates.update(field_info.json_schema_extra)


why don't we call field_info.json_schema_extra if it's a function here?

okay so, in the update func (def json_schema_update_func below), we have:

json_schema = {**handler(schema), **json_schema_updates}

and handler(schema) may result in a different schema than just the updates; json_schema_updates is not the whole json schema. In particular, getting stuff like "type": "object" (or even other types for RootModel subclasses), you probably want that to be accessible in the json_schema_extra call, which is why that needs to happen after the call to handler in the json_schema_update_func below.

samuelcolvin · 2023-07-24T19:38:09Z

+            if callable(field_info.json_schema_extra):
+                field_info.json_schema_extra(json_schema)


again, I'm a bit lost why we don't update do json_schema.update(field_info.json_schema_extra) here if it's a dict?

If you sure these are fine, great.

I think we can clean this up, I'll push a commit doing that shortly

…ds and dataclasses

Make it so callable JSON schema extra works

7798274

dmontagu mentioned this pull request Jul 21, 2023

✨ Add SkipJsonSchema annotation #6653

Merged

dmontagu added 2 commits July 21, 2023 14:38

Merge branch 'main' into field-json-schema-extra-callable

d94a494

Add test for removing default from JSON schema

bfd0722

dmontagu mentioned this pull request Jul 21, 2023

anyOf causing problems with Swagger/ReDoc on primitive types #6647

Closed

1 task

pydantic-hooky Bot added the ready for review label Jul 21, 2023

pydantic-hooky Bot assigned samuelcolvin Jul 21, 2023

dmontagu commented Jul 21, 2023

View reviewed changes

samuelcolvin approved these changes Jul 24, 2023

View reviewed changes

Simplify json_schema_extra application, and fix some issues with fiel…

6745019

…ds and dataclasses

dmontagu merged commit 2181d12 into main Jul 24, 2023

dmontagu deleted the field-json-schema-extra-callable branch July 24, 2023 20:59

		if isinstance(field_info.json_schema_extra, dict):
		json_schema_updates.update(field_info.json_schema_extra)

		if callable(field_info.json_schema_extra):
		field_info.json_schema_extra(json_schema)

Uh oh!

Conversation

dmontagu commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Pages

Uh oh!

dmontagu commented Jul 21, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmontagu Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmontagu Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Jul 22, 2023

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmontagu Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dmontagu commented Jul 21, 2023 •

edited

Loading

cloudflare-workers-and-pages Bot commented Jul 21, 2023 •

edited

Loading

dmontagu Jul 21, 2023 •

edited

Loading

dmontagu Jul 21, 2023 •

edited

Loading

dmontagu Jul 24, 2023 •

edited

Loading