Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 92 additions & 15 deletions docs/concepts/serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -563,9 +563,10 @@ print(m.model_dump_json())

### Subclasses of model-like types

When using model-like classes (Pydantic models, dataclasses, etc.) as field annotations, the default behavior is to
serializer the field value as though it was an instance of the class, even if it is a subclass. More specifically,
only the fields declared on the type annotation will be included in the serialization result:
When using model-like classes (Pydantic models, [dataclasses](./dataclasses.md), etc.) as field annotations,
the default behavior is to serialize the field value as though it was an instance of the class used as the
annotation, even if it is a subclass. More specifically, only the fields declared on the type annotation
will be included in the serialization result:

```python
from pydantic import BaseModel
Expand Down Expand Up @@ -602,27 +603,103 @@ print(m.model_dump()) # (1)!
when adding sensitive information like secrets as fields of subclasses. To enable the old V1 behavior, refer
to the next section.

### Serializing with duck typing 🦆
<!-- old anchor added for backwards compatibility -->
<!-- markdownlint-disable-next-line no-empty-links -->
[](){#serializing-with-duck-typing}

### Polymorphic serialization

/// version-added | v2.12
Polymorphic serialization was added as an better alternative to the [serialize as any](#serializing-as-any) behavior, and only
applies to Pydantic models and Pydantic dataclasses.
///

Polymorphic serialization is the behavior of serializing a model (or [Pydantic dataclass](./dataclasses.md)) instance
according to the serialization schema of such instance, rather that the schema of the class used as the type.
This will expose all the data defined on the subclass in the serialized payload.

This behavior can be configured in the following ways:

* Configuration level: use the [`polymorphic_serialization`][pydantic.config.ConfigDict.polymorphic_serialization] setting
in the model/dataclass [configuration](./config.md).
* Runtime level: use the `polymorphic_serialization` argument when calling the [serialization methods](#serializing-data).
This will apply to all (nested) types, overriding any configuration.

!!! note "Duck-typed serialization"
This behavior (and the ["any" serialization](#serializing-as-any) discussed below) was previously referred
to as duck-typed serialization. This was a misnomer; it did not function like
[duck typing](https://en.wikipedia.org/wiki/Duck_typing) in the conventional programming language sense.

!!! warning Polymorphic serialization of standard library dataclasses
Polymorphic serialization is only supported for Pydantic models and [Pydantic dataclasses](./dataclasses.md).
When using [standard library dataclasses][dataclasses], polymorphic serialization is *not* supported,
even if the dataclass is a subclass of a Pydantic dataclass. This may be fixed in a future Pydantic release.
Comment on lines +633 to +636
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How hard would it be to fallback to inference (as serialize_as_any does) in this case? I'm worried we recommend switching from serialize_as_any to polymorphic_serialization but then users discover that it's "less powerful" in this case.

I guess it's a trade-off between not having it supported for now (and being able to implement it properly later), and having the limited inference behavior now (and users could as such rely on it) but such behavior changing when we implement it properly (and so is a behavior change for users already relying on it).


The example below defines a type `User` and a subclass of it, `UserLogin`. A second pair of types, `PolymorphicUser`
and `PolymorphicUserLogin` are defined as equivalents with `polymorphic_serialization` enabled.

We can then see the effect of serializing each of these types, and the interaction of this config with the runtime
`polymorphic_serialization` setting:

```python
from pydantic import BaseModel


class User(BaseModel):
name: str


class UserLogin(User):
password: str


class OuterModel(BaseModel):
user: User


outer_model = OuterModel(
user=UserLogin(name='pydantic', password='password'),
)


print(outer_model.model_dump()) # (1)!
#> {'user': {'name': 'pydantic'}}
print(outer_model.model_dump(polymorphic_serialization=True)) # (2)!
#> {'user': {'name': 'pydantic', 'password': 'password'}}
```

1. With polymorphic serialization disabled, `user` serializes as the base type.
2. With polymorphic serialization enabled, `user` serializes as the actual runtime subclass.

As seen in the example, by having polymorphic serialization enabled, the `User.model_dump()` method will by respect the value
of the `UserLogin` subclass when it is provided instead of a `User` value, and serialize the full `UserLogin` type. This
behavior can be globally overridden with the `polymorphic_serialization` runtime setting; in this case setting it to `False`
causes the `UserLogin` value to serialize just as a `User` value, ignoring the subclass' `password` field.

## Serializing "as Any"

A more extreme form of [polymorphic serialization](#polymorphic-serialization) is "any" serialization. In this
mode, Pydantic does *not* make use of any type annotation (more precisely, the serialization schema derived from
the type) to infer how the value should be serialized, but instead inspects the actual type of the value at runtime
to do so (and this applies to *all* types, not only Pydantic models and dataclasses).

Duck typing serialization is the behavior of serializing a model instance based on the actual field values, rather
than the field definitions. This means that for a field annotated with a model-like class, all the fields present
in subclasses of such class will be included in the serialized output.
This means that every value will be serialized exactly based on its runtime type and any knowledge Pydantic has
of how to serialize the type. Pydantic can infer how to serialize the following types:

To achieve duck typing serialization, Pydantic can apply *serialize as any* behavior. In this mode, Pydantic does
*not* make use of the type annotation (more precisely, the serialization schema derived from the type) to infer
how the value should be serialized, but instead inspects the actual type of the value at runtime to do so.
* Many Python standard library types (exact set may be expanded depending on Pydantic version).
* Types with a `__pydantic_serializer__` attribute.
* Any type serializable with the `fallback` function passed as an argument to [serialization methods](#serializing-data).

When a subclass of a model is used as a value, Pydantic will *not* serialize it according to the schema of the
parent class, but rather use the value itself and preserve all of its fields.
In most cases, you will want to use the [polymorphic serialization](#polymorphic-serialization) behavior instead.

This behavior can be configured at the field level and at runtime, for a specific serialization call:

* Field level: use the [`SerializeAsAny`][pydantic.functional_serializers.SerializeAsAny] annotation.
* Runtime level: use the `serialize_as_any` argument when calling the [serialization methods](#serializing-data).

We discuss these options below in more detail:
These options are discussed below in more detail.

#### `SerializeAsAny` annotation
### `SerializeAsAny` annotation

If you want duck typing serialization behavior, this can be done using the
[`SerializeAsAny`][pydantic.functional_serializers.SerializeAsAny] annotation
Expand Down Expand Up @@ -661,7 +738,7 @@ annotated as `<type>`, and static type checkers will treat the annotation as if
When serializing, the field will be serialized as though the type hint for the field was [`Any`][typing.Any],
which is where the name comes from.

#### `serialize_as_any` runtime setting
### `serialize_as_any` runtime setting

The `serialize_as_any` runtime setting can be used to serialize model data with or without duck typed serialization behavior.
`serialize_as_any` can be passed as a keyword argument to the various [serialization methods](#serializing-data) (such as
Expand Down
10 changes: 5 additions & 5 deletions pydantic-core/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -141,11 +141,11 @@ branch = true

[tool.coverage.report]
precision = 2
exclude_lines = [
'pragma: no cover',
'raise NotImplementedError',
'if TYPE_CHECKING:',
'@overload',
exclude_also = [
'raise\sNotImplementedError',
'@(typing\.)?overload',
'class .*\bProtocol\):',
'(typing\.)?.assert_never',
]

# configuring https://github.com/pydantic/hooky
Expand Down
8 changes: 8 additions & 0 deletions pydantic-core/python/pydantic_core/_pydantic_core.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,7 @@ class SchemaSerializer:
warnings: bool | Literal['none', 'warn', 'error'] = True,
fallback: Callable[[Any], Any] | None = None,
serialize_as_any: bool = False,
polymorphic_serialization: bool | None = None,
context: Any | None = None,
) -> Any:
"""
Expand All @@ -347,6 +348,7 @@ class SchemaSerializer:
fallback: A function to call when an unknown value is encountered,
if `None` a [`PydanticSerializationError`][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any: Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization: Whether to use model and dataclass polymorphic serialization for this call.
context: The context to use for serialization, this is passed to functional serializers as
[`info.context`][pydantic_core.core_schema.SerializationInfo.context].

Expand All @@ -373,6 +375,7 @@ class SchemaSerializer:
warnings: bool | Literal['none', 'warn', 'error'] = True,
fallback: Callable[[Any], Any] | None = None,
serialize_as_any: bool = False,
polymorphic_serialization: bool | None = None,
context: Any | None = None,
) -> bytes:
"""
Expand All @@ -397,6 +400,7 @@ class SchemaSerializer:
fallback: A function to call when an unknown value is encountered,
if `None` a [`PydanticSerializationError`][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any: Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization: Whether to use model and dataclass polymorphic serialization for this call.
context: The context to use for serialization, this is passed to functional serializers as
[`info.context`][pydantic_core.core_schema.SerializationInfo.context].

Expand Down Expand Up @@ -427,6 +431,7 @@ def to_json(
serialize_unknown: bool = False,
fallback: Callable[[Any], Any] | None = None,
serialize_as_any: bool = False,
polymorphic_serialization: bool | None = None,
context: Any | None = None,
) -> bytes:
"""
Expand Down Expand Up @@ -455,6 +460,7 @@ def to_json(
fallback: A function to call when an unknown value is encountered,
if `None` a [`PydanticSerializationError`][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any: Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization: Whether to use model and dataclass polymorphic serialization for this call.
context: The context to use for serialization, this is passed to functional serializers as
[`info.context`][pydantic_core.core_schema.SerializationInfo.context].

Expand Down Expand Up @@ -512,6 +518,7 @@ def to_jsonable_python(
serialize_unknown: bool = False,
fallback: Callable[[Any], Any] | None = None,
serialize_as_any: bool = False,
polymorphic_serialization: bool | None = None,
context: Any | None = None,
) -> Any:
"""
Expand All @@ -538,6 +545,7 @@ def to_jsonable_python(
fallback: A function to call when an unknown value is encountered,
if `None` a [`PydanticSerializationError`][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any: Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization: Whether to use model and dataclass polymorphic serialization for this call.
context: The context to use for serialization, this is passed to functional serializers as
[`info.context`][pydantic_core.core_schema.SerializationInfo.context].

Expand Down
7 changes: 7 additions & 0 deletions pydantic-core/python/pydantic_core/core_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ class CoreConfig(TypedDict, total=False):
validate_by_alias: Whether to use the field's alias when validating against the provided input data. Default is `True`.
validate_by_name: Whether to use the field's name when validating against the provided input data. Default is `False`. Replacement for `populate_by_name`.
serialize_by_alias: Whether to serialize by alias. Default is `False`, expected to change to `True` in V3.
polymorphic_serialization: Whether to enable polymorphic serialization for models and dataclasses. Default is `False`.
url_preserve_empty_path: Whether to preserve empty URL paths when validating values for a URL type. Defaults to `False`.
"""

Expand Down Expand Up @@ -120,6 +121,7 @@ class CoreConfig(TypedDict, total=False):
validate_by_alias: bool # default: True
validate_by_name: bool # default: False
serialize_by_alias: bool # default: False
polymorphic_serialization: bool # default: False
url_preserve_empty_path: bool # default: False


Expand Down Expand Up @@ -181,6 +183,11 @@ def serialize_as_any(self) -> bool:
"""The `serialize_as_any` argument set during serialization."""
...

@property
def polymorphic_serialization(self) -> bool | None:
"""The `polymorphic_serialization` argument set during serialization, if any."""
...
Comment thread
Viicos marked this conversation as resolved.

@property
def round_trip(self) -> bool:
"""The `round_trip` argument set during serialization."""
Expand Down
1 change: 1 addition & 0 deletions pydantic-core/src/errors/validation_exception.rs
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,7 @@ impl ValidationError {
None,
false,
None,
None,
);
let mut state = SerializationState::new(config, WarningsMode::None, None, None, extra)?;
let mut serializer = ValidationErrorSerializer {
Expand Down
7 changes: 7 additions & 0 deletions pydantic-core/src/serializers/extra.rs
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,8 @@ pub(crate) struct Extra<'py> {
pub serialize_unknown: bool,
pub fallback: Option<Bound<'py, PyAny>>,
pub serialize_as_any: bool,
/// Whether `polymorphic_serialization` is globally enabled / disabled for this serialization process
pub polymorphic_serialization: Option<bool>,
pub context: Option<Bound<'py, PyAny>>,
}

Expand All @@ -202,6 +204,7 @@ impl<'py> Extra<'py> {
serialize_unknown: bool,
fallback: Option<Bound<'py, PyAny>>,
serialize_as_any: bool,
polymorphic_serialization: Option<bool>,
context: Option<Bound<'py, PyAny>>,
) -> Self {
Self {
Expand All @@ -217,6 +220,7 @@ impl<'py> Extra<'py> {
serialize_unknown,
fallback,
serialize_as_any,
polymorphic_serialization,
context,
}
}
Expand Down Expand Up @@ -260,6 +264,7 @@ pub(crate) struct ExtraOwned {
serialize_unknown: bool,
pub fallback: Option<Py<PyAny>>,
serialize_as_any: bool,
polymorphic_serialization: Option<bool>,
pub context: Option<Py<PyAny>>,
include: Option<Py<PyAny>>,
exclude: Option<Py<PyAny>>,
Expand Down Expand Up @@ -294,6 +299,7 @@ impl ExtraOwned {
fallback: extra.fallback.clone().map(Bound::unbind),
serialize_as_any: extra.serialize_as_any,
context: extra.context.clone().map(Bound::unbind),
polymorphic_serialization: extra.polymorphic_serialization,
include: state.include().map(|m| m.clone().into()),
exclude: state.exclude().map(|m| m.clone().into()),
}
Expand All @@ -314,6 +320,7 @@ impl ExtraOwned {
fallback: self.fallback.as_ref().map(|m| m.bind(py).clone()),
serialize_as_any: self.serialize_as_any,
context: self.context.as_ref().map(|m| m.bind(py).clone()),
polymorphic_serialization: self.polymorphic_serialization,
}
}

Expand Down
29 changes: 20 additions & 9 deletions pydantic-core/src/serializers/infer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ pub(crate) fn infer_to_python_known<'py>(
let uuid = super::type_serializers::uuid::uuid_to_string(value)?;
uuid.into_py_any(py)?
}
ObType::PydanticSerializable => call_pydantic_serializer(value, state, serialize_to_python(py))?,
ObType::PydanticSerializable => serialize_pydantic_serializable(value, state, serialize_to_python(py))?,
ObType::Dataclass => infer_serialize_dataclass(value, state, serialize_to_python(py))?,
ObType::Enum => {
let v = value.getattr(intern!(py, "value"))?;
Expand Down Expand Up @@ -236,7 +236,7 @@ pub(crate) fn infer_to_python_known<'py>(
let dict = value.cast::<PyDict>()?;
serialize_pairs(dict.iter().map(Ok), state, serialize_to_python(py))?
}
ObType::PydanticSerializable => call_pydantic_serializer(value, state, serialize_to_python(py))?,
ObType::PydanticSerializable => serialize_pydantic_serializable(value, state, serialize_to_python(py))?,
ObType::Dataclass => infer_serialize_dataclass(value, state, serialize_to_python(py))?,
ObType::Generator => {
let iter = super::type_serializers::generator::SerializationIterator::new(
Expand Down Expand Up @@ -421,7 +421,7 @@ pub(crate) fn infer_serialize_known<'py, S: Serializer>(
| ObType::Ipv4Network
| ObType::Ipv6Network => serialize_via_str(value, serialize_to_json(serializer)).map_err(unwrap_ser_error),
ObType::PydanticSerializable => {
call_pydantic_serializer(value, state, serialize_to_json(serializer)).map_err(unwrap_ser_error)
serialize_pydantic_serializable(value, state, serialize_to_json(serializer)).map_err(unwrap_ser_error)
}
ObType::Dataclass => {
infer_serialize_dataclass(value, state, serialize_to_json(serializer)).map_err(unwrap_ser_error)
Expand Down Expand Up @@ -592,25 +592,36 @@ pub(crate) fn infer_json_key_known<'a, 'py>(
}
}

pub(crate) fn get_pydantic_serializer<'py>(value: &Bound<'py, PyAny>) -> PyResult<Bound<'py, SchemaSerializer>> {
let py = value.py();
let py_serializer = value.getattr(intern!(py, "__pydantic_serializer__"))?;
py_serializer.cast_into_exact().map_err(Into::into)
}

/// Serialize `value` as if it had a `__pydantic_serializer__` attribute
///
/// `do_serialize` should be a closure which performs serialization without type inference
pub(crate) fn call_pydantic_serializer<'py, S: DoSerialize>(
fn serialize_pydantic_serializable<'py, S: DoSerialize>(
value: &Bound<'py, PyAny>,
state: &mut SerializationState<'py>,
do_serialize: S,
) -> Result<S::Ok, S::Error> {
let py = value.py();
let py_serializer = value.getattr(intern!(py, "__pydantic_serializer__"))?;
call_pydantic_serializer(py_serializer.cast().map_err(Into::into)?, value, state, do_serialize)
}

let extracted_serializer: PyRef<SchemaSerializer> = py_serializer.extract().map_err(Into::into)?;

// Use current serialization state, except with the config from the extracted serializer
let state = &mut state.scoped_set(|s| &mut s.config, extracted_serializer.config);
pub(crate) fn call_pydantic_serializer<'py, S: DoSerialize>(
serializer: &Bound<'py, SchemaSerializer>,
value: &Bound<'py, PyAny>,
state: &mut SerializationState<'py>,
do_serialize: S,
) -> Result<S::Ok, S::Error> {
let state = &mut state.scoped_set(|s| &mut s.config, serializer.get().config);

// Avoid falling immediately back into inference because we need to use the serializer
// to drive the next step of serialization
do_serialize.serialize_no_infer(&extracted_serializer.serializer, value, state)
do_serialize.serialize_no_infer(&serializer.get().serializer, value, state)
}

fn serialize_pattern<S: DoSerialize>(value: &Bound<'_, PyAny>, do_serialize: S) -> Result<S::Ok, S::Error> {
Expand Down
Loading
Loading