from_builtins, but for objects? (e.g instantiate from SQLAlchemy models)

Description

Is there an equivalent to from_builtins, but for objects? Or maybe msgspec.obj.encode / msgspec.obj.decode ?

I'd like to be able to encode/decode into regular Python objects, specifically SQLAlchemy.

Given:

tables.py

from sqlalchemy import ForeignKey
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
from typing import List


# SQLAlchemy 2.0 Models
class Base(DeclarativeBase):
    pass


class GrommetTable(Base):
    __tablename__ = 'grommet'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    widget_id: Mapped[int] = mapped_column(ForeignKey("widget.id"))

    widget: Mapped["WidgetTable"] = relationship(back_populates="grommets")


class WidgetTable(Base):
    __tablename__ = 'widget'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]

    grommets: Mapped[List["GrommetTable"]] = relationship(back_populates="widget")

and:

models.py

from msgspec import Struct


class Grommet(Struct):
    id: int
    name: str
    widget_id: int


class Widget(Struct):
    id: int
    name: str
    grommets: List[Grommet]

and (create test data):

from sqlalchemy import create_engine
from sqlalchemy.orm import Session

from models import Grommet, Widget
from tables import GrommetTable, WidgetTable

engine = create_engine("sqlite+pysqlite:///:memory:", echo=True)

# Create table structure
Base.metadata.create_all(engine)

# Create demo data
g1 = GrommetTable(id=1, name="Grommet 1")
g2 = GrommetTable(id=2, name="Grommet 2")
g3 = GrommetTable(id=3, name="Grommet 3")
g4 = GrommetTable(id=4, name="Grommet 4")
g5 = GrommetTable(id=5, name="Grommet 5")
g6 = GrommetTable(id=6, name="Grommet 6")

w1 = WidgetTable(id=1, name="Widget 1", grommets=[g1, g2, g3])
w2 = WidgetTable(id=2, name="Widget 2", grommets=[g4, g5, g6])

with Session(engine) as session:
    session.add_all([w1, w2])
    session.commit()

It would be great to be able to do something like the below:

from sqlalchemy import select
from sqlalchemy.orm import selectinload

from models import Widget
from tables import WidgetTable

with Session(engine) as session:
    query = select(WidgetTable).options(selectinload(WidgetTable.grommets))
    result = session.execute(query)

    for widget in result.scalars():
        widget = from_obj(widget, Widget)   # NEW FEATURE
        print(widget)

Output:

Widget(id=1, name='Widget 1', grommets=[Grommet(id=1, name='Grommet 1', widget_id=1), Grommet(id=2, name='Grommet 2', widget_id=1), Grommet(id=3, name='Grommet 3', widget_id=1)])
Widget(id=2, name='Widget 2', grommets=[Grommet(id=4, name='Grommet 4', widget_id=2), Grommet(id=5, name='Grommet 5', widget_id=2), Grommet(id=6, name='Grommet 6', widget_id=2)])

The current way to do this would be something like the below implementation, which does work, but feels like there should be something more elegant.

from msgspec import from_builtins
from msgspec.inspect import ListType, type_info

def from_obj(obj, type, _data_only=False):
    data = {}
    for field in type_info(type).fields:
        if isinstance(field.type, ListType):
            value = [from_obj(o, field.type.item_type.cls, _data_only=True) for o in getattr(obj, field.name)]
        else:
            value = getattr(obj, field.name)

        data[field.name] = value

    if _data_only:
        return data
    else:
        return from_builtins(data, type)

I think it's most closely related to the Encode/Decode because the input could have more attributes than the Struct, just as JSON input could have more keys than the Struct when utilizing Schema Evolution

This feature would be useful because it allows a common, very fast internal data structure to be used when reading/writing from common data serialization formats (json, toml, yaml msgpack), but also from a large number of common data storage engines (SQLite, MySQL / MariaDB, PostgreSQL, etc..)

Pydantic v1 has a similar system with from_orm(), which I believe has now been replaced with model_validate() in v2.

Thanks for opening this, this is definitely a feature we could add support for.

If I was to add this, I'd add a new boolean keyword to from_builtins, defaulting to disabled . I'm not sure yet what a good keyword name would be (coerce_objects, attributes, ...), any thoughts appreciated. When this option is enabled, objects may be coerced to the specified types by accessing their attributes.

Two open questions:

msgspec supports renaming fields, which is useful for keys that aren't valid python identifiers (e.g. from) or don't match common naming conventions (e.g. myFieldName ). The original attribute name is still used by python, but the new name is used when encoding/decoding. I think when coercing an object with attributes we'd want to use the original names (my_field_name instead of myFieldName), since that's more likely to match the input objects. But when coercing from a dict the renamed names are used, which may be confusing to users. Alternatively we could make this configurable using the keyword to from_builtins.
How should we handle conversions most efficiently? There's two methods we could take:
- Iterate over all fields in __dict__/__slots__ on the input object, only accessing those that are used by the specified type.
- Iterate over all fields in the specified type, and try to extract them from the object
The first will be more efficient if the output type (e.g. Struct) has more fields than the input object, but won't support extracting attributes defined as properties (though we can work around this). The second will be more efficient if the input object has more attributes than the output type (struct), as we'd only be accessing the requested fields - however each of these will hit a getattr op which will be slightly slower, and any missing fields would still have a getattr call to check for their existence on the input object. In your use case would you expect the attributes on the input type to be a superset or subset of attributes on the output type?

All that said, I'm curious what you need this for. If the intent is to be able to serialize ORM objects to JSON, you can do that more efficiently using an enc_hook, avoiding the need to ever convert them to Struct instances (https://jcristharif.com/msgspec/extending.html). I have some thoughts on how to make this more efficient too in the near future.

The main reason I could see wanting to use from_builtins to convert ORM objects to structs would be if you wanted to write all your business logic on the Struct types rather than the ORM instances.

A keyword argument on from_builtins sounds great. I like attributes=, or maybe attr= for short, given getattr and setattr.

Generally speaking I'm looking for 1 to 1 parity with my SQLAlchemy instances and msgspec instances, so it's hard for me to say which would be bigger -- it's neither option.

For me, your interesting comment was about using getattr vs. not using it. One really great feature of SQLAlchemy is a feature called @hybrid_property

Hybrid properties allow a structure like this:

from sqlalchemy import func
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


# SQLAlchemy 2.0 Models
class Base(DeclarativeBase):
    pass


class UserTable(Base):
    __tablename__ = 'user'

    id: Mapped[int] = mapped_column(primary_key=True)
    first_name: Mapped[str]
    last_name: Mapped[str]

    @hybrid_property
    def full_name(self):
         """
         Evaluated if run by Python, after model instance is instantiated/loaded.
         """
         return f"{self.first_name} {self.last_name}"

   @full_name.expression
    def full_name(cls):
        """
        Evaluated if run by SQL engine in a select() or where() -- great when combined with combined indexes.
        """
        return func.concat(cls.first_name, " ", cls.last_name)

I've found these kinds of properties very useful for doing schema migration across versions, if columns in the database need a layout change, for whatever reason.

So property support in this use case is a big +1 for me.

I love performance as much as the next person, but given that this use case is already pulling from a SQL database, it's never going to be as fast as something like JSON serialization -- I'm willing to sacrifice a little performance for the ability to load data into the more flexible/lightweght Structs in msgspec, which can be fast after initial load.

Another thing to take into account is that SQLAlchemy stores metadata in objects, which looks like this:

{
    '_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x1064ff9b0>, 
   ...
}

So SQLAlchemy model instances will always have at least 1 more key than most users would want to translate to a Struct.

In the end, the idea is to put all the business logic on the Struct types, because it's a more portable place for the logic to live.

I'm targetting an architecture like this:

flowchart LR
  edge--api-->user
  edge--encode-->cache
  cache--decode-->edge
  backend--json api / Struct-->edge
  db--from_builtins-->backend

Maybe edge and backend are in the same process (an ops decision), but they could also be split across some distance as well (a different ops decision). By having a consistent data shape in a Struct it allows for consistent business logic method signatures and that gives some advantage over just attaching business logic to ORM models, as I (and many others) have done in the past.

Background: pydantic uses by_alias= for this use case. (Link - Search by_alias)

I prefer the use of the word rename though. (Which you already use on Struct)

When it comes to aliases / field names, I think it's hard to anticipate why people need them. There are nice clean cases like just translating snake case to camel case, but then there are also situations where either the SQL schema or a JSON API schema are fixed, and the developer just needs to adapt. (The human condition of the developer)

In the most pathological case, a developer might be stuck with both a fixed SQL Layout and a fixed JSON API, and our poor developer is just trying to create sanity in the middle.

In a structure like this:

flowchart LR
    db -->app
    app-->api

In that scenario you might have two parallel msgspec.Struct:

# SQL layout definition
class UserDB(Struct):
    id: int  = field(name='seq')
    first_name: str = field(name='first')
    last_name: str = field(name='last')

# JSON API definition
class UserAPI(Struct):
    id: int  = field(name='num')
    first_name: str = field(name='firstName')
    last_name: str = field(name='lastName')

and in that case, from_builtins() already uses the field renaming when the source is a dict:

user = from_builtins({'seq': 1, 'first': "bob", "last": "smith"}, UserDB)
print(user). # UserDB(id=1, first_name='bob', last_name='smith')

When going from db to app one could use user = from_builtins(user_row, UserDB, attr=True, rename=True), then could "swap aliases/names" using from_builtins(user, UserAPI, attr=True, rename=False) to switch from one Struct to another -- same attributes, different meta -- and finally could output to json for the JSON API: json.encode(user, type=UserAPI).

Ideally no one has to do that, but I think by just providing a fast way to swap metadata on a common set of attributes, then it could be possible without adding noise for the simple/common cases.

Simulating all that with current functionality would be:

# From DB to App
user = from_builtins({'seq': 1, 'first': "bob", "last": "smith"}, UserDB)  # Simulate from_builtins(user_row, UserDB, attr=True, rename=True)
print(user)  # UserDB(id=1, first_name='bob', last_name='smith')  

# From App to JSON API
public_user = UserAPI(id=1, first_name="bob", last_name="smith")  # Simulate from_builtins(user, UserAPI, attr=True, rename=False)
user_json = encode(public_user)
print(user_json)  # b'{"num":1,"firstName":"bob","lastName":"smith"}'

Summary of both cases to take into account:

SQLAlchemy, which favours using props and maybe field name translation (if you couldn't do field name translation with SQLAlchemy models)
"Struct Swap" which probably requires performance (so whichever of the two is fastest, and doesn't use field name translation)

Which probably lends itself to. attr= and rename=, both booleans. Default of attr=False and rename=True

Thanks for the feedback! This has been added in #419 - if you have the time I'd appreciate you trying it out before the next release. You can install from GitHub using the instructions here.

Navigation Menu

Repository navigation

from_builtins, but for objects? (e.g instantiate from SQLAlchemy models) #416

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

from_builtins, but for objects? (e.g instantiate from SQLAlchemy models) #416

Description

Description

Activity

jcrist commented on May 27, 2023

dmckeone commented on May 28, 2023

dmckeone commented on May 28, 2023

dmckeone commented on May 28, 2023

jcrist commented on May 30, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions