Support for dataclasses

Given the close similarity between msgspec.Struct and dataclasses.dataclass, am curious to what extent dataclasses can be used with msgspec and what this entails.

Pretty sure you can write a custom codec to handle them as per: https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types

Dataclasses aren't natively supported, but could be handled with a custom enc_hook and dec_hook. However, this will be slow (much slower than using a msgspec.Struct or using a raw dict). While structs and dataclasses have similar apis, its the internal design of struct types that makes them so fast. We could add native support for dataclasses which would put them somewhere between structs and dicts in encode speed (faster than a dict, slower than a struct), but for decoding they'd still be slower than a dict. If possible I recommend people just use structs - they're fast and support most of the things people want in dataclasses.

they're fast and support most of the things people want in dataclasses

This is true except there is an ecosystem of tooling around dataclasses, adopting msgspec.Struct means disconnecting from that ecosystem

What libraries/tools are you thinking of that require dataclasses that you'd also like to use with msgspec?

Hi there. So, while trying to make the switch from orjson to msgspec for Starlite, I encountered this issue. The main concern here is performance really. Dataclasses are natively supported by orjson; Their serialization is therefore quite fast.
What this means for us is, that making the switch to msgspec would be a massive drop in performance for our users when dataclasses are being used.

You can see below a simple benchmark result comparing the branch with orjson and the branch with msgspec.

I'm not sure how much work implementing dataclasses support would be, I just wanted to let you know of this use case where dataclass support is kinda vital, so you can consider if you deem it to be worth the effort.

Thanks for the ping (and the nice benchmark plot). I've actually been revisiting this opinion, and I think you've convinced me that adding simple support for dataclasses is worth it. Now that we have support for TypedDict and NamedTuple objects, asking for dataclass support isn't that far off. And compatibility with orjson is a convincing use case. Users really should use msgspec.Struct objects instead when possible (they're much faster, and have fewer weird edge cases than dataclasses), but we can do a decent job with encoding/decoding dataclasses too.

I spent some time this evening experimenting with an implementation, and I'm pretty happy with the results. Encoding time is already much faster than what orjson provides, especially when slots=True is set:

In [1]: import msgspec, orjson

In [2]: from dataclasses import dataclass

In [3]: enc = msgspec.json.Encoder()

In [4]: @dataclass
   ...: class NoSlots:
   ...:     field_one: int
   ...:     field_two: int
   ...: 

In [5]: @dataclass(slots=True)
   ...: class Slots:
   ...:     field_one: int
   ...:     field_two: int
   ...: 

In [6]: no_slots = [NoSlots(i - 1, i + 1) for i in range(10000)]

In [7]: with_slots = [Slots(i - 1, i + 1) for i in range(10000)]

In [8]: %timeit enc.encode(no_slots)  # msgspec, no slots
561 µs ± 2.04 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [9]: %timeit orjson.dumps(no_slots)  # orjson, no slots
834 µs ± 1.69 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [10]: %timeit enc.encode(with_slots)  # msgspec, with slots
779 µs ± 20 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [11]: %timeit orjson.dumps(with_slots)  # orjson, with slots
3.71 ms ± 90.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [12]: class Struct(msgspec.Struct):
    ...:     field_one: int
    ...:     field_two: int
    ...: 

In [13]: structs = [Struct(i - 1, i + 1) for i in range(10000)]

In [14]: %timeit enc.encode(structs)  # msgspec structs
356 µs ± 307 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

I have JSON encoding done already. Decoding will take a bit more work, but I'd estimate done in the next week or two. I'll reopen this for now as a reminder.

fantastic! were are eagerly awaiting 😉

@jcrist

@jcrist Thank you for your fantastic work on this and your responsiveness to the issues we raised here in general!

Ok, #218 is in - msgspec now supports encoding/decoding dataclasses. This came together a lot quicker than I expected, and was pretty fun to work on. They're not as performant or featureful as Struct types, but perforamance isn't bad (especially for encoding).

I have a few other small fixups I'd like to get in, I'd expect a release sometime next week.

Thanks Jim! 🙏

It's working fantastic! I ran the benchmark again and we can see some real improvements!

Fix `Meta(pattern=..)` memory leak / corruption (msgspec#1256)

Navigation Menu

Repository navigation

Support for `dataclass`es #57

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support for dataclasses #57

Description

Activity

goodboy commented on Feb 6, 2022

jcrist commented on Feb 8, 2022

old-ocean-creature commented on Jun 19, 2022

jcrist commented on Jun 19, 2022

provinzkraut commented on Nov 30, 2022

jcrist commented on Dec 1, 2022

Goldziher commented on Dec 1, 2022

provinzkraut commented on Dec 1, 2022

jcrist commented on Dec 2, 2022

jakirkham commented on Dec 2, 2022

provinzkraut commented on Dec 2, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Support for `dataclass`es #57