Support omitting Struct attributes from encoding/decoding

Since Struct types force users to declare all Struct attributes, there's currently no way to hold extra state on a Struct without having it also be encoded/decoded.

Two ways to handle this:

Add omit=True to Meta annotations. This could also be packaged up as a parametrized type, so users could write

class Point(Struct):
    x: int
    y: int
    _private: Omit[some_type]

Excluded attributes with a leading underscore by default? I'm not sure if we should support this one, it may be confusing behavior. As a reference point, golang's json doesn't encode/decode private struct attributes by default.

Yes please! I would like to cache some (expensive) computed values in my objects, but not serialize them.

Thanks for the motivating use case (and apologies for the delayed response)! I hope to tackle this issue in the next few weeks before issuing a new release.

Since you commented here - as a user, would you want omitted attributes to also be excluded from __init__? Giving a concrete example:

class Point(Struct):
    x: int
    y: int
    _private_1: Omit[int]
    _private_2: Omit[int] = 0

Should __init__ include _private_1 and _private_2 arguments? Or should they be excluded from __init__, and only default values filled in? If they're excluded, that would mean that _private_1 would raise an AttributeError until it's explicitly set, and _private_2 would have an initial value of 0 after __init__.

Honestly I haven't been using the constructors much (decoding from a web request, encoding to a disk cache, decoding from the disk cache) so I don't have a strong opinion. I guess given that the object is valid without those values, the constructor shouldn't require them, but it would be nice if you could include them.

Also I would lean towards excluding attributes with leading underscores by default (without the Omit type), but I could see it being more annoying to have to include them if some API you're using has names like that.

Ok, we finally have the internals setup to make this fairly straightforward to implement.

First, I plan to call these "private" attributes, since that's effectively what they are. A private attribute won't be encoded or decoded, but will have space allocated on its respective struct.

There are some open questions though.

How should this be spelled?

There are two places that field configuration like this can go:

In the type annotation

class Example(Struct):
    x: int
    y: Private[int]
    z: Private[float] = 0.0

An option in msgspec.field

class Example(Struct):
    x: int
    y: int = msgspec.field(private=True)
    z: int = msgspec.field(default=0.0, private=True)

The former I find a bit easier to read, but it adds some complications:

The type annotations would have to be parsed at struct definition time, same as they would for ClassVar. This is doable, but to remain efficient can't handle all cases in the case of postponed annotations.
A Private type annotation by its nature must be applied to a field - a type list[Private[int]] is invalid, but python would happily let the user spell such a type. Using a default value with field ties this annotation to struct fields alone, avoiding this issue.
It's much harder to get this approach to play well with dataclass_transform, treating Private fields differently from public ones.

Right now I'm leaning towards the msgspec.field approach. Many users will already be familiar with dataclasses.field, which includes support for omitting fields from the generated __init__ via dataclasses.field(init=False). This is similar enough to private=True that it shouldn't be that wild of a concept.

Should fields with leading underscores be private by default?

Since we're providing a way to set whether a field is public/private explicitly, users can always override any default behavior chosen here. Since fields with leading underscores are usually treated as private in python by convention, I'm tempted to do so here as well. So the following would be equivalent:

class Example1(Struct):
    _field: int = 0  # private, since the leading underscore

class Example2(Struct):
    _field: int = field(default=0, private=True)  # explicitly private

On the other hand, I'd expect most users to never need to define private fields, so maybe inferring private/public status from the field name itself is overreaching.

Are private attributes excluded from `init`?

Right now I'm planning on excluding private attributes from basically everything in msgspec. They'll have space allocated in the __slots__ of the generated Struct type (and corresponding attributes defined), but will be excluded from:

__init__
__copy__
__reduce__ (pickling)
__repr__
__eq__ and other comparison methods
__hash__
__match_args__
encode
decode

This makes them most useful for including extra state during runtime that is initialized after the Struct instance is already created. cached_property usage is related here (#204), since you'd need a slot allocated to store the cached value.

If you have a use case for including private attributes in __init__, please let comment below. If needed, worst case we can add another init parameter for including/excluding a field from __init__. private=True will imply init=False by default, but you can manually re-include it as field(private=True, init=True).

Should private fields require a default value/factory?

Right now all msgspec.Struct types require all fields be initialized as part of the generated __init__. The only way to get an AttributeError for a field on a struct is if you manually del the attribute off an existing struct. I think this makes sense. mypy/pyright type annotations are designed to handle types (or unions of types), but not the presence/absence of an attribute.

Keeping with this design (and assuming private fields are excluded from __init__), it may make sense to enforce that all private fields have a default value (since there's no other way a value could be set).

class Example(Struct):
    private1: int | None = field(private=True, default=None)  # this is ok, there's a default value
    private2: int | None = field(private=True)  # this is invalid, there's no default value

Summary

In summary, my current leanings on these questions are:

Use msgspec.field for configuring if a field is private.
Make fields with a leading underscore private by default.
Exclude private fields from __init__. We can add a way to override this later if needed.
Require all private fields set a default value.

I don't plan on adding this before the next release (hopefully out next week), but after that whatever choices made here should (hopefully) be fairly straightforward to implement 🤞 .

This would be of great use to my current project --- like IcyMidnight, we've got a cache that would be really pointless to encode. The current plan is to clear the cache before encoding, which will work as long as we remember to do it.

Use msgspec.field for configuring if a field is private.

Make fields with a leading underscore private by default.

What happens if you do _something = field(private=False)? I think this should be allowed: a private field (in the Python convention sense) that's not private (in the msgspec sense).

Additionally, automatically treating fields with a leading underscore as private would break backwards compatibility. I found a few examples of Structs with _private fields which makes me think that making private fields an explicit, opt-in choice is a better idea.

Really looking forward to this feature, I hope you have time to work on it soon.

Adding a private=True parameter to field() seems fine to me.
Private omitted fields are the only way to have self-referential Structs.
e.g. a nested and occasionally recursive data structure with access to the _root node.

We can also use it to compute and store the path in that data structure because Structs aren't aware of their parent nodes.

Hi. Any update on this? I'm confused whether the library has this attribute.

Right now I'm leaning towards the msgspec.field approach. Many users will already be familiar with dataclasses.field, which includes support for omitting fields from the generated init via dataclasses.field(init=False). This is similar enough to private=True that it shouldn't be that wild of a concept.

If this is still up for debate, I concur that this is the right approach.

If you have a use case for including private attributes in init, please let comment below. If needed, worst case we can add another init parameter for including/excluding a field from init. private=True will imply init=False by default, but you can manually re-include it as field(private=True, init=True).

class Primitive(Struct):
    iid: int = field(private=True)    # internal ID, think auto-incrementing sequence meant to be private
    eid: str = field(private=False)   # external ID, some sort of public-exposed string identifier

In this use case, the object when fetched from the DB would have both IDs so the application layer has them available. But, when serialized as an HTTP response, we strip out the private iid.

Use msgspec.field for configuring if a field is private.

Agreed.

Make fields with a leading underscore private by default.

This is a nice API 😎

Exclude private fields from init. We can add a way to override this later if needed.

See my comment above. I am not sold on this one. The two paramater approach makes more sense to me as they are different use cases. I am fine if private=True implies init=False. But, I want to have the flexibility for private=True, init=True.

Require all private fields set a default value.

I guess I am okay with this, as long as I can still require it as an __init__. But once that is allowed, it seems to me this requirement falls off.

Would private fields also be excluded from __struct_fields__?

I came here looking for this functionality so I can store some extra metadata on a Struct that I don't want to be serialized or part of the __init__.

I'm envisaging an api where I can have an alternative constructer which can take optional (metadata) kwargs and store them on the struct without having them be included in the serialized representation - e.g.

class MyStruct(msgspec.Struct, tag=True):
    metadata: dict = field(default={}, private=True)
    
    @classmethod
    def create(cls, **kwargs) -> MyStruct:
        instance = MyStruct()
        instance.metadata.update(**kwargs)
        return instance

>>> obj = MyStruct.create(key='value')

>>> obj
MyStruct(metadata={'key': 'value'})

>>> msgspec.json.encode(obj)
b'{"type":"MyStruct"}'

>>> msgspec.json.decode(msgspec.json.encode(obj), type=MyStruct)
MyStruct(metadata={})

My actual use-case is a bit more complex but it boils down to wanting metadata associated with my objects but only for their lifetime in the current interpreter.

Another example, of something I'd like to do is to attach an fsspec.AbstractFileSystem to the class which would then be used in a save method to serialize itself and write to disk - e.g.

class MyStruct(msgspec.Struct, tag=True):
    metadata: dict = field(default={})
    fs: fsspec.AbstractFileSystem = field(private=True)

    @classmethod
    def create(cls, fs: fsspec.AbstractFileSystem, **kwargs) -> MyStruct:
        instance = MyStruct()
        instance.metadata.update(**kwargs)
        instance.fs = fs
        return instance

    def save(self, filepath: str) -> None:
        self.fs.write_bytes(
            path=filepath,
            value=msgspec.msgpack.encode(self),
        )

msgspec doesn't know how to serialize an AbstractFileSystem, and I wouldn't want it to be serialized along with my actual data, however it would be convenient to be able to store it on the instances.

Any updates on this?

pretty crucial feature to have

@jcrist

Make fields with a leading underscore private by default.

@jcrist wouldn't this break existing code? For example:

class MyState(msgspec.Struct):
    name: str
    _timestamp: datetime = msgspec.field(name="timestamp", default=None)

Assuming fields starting with underscore are going to be private by default, what would happen in the case of _timestamp? Would an explicit name, different from the one declared in the class, be able to differentiate and force a non-private flag by default?

A more extreme example:

class ApiUser(msgspec.Struct):
    name: str
    _api_type: str

In this case, assuming prior behaviour, without code intervention, there's no way for the user to force _api_type to be serialized without rewriting the class and passing private=False. This would break existing code/software in unexpected ways, I suppose.

Having said all that, I'm eagerly awaiting such a functionality. I've already stumbled upon two or three use cases where it would be useful to disable serialization of specific fields. The current alternative is to just create two different classes, one being specifically for serialization, but that may be too much of a hassle in most cases.

Any updates？

Hi,
This would be very useful as others have said, I am needing it today and that would make my code much cleaner ;)

Just one important thing, please don't do the same kind of shenanigans as attrs by interpreting _xxx has something, in that case 'private'.

It can and will break previous code, lead to unintended behavior when someone is following an external spec that has _xxx and do not know this.

And more generally it is against the python core tenet of "Explicit is better than implicit". Keystroke aversion is a curse :)

Finally. the = field( construct is used everywhere.

And also msgspec is awesome :)

@jcrist

I spent 3 days stuck on this (including burning $5 asking claude code to implement it). Turns out this is pretty trivial:

This is possible since version 0.14.0.

just use dict=True when constructing a Struct:

import msgspec


class User(msgspec.Struct, dict=True):
    name: str
    id: int

    def __post_init__(self):
        self.secret: str | None = None


user = User(name="John Doe", id=1234)
user.secret = "hunter2"

print(user)  # User(name='John Doe', id=1234)
print(user.secret)  # hunter2
print(msgspec.json.encode(user)) # b'{"name":"John Doe","id":1234}'

for more info on dict parameter see #331 #324 #204

@jcrist Should I write a short guide for the docs?

EDIT:
You can add a set_secret function if you want:

    def set_secret(self, secret: str):
        self.secret = secret
        return self

This creates an even nicer syntax for initializing the Struct:

user = User(name="John Doe", id=1234).set_secret("hunter2")

Or with a decoder:

decoder = msgspec.json.Decoder(User)
json_data = b'{"name":"John Doe","id":1234}'
decoder.decode(json_data).set_secret("hunter2") # User(name='John Doe', id=1234)

dict doesn't work with frozen structs.

for frozen structs you can use a reference dict. It wont perform is nicely as though.

import weakref

import msgspec

_secrets_dict: dict[int, str] = {}


class User(msgspec.Struct, frozen=True, weakref=True):
    name: str
    id: int

    def __post_init__(self):
        weakref.finalize(self, _secrets_dict.pop, id(self), None)

    def set_secret(self, secret: str):
        _secrets_dict[id(self)] = secret
        return self

    @property
    def secret(self):
        return _secrets_dict.get(id(self), None)


user_1 = User(name="John", id=12345).set_secret("hunter2")
print(user_1, user_1.secret)  # User(name='John', id=12345) hunter2


user_2 = User(name="Alice", id=21345).set_secret("password")
print(user_2, user_2.secret)  # User(name='Alice', id=21345) password


print(_secrets_dict)  # {2941001576528: 'hunter2', 2941001577616: 'password'}

# when the struct is garbage collected, the secret will be removed from the dict
del user_1
print(_secrets_dict)  # {2941001577616: 'password'}

let me know if anything is unclear.
Cheers,
Shlomo

Actually, if your Struct is frozen, you can use a weakkeydictionary. and it has some performance gains compared to the regular dictionary

from weakref import WeakKeyDictionary

import msgspec

_secrets_data: WeakKeyDictionary[msgspec.Struct, str] = WeakKeyDictionary()


class User(msgspec.Struct, frozen=True, weakref=True):
    name: str
    id: int

    def set_secret(self, secret: str):
        _secrets_data[self] = secret
        return self

    @property
    def secret(self):
        return _secrets_data.get(self, None)


user_1 = User(name="John", id=12345).set_secret("hunter2")
print(user_1, user_1.secret)  # User(name='John', id=12345) hunter2


user_2 = User(name="Alice", id=21345).set_secret("password")
print(user_2, user_2.secret)  # User(name='Alice', id=21345) password


print(
    list(_secrets_data.items())
)  # [(User(name='John', id=12345), 'hunter2'), (User(name='Alice', id=21345), 'password')]

# when the struct is garbage collected, the secret will be removed from the dict
del user_1
print(list(_secrets_data.items()))  # [(User(name='Alice', id=21345), 'password')]

A very basic, chatgpt written benchmark showed this:

=== Detailed Timings ===
Implementation      Create(s)    SetSecret(s)    GetSecret(s)    Delete(s)    Total(s)
----------------  -----------  --------------  --------------  -----------  ----------
OuterDictUser        0.855143        0.056949        0.051307     0.371720    1.335119
WeakrefUser          0.045294        0.153226        0.134727     0.021986    0.355233
UnHashableUser       0.134465        0.056266        0.035185     0.063592    0.289508

=== Per-Metric Winners ===
Metric        Winner          RunnerUp        Loser            WinTime(s)    PctVsRunner(%)    PctVsLoser(%)
------------  --------------  --------------  -------------  ------------  ----------------  ---------------
Create(s)     WeakrefUser     UnHashableUser  OuterDictUser        0.0453          196.8681        1787.9628
SetSecret(s)  UnHashableUser  OuterDictUser   WeakrefUser          0.0563            1.2139         172.3235
GetSecret(s)  UnHashableUser  OuterDictUser   WeakrefUser          0.0352           45.8194         282.9081
Delete(s)     WeakrefUser     UnHashableUser  OuterDictUser        0.0220          189.2373        1590.7046
Total(s)      UnHashableUser  WeakrefUser     OuterDictUser        0.2895           22.7023         361.1680

Ultimately, these were extremely simple structs, and the benchmarks didn't include encoding and decoding. Also, I didn't really verify that the benchmark is written well.

Note: WeakrefUser refers to User struct with a weakkeydictionary. Unhashable user uses direct access to self with dict = True, and OuterDictUser is the one that uses a dict with objects ids to keep track. All the structs are above.

This feature is useful and should be implemented like it was being discussed in the start, I don't think we should just do some work around to achieve this, also i don't recommend the idea of private, i would rather have a boolean encode=True/False or exclude_from_encoding=True/False rather than this private thing where you add the additional complexity of variables starting with _. I also have a use case for this that's why i am here

class TokenBucket(Struct):
    capacity: int = field(encode=False)
    refill_rate: int = field(encode=False)
    tokens: float
    last_refill: float = field(default_factory=time.monotonic)

    def allow_request(self) -> bool:
        now = time.monotonic()
        tokens_to_add = (now - self.last_refill) * self.refill_rate
        self.tokens = min(self.capacity, self.tokens + tokens_to_add)
        self.last_refill = now

        if self.tokens < 1:
            return False

        self.tokens -= 1
        return True

    def __lt__(self, other: TokenBucket) -> bool:
        return self.tokens < other.tokens

i just want to encode tokens and last_refill since that's all i need to save in redis but not the capacity or the refill rate

@jcrist

In summary, my current leanings on these questions are:

Use msgspec.field for configuring if a field is private.

Make fields with a leading underscore private by default.

Exclude private fields from __init__. We can add a way to override this later if needed.

Require all private fields set a default value.

@jcrist Very excited to see this proposal happen, and I agree with everything here except "make fields with a leading underscore private by default". A few reasons I think this should not be included, besides being a breaking change:

I think your own counter-argument is 100% correct: "I'd expect most users to never need to define private fields, so maybe inferring private/public status from the field name itself is overreaching."
Explicit is better than implicit: I would expect all cases where a developer is storing cached data on the struct and does not want that field to participate in the normal encoding/decoding process to explicitly add private=True to the field, regardless of the leading underscore, so that it's clear to anyone reading (incl. future self) that this field is not participating in encoding/decoding without having to look up the current convention in the msgspec docs.
In most cases of storing cached data on the field, I suspect that we actually don't want the cached data to be "python-private". For example: we have raw polygon coordinates as the decoded field, and a shapely Polygon as the computed field. In this case, we are actually going to be using the shapely Polygon, not the raw coordinates, throughout the code, so we don't want it "python-private" at all! Which removes all the value of a private convention for most use-cases, I think, as we'd have to specify private=True anyway.
There should be one way to do it: what would be the worst for readability is a mixture of the 2: some fields declared private=True because we don't want them prefixed by underscores, and some not since we do want them prefixed by underscores and then potentially others with private=False to opt out: that would be very weird. Whether or not the field is python-private is totally orthogonal to whether or not we want it decoded/encoded.
A convention like this would inappropriately mix python conventions with json conventions. Some users will nix the python conventions and make the field name the same as the json key to avoid having to rename it (this would be especially true for dynamically generated models), and some JSON keys are prefixed by underscores. This does not necessarily have anything to do with Python conventions.
There are less-surprising ways to reduce boilerplate: we can simply define Private = field(default=None, private=True) once and then do foo: Bar = Private from then on. Or, if there were to be a built-in convention, a default of None on private fields (allowing foo: Bar = field(private=True)) would be far less surprising than a "decoder skips _data" convention.

Just to add in another workaround suggestion for anyone wanting to skip fields during encoding, using some undocumented behavior:

Any field with its value set to msgspec.UNSET will be skipped during encoding. Caveats to note:

Does not work for encoding in array_mode
Any field you skip encoding must have a default value present in its Struct def or it will fail to decode (missing required parameter.)

class Test(msgspec.Struct):
    a: int
    b: float = 1.0
    c: str = 'string'

t= Test(0)
tt = copy.replace(t, b=msgspec.UNSET)
enc = msgspec.json.encode(tt)
enc
# b'{"a":0,"c":"string"}'
dec = msgspec.json.decode(enc, type=Test)
dec
# Test(a=0, b=1.0, c='string')

I would also like to add that sometimes its useful to specify which fields to omit when using the encoder/converters, for example:

msgspec.toml.encode(struct, exclude=("field_a", "field_b"))

Pydantic supports this in a similar way: https://pydantic.dev/docs/validation/latest/concepts/serialization/#as-parameters-to-the-serialization-methods

Navigation Menu

Repository navigation

Support omitting Struct attributes from encoding/decoding #199

How should this be spelled?

Should fields with leading underscores be private by default?

Are private attributes excluded from `init`?

Should private fields require a default value/factory?

Summary

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support omitting Struct attributes from encoding/decoding #199

Description

Activity

IcyMidnight commented on Nov 14, 2022

jcrist commented on Nov 28, 2022

IcyMidnight commented on Nov 28, 2022

jcrist commented on Jan 26, 2023

How should this be spelled?

Should fields with leading underscores be private by default?

Are private attributes excluded from __init__?

Should private fields require a default value/factory?

Summary

Solumin commented on Jul 11, 2023

illeatmyhat commented on Aug 10, 2023

FeryET commented on Sep 19, 2023

ahopkins commented on Sep 24, 2023

Solumin commented on Nov 21, 2023

dhirschfeld commented on Nov 23, 2023

dhirschfeld commented on Nov 23, 2023

c0x65o commented on Apr 22, 2024

oxysoft commented on Nov 30, 2024

flisboac commented on Dec 18, 2024

dream2333 commented on Jan 13, 2025

Annakan commented on Jan 24, 2025

SZRabinowitz commented on Apr 30, 2025

Solumin commented on Apr 30, 2025

SZRabinowitz commented on Apr 30, 2025

SZRabinowitz commented on Apr 30, 2025

SZRabinowitz commented on Apr 30, 2025

iyadzargar commented on Aug 15, 2025

HansBrende commented on Oct 18, 2025

tbronson commented on Dec 9, 2025

theacodes commented on May 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Are private attributes excluded from `init`?