I thought about removing the convert_*-logic out of data.rs into a real Deserialize-Implementation

I thought about removing the convert_*-logic out of data.rs into a real Deserialize-Implementation

If that works, that'd be great, especially considering the existing Serialize implementation.

There are a few problems:

• Serializing Bytes as readable is just Bytes(some size) as a string, probably not what you want.
• everything serialized as string (..., via repr) will be deserialized as string.
• supporting $__toml_private_datetime is nice, but not universal ... do we need this?

Maybe we should use some kind of enum tagging for all non-native types (numbers, strings, bool, none)?

Serializing Bytes as readable is just Bytes(some size) as a string, probably not what you want.

I think that's fine to be honest. You don't really want to serialize a large byte buffer to a JSON array, it will be horribly slow (which is precisely why it's serialized that way, otherwise typst query would fill your whole terminal with image data when you query for a figure). If you need to have binary data in JSON, you can explicitly convert it to an array. Otherwise, just don't serialize binary data in a human-readable format.

everything serialized as string (..., via repr) will be deserialized as string

Maybe we should use some kind of enum tagging for all non-native types (numbers, strings, bool, none)?

We could but I'm not sure it would be good. It is not to be expected that serialize -> deserialize roundtrip of arbitrary Typst values is lossless. I think adding tags would be quite heavy. For plugin usage, you generally want to have your input data expressed mostly in terms of primitives that just work with JSON.

supporting $__toml_private_datetime is nice, but not universal ... do we need this

I am not familiar with the $__toml_private_datetime notation. Is this some serde way to have TOML datetimes deserialize into Typst datetimes? In general, I think it would be unfortunate to regress on that being supported.

Serializing Bytes as readable is just Bytes(some size) as a string, probably not what you want.

I think that's fine to be honest. You don't really want to serialize a large byte buffer to a JSON array, it will be horribly slow (which is precisely why it's serialized that way, otherwise typst query would fill your whole terminal with image data when you query for a figure). If you need to have binary data in JSON, you can explicitly convert it to an array. Otherwise, just don't serialize binary data in a human-readable format.

With that said (and going on a bit of a tangent), would it make sense to add native BSON conversion to Typst (with the same API as json(), including the proposed functions from this PR)? Perhaps that could be yet another handy tool for plugin creators.

With that said (and going on a bit of a tangent), would it make sense to add native BSON conversion to Typst (with the same API as json(), including the proposed functions from this PR)? Perhaps that could be yet another handy tool for plugin creators.

I would be ok with that. Although I wonder if bson or bincode is the more "blessed" binary encoding.

I'm just experimenting with the following:
In JSON/TOML serialize normally; butin yaml, where !tags are supported, insert that type information as tag

JSON

{
  "a": 1,
  "b": [
    1,
    2,
    3.1415,
    "Test"
  ],
  "c": {
    "func": "text",
    "text": "Aha"
  },
  "d": [
    true,
    false
  ],
  "e": null,
  "f": [
    "28.35pt",
    "90deg"
  ]
}

TOML

a=1
b=[
  1,
  2,
  3.1415,
  "Test",
]
d=[
  true,
  false,
]
[c]
func = "text"
text = "Aha"
f = ["28.35pt", "90deg"]

YAML

a: 1
b:
  -1
  -2
  - 3.1415
  - Test
c: !Content
  func: text
  text: Aha
d:
  - true
  - false
e: null
f:
  - !length 28.35pt
  - !angle 90deg

Interesting, I didn't know of that YAML feature.

Is there a convenient way to get back from a repr()-string to a Value? I would use eval(), but don't have a World.

Is there a convenient way to get back from a repr()-string to a Value? I would use eval(), but don't have a World.

No. But this would only be useful for the tagged YAML thing anyway right? Because otherwise you can't know whether it was just a normal string. To be honest, I think nobody will use the tagged YAML thing, I'd just skip it and not try to do anything smart with unrepr-ing a string. repr also doesn't always produce evaluatable output.

@PgBiel do you still think that allowing bytes as input to json.decode and friends is problematic? because it's currently implemented that way. one upside is that this way it is more efficient to deserialize plugin output. converting to a string has not only the UTF-8 check but also a full copy from a wonderful prehashed byte buffer to a non-prehashed EcoString.

I think it's fine (I'm guessing it will check for UTF-8 validity either way). Ship it 🚀

(Although string should still be an option of course)

With that said (and going on a bit of a tangent), would it make sense to add native BSON conversion to Typst (with the same API as json(), including the proposed functions from this PR)? Perhaps that could be yet another handy tool for plugin creators.

I would be ok with that. Although I wonder if bson or bincode is the more "blessed" binary encoding.

I think we can have support for both in the future if there is demand for that. I just thought BSON (for now) would be more generic and probably easier to use across different languages.

The discussion might be a different one, but the implementation is related.

The discussion might be a different one, but the implementation is related.

Of course, but the PR becomes easier to approve and reason about when it's more self-contained.

I tend to agree that a new format should be a separate PR.

2 : 1 ok. ;-)

Since the separated PR with new format(s) will depend on this, I will place it when this one is done.

Please consider #1997 to calm clippy down even more and get a test pass...

Thank you!