Skip to content

v2 message schema tweaks #351

@rvagg

Description

@rvagg

A fully-loaded current v2 message format looks like this when represented as semi-pretty-printed dag-json (data taken from the roundtrip test, this is not representative of a real message, it contains everything)):

{
  "Blks": [
    {
      "Data": {"/":{"bytes":"QgTLmh40xfCOmyCqdgkOcAILtWwMo9OvcpbNEFilESiQ/tIYSI8ITY355INftUrQRf/ZNuO/cmGwQmxRNSoJeBbtdEgruQhLSn7YrcUX8zceDgQ0tRFiXNGkF5IkPczc/ogJSw"}},
      "Pre": {"/":{"bytes":"AVUSIA"}}
    },
    {
      "Data": {"/":{"bytes":"xfPTKlWZ2kO4US4NsU11HtHWQJ2dy3rtBdbMbAmHxjTufEI28FF8INzGqZ6G5LrcAl5fXoG0WWEoUimfNvEQ6Qa6IW4ByDcDUq0ziUrrH+WNzNCC9FaSKBovz42VowZ4zaRVhw"}},
      "Pre": {"/":{"bytes":"AVUSIA"}}
    }
  ],
  "Reqs": [
    {
      "Canc": false,
      "Ext": {
        "AppleSauce/McGee": "yee haw"
      },
      "ID": {"/":{"bytes":"v8HW/fqRRd65WDICpHzhIw"}},
      "Pri": 101,
      "Root": {"/":"bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"},
      "Sel": {"R":{":>":{"|":[{".":{}},{"a":{">":{"@":{}}}}]},"l":{"none":{}}}},
      "Updt": false
    },
    {
      "Canc": false,
      "Ext": {},
      "ID": {"/":{"bytes":"wSjEHwARQ66IR7wP7nOl0g"}},
      "Pri": 202,
      "Root": {"/":"bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy"},
      "Sel": {"R":{":>":{"a":{">":{"@":{}}}},"l":{"none":{}}}},
      "Updt": false
    }
  ],
  "Rsps": [
    {
      "Ext": {},
      "ID": {"/":{"bytes":"v8HW/fqRRd65WDICpHzhIw"}},
      "Meta": [
        [
          {"/":"bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy"},
          "m"
        ]
      ],
      "Stat": 34
    },
    {
      "Ext": {
        "Hippity+Hoppity": {"/":{"bytes":"9V/48SUItj7yv+ynVXrpDfYxGl7BYxtKH6hDMQvZw6cQ6qzlob3XKtC/4El3HBHnVjOL2Thl5kXxreybnJnvQH+9T8aFnnkExa19yb0QpcwWlz1bKOwabdQ9n4L58Yw9A0GONQ"}}
      },
      "ID": {"/":{"bytes":"wSjEHwARQ66IR7wP7nOl0g"}},
      "Meta": [],
      "Stat": 14
    }
  ]
}

Current IPLD schema is https://github.com/ipfs/go-graphsync/blob/83c3860142cfbed18b8d87a6b3d375d017ef4fc8/message/ipldbind/schema.ipldsch

There's two primary things that I'd like to address here in a follow-up PR with a more concrete proposal:

  1. Size: the current format is a bit excessive with the string keys, I'd like to try and minimise this a bit more, mainly by leaning on tuple representation for most of the important bits.
  2. Additional versioning options: we have protocol version switching at the libp2p layer, but we could also be providing ourselves some optionality at the IPLD layer (https://ipld.io/docs/schemas/using/migrations/ is a good reference for this).

On this second item—by wrapping in a top-level single key keyed union we can iterate on this protocol without being forced to do it only at the libp2p layer. Wrapping in something like {"gs2":{....}} lets us do this. This way we can even iterate within the single, main schema, just by adding another key to the union. We get to reuse any of the overlapping types and make breaking adjustments without actually breaking the protocol. Implementations (this and Myel's one and anyone else's implementation) would just need to be aware that a version signal exists at this top-level union as well so any "can't decode according to schema" could optionally be translated as "don't know how to deal with protocol v2 / gsX" or some such.

So we could evolve the schema in 3 ways over time:

  1. Backward compatible within the current schema, which for tuples would mostly mean adding new items to the end and inserting null entries where things go missing.
  2. Breaking the schema and adding a new key to the top-level union, which gives us a lot of flexibility to move things around - perhaps we've extended the tuples a whole lot and nullified existing entries and they're getting out of control so we reset by doing this.
  3. Adding a new libp2p protocol version and doing an even harder break.

Having the optionality and doing some additional descriptive in-protocol signalling are the main goals though.

Metadata

Metadata

Assignees

No one assigned

    Labels

    need/triageNeeds initial labeling and prioritization

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions