Repository navigation
Closed
Description
When designing in memory model for iceberg spce(metadata, snapshots, manifests etc), there are some problems to consider:
API safety.
For a field in spec, there are several cases to consider:
- Exists in both v1 and v2, and both are required
- Exists in both v1 and v2, and both are optional
- Exists in both v1 and v2, but optional in one and require in another.
- Exists in v1, but deprecated v2.
- Exists in v2, and have default value in v1.
Let's use the summary field in snapshot spec as an example. There are some potential solutions to this:
- Just like in https://github.com/JanKaul/iceberg-rust, we have different models for v1 and v2 spec, e.g.
SnapshotV1andSnapshotV2. This way we can guarantee the api safety of model, but it introduces extra maintaince burden when spec evolves, since most fields are common in both spec. - Use one struct for both version(https://github.com/icelake-io/icelake/), and let users to do the check at runtime. This way we don't have extra maintaince effort for different specs, but we can't guarantee the api safety.
Field access
Currently, in both two repos, all fields are accessed through the public field. Personally, I'm not in favor of this approach since it may cause some problems:
- Misuse of fields. For example, for
TableMetadata, when we want to append a new snapshot, we should both append snapshot and snapshot log, but exposingVec<Snapshot>may lead to wrong usage. - In-memory data structure should be designed for high-performance access. For example,
TableMetadatashould contain a map of snapshot id to snapshot, rather than a vector of snapshots, so that we can access snapshot by id fast. - The in-memory data struct fields should not have a one-one mapping as spec, but should provide more friendly access methods to user. For example, there should no
ManifestListdata structure, and user should access manifest fiels fromSnapshot's method.
So for in-memory data structure, I would propose hiding all public fields and provide public methods to access necessary fields.
Reacted by Xuanwo, Jan Kaul, ZENOTME and lmatz
Metadata
Metadata
Assignees
Labels
No labels
Activity
Xuanwo commentedon Jul 21, 2023
Agreed.
Xuanwo commentedon Jul 21, 2023
Please correct me if my understanding is wrong: it's safe to convert all v1 table to v2 table during reading. And we only need to care about writing.
JanKaul commentedon Jul 21, 2023
I think the solution that @Xuanwo is hinting at could be really cool. We could store the data internally similar to the v2 metadata and we only need to care about writing of v1 metadata. If I understood @Fokko right, he implemented it similarly in pyiceberg.
Regarding the field access, I agree with all your points.
liurenjie1024 commentedon Jul 21, 2023
I agree the reading/writing part, but has little concern about the api. Still the
summaryexample(optional in v1, required in v2), what's the return type ofSnapshot::summary()?Summarystruct. Then what should be the value filled for v1 format?Option<Summary>. Then we have to force user to unwrap it when using v2 format.liurenjie1024 commentedon Jul 21, 2023
Or should we provide a method to return
Result<Summary>when so that v2 format user can return an error when there is noSummary?JanKaul commentedon Jul 21, 2023
Regardless of the implementation we have the problem that there is a
summaryin v2 and there is none in v1. So if we want to support v1 we have to useOption/Result. I would preferOptionbecause I don't see any drawbacks compared toResult, the user has to "unwrap" in either case.JanKaul commentedon Jul 21, 2023
Another issue, that @liurenjie1024 pointed out in #2, is that Avro requires a strict schema when reading v1 and v2 values. So we might need two representations for v1 and v2 anyway.
ZENOTME commentedon Jul 21, 2023
Does this means that we need two on-disk representation, but we only need one unique in-memory representation.
liurenjie1024 commentedon Jul 21, 2023
In fact, we can still have one in memory struct representation, and have two serialize newtypes, like this:
JanKaul commentedon Aug 2, 2023
I just found a serde feature which might simplify the serialization/deserialization for in-memory/on-disk types. Serde let's you you define a "from" type which specifies that the annotated type first get's deserialized into the "from" type and then converted using the
Fromimplementation. And it works similarly with an "into" type. This would allow something like the following:What do you think?
liurenjie1024 commentedon Aug 2, 2023
How to implement this? I guess that we can depend on a field for table format version? So we need to maintain the same field for all other structs such as snapshot, manifest?
Generally LGTM
JanKaul commentedon Aug 2, 2023
You are right, we probably need a field for the format version in the
TableMetadatastruct.I don't think we need a field in every other struct. When we serialize the
TableMetadatawe know which format version it is based on the aforementioned field. We can then do the conversion for snapshots, schema,... ourselves inside theimpl Into<TableMetadataUnion> for TableMetadata.I think you would only need a format version field for every struct that you explicitly serialize/deserialize and that is not part of another struct. These could be
TableMetadata,ManifestList,ManifestEntry.liurenjie1024 commentedon Dec 14, 2023
Close for now. Feel free to reopen when necessary.
Merge pull request apache#3 from risingwavelabs/xxh/test-moka
Merge pull request apache#3 from dqhl76/bump-opendal
Auto release (apache#3)
fix(arrow): error on required field absent from data file with no def…