Skip to content

New types can break older Iceberg agents #10775

@jacobmarble

Description

@jacobmarble

After #9008 is merged, it becomes possible for an Iceberg agent to create schemas with timestamp_ns columns. The behavior of Iceberg agents not aware of this new type is undefined.

A simple solution: new types require format-version=3


Originally posted by @nastra in #9008 (comment)

Given that this type is for v3, wouldn't a v2 writer just blindly write this new type if it's being used anywhere and thus break forward compability?

@nastra I don't think I understand your question, would you mind sharing an example?

It seems to me that any writer that doesn't recognize a particular type (in this case timestamp_ns or timestamptz_ns) should fail when it reads the schema in the table metadata. How does the writer "blindly" write to an unimplemented type?

@jacobmarble currently nothing is preventing this new type to be written with a V2 writer, meaning that one could use this new type in a schema and write it with a V2 writer. Now an older Iceberg client tries to read this using a V2 reader and will fail.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions