Skip to content

V2 File writers and readers can support columns of varying lengths #7404

Description

@wjones127

Context

Sparse data overlays (one file carrying a different set of rows per field) require the v2 file writer to emit columns whose item counts differ within a single file. The current v2 writer advances all columns from one global row counter and cannot do this. It is technically allowed by the file format but unused today. This is the foundation for Data Overlay Files (see "Physical layout" and "Writer support" in the spec).

Acceptance criteria

  • The v2 file writer can write a single file in which columns have differing item counts (no shared global row counter).
  • The v2 file reader can read each column back independently at its own length.
  • Random access (by rank/position within a column) returns the correct value for a column shorter than the file's longest column.
  • Per-column row counts are recorded in / derivable from file metadata.
  • Round-trip tests: write columns of unequal length (including a zero-length column and a single-row column alongside longer ones); read each back fully and via random access; assert values and lengths.
  • Existing equal-length files round-trip unchanged (backwards compatible).

Out of scope

  • The DataOverlayFile concept, coverage bitmaps, and rank-vs-offset resolution semantics — this issue delivers only the file-level capability; overlay write/read is OSS-1322.
  • Scanner / take / index / compaction integration.
  • Dense (rectangular) overlays, which already write with the existing equal-length writer and do not depend on this work.

Dependencies

Metadata

Metadata

Assignees

Labels

A-encodingEncoding, IO, file reader/writer

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions