[Feature] Support dedicated Vector file compaction for Data Evolution tables #1381

Search before asking

• I searched existing issues and pull requests and found no issue tracking dedicated Data Evolution Vector file compaction.

Motivation

Multimodal embedding updates create small dedicated Vector files and overlapping field versions. Rust's normal Data Evolution compaction leaves them untouched, so applications cannot consolidate this storage through the Rust maintenance pipeline.

The existing normal-file implementation (#1249) explicitly excludes dedicated BLOB/Vector compaction. This issue is the suggested follow-up to #1380; reuse the dedicated-task and commit infrastructure established for BLOB first.

Scope: Data Evolution append tables with row tracking, no primary key, bucket=-1, and vector.file.format configured. This concerns row-ID-aligned Vector storage, separately from primary-key managed BLOB compaction (#1369), primary-key vector indexes, or a new ANN indexing feature.

Solution

Port Java's DataEvolutionVectorCompactTask and the Vector planning rules into Rust's existing compaction pipeline.

• Plan a complete set of Vector-store files for the selected normal row-ID group, including all overlapping versions and field projections. Only span adjacent normal groups when their normal files are compacted together in the same batch.
• Resolve write columns using each file's schema and stable field IDs. For each current field represented in the inputs, require coverage of the entire task range. Skip unsafe groups with partial field coverage: filling missing coverage with NULL can overwrite values held elsewhere. Ignore dropped fields and do not materialize newly added fields absent from all inputs.
• Read effective column values through the existing Data Evolution reader. Preserve field-wise version ordering: output files inherit the maximum input sequence, so omitting an overlapping input can change which values win.
• Write the configured dedicated Vector format, rolling by vector.target-file-size. Assign consecutive first-row IDs to the outputs and verify that their combined range and physical row count exactly match the input union. Preserve sequence bounds and applicable per-column sequence metadata, including global-index column-update handling; validate winner selection again after subsequent field updates.
• Integrate task serialization, output validation, snapshot publication, and failure cleanup. Preserve normal/BLOB files, logical deletions, and valid global indexes. Avoid repeatedly selecting already compacted rolled outputs when no consolidation is useful.

Acceptance criteria

• Row-ID ranges: test multiple rolling outputs, adjacent groups, overlapping versions, gaps, incomplete task inputs, and normal/Vector tasks committed together. Every physical slot retains its row ID, including deleted slots.
• Field versions: cover multiple Vector fields updated independently, overlapping row ranges, NULL values, renames, dropped/recreated names, and newly added fields. Partial field coverage is skipped or rejected without publication; omitted versions cannot alter winners.
• Historical snapshots: retained snapshots still return their original embeddings. Latest projected/full reads and existing vector-search/index paths return the same logical results after compaction.
• Commit/recovery: test concurrent Vector/normal updates, disjoint appends, deletion-vector changes, retries, serialized task subsets, and abort/failure cleanup. Retained snapshots keep their referenced inputs.
• Add Java-compatible fixtures or interoperability coverage, verify output format and size controls, show useful file consolidation, and confirm a second run does not endlessly rewrite completed outputs.
• Update compaction documentation with supported configurations, invocation, and eligibility rules.

Anything else?

Suggested order: BLOB (#1380) first, then this Vector issue. Rough effort: several weeks for Vector, including field-coverage, compatibility, and recovery validation; this is a planning estimate.

New ANN index formats, primary-key vector compaction, inline-to-dedicated migration, large normal-file splitting, and row-ID reassignment are outside this issue.

Source baseline: paimon-rust f91fdb8ab034887547a5ee2f26c1979e30250558, compared with apache/paimon 3821db1972b87d5cd3ac1d025ec214cc3842ae9a.