[Feature] Support dedicated BLOB file compaction for Data Evolution tables #1380

Search before asking

• I searched existing issues and pull requests and found no issue tracking dedicated Data Evolution BLOB file compaction.

Motivation

Multimodal Data Evolution tables accumulate small dedicated BLOB files and overlapping column-update versions. Rust's normal-column compaction preserves those files, so it cannot consolidate their payload storage or reduce the associated file-open and metadata overhead.

The existing Data Evolution implementation (#1249) supports normal files only; the documented limitation explicitly excludes dedicated BLOB/Vector compaction.

This issue concerns append tables with data-evolution.enabled=true, row-tracking.enabled=true, bucket=-1, and no primary key. They reconstruct columns by row-ID range and field version. This is a separate mechanism from primary-key managed BLOB compaction (#1369), which merges descriptors and rebuilds .blobref sidecars without rewriting payload packs.

Solution

Port Java's DataEvolutionBlobCompactTask and the BLOB planning rules in DataEvolutionCompactCoordinator into the existing Rust compaction pipeline.

• Add BLOB task planning, execution, serialization, and commit validation alongside normal tasks. Reuse snapshot capture, task subsets, retries, and output cleanup.
• Resolve each input's single write column through its historical schema and group by stable field ID. Handle renames correctly and keep dropped/recreated columns with the same name distinct. Only compact fields still configured for dedicated BLOB storage.
• Keep overlapping versions together and form contiguous row-ID ranges. Follow Java's dedicated-file association with normal row groups, two-file minimum, and blob.target-file-size selection. Consolidation across normal groups must respect coordinated normal-file rewrites; never bridge a row-ID gap.
• Read the effective BLOB values in row-ID order using the existing version/fallback rules and stream them into one dedicated output per task, as Java does. Cover scalar BLOB and the collection forms already supported by the Blob format; explicitly reject unsupported dedicated layouts.
• Preserve the exact physical row-ID range, row count, input sequence bounds, write-column identity, and applicable column sequence metadata without changing field-version selection. Retain deleted slots and logical deletions. Keep normal-file/DV handling and unrelated fields correct when normal and BLOB tasks share a commit.

Acceptance criteria

• Row-ID ranges: adjacent segments and overlapping versions compact correctly; gaps, invalid ranges, and incomplete input groups cannot publish. Verify independent BLOB tasks and combined normal/BLOB commits.
• Field versions: validate partial and sparse BLOB updates, NULL versus fallback values, multiple BLOB fields, historical schemas, renames, and name reuse. Logical values and latest-version selection remain correct after compaction and subsequent updates.
• Historical snapshots: retained snapshots still read their original values and payloads. Current reads preserve logical deletions, unaffected Vector files, and valid global-index state.
• Commit/recovery: revalidate relevant inputs against concurrent updates; test commit retries, serialized task subsets, abort/failure cleanup, and idempotent recovery without deleting files referenced by retained snapshots.
• Add Java-compatible fixtures or interoperability coverage, document supported layouts and invocation, and show file-count reduction on a representative multimodal workload.

Anything else?

Suggested order: implement BLOB first, then dedicated Vector compaction in #1381. Rough effort: several weeks for BLOB, including compatibility and recovery validation; this is a planning estimate.

Vector compaction, primary-key managed payload-pack rewriting, inline-to-dedicated migration, large normal-file splitting, and row-ID reassignment are outside this issue.

Source baseline: paimon-rust f91fdb8ab034887547a5ee2f26c1979e30250558, compared with apache/paimon 3821db1972b87d5cd3ac1d025ec214cc3842ae9a.