Skip to content

Commit 2e5f393

Browse files
morningmanclaude
andcommitted
[perf](catalog) fe-connector-iceberg: reconnect the manifest cache to streaming + COUNT(*) planning, lazily (PERF-04)
Root cause: the opt-in IcebergManifestCache (meta.cache.iceberg.manifest.enable, default off) was consulted only on the synchronous planning path (planFileScanTaskWithManifestCache). Two paths bypassed it: (C17) the streaming split path for tables >= num_files_in_batch_mode -- the OOM-protection lazy path, which is exactly the large tables the cache targets -- read scan.planFiles() (SDK, uncached); (C18) COUNT(*) pushdown read scan.planFiles() just to grab one placeholder file (the count itself comes from the snapshot summary), while iceberg's ParallelIterable eagerly submitted every manifest reader. On a cache-enabled catalog the cache delivered 0% hit rate on its target tables. Legacy contrast (verified against 6fef25709d3^ fe-core IcebergScanNode): batch mode DID honor the cache -- but the cache-on path MATERIALIZED the full task list (no OOM protection); OOM safety existed only with the cache off. The SPI cutover traded the cache for unconditional streaming (OOM-safe, uncached). Rather than restore legacy (reintroducing the OOM risk) or leave it, this makes the cache-backed planning itself LAZY so both paths get cache hits AND stay bounded. Fix (connector-side, no fe-core change): extract cacheBackedFileScanTasks(), a lazy CloseableIterable<FileScanTask> shared by all three consumers. Phase 1 (eager) reads the delete manifests through the cache and builds the DeleteFileIndex -- required before any data task and bounded like the SDK planFiles(), which also reads all delete manifests up front. Phase 2 (lazy) is a flat-map iterator over the data manifests, reading each through the cache on demand and yielding pruned whole-file BaseFileScanTasks WITHOUT materializing the list (peak heap = delete index + one manifest's files + the split queue). - planFileScanTaskWithManifestCache now materializes that iterable (byte-identical to before; the sync/small-table path keeps the per-table heuristic split size). - streamSplits, when the cache is enabled, feeds the lazy iterable to the existing IcebergStreamingSplitSource at a fixed slice size (unchanged for big tables) -- now cache-hitting and still OOM-safe. - planCountPushdown, when enabled, iterates the lazy iterable and takes the FIRST file (lazy early stop, no ParallelIterable fan-out). The batch/stream DECISION (streamingSplitEstimate) is unchanged. A cache-read failure records it and falls back to scan.planFiles() (catch Exception, since the cache rethrows failures as RuntimeException). statsQueryId is nullable: streaming passes null (no-stats overload) because its Phase 2 runs on the engine pump thread while Phase 1 ran on the caller -- tallying the per-query ScanStats from two threads would race; sync/count are single-threaded and keep stats. Parity: the lazy path reuses the exact per-file prune/task-construction of the old materialized path (which already claims legacy planFiles parity), so no new planning is invented. The COUNT placeholder file may differ from the SDK path's first file (ParallelIterable order is non-deterministic) but the count (snapshot summary) is identical and BE ignores the file. Big-table streaming now populates the cache (its purpose; capacity-bounded, no OOM). Tests: streaming cache-parity vs SDK + cache consumed; partition-prune parity; flat-map across multiple data manifests; COUNT cache count-parity + lazy early stop (reads < all manifests); empty null-snapshot COUNT guard. Sync cache-path tests stay green. Full iceberg module UT: 962 run / 0 fail / 0 error / 1 skip; checkstyle 0 violations. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
1 parent 584ef38 commit 2e5f393

2 files changed

Lines changed: 329 additions & 46 deletions

File tree

0 commit comments

Comments
 (0)