Repository navigation
Commit 2e5f393
[perf](catalog) fe-connector-iceberg: reconnect the manifest cache to streaming + COUNT(*) planning, lazily (PERF-04)
Root cause: the opt-in IcebergManifestCache (meta.cache.iceberg.manifest.enable,
default off) was consulted only on the synchronous planning path
(planFileScanTaskWithManifestCache). Two paths bypassed it: (C17) the streaming
split path for tables >= num_files_in_batch_mode -- the OOM-protection lazy path,
which is exactly the large tables the cache targets -- read scan.planFiles()
(SDK, uncached); (C18) COUNT(*) pushdown read scan.planFiles() just to grab one
placeholder file (the count itself comes from the snapshot summary), while
iceberg's ParallelIterable eagerly submitted every manifest reader. On a
cache-enabled catalog the cache delivered 0% hit rate on its target tables.
Legacy contrast (verified against 6fef25709d3^ fe-core IcebergScanNode): batch
mode DID honor the cache -- but the cache-on path MATERIALIZED the full task list
(no OOM protection); OOM safety existed only with the cache off. The SPI cutover
traded the cache for unconditional streaming (OOM-safe, uncached). Rather than
restore legacy (reintroducing the OOM risk) or leave it, this makes the
cache-backed planning itself LAZY so both paths get cache hits AND stay bounded.
Fix (connector-side, no fe-core change): extract cacheBackedFileScanTasks(), a
lazy CloseableIterable<FileScanTask> shared by all three consumers. Phase 1
(eager) reads the delete manifests through the cache and builds the
DeleteFileIndex -- required before any data task and bounded like the SDK
planFiles(), which also reads all delete manifests up front. Phase 2 (lazy) is a
flat-map iterator over the data manifests, reading each through the cache on
demand and yielding pruned whole-file BaseFileScanTasks WITHOUT materializing the
list (peak heap = delete index + one manifest's files + the split queue).
- planFileScanTaskWithManifestCache now materializes that iterable (byte-identical
to before; the sync/small-table path keeps the per-table heuristic split size).
- streamSplits, when the cache is enabled, feeds the lazy iterable to the existing
IcebergStreamingSplitSource at a fixed slice size (unchanged for big tables) --
now cache-hitting and still OOM-safe.
- planCountPushdown, when enabled, iterates the lazy iterable and takes the FIRST
file (lazy early stop, no ParallelIterable fan-out).
The batch/stream DECISION (streamingSplitEstimate) is unchanged. A cache-read
failure records it and falls back to scan.planFiles() (catch Exception, since the
cache rethrows failures as RuntimeException). statsQueryId is nullable: streaming
passes null (no-stats overload) because its Phase 2 runs on the engine pump thread
while Phase 1 ran on the caller -- tallying the per-query ScanStats from two
threads would race; sync/count are single-threaded and keep stats.
Parity: the lazy path reuses the exact per-file prune/task-construction of the old
materialized path (which already claims legacy planFiles parity), so no new
planning is invented. The COUNT placeholder file may differ from the SDK path's
first file (ParallelIterable order is non-deterministic) but the count (snapshot
summary) is identical and BE ignores the file. Big-table streaming now populates
the cache (its purpose; capacity-bounded, no OOM).
Tests: streaming cache-parity vs SDK + cache consumed; partition-prune parity;
flat-map across multiple data manifests; COUNT cache count-parity + lazy early
stop (reads < all manifests); empty null-snapshot COUNT guard. Sync cache-path
tests stay green. Full iceberg module UT: 962 run / 0 fail / 0 error / 1 skip;
checkstyle 0 violations.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG31 parent 584ef38 commit 2e5f393
2 files changed
Lines changed: 329 additions & 46 deletions
File tree
- fe/fe-connector/fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
0 commit comments