Repository navigation
Commit 5ff7af6
POC: streaming (batch-granular) parquet scan via a persistent sync reader
Adds DF_FETCH_POLICY=streaming: instead of buffering whole row groups,
one long-lived sync ParquetRecordBatchReader pulls bytes through a shared
in-memory ChunkReader. The stream driver computes from the offset index
exactly which page ranges the next batch needs, awaits their fetch, and
keeps up to DF_FETCH_BUDGET bytes of background readahead in flight;
page bytes are dropped as soon as the decode cursor passes them.
Because the reader persists for the whole file, dictionary pages are
fetched and decoded exactly once, there is no per-row-group reader
rebuild, and resident memory is bounded by the readahead window rather
than row-group size. Supports projections, RowSelections (page skipping
preserved via selected-row prefix sums), limits, and multi-RG files;
falls back to the push-decoder path when row filters are active or the
offset index is unavailable.
Local simulated-latency results (300MB single-row-group file, 50ms
latency, 100MB window, streaming scan): time-to-first-batch 112ms vs
~700ms for all row-group-granular policies, total 708ms vs ~845ms, peak
staged 100MB vs 283MB.
Co-Authored-By: Claude Fable 5 <[email protected]>1 parent d38a4ee commit 5ff7af6
2 files changed
Lines changed: 602 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1387 | 1387 | | |
1388 | 1388 | | |
1389 | 1389 | | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
| 1424 | + | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
| 1429 | + | |
| 1430 | + | |
| 1431 | + | |
| 1432 | + | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
1390 | 1438 | | |
1391 | 1439 | | |
1392 | 1440 | | |
| |||
0 commit comments