Skip to content

Comments

Mark Task #4 as complete#14

Merged
cbb330 merged 1 commit intomainfrom
update-task-4-status
Feb 20, 2026
Merged

Mark Task #4 as complete#14
cbb330 merged 1 commit intomainfrom
update-task-4-status

Conversation

@cbb330
Copy link
Owner

@cbb330 cbb330 commented Feb 20, 2026

Updates task_list.json to mark Task #4 as complete.

@cbb330 cbb330 merged commit fcd92ef into main Feb 20, 2026
7 of 10 checks passed
@cbb330 cbb330 deleted the update-task-4-status branch February 20, 2026 22:32
cbb330 added a commit that referenced this pull request Feb 20, 2026
cbb330 added a commit that referenced this pull request Feb 20, 2026
Implements predicate pushdown in the scan path by filtering stripes
based on column statistics before reading data. This is the core
integration that enables I/O reduction.

Changes:
1. Modified OrcScanTask to accept selected stripe indices
   - Changed Execute() to read only selected stripes using ReadStripe()
   - Iterator returns one batch per stripe (stripe = unit of parallelism)

2. Modified OrcScanTaskIterator to call FilterStripes
   - Applies predicate pushdown automatically during scan
   - FilterStripes ensures metadata is loaded
   - Returns task with selected stripes (empty if none match)

3. Stripe-selective reading
   - Task reads selected stripes one at a time using ReadStripe()
   - Replaces previous GetRecordBatchReader() which read all stripes
   - Implements stripe-level granularity from Task #0.5

Benefits:
- Skips stripes where predicate is known to be unsatisfiable
- Skips empty stripes (num_rows == 0)
- Reduces I/O by avoiding reads of filtered stripes
- Statistics loaded lazily and cached incrementally

Example: Query "WHERE x > 1000" on file with 10 stripes
- Stripe 0: x in [0, 100] -> Skip (literal(false))
- Stripe 5: x in [500, 600] -> Skip (literal(false))
- Stripe 9: x in [900, 1100] -> Scan (may have x > 1000)
Result: Only scans stripes that may contain matching rows

Verified: Mirrors cpp/src/arrow/dataset/file_parquet.cc lines 619-636

Co-authored-by: Claude Sonnet 4.5 <[email protected]>
cbb330 added a commit that referenced this pull request Feb 20, 2026
@github-actions
Copy link

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

cbb330 added a commit that referenced this pull request Feb 24, 2026
cbb330 added a commit that referenced this pull request Feb 24, 2026
Implements predicate pushdown in the scan path by filtering stripes
based on column statistics before reading data. This is the core
integration that enables I/O reduction.

Changes:
1. Modified OrcScanTask to accept selected stripe indices
   - Changed Execute() to read only selected stripes using ReadStripe()
   - Iterator returns one batch per stripe (stripe = unit of parallelism)

2. Modified OrcScanTaskIterator to call FilterStripes
   - Applies predicate pushdown automatically during scan
   - FilterStripes ensures metadata is loaded
   - Returns task with selected stripes (empty if none match)

3. Stripe-selective reading
   - Task reads selected stripes one at a time using ReadStripe()
   - Replaces previous GetRecordBatchReader() which read all stripes
   - Implements stripe-level granularity from Task #0.5

Benefits:
- Skips stripes where predicate is known to be unsatisfiable
- Skips empty stripes (num_rows == 0)
- Reduces I/O by avoiding reads of filtered stripes
- Statistics loaded lazily and cached incrementally

Example: Query "WHERE x > 1000" on file with 10 stripes
- Stripe 0: x in [0, 100] -> Skip (literal(false))
- Stripe 5: x in [500, 600] -> Skip (literal(false))
- Stripe 9: x in [900, 1100] -> Scan (may have x > 1000)
Result: Only scans stripes that may contain matching rows

Verified: Mirrors cpp/src/arrow/dataset/file_parquet.cc lines 619-636

Co-authored-by: Claude Sonnet 4.5 <[email protected]>
cbb330 added a commit that referenced this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant