Repository navigation
[fix](orc) Advance past SARG-filtered stripe suffixes when seeking - #431
Merged
Merged
Conversation
When seekToRow advances beyond every selected row group of an initialized stripe, the returned row offset is the stripe-end sentinel. Treating it as a row-group index can access past RowIndex entries or prematurely stop reading a partially filled final row group. Initialize the next selected stripe, respecting the configured stripe range, before positioning column readers. Add 32 cases covering initialized and fresh seeks, exact and partial index groups, lazy followers, filtered intermediate stripes, split boundaries, EOF and backward seeks. Validation: 56 focused ORC reader and predicate-pushdown tests passed under AddressSanitizer. All 16 initialized-stripe cases fail without the fix; all 32 new cases pass with it. clang-format 16 checks passed.
5 of 7 tasks
Gabriel39
added a commit
to apache/doris
that referenced
this pull request
Nov 4, 2026
…#70079) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#431 (merge the ORC dependency fix first) A condition-cache granule can start in a stripe's SARG-filtered suffix while containing a matching row in the next stripe. On an initialized stripe, ORC's `seekToRow()` uses the stripe-end sentinel as a row-group index. This can access past the index when the stripe ends on an index boundary, or prematurely return EOF when the last row group is partial. The same-stripe fast path originates in [Apache ORC ORC-1170](apache/orc#1114); the same missing stripe-end handling remains in the current upstream C++ reader. Pin the ORC fix that initializes the next selected stripe before positioning its column readers. Preserve SARG pruning, lazy reads and split boundaries. Add a Doris reader regression and a 1.5 KB synthetic ORC fixture with its generator. The fixture has two 20,000-row stripes, a 10,000-row index stride, and one matching row at the start of each stripe. A cache hit seeks to row 18,432 in the first stripe's filtered suffix. The SQL regression compares cache-disabled and cached results with lazy reads and min/max pruning enabled or disabled, and checks the profile for actual cache hits and skipped rows. ### Release note Fix potential crashes and missing rows when ORC condition-cache seeks reach a SARG-filtered stripe suffix. ### Check List (For Author) - Test: - [x] Unit Test: 247 ORC-related BE tests passed under AddressSanitizer, including the new Doris condition-cache case and 32 dependency boundary cases. - [x] Unit Test: 56 focused standalone ORC tests passed under AddressSanitizer. Reverting the fix makes all 16 initialized-stripe boundary cases fail. - [ ] Regression test: SQL suite added; Groovy compilation checked. The external SQL suite has not been executed locally. - [x] clang-format 16 and BE build-hygiene checks passed. - [ ] Local full-file clang-tidy analysis did not complete within ten minutes; no completed result is available. - Behavior changed: - [x] Yes. Safely advance to the next matching stripe or EOF. - Does this need documentation? - [x] No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On an initialized stripe,
seekToRow()can receive the stripe-end sentinel when SARG rejects the target and every remaining row group. Treating the sentinel as a row-group index accesses pastRowIndexentries for an exact index boundary. A partial final group can also cause a premature EOF before later matching stripes. Doris condition-cache granule seeks can trigger this boundary.Change
Initialize the next selected stripe before positioning its column readers, respecting the configured stripe range and EOF. This follows the same initialization path as a fresh seek.
Tests