Repository navigation
branch-4.2: [fix](be) Preserve JSON and WAL rows with empty file projections #70154 - #70180
Merged
Merged
Conversation
) ### What problem does this PR solve? Issue Number: close #69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
yiguolei
approved these changes
Nov 5, 2026
Contributor
Author
|
PR approved by anyone and no changes requested. |
Contributor
Author
|
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #70154