Skip to content

branch-4.2: [fix](be) Preserve JSON and WAL rows with empty file projections #70154 - #70180

Merged
yiguolei merged 1 commit into
branch-4.2from
auto-pick-70154-branch-4.2
Nov 5, 2026
Merged

yiguolei merged 1 commit into
branch-4.2from
auto-pick-70154-branch-4.2

Conversation

@github-actions

@github-actions github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Cherry-picked from #70154

)

### What problem does this PR solve?

Issue Number: close #69926

Problem Summary:

Hive JSON queries that only need partition columns can return zero rows
even when the selected files contain records. For example, `SELECT * ...
WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt =
'p1'` returns zero. The V2 JSON reader derived its row count from the
output block, which has no columns after physical field pruning.
Writer-backed reader tests reproduce the same row-count loss for empty
WAL projections.

Track accepted JSON records independently of materialized columns,
including batch progress, array expansion, error rollback, and
ignore-malformed NULL rows. Preserve JSON-path record selection for
empty projections, and let the existing TableReader materialize
partition values using the returned row count.

For WAL, save the deserialized source row count before materialization
transfers column ownership, then pass it through the existing filter
pipeline. This preserves the row count when no file columns are
requested. The empty-projection reproduction is at the reader interface;
production replay reachability of an empty projection remains
unverified.

Add ten JSON BE tests covering empty file schemas and projections,
arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and
actual TableReader partition materialization. Add eight writer-backed
WAL cases covering empty, single-column, and reordered full projections
across multiple blocks, with and without schema discovery, plus repeated
EOF reads. Add a WAL SQL regression that forces two original async-load
transactions to abort, then verifies row counts, NULLs, aggregates,
filters, and actual values after replay. Add a Hive SQL regression using
the existing partitioned JSON fixture for COUNT, partition projections,
grouping, and file-column filters with external COUNT pushdown enabled
and disabled.

### Release note

Fix incorrect zero counts and missing rows when querying only partition
columns of Hive JSON tables. Preserve logical row counts for empty WAL
reader projections.

### Check List (For Author)

- Test
- [x] Unit Test: 166 ASAN tests passed, including JSON reader,
TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text
empty-projection controls. Before the JSON fix, nine of ten new tests
failed; before the WAL fix, both empty-projection cases failed while six
projection controls passed.
- [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice
on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out`
file with `run-regression-test.sh`, then reran for result comparison.
Execution on master is pending CI.
- [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy
compilation passed. Full execution against the external Hive fixture is
pending CI.
- Behavior changed:
- [x] Yes. Empty JSON and WAL file projections preserve logical row
counts. Hive JSON partition-only queries produce the correct rows and
aggregates.
- Does this need documentation?
    - [x] No.
@github-actions
github-actions Bot requested a review from yiguolei as a code owner November 4, 2026 06:02
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor

run buildall

@yiguolei
yiguolei merged commit 9ddd6f8 into branch-4.2 Nov 5, 2026
34 of 35 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants